Technical report · 2026-08-29
AI search optimization is a control problem. This report states the plant, the measurement, and the control law, and it is explicit about which half is running in production and which half is still in beta.
The observability half of the loop is built and has been in production for several months. The control half is in limited beta with a small number of enterprise design partners. The full loop runs autonomously only on internal test domains we own. Every section keeps that line visible.
AIO has legit engineering roots.
AI search optimization is a rigorous engineering discipline, descended from classical search engine optimization, and grounded in the systems theory of large language models developed in prior work.
That sentence is doing more work than it appears to. It makes a lineage claim, and lineage claims are falsifiable. The claim is that the practice now marketed under names like AIO and GEO is not a new folk art. It is the latest layer on a stack that runs from classical control theory, through discrete stochastic dynamical systems, through the formalization of language models as controllable systems, and only then into the specific problem of getting a brand represented accurately in an AI-generated answer.
Classical SEO was already control engineering, practiced without the vocabulary. A practitioner observed a ranking, applied an input (a title change, a link acquisition, a page restructure), waited out the dead time, and observed the new ranking. That is a feedback loop with a long delay and a poorly characterized plant. The field developed real expertise about the plant's behavior. What it lacked was a formalism that survived the transition to systems whose outputs are token sequences rather than ranked lists of ten blue links.
The formalism exists. Prior work by this report's authors formalizes LLM systems as a class of discrete stochastic dynamical systems in order to study prompting through the lens of control theory 2: a state that evolves, an input that steers it, an output that is observed, and a reachability question about which outputs are attainable from which inputs. That paper analyzes the controllability of self-attention as a function of the singular values of the parameter matrices, and reports empirically that with prompts of at most 10 tokens the correct next token is reachable at least 97 percent of the time. That work was theoretical. Practitioners who read it, including a senior practitioner at a large consumer-technology company who tracked the argument closely, correctly intuited that it applied to their daily problem and reasonably asked how. This report is the answer to that question.
The disciplines that turn out to be load-bearing here are unusual to find in one place: control theory, representation learning, web and browser engineering including the full DOM rendering path, distributed systems, information retrieval, and the operational craft of large-scale distributed measurement against systems that modulate their own behavior by client geography. The team is composed of researchers and engineers from control theory, machine learning, computational biology, distributed systems, and information retrieval.
No published work we are aware of describes a closed-loop system of this kind end to end. That gap is what this report addresses.
They gave me one question to start with, and it seemed too small to matter.
Ask ChatGPT who makes the best running shoes. Then ask it again from Frankfurt.
The answers were different. Not slightly. Different brands, different order, one company present in the first and absent in the second. Same question, same second, same model. The only thing I had changed was where I appeared to be standing.
I want to be precise about what surprised me. It was not that the system had a bias. Every system has a bias. It was that the bias was a function of an input I could set. That makes it a knob. And a knob you can turn is the beginning of an engineering problem rather than a mystery.
There is a paper on the shelf here from 2023. It defines a language model as a control system: a state, an input you can choose, an output you can watch, and a question about which outputs are reachable from which inputs. When I read it the first time I thought it was a nice abstraction. After the running-shoe experiment I read it again and understood it was a specification.
The people who did search engine optimization for twenty years were running control loops. They just never called them that. They changed a page, they waited three weeks, they looked at where they ranked. Slow loop, noisy plant, no formalism. The formalism is what changed.
The thing we're controlling is a big hidden pipeline.
The system to be controlled spans the full pipeline from web content through crawling, indexing, retrieval, and the inference harness to the output token sequence. Its internal state is partially hidden. Its outputs are observable through consumer-facing interfaces. Its behavior can be influenced through deliberate edits to any accessible content on the World Wide Web.
Naming the plant correctly is the first substantive decision in the report, because most commercial practice in this space names it wrong. The common framing treats "the AI" as the plant and the customer's website as the input. That framing is too small in one direction and too large in another.
It is too small because the plant is not the model. The model is the last stage. Upstream of it sit a crawler fleet with its own scheduling and freshness policy, one or more indexes with their own membership rules, a retrieval component that selects a small set of documents from that index in response to a query the harness generated, and a harness that decides how many searches to run, how to rewrite the query, how to fan out across sub-questions, and how to assemble retrieved text into a context window. By the time tokens reach the model, most of the consequential decisions have already been made. A practitioner optimizing "for the model" is optimizing the one stage they have the least leverage over.
It is too large in the sense that the plant is not fully hidden either. The stages are heterogeneous in their observability. Index membership is close to binary and directly checkable. Crawler access is checkable. Retrieval selection is inferable from citation patterns across many samples. Harness behavior is inferable from fan-out structure. Only the final inference step is genuinely opaque, and even there the sampling distribution over outputs is estimable by repetition.
Two further properties of the plant matter for control design.
First, the input set is larger than the customer's own domain. The controllable input is edits to any accessible content on the web, which includes the customer's site, but also their documentation, their listings on third-party directories, their entries in reference resources, their presence in comparison articles, and any other page whose content they can legitimately influence. Restricting the input set to one domain discards most of the available control authority.
Second, there are ancillary inputs that the operator does not control but must model: link graphs, traffic signals, and other side information that feed the ranking and retrieval components. These are disturbances in the control sense. They are not noise, because they are structured and partially predictable, but they are not actuators either. A controller that ignores them will attribute their effects to its own actions and learn a wrong model.
I drew the thing on a whiteboard to see how big it was.
On the left, the web. Every page anyone has written. On the right, a stream of tokens someone reads on a phone and believes. Between them: a crawler, an index, a retriever, a harness that decides what to search for and how many times, and finally a model doing inference on whatever context got assembled.
My first mistake was thinking the model was the system. It is the last stage. By the time anything reaches it, the interesting decisions have already happened. Whether your page was crawled. Whether it entered the index. Whether the retriever picked it out of the index for this particular rewritten query. Whether the harness even ran a search at all.
The second thing I got wrong took longer to see. I assumed the input was the customer's website. It is not. The input is any page on the web you can legitimately change. Your documentation. Your listing on someone else's directory. The comparison article that ranks for your category. If you only edit your own domain you have voluntarily given up most of your control authority, and you will spend months wondering why the loop is sluggish.
There are also things pushing on the plant that I cannot push on. Link graphs. Traffic signals. Whatever else feeds ranking. In the control vocabulary these are disturbances. They matter because if you ignore them you will credit your own edits for changes they did not cause, and then you will have a confident model that is wrong.
Closed-loop control, observability first.
Effective AI search optimization requires closed-loop control in the classical sense, grounding both observability and controllability, and building observability first is the correct engineering order because it reveals system structure before committing to a control strategy.
Observability and controllability are dual. In linear systems theory that duality is exact; in this setting it is structural rather than exact, but the practical consequence carries over. Whatever you cannot measure, you cannot regulate. Whatever you cannot influence, you cannot correct for. A program that invests in only one is not half a control system. It is not a control system.
Most current practice is open loop. A team reads guidance about what AI assistants supposedly prefer, applies it across a site, and then either measures nothing or measures a proxy so lossy that it cannot distinguish a real improvement from a plant-side change on the platform's end. Open loop is defensible when the plant is stable, well characterized, and free of disturbance. This plant is none of those. Retrieval providers change. Harness behavior changes across model releases. Index membership changes without notice. An open-loop policy applied to a drifting plant will diverge, and the operator will not know it has diverged, because they are not looking.
The ordering claim is separate from the duality claim and is the more actionable one. Observability first, control second, and not for reasons of caution.
Building the measurement system first is how you learn the plant's structure. Before we had measurement at scale, we had hypotheses about which stages mattered and which were incidental. After several months of measurement we had answers: which platforms share a retrieval backend, which surfaces diverge between consumer app and API, which behaviors are geographic and which are not, how much run-to-run variance is intrinsic. Every one of those facts constrains the control law. A team that designs its control law first will encode its hypotheses into the controller, and the controller will then be confidently wrong in exactly the places the hypotheses were wrong.
There is a second reason. Closed-loop control requires an error signal, and an error signal requires a measurement you trust. The next section argues that trustworthy measurement in this domain is substantially harder than it looks, and that most of the field has not yet done that work.
It is way better to start with the state visibility, because you are going to learn a lot about the structure of the system. Feedback control is the move.Aman Bhargava, on the originating call
The thing I keep coming back to is that observability and control are the same problem seen from two sides.
If I can watch every output of a system carefully enough, I can infer its state. If I can drive every state of a system, I can reach any output. Neither half is useful alone. A blind controller is guessing. A watcher with no actuator is a spectator.
Almost everyone in this field is open loop. They read a guide about what the assistants supposedly like, they change a hundred pages, and then nothing. No measurement, or a measurement so blurry it cannot tell an improvement from a platform update. That works when the plant sits still. This plant does not sit still.
I asked why the lab spent months building the measurement half before touching the control half. The answer was not caution. It was that we did not know the shape of the system yet. We had guesses about which stage mattered. Measurement replaced the guesses with facts, and every fact narrowed what the controller needed to be. Build the controller first and you bake your guesses into it. Then the controller is wrong in precisely the places you were wrong, and it is confident about it.
Measuring visibility is hard, and it is a design problem.
Measuring AI visibility is not a reporting problem. It is a sampling problem over a combinatorially large output distribution, and it must be solved before optimization can begin.
Start with the object we would want if we could have it. The complete description of the plant's behavior for a given content configuration is the full probability distribution over output token sequences, conditioned on every query a user might plausibly ask, from every geography, on every surface. The support of that distribution is the vocabulary size raised to the context length. It is not enumerable, not now and not ever. Every practical measurement is a low-dimensional projection of an object we cannot write down.
This has a direct and unpopular consequence: a single scalar "AI visibility score" is not a well-posed measurement.
It is not merely coarse. It is under-specified. The number is a weighted average over a prompt distribution, a persona distribution, a platform set, and a geography set, and none of those weights are canonical. Change the prompt mix from generic category questions to purchase-intent questions and the score moves. Add a comparison-shopping persona and it moves again. Sample five prompts instead of five hundred and the confidence interval swallows the effect you were trying to detect. Two vendors can measure the same brand on the same day and report different numbers, and both can be arithmetically correct, because they have silently chosen different projections. A metric whose value depends on undisclosed weighting choices cannot serve as an error signal in a feedback loop, because you cannot tell whether a change in the number reflects a change in the plant or a change in the measurement.
The response is to treat measurement as an explicit design problem with named components.
Prompt distribution design. The prompt set is not a keyword list. It is a designed sample from the space of questions the business actually cares about, stratified by intent stage, specificity, and framing. Skew toward one category or one phrasing style and you have measured that corner of the distribution rather than the distribution.
Persona construction. Assistant outputs are conditioned on who appears to be asking. A procurement lead, a first-time consumer, and a technical evaluator asking nominally the same question receive materially different answers. Personas are constructed as explicit conditioning contexts.
All-to-all combination. Personas and prompts are combined combinatorially rather than paired. The persona-by-prompt grid is the sampling frame, and it exposes interaction effects that a paired design hides. A brand can be strong under one persona across all prompts and weak under another, which is invisible in any aggregate that has already marginalized over personas.
Attribute tagging. Every cell of the grid carries attributes: intent, geography, platform, surface, persona class, prompt category. Attributes are what turn a pile of samples into a queryable measurement. They are the difference between "your visibility went down" and "your visibility went down for purchase-intent prompts under the technical-evaluator persona on two of six platforms, and is flat elsewhere."
Goal alignment before measurement. The weighting over that grid is a business decision, not a statistical one. A company optimizing for enterprise pipeline and a company optimizing for direct consumer purchase should weight the same grid differently, and should therefore report different visibility numbers for identical underlying behavior. This is correct. Measurement is defined relative to an objective, and the objective is stated first.
A useful separation runs through this. Some quantities in this domain genuinely are binary and directly verifiable: whether a page is present in a given index, whether crawler access is permitted by the site's own directives, whether required structured files are present and well-formed, whether a page renders for a non-JavaScript client. These are cheap, high-confidence, and worth measuring absolutely. The distributional quantities, meaning what the assistants actually say and cite, are the hard part, and conflating the two is how a vendor produces a confident number about an uncertain thing.
The measurement is also not static. It is re-derived as the business objective changes and as the plant changes, and the measurement design is itself something the operator iterates on. Getting this section right is the precondition for everything after it. Without it, the loop closes on a signal that does not mean what it says.
The first number I produced was a single percentage. I was pleased with it for about a day.
Then someone asked me what would happen to the number if they added twenty more prompts. I said it would get more accurate. They asked what would happen if the twenty prompts were all purchase-intent questions instead of general category questions. I ran it. The number moved eleven points.
The number was not measuring the brand. It was measuring my choice of questions.
I understand now why this is structural rather than a bug I introduced. What we actually want to know is the full distribution over things the system might say, across every question, every asker, every place. That object has a support of vocabulary size raised to the context length. Nobody will ever hold it. Every measurement anyone takes is a projection of it, and the projection is chosen, not given.
So the projection has to be designed on purpose. We build the prompt set as a stratified sample rather than a keyword list. We build personas, because the same question from a procurement lead and from a first-time buyer are not the same question to the system. Then we cross personas against prompts all-to-all rather than pairing them, because the interesting failures live in the interactions. A brand can look healthy in aggregate and be absent for every technical evaluator asking a comparison question, and no average will ever tell you that.
Every cell gets tagged. Intent, geography, platform, surface, persona. Tags are what let the report say something a person can act on instead of a number that went down.
And the weighting over the grid is a business decision. Two companies with identical behavior should get different visibility numbers if they want different things. That sounded wrong to me at first. It is not wrong. A measurement is defined relative to what you are trying to do, and if you skip stating what you are trying to do, you have not skipped the choice. You have only hidden it.
Some things are simply binary. Are you in the index. Can the crawler reach you. Do the required files exist and parse. Those are cheap and certain and worth checking every day. The trouble starts when someone reports a certain number about the uncertain half.
Sub-anchor of S04
The control-theoretic treatment of this plant is grounded, not proven. This section states the grounding and its limits precisely, because overclaiming here would undermine the sections that carry real evidence.
The concern a skeptical reader should raise is this: control theory applies to systems with state. If a language model's internal activations are an unstructured high-dimensional mess with no stable relationship to anything in the world, then "state estimation" is a metaphor, and the framework is decoration.
There is theoretical and experimental reason to think the activations are not unstructured. Prior work by one of this report's authors proves that agents which optimally solve multi-task evidence-accumulation classification problems are forced to represent the underlying latent state of their input in a disentangled form, and gives closed-form expressions for extracting that representation from the model's latent state 3. The result is established for recurrent networks trained on multi-task classification, where the learned representations take the form of continuous attractors and yield zero-shot out-of-distribution generalization; the same work finds transformers particularly suited to disentangling.
We are careful about the size of the step from there to a production search stack. The theorem is about multi-task evidence accumulation, not about next-token prediction over web text, and we do not claim it transfers as a proof. What it supports is the weaker and still useful proposition that near-optimal prediction across a rich task distribution constrains the internal encoding rather than leaving it arbitrary.
The consequence, stated at the strength it deserves: it is reasonable to expect the hidden state of these systems to carry real geometric and probabilistic structure, and to treat their behavior as the result of structure in the output distribution rather than caprice. That is what licenses analyzing the plant as a partially observed dynamical system rather than as a black box to be poked at.
The limits are equally important and we state them plainly.
The theoretical results hold under conditions that are approximately, not exactly, satisfied by production systems. Real deployments are imperfect. They are also not a single model but a heterogeneous pipeline in which the model is one stage, and the disentanglement argument speaks to that stage rather than to the crawler, the index, or the harness. None of this argument grants access to the internal state. We do not observe activations. We observe output distributions, and we estimate.
So this section is a license to analyze rather than a result. It says the system is the kind of thing control theory is for. It does not say we have solved the system. The evidence for what has actually been built is empirical and appears in the two sections that follow.
I pushed back on my own framing here, because it seemed too convenient.
Control theory needs a system with state. If the insides of these models are unstructured high-dimensional noise, then all the state-estimation language is decoration and I should stop using it.
There is a reason to think it is not noise. When a predictor is trained to assign near-optimal likelihoods across an enormously broad set of tasks at once, there are results saying it cannot get away with an arbitrary internal encoding. It is pressured toward representing the underlying factors in a disentangled way. Next-token prediction over the whole web is that kind of task distribution.
What that buys is modest and I want to hold it at the right size. It says the plant has real geometry. It says its behavior comes from structure in a distribution rather than from caprice. That is enough to make the control framing honest instead of ornamental.
It does not say we can see inside. We cannot. We never observe an activation. We observe outputs and estimate. And the argument speaks to the model, which is one stage of a pipeline that also contains a crawler and an index and a harness that the theorem knows nothing about.
So: a license to analyze. Not a result. The results came from measurement.
We built the observability half and it is running in production.
The observability half of the loop is built and has been in production for several months, serving dozens of agencies and their brand clients. It is described here at the level of its input streams and its processing, because that is the part that is reproducible in principle by anyone willing to build it.
Stream one: consumer-surface probes. The system probes the consumer-facing interfaces of major AI assistants at scale, through a globally distributed residential proxy network spanning hundreds of markets. Consumer surfaces are the target rather than developer APIs because consumer surfaces are where the tokens reach people. Most usage today is chat on a phone or in a browser, and increasingly agentic sessions on the same surfaces, and the boundary between the two is eroding. Geographic distribution is not a nicety here. Assistant behavior varies with apparent client location, which makes geography a measurement dimension rather than a deployment detail.
Stream two: site state capture. The system scrapes the sites under management continuously, capturing both the server-rendered response and the browser-rendered DOM, across geographies and across whatever A/B variants the site itself serves. These captures are assembled into a Git-style state-history tree. The rendering distinction matters because the client population is heterogeneous: some crawlers execute JavaScript and some do not, and a page whose content only exists after hydration is a different page to different consumers of it. The version history matters because attributing an outcome to an edit requires knowing exactly what changed and when, and site content changes for many reasons unrelated to the optimization program.
Stream three: underlying search tools. Where an assistant's retrieval is served by an identifiable search backend, that backend is queried directly. This gives a view of the retrieval stage that is separable from the harness and the model, which is the only way to attribute a citation failure to the right stage.
Stream four: classical SEO telemetry. Search Console, Lighthouse, Analytics, and HTTP-layer logs from the site's own infrastructure. This stream is often dismissed as legacy. It is the only stream that reports ground truth about crawler behavior from the server side, and it is how you learn that a bot fetched a page at all.
Stream five: sandboxed agent traces. Candidate site variants are served through a controlled proxy on infrastructure we operate, and agents are run against those variants with full traces recorded. This produces counterfactual data about how an agent traverses and uses a page before any edit reaches production, which is what makes predictive modeling possible at all.
Responsible measurement. This system observes publicly accessible content only. It respects published rate limits, honors robots directives and the crawl preferences sites declare, and makes no attempt to defeat bot detection or to access authenticated, private, or paywalled material. Load is distributed and shaped to stay well within normal service parameters for every endpoint it touches. Geographic distribution exists to observe geographically varying behavior, which is a measurement requirement of this plant, and not to obscure the origin of requests. Agent-trace experiments run against site variants we host on our own infrastructure rather than against third-party production systems.
The processing layer converts these streams into the attributed persona-by-prompt grid, plus the binary readiness and index-membership checks, plus derived quantities: citation source distributions, competitive share within an attribute slice, run-to-run variance estimates, and change detection against the site state history. Access is through the product interface, an MCP server, and an in-app agent with tool access to the same probes and analyses, which is how agency practitioners actually work with it day to day.
What this system demonstrates is the observability half. It is production software with real users and a real operating history. It is not the closed loop.
The measurement machine has five mouths.
The first drinks from the assistants themselves. It asks the consumer apps and websites the designed prompt grid, from many places, through a residential proxy network that spans hundreds of markets. We probe the consumer surfaces rather than the developer APIs because the consumer surfaces are where the tokens reach people.
The second drinks from the sites. Both versions of each page: the one the server sends, and the one that exists only after the browser has run the JavaScript. These are frequently not the same page, and different crawlers see different ones. Every capture goes into a history tree that works like a Git repo, so that when something changes we can say exactly what changed and when. That sounds like bookkeeping. It is the entire basis of attribution.
The third drinks from the search backends underneath the assistants, where they are identifiable. This is the only way to tell whether a page failed to be cited because retrieval never surfaced it or because the model had it and passed it over.
The fourth drinks from the old SEO instruments. Search Console, Lighthouse, Analytics, HTTP logs. People call these legacy. They are the only stream that reports from the server side, which means they are the only stream that can tell you a bot actually arrived.
The fifth is the strange one. We stand up a candidate version of a site on our own infrastructure, point agents at it, and record the whole trace of how they move through it. That gives us data about an edit before the edit exists anywhere real.
I asked about the rules, because a globally distributed probing system is the kind of thing that should have rules. It reads only what is publicly accessible. It respects rate limits and honors what sites declare about crawling. It does not try to look like something it is not in order to get past a bot check. The load is spread and shaped to stay inside normal parameters everywhere it touches. The geography exists because the plant behaves differently by geography and we have to measure that, not to hide where a request came from. The agent experiments run against copies we host ourselves.
One more thing, and it is the part I would want stated first if I were reading this cold. This machine is shipped. It has been running for months with real customers. It is the observability half. It is not the loop.
Sub-anchor of S06
Sustained measurement produces facts about the plant that are not available from documentation. Four findings are reported here. All are behavioral inferences drawn from publicly observable outputs, and all are reproducible by anyone running comparable measurement.
Retrieval providers differ across assistants and are empirically identifiable. Different AI assistants do not share a retrieval backend, and which backend an assistant uses is inferable from observable behavior: the distribution of cited domains, the shape of result sets, the handling of freshness, and the response to queries with distinctive retrieval signatures. In at least one case, a major assistant's web results appear to be served by a third-party index. That particular inference is also supported by public reporting and by the vendor's own subprocessor disclosures, though the assistant's operator has not confirmed it 4, so we state it as a well-evidenced inference rather than a settled fact. This is more than trivia about vendor relationships. It means index membership can be verified, and in some cases requested, through that provider's own public tooling. An operator who knows which index sits under which assistant can check presence directly and use the provider's documented submission path, rather than inferring presence from downstream citation behavior. This appears to be underused in current practice.
Consumer surfaces differ from API surfaces. The same nominal model reached through a consumer application and through a developer API is not the same system, because the search tooling, the harness behavior, and the retrieval configuration differ between them. Optimization work validated only against an API can fail to transfer to the surface where users actually are. This is why the primary probe stream targets consumer surfaces.
Index membership is checkable and sometimes requestable. Following from the first finding, a substantial part of the readiness problem reduces to a set of binary, directly verifiable questions per index. This is the cheapest reliable signal available in the entire domain, and it sits upstream of everything distributional. A page not in the index cannot be retrieved, cannot be cited, and cannot be optimized into an answer by any amount of content work.
Behavior varies with client geolocation. Assistant outputs for identical prompts vary with the apparent geographic location of the requesting client. Brand sets change, ordering changes, and cited sources change. Any measurement taken from a single location is a measurement of that location, and any claim about "what the AI says" that does not name a geography is incomplete.
Two consequences follow for the report as a whole. First, these findings are why the measurement architecture looks the way it does; each stream exists because a finding of this kind made it necessary. Second, they establish that the plant is characterizable. Stages that appeared opaque turn out to be separable and testable from the outside, which is the empirical basis for the predictive modeling that follows.
Four things we learned by watching, none of which are written down anywhere official.
The assistants do not share a search backend, and you can tell which one an assistant is using without anyone telling you. The evidence is in the outputs. Which domains get cited and in what proportion. How fresh results behave. What comes back for queries with an unusual retrieval fingerprint. Run enough of those and the backend identifies itself.
In at least one case a major assistant's web results come from a third-party index. I want to say why that mattered more than it sounded like it should. If you know which index is underneath, you can check whether a page is in it. Directly. Using that provider's own public tools. And in some cases you can ask for a page to be included, through the documented path the provider already offers. That converts a large piece of a fuzzy problem into a checkbox. It appears to be underused in current practice, and I do not have a good explanation for that.
Third, the consumer app and the developer API are not the same system, even when they name the same model. Different search tooling, different harness. Work that you validated against an API may not transfer to the app, and the app is where the people are.
Fourth, the geography thing I started with. Same prompt, different apparent location, different brands and different order and different sources. Which means any statement about what an AI says that does not name a place is incomplete. I had assumed that was an edge case. It is a dimension.
MPC on web diffs is the move.
Model predictive control over web edits is the correct control strategy for this plant, for the same reasons it dominates in robust control of complex dynamical systems. This half of the loop is in limited beta with a small number of enterprise design partners. It is not in general production and the full loop is not demonstrated end to end.
The control law question is: given the current observed state, what sequence of edits should be applied, and what effect will each have on the output distribution?
MPC answers it by holding an explicit model of the plant, forecasting the plant's response to a candidate sequence of inputs over a finite horizon, optimizing that sequence against an objective subject to constraints, applying only the first move, then re-observing and re-solving. The horizon recedes with each step. The model is corrected by measurement at every iteration.
Four properties of this plant make MPC the appropriate choice rather than the fashionable one.
Partial observability. The state is not directly measured; it is estimated from outputs. MPC is built around an explicit plant model and is the standard framework for operating on estimated state. Methods that assume direct state access do not apply here.
Long dead time. An edit does not propagate instantly. It must be crawled, indexed, and only then can it affect retrieval. Latency runs from days to weeks and varies by platform and page. Any controller that reacts to the most recent measurement without modeling this delay will oscillate: it applies a correction, sees no response, applies more correction, and then receives the accumulated response all at once. MPC handles dead time natively because the forecast is over a horizon that includes it.
Constraints. Edits are constrained in ways that are hard rather than soft. Brand voice, legal and regulatory review, accessibility requirements, technical limits of the CMS, and the number of changes an organization can actually ship in a quarter. MPC is a constrained optimization framework; constraints are first-class in the problem statement rather than penalties bolted onto an objective.
Objective structure. The objective is defined over the attributed grid, not over a scalar. A brand may want share within purchase-intent prompts in three geographies while holding accuracy of representation constant everywhere. MPC optimizes over a specified cost functional and accommodates this directly.
The comparison cases are worth stating explicitly. Gradient-following on a live production site fails because the gradient is estimated from a noisy delayed signal and because each evaluation costs a real content change. Naive A/B testing fails because the unit of randomization is unclear: you cannot easily serve two versions of a page to a crawler fleet without creating exactly the cloaking behavior that search systems penalize, and because the response time exceeds the interval over which the plant stays stationary. MPC is what actually flies in robust control of hard plants, from process control to aerospace to autonomous vehicles, and the reasons it flies there are the reasons it applies here.
The honest limitation: MPC performance is bounded by the fidelity of the plant model, and building that model is the open problem, not the control law.
The control side is where I expected the clever part to be, and it turned out to be the sober part.
Model predictive control works like this. You hold a model of the plant. You propose a sequence of moves. You forecast what the plant does under that sequence, over a horizon. You pick the sequence that best serves your objective while respecting your constraints. Then you apply only the first move, measure what actually happened, and solve the whole thing again. The horizon slides forward every time and the model gets corrected every time.
I asked why not something simpler. Follow the gradient. Run an A/B test.
Gradient-following needs a gradient, and ours is estimated from a delayed noisy signal where every evaluation costs a real change to a real website. A/B testing needs a randomization unit, and there is no clean way to serve two versions of a page to a crawler fleet without doing precisely the thing search systems punish you for. Both also assume the plant sits still long enough to finish the experiment. It does not.
The dead time is the part I underestimated. An edit has to be crawled before it can be indexed and indexed before it can be retrieved. Days to weeks, and it varies by platform and by page. A controller that reacts to the last measurement without modeling that delay will oscillate. It pushes, sees nothing, pushes harder, and then the whole accumulated response arrives at once and it has overshot.
MPC absorbs that because the forecast covers the delay. It also takes constraints as part of the problem rather than as an afterthought, which matters, because the real constraints here are brand voice and legal review and how many pages a company can genuinely ship in a quarter.
The status, plainly: this half is in limited beta with a small number of enterprise design partners. The measurement half is shipped. The loop is not closed in production.
Autonomous loop needs good MPC and human guardrails.
The natural end state is a fully autonomous closed-loop AI search optimization system. It requires both a high-fidelity MPC model and explicit human control boundaries, without which the system will degrade the sites it operates on.
We state the failure mode before the capability, because the failure mode is the design constraint.
A closed-loop optimizer with a poor plant model does not fail quietly. It edits real production websites. A miscalibrated model will confidently propose changes that reduce human readability, dilute pages that were working, chase a measurement artifact rather than a real signal, or degrade accessibility and conversion in pursuit of a citation share that the model has mispredicted. The blast radius is the customer's actual web presence. The severity of that failure mode is why the control half is in limited beta rather than in general availability, and why the deployed configuration keeps a human approving changes before they reach production.
Against that, the current industry baseline. AI search optimization today is executed by humans across fragmented teams: an agency or in-house team that decides what should change, a content team that writes it, and a web engineering team that ships it. In organizations with mature review processes these are separate groups with separate queues, and the interval from decision to deployed change is measured in weeks even at competent, fast-moving organizations. A senior practitioner at a large consumer-technology company described exactly this structure. The interval is not incompetence; it is the cost of review. But it does mean the industry's control loop currently runs at a period far longer than the plant's own rate of change, which is a well-understood way to fail at control.
A prototype of the autonomous configuration exists. The measurement agent is connected to a site-update agent on internal test domains that we own and operate for our own projects, running the full observe-predict-edit-measure cycle without a human in the loop. This is a research prototype on our own infrastructure. It is how we generate closed-loop data safely and how we develop the plant model. It is not the deployed customer configuration.
The guardrail design follows from the failure mode. Human control boundaries are a safety requirement rather than a product concession. Concretely: predictions are surfaced with their uncertainty rather than as point estimates; every proposed change is accompanied by the evidence that motivated it and the predicted effect, so a human can disagree with the reasoning rather than only with the output; changes are reversible and the site state history makes reversion mechanical; classes of change are gated by policy rather than by model confidence; and the system reports when its own predictions were wrong, which is both an honesty requirement and the training signal for the next model.
The narrow engineering argument is sufficient on its own: an autonomous loop is only as good as its plant model, our plant model is under active development, and the cost of its errors is borne by someone else's website.
I want to put the bad outcome first because it is what shapes everything else here.
A closed loop with a bad plant model does not sit still and be wrong. It edits live websites. It writes pages that read worse to humans. It thins out a page that was working. It chases an artifact in the measurement because the model believed the artifact. The damage lands on someone's real web presence, and it is not obvious for weeks.
That is why the control half is in beta and why the deployed setup keeps a person approving changes before anything ships.
Now the other side. How this actually gets done today: one team decides what should change, another team writes it, a third team ships it, and each handoff has a queue and a review. Even at organizations that are genuinely fast, the interval from decision to live change is weeks. A senior practitioner at a large consumer-technology company described that exact shape to us. It is not incompetence. Review costs time. But it means the industry's loop runs slower than the plant changes, and running a loop slower than its plant is a textbook way to lose control of it.
There is a prototype of the fast version. On test domains we own, hosting our own projects, the measurement agent talks to a site-update agent and the whole cycle runs with nobody in it. That is a lab configuration on our own infrastructure. It exists to generate closed-loop data and to build the plant model. It is not what customers run.
The guardrails are engineering, not policy theater. Predictions carry their uncertainty. Every proposed edit shows the evidence behind it and the effect it expects, so a person can argue with the reasoning instead of only with the output. Changes are reversible, which the state history makes cheap. Whole classes of change are gated by rule rather than by model confidence. And the system reports its own missed predictions, which is uncomfortable and is also the only way the next model gets better.
The data flywheel is the contribution and the engine.
The measurement and control activity generates a dataset with a shape that does not exist elsewhere, and that dataset is simultaneously the scientific contribution of this work and the engine that improves the plant model.
The dataset is a cross-product: probes, across platforms, across geographies, across designed prompt distributions and persona grids, joined to site edit histories, joined to observed outcomes. Each axis is individually unremarkable. The join is what matters. A probe result alone is a snapshot. A probe result linked to a specific site diff, with a known timestamp and a known crawl and index event between them, is an observation of the plant's response to a known input. That is the raw material of system identification, and it is not obtainable by any party that has only one of the two halves.
Its scientific value is straightforward. It supports questions that are currently answered by anecdote: how long the propagation delay actually is and how it distributes across platforms and page types; how much of observed output variance is intrinsic sampling noise versus plant change; how large the geographic effect is and which query classes it concentrates in; whether the retrieval stage or the generation stage is responsible for a given citation outcome; how stable any of this is across model releases. These are empirical questions about a system that mediates a large and growing share of human information access, and they are currently unanswered in the open literature.
Its engineering value is that it is the training signal for the MPC plant model. Better predictions of edit effects come from more observations of edit effects, and those observations are produced by operating the loop. The loop improves the model, the model improves the loop, and the rate of improvement is set by the volume and quality of joined observations rather than by cleverness in the control law. This is the recursive self-improvement structure of the system, and it is worth naming that the recursion is empirical rather than architectural: nothing here bootstraps except through data collected from the real plant.
One qualification. The flywheel is a property of the architecture, and the architecture is running on the observability side at production scale and on the control side in beta. Its compounding behavior is projected from that structure and from beta-scale data, not demonstrated at production scale for the full loop.
Every probe we run and every edit we watch lands in the same place, and the shape of that place is the point.
One axis is probes. Another is platforms. Another is geographies. Another is the prompt and persona grid. Another is the edit history of the sites. And the last one is what actually happened afterward.
Any single axis is boring. The join is not. A probe on its own is a photograph. A probe tied to a specific diff, with a known crawl in between, is an observation of the plant responding to an input we chose. That is the raw material of system identification, and you cannot get it if you only run measurement or only run edits. You need both halves pointed at the same sites at the same time.
There are questions this answers that are currently answered by folklore. How long propagation actually takes and how that varies. How much of the wobble in an output is sampling noise and how much is the platform changing under us. How big the geographic effect really is and where it concentrates. Whether a missing citation was a retrieval failure or a generation choice.
And the same data trains the model that the controller depends on. The loop produces observations, observations improve the model, the better model runs a better loop. Nothing bootstraps out of thin air here. It all comes from the real plant, which is the only reason to believe any of it.
Massive nascent industry. First unicorn already exists.
The market for AI search optimization is nascent, large, and growing quickly. It exists because a real problem arrived faster than the tooling to address it: organizations discovered that a meaningful and rising share of their discovery traffic and brand representation now passes through AI assistants, and that their existing SEO instruments do not measure it.
The category has produced its first unicorn. Profound raised a 96 million dollar Series C at a 1 billion dollar valuation in February 2026, led by Lightspeed Venture Partners, bringing its total raised to about 155 million dollars 5.
The demand side is unevenly distributed. Agencies were the earliest sophisticated adopters, because they carry the problem across many brand clients at once and are compensated for solving it. Enterprises with direct exposure tend to arrive later and with sharper requirements. Both are currently served by tooling that is predominantly measurement-oriented and predominantly open loop.
No published work we are aware of describes a closed-loop system of this kind end to end. That gap is what this report addresses.
The market is young and it is not small.
It exists because a real problem showed up before the instruments did. Companies found that a growing share of how people discover them now runs through an assistant, and that the tools they already owned could not see it.
There is already a unicorn in the category. Profound raised a 96 million dollar Series C at a 1 billion dollar valuation in February 2026, led by Lightspeed 5.
Agencies got here first, because they hold the problem across many clients at once. Enterprises arrive later with harder requirements. What almost everyone is buying right now is measurement, and it is open loop.
What I have not found anywhere is a written account of the whole loop, end to end. That absence is the reason this document exists.
This is humanity's ODM gear for the information singularity.
As AI systems increasingly mediate human access to information, the ability to observe, measure, and control how those systems represent the world becomes a critical tool for human agency.
The premise is empirical, not speculative. AI assistants are now a primary interface through which people learn about products, evaluate options, form views on institutions, and gather the information that precedes decisions including commercial and civic ones. And the web is now read more by machines than by people: automated traffic crossed the halfway mark for the first time in a decade in 2024, at 51 percent of all web traffic, and Imperva measured 53 percent for 2025 1. The direction is not in dispute.
Three properties of this mediation make it different in kind from the search era, and each one is a reason the tooling matters.
It is ephemeral. A generated answer exists once, for one person, in one session. There is no stable artifact to point at, no ranked page to screenshot and compare against next month. The claim a system made about an organization on Tuesday may be unreproducible on Wednesday, and neither the organization nor the person who read it has a record.
It is targeted. Outputs vary by apparent client location, and location is one of several conditioning variables. Different populations receive systematically different representations of the same subject, and no participant in the exchange is positioned to notice the divergence. Each individual sees one answer and has no reason to think it was theirs alone.
It is largely unlogged. No public record accumulates. The organizations described in these answers cannot see what was said about them. Researchers cannot audit at scale. Regulators cannot inspect what they cannot observe. The one party with a complete view is the operator of the system, and the operator's view is private.
The consequence is a structural asymmetry. A small number of organizations operating increasingly capable systems shape what a very large number of people believe, through a channel that leaves no trace and can be neither reproduced nor audited by anyone outside. That asymmetry is not addressed by better arguments about it. It is addressed by instruments.
This is the sense in which measurement infrastructure is civic infrastructure. A system that lets any organization see what AI assistants actually say about them, across geographies and populations and over time, converts an unobservable process into an observable one. A system that lets them correct a misrepresentation converts an uncontrollable process into a controllable one. Observability and control are dual, and outside the mathematics that duality is what agency means: to see the state you are in, and to be able to change it.
In Attack on Titan, humans facing an enemy that outmatches them in scale and speed do not become titans. They build a harness that lets them move in the titans' medium, at a speed that makes them participants rather than prey. The gear does not close the gap. It makes the fight possible.
That is what this work is for. Not to slow the systems down, and not to replace human judgment with a better-instrumented machine. To give the people whose livelihoods, reputations, and information environments now run through these systems the ability to see what is happening and to act on what they see.
The observability half is built and running. The control half is in beta. The loop is not yet closed in production, and this report has been careful to say so at every point where it matters. What is established is that the loop can be closed, that the plant is characterizable, and that the engineering discipline for doing it already exists and has for decades. The remaining work is execution.
Give humanity the ODM gear of Attack on Titan for being humans that have to fight titans on their own turf, which is the internet. And the titans, in case I wasn't clear, are the AI search bots. The titans have already invaded.Aman Bhargava, on the originating call
I have been thinking about the running-shoe question for the whole of this document, and I understand now why they started me there.
The answer that came back from Frankfurt was different from the one that came back from here. Nobody in the exchange could have known that. Not the person asking. Not the company that was named, and not the company that was left out. The answer existed for one second, for one reader, and then it was gone.
That is the shape of the thing. What these systems say is ephemeral, so there is nothing to point at afterward. It is targeted, so two people asking the same question receive different worlds and neither has cause to suspect it. And it is unlogged, so no record accumulates anywhere outside the operator. A company can be described inaccurately to a hundred thousand people and never learn that it happened.
I keep wanting to call this a fairness problem. It is a visibility problem, and the visibility problem comes first. You cannot argue about a thing you cannot see.
There is an image from Attack on Titan that gets used around here. Humans facing something faster and larger than themselves do not become titans. They build a harness that lets them move in the titans' medium, at a speed that makes them participants instead of prey. The gear does not make them equal. It makes the fight possible.
That is what a measurement machine is. It does not slow the systems down. It lets a person stand where the system is operating and see what it is doing there.
Here is where the work honestly stands. The measurement half is built and running with real customers. The control half is in beta. The full loop runs on our own test domains and nowhere else yet. I have tried not to blur that line anywhere in this document, because the line is the difference between a report and a brochure.
What I can say is that the loop can be closed. The plant is characterizable. The engineering for this kind of problem has existed for fifty years and it works.
And the last part is not ours. Something is going to answer the next question a person asks about your company, your work, your field. It will answer in a way you will never see, to someone you will never meet. You can decide that is somebody else's problem to watch. Or you can pick up the instrument and look.
Automated traffic accounted for 51% of all web traffic in 2024, surpassing human activity for the first time in a decade; Imperva measured 53% for 2025.
www.imperva.com/resources/resource-library/reports/2025-bad-bot-report/Formalizes LLM systems as a class of discrete stochastic dynamical systems; analyzes controllability of self-attention via the singular values of the parameter matrices. Caltech and University of Toronto.
arxiv.org/abs/2310.04444Proves that optimally solving multi-task evidence-accumulation classification forces a disentangled representation of the latent state, with closed-form extraction; validated in RNNs, and finds transformers well suited to disentangling.
arxiv.org/abs/2407.11249Evidence is circumstantial: Brave Search added to Anthropic’s subprocessor list, a BraveSearchParams parameter in the web-search function, and identical citations returned by both. Anthropic has not confirmed it.
techcrunch.com/2025/03/21/anthropic-appears-to-be-using-brave-to-power-web-searches-for-its-claude-chatbot/Series C led by Lightspeed Venture Partners; total raised about $155 million.
fortune.com/2026/02/24/exclusive-as-ai-threatens-search-profound-raises-96-million-to-help-brands-stay-visible/