Managed Operations

Stage seven, the learning loop, is the stage that quietly stops running. Evaluation suites go stale. Sources drift. A policy library gets superseded and the retrieval index does not hear about it. Nobody notices until an agent answers confidently from material that was replaced a year ago — and by then the question of who owns context health has the worst possible answer, which is everyone, meaning no one.

Operationally, most of the month is spent keeping the Core Model true: a system gets added and needs a connector, a team changes a schema and the change log shows the drift before an agent reads it, an incident traces back to a mapping and the mapping is fixed in the model rather than in a prompt. The learning loop is not a report; it is the list of changes made to the model, with reasons.

Managed Operations exists because that stage needs a named owner. It is a monthly engagement, on a quarterly rolling commitment, in two SKUs. Care, at $9,000 a month, keeps the Core Model true: source freshness monitored, new connectors added as systems change, mappings and terminology kept in step, drift caught in the change log before an agent reads it. Ops + Eval, at $18,000 a month, adds what Care does not: the evaluation harness run on schedule against ground truth, every production incident attributed to its layer — context, model, or application — and on-call for the named agent workflows.

It is deliberately the smallest rung by monthly cost and the longest by relationship, because context health is not a project. It is operated infrastructure, and the alternative to operating it is watching a well-built chain drift back to ad hoc over four quarters — a decay we have watched often enough to price the prevention.

• Teams whose go-live went well and who intend for that sentence to still be true in eighteen months
• Organizations whose content estate changes weekly — product catalogs, policy libraries, clinical documentation — where the chain must keep pace with the sources or become a liability with a search box
• Internal platform teams that own the agent runtime but want the learning loop externally owned and externally accountable
• Anyone who has discovered that keeping the eval suite updated ourselves is the most confidently made and least kept promise in production AI

• Care — Core Model stewardship: a system gets added and needs a connector; a team changes a schema and the change log shows the drift before an agent reads it; mappings and terminology are kept in step
• Care — source freshness monitoring: sampling the content estate for supersession and drift, so stale material is found by us before it is found by an agent
• Care — backlog and cadence: a maintained continuous-improvement backlog, groomed monthly, drawn on quarterly
• Ops + Eval — retrieval fitness re-testing: the ground-truth question set run against production retrieval on a schedule, so quality regression is a detected event with a date, not a slow suspicion
• Ops + Eval — evaluation suite maintenance: new failure modes become new test cases; the suite grows with the estate instead of fossilizing at go-live
• Ops + Eval — incident attribution and on-call: production behavior signals triaged, each incident attributed to its actual layer — context, model, or application — with an on-call owner for the named agent workflows. The attribution matters, because the three have different owners and different fixes
• Ops + Eval — learning-loop execution: validated operational findings routed back into sources and structure, with the change recorded. The loop’s output is changes made, not observations filed

ROKA — Return on Knowledge Assets — is the quarterly statement of account: what changed in the chain, what it cost, and what it returned, in terms a business reader can check. Freshness posture by source. Connectors added and drift caught in the change log. On Ops + Eval, the retrieval fitness trend against the ground-truth set and incidents by layer with time-to-attribution. Changes shipped through the learning loop, each traceable from the failure that prompted it to the change that resolved it.

It is written so the executive who funded the chain can see whether it is still earning its keep — and so that if the honest trend line is flat, the report says flat, and the review meeting is about why.

• A context supply chain whose health is measured on a cadence instead of assumed
• An evaluation harness that is alive — growing with the estate, run on schedule, trusted enough to gate changes (Ops + Eval)
• Incident attribution that ends the is-it-the-model-or-is-it-us argument with evidence, each time — and an on-call owner for the named agent workflows (Ops + Eval)
• A ROKA report your finance and leadership can read without translation
• One accountable partner for long-running context work — a named owner for the stage that otherwise has none

A chain built well at go-live degrades on no particular day. Sources turn over, the eval suite stops being run, and two years in, answer quality is visibly worse with no visible cause. Managed operations is priced against exactly this decay curve — the monitoring exists so the drift is caught in month two, not month eighteen.

A commerce platform's catalog changes daily; an insurer's policy library changes with every regulatory cycle. For these estates the chain is never done, and the question is only whether keeping it current is someone's job or everyone's assumption. This rung makes it a job.

A strong internal team runs the agents and wants to keep running them — but wants the learning loop owned by a party whose quarterly report will say flat when the trend is flat. External accountability for stage seven, internal ownership of everything else, is a clean and common split, and it is usually Ops + Eval: the harness and the incident attribution are exactly what the internal team wants externally owned.

• It is not a helpdesk or a NOC for your AI stack. Coverage is the agreed stages of the context supply chain, and on-call on Ops + Eval is for the named agent workflows — model provider incidents, application bugs, and infrastructure outages have their own owners, and the attribution work exists partly to route things to them quickly.
• It is not a substitute for source ownership. Your people still own the content and make the editorial calls; we detect, attribute, route, and verify. A managed loop with no engaged source owners degrades into a very well-documented backlog, and we will say so if we see it happening.
• It is not open-ended. Quarterly rolling means each quarter is a renewal decision made against a ROKA report — which is exactly the discipline we would want if we were the buyer.