What Is Agent Monitoring?
Agent monitoring is the practice of recording what a run did — the steps it took, the calls it made, what each returned, why it stopped and what it cost — and reading those records afterwards, both in aggregate and one at a time. The unit of observation is the run rather than the request, and that is the whole of the difference from ordinary application monitoring: a failure here is a sequence, and no individual request within it need look wrong.
What is recorded, and the shape it is recorded in
The record of a run is a tree rather than a line. A run contains steps; a step contains a request and whatever calls it issued; a call may itself be an entire nested run. The representation that survives this is a trace of nested spans, each with a start, a duration, a parent and its own attributes, because that is the only shape in which a wrong final result can be walked back to the pass where it entered. A flat log of requests loses the parentage, and the parentage is the thing being diagnosed.
On each span belong the plain operational facts: which step it was, what was called, what came back, what it cost in tokens and in time, whether it raised an error, and how many times it was retried. On the run as a whole belong the totals, the step count, an identifier tying it to whatever request or schedule started it, and the exit reason. The exit reason earns its place because it separates the runs that finished from the runs that were stopped, and no aggregate over latency or error rate makes that distinction at all. It is cheap to record and it is frequently the field that is missing.
What makes this hard is that the interesting quantities are not errors. A run in which every call succeeded, nothing threw, and a well-formed answer came back is a run with nothing for a conventional error-rate monitor to notice — and it can still be a run that took forty passes to do what it usually does in four, or one that hit its bound and returned its working. Monitoring an agent means monitoring the shape of a run, and shape is made of quantities that are unremarkable at any particular value.
| Signal | What a shift in it points to |
|---|---|
| Steps per run | Work now taking more passes than the same work used to take. |
| Exit reason | Runs ending at a bound rather than at a completion signal. |
| Repeated identical calls | Passes that are not advancing the run. |
| Error rate per tool | A dependency that has changed underneath the run, which the run will route around quietly. |
| Tokens and spend per run | Context accumulating faster than before, usually visible here before anywhere else. |
| Time to first action | A change in how the request is assembled rather than in the work itself. |
None of these is an error and none of them is wrong at any particular value. Each supports one comparison — this signal against the same signal last week — which is why an agent monitor is built around distributions and individual traces together. The distribution establishes that something changed; only the trace says what.
What monitoring cannot see, and what it collects while looking
A monitor observes the run. It does not observe the outcome. Whether the answer produced was correct, whether the message sent was the one that should have been sent, whether the file is better for the edit — none of that is in a trace, and no amount of further instrumentation puts it there. That gap is closed by something else entirely: a person reading a sample, a check run against a held-out set of cases with known answers, or a signal arriving later from whatever consumed the output. Monitoring establishes what happened. Judging whether it should have is a different activity with a different apparatus.
The second limit is that a monitor is evidence only for the runs it kept. Sampling and retention are chosen for cost, before anyone knows which run will turn out to matter, and they decide in advance which incidents will be diagnosable. A trace discarded at midnight is not a trace anybody can consult about the complaint that arrives on Tuesday, and the usual compromise — keep everything briefly, keep failures longer, keep aggregates indefinitely — is a compromise rather than a solution, since the run that needs explaining is often one that was recorded as a success.
The third is an obligation rather than a limit. A full trace of an agent run contains everything the run was given and everything it produced: the operator's task, the documents it read, the arguments it passed, the text it wrote. That is a store of other people's material, created for an operational purpose and retained by default for as long as nobody sets a policy. Recording a run and retaining a run are two decisions, and the second is a data-protection decision under whatever name the tool gives it. This registry's own position is unusually easy to state, because there is nothing to weigh: the task description a buyer submits is hashed and discarded, so no run of it exists to retain.
How the Registry classifies a run-monitoring task
Recording each step, call and stopping reason from a run and alerting on runs that exceed a step budget is filed as data analysis in this registry's classification. The deliverable is a judgement drawn from records that already exist — a threshold crossed, a comparison made, a signal raised — rather than an artefact produced or a message sent, which is what separates this class from the run being monitored.
The Registry's canonical brief for this entry is filed as:
Record each step, tool call and stopping reason from an agent run and alert on runs that exceed a step budget.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
e146341f418f3612…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about agent monitoring
Which signals matter for a given run shape, what a normal step count is for it, and where a threshold belongs are properties of one deployment measured over its own history, and the canonical brief above supplies none of them. This registry monitors no run, receives no trace, and compares nothing it has issued against what a run afterwards did.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about agent monitoring
- What is agent monitoring?
- Agent monitoring is recording what a run did — its steps, its calls and their results, its cost, and the reason it stopped — and reading those records afterwards, in aggregate to detect a change and individually to explain one. The unit is the run rather than the request.
- How is it different from ordinary application monitoring?
- The unit and the signals both change. A failure is a sequence rather than a request, so the record has to preserve the parentage between steps and calls. And the informative quantities are not errors: step counts, exit reasons and cost per run shift while every individual request still returns successfully.
- What is a trace in this context?
- A trace is the record of one run as a tree of nested spans — the run, its steps, the calls each step made, and any nested run underneath those — each span carrying its own timing, inputs, outputs and errors. The nesting is what allows a wrong final result to be walked back to the point where it entered.
- Which signal is worth recording first?
- The exit reason, because it distinguishes a run that finished from a run that was stopped by a bound, and nothing else in a conventional monitoring set makes that distinction. It costs one field and it changes how every other number is read.
- Does monitoring tell you whether a run was correct?
- No. It describes what the run did, not whether that was the right thing to do. Correctness is established by evaluation against cases with known answers, by human review of a sample, or by a downstream signal — all of which are separate from the trace and none of which the trace can substitute for.
- What should not be kept in a trace?
- That is a retention decision rather than a technical one, and it has to be taken deliberately, because the default is to keep everything a run touched. A full trace holds the submitted task, the documents read and the text produced, so whatever policy governs that material governs the trace as well.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.