With a cloud agent: what stops costing tokens
On this page
- Per-operation context estimate
- It works by default, and you can see it
- If something does not work the way you want
Per-operation context estimate
After a local model call, the main notice estimates full operation input minus the context returned to the cloud agent. Input includes the observed source material and request instructions. Output includes the complete visible response, its wrappers and notice. Both sides use the same named cloud-token profile; backend prompt/eval counters are separate diagnostics.
This hypothetical comparison treats the supplied material as input even when sourceContext is AlreadyPresent or Unknown. That flag remains truthful diagnostic metadata. The estimate does not justify redundant local calls. Retries and overlapping source ranges do not earn duplicate credit. Negative and zero differences are valid results.
Usage schema 2 adds contextEstimate with inputTokens, returnedTokens, savedTokens, methodology and tokenizer/vision profiles. Historical schema 1 records and the original cloudTokens comparisons keep their meanings. The original comparison fields described below remain available for diagnostics; the main notice prefers the new estimate after a model call.
The CLI JSON fields savedTokensEstimate and translation’s netContextTokensSavedEstimate use this same difference, including negative values. They are null when the estimate is unavailable or outside their legacy 32-bit range; the usage report retains the full value. The final JSON and notice are counted together before delivery, while progress remains live.
The count measures a prepared visible response when delivery is not yet established. Failed or unattempted delivery leaves the returned count and difference unknown. Missing/conflicting source evidence and unsupported image accounting remain explicit unknowns, not fabricated zeroes. Administrative calls and model loading without a task do not gain this estimate. A numerical difference is not proof that the model’s answer is correct and is not measured provider billing.
It is not only about volume. A question asked in words rarely maps onto a keyword — and a keyword is exactly what an agent is reduced to searching for when there is no index.
Measured on this repository on 2026-09-06. The question “who is subscribed to a broker job finishing”:
- Grepping for
Subscribefinds one file undersrc, and it is not the answer. search_codereturnsIBrokerQueue.CompleteAsyncandBrokerJobPending— neither of which contains the word “subscribe” anywhere. It matched by meaning.
| Question | Without Local AI: where the tokens go | With Local AI: where the tokens go |
|---|---|---|
| “Who is subscribed to a job finishing?” | locate the relevant declarations and inspect enough surrounding context; the required volume depends on the question | three fragments, with paths and line ranges |
| “How does the order pipeline work?” | inspect the relevant pipeline definitions and relationships | a fragment per link in the chain, each with its place |
| “What does this codebase do?” | inspect representative entry points and documentation | use the index to locate them, then request a bounded overview |
The difference is not that the agent reads faster. It is that only the fragments that matched enter its context, and the hunt through candidates is paid for by your GPU rather than by your subscription.
It works by default, and you can see it
Nothing to switch on. The installer tells the agent these tools exist and that they come
first. From there the agent decides for itself: it sees a “where does this live” question and
reaches for search_code instead of opening files.
Runtime token reports use one shared record per call. A positive figure is a conditional estimate against the necessary context for that request, not the total size of files read locally. Unknown evidence stays unknown. A negative comparison is displayed as an amount of additional tokens, rather than a negative savings counter. Administrative calls are inapplicable. This is not account usage or a bill. See Understanding token estimates for the assumptions, a worked example and the difference between installed runtime versions.
localai sync announces each indexing phase and repeats its current status every 15 seconds.
Known work reports completed/total and a percentage for that phase; embeddings also report
rate and ETA. Work with no known total reports its activity and elapsed time, without an
invented percentage. localai repo status --follow uses the same phase descriptions.
The shared method compares cloud input and output separately. It requires an explicit
alternative reading scenario, source provenance, the complete returned payload, and a named
tokenizer or vision profile. Previously retrieved text and overlapping chunks are not counted
twice. Missing evidence produces unknown with a reason; unsupported image profiles do not
fall back to a universal area formula.
Existing numeric response fields retain their JSON shapes. New broker attempt records use
schema 2, with attempt/workflow identities, observed backend counts where available, and the
residency proof of the load the attempt ran on. A model’s offload in the call report comes
from its own attempt’s proof or, when the attempt carries none, from the job receipt’s proof
for the same model and context. Without either it is unknown with a reason:
no_residency_check for a job whose receipt carries no routing, such as embeddings, and
not_recorded otherwise. A proof made for another model or context is never used.
Attempts cannot prove delivery or cloud savings: their versioned value is unknown, and
their legacy scalar savings fields are zero. Older telemetry remains readable; its unsupported
scalar estimates are excluded from totals. Final call reports are stored separately, once
per call, and account for the complete returned volume. Multiple jobs in one call do not
multiply its savings. Incompatible comparison profiles are not summed. localai prune
applies the telemetry retention policy to both attempts and final reports.
MCP results carry schema-versioned data in _meta["localai/usage"]. Text tools include the
notice outside untrusted-content boundaries; JSON tools keep their content unchanged as the
first block and add the notice after it as a separate text block, with the same text in
_meta["localai/usageNotice"]. Cancellation remains protocol cancellation. Supply a unique
GUID in request _meta["localai/callId"] before starting a cancellable call, then retrieve
the partial report with localai telemetry --call-id <guid>.
CLI stdout keeps its existing text or JSON contract; the compact report goes to stderr.
The installer’s version-one localai model protocol permits no extra stderr records other
than the reason a command failed with broker_failure, so those reports are only persisted.
Reports are stored at %LOCALAPPDATA%\LocalAi\telemetry\usage; they contain technical
measurements and identifiers, never prompts or document contents.
A storage failure preserves the command result and is reported separately where the protocol
permits it.
Response size counts the full serialized MCP result, including its notice and metadata,
or all CLI stdout/stderr text in UTF-8. JSON-RPC framing is outside this boundary. The
self-inclusive MCP calculation is bounded; nonconvergence produces an unknown size.
Delivery records a completed file write or handoff to the output surface, not acknowledgement
by a remote client. Missing measurements remain unknown; partial reports describe the
evidence available at cancellation. An index_memory_cache hit describes cached index data
and does not imply that inference was skipped.
The synthetic calibration corpus has
reference counts and comparison results
from tiktoken 0.12.0 for cl100k_base and o200k_base. It covers EN/RU, code, mixed text,
an empty input, a long response, source/path/outPath, prior and repeated reads, PNGs, and a
two-page PDF. Error statistics describe this corpus and these encodings; they do not establish
a universal tokenizer or provider-reported usage. The long sample repeats shorter fixtures
to exercise response size, not to add independent language coverage.
Image references use the documented gpt-4o tile formula and are labeled formula estimates.
PDF references include text counts but leave the full input unknown because provider page
rendering is not observed. Unsupported vision profiles remain unknown at runtime. Character
estimates have no statistical interval; the search evaluator’s existing numeric bounds are
explicitly labeled heuristic_not_confidence_interval in the versioned
responseTokenEstimatePolicy companion. Calibration does not replace comparison scenarios,
source provenance, delivery evidence, or complete response accounting.
To reproduce the references, install tiktoken==0.12.0 and run
python tools/calibrate-tokens.py --verify. tiktoken may download its encoding tables on
first use; the script calls no model or provider API. --write deliberately
regenerates the synthetic assets and reference JSON. The normal .NET suite checks the retained
measurements, scenario arithmetic and report privacy without Python, a network connection, or a GPU.
If something does not work the way you want
Nothing is nailed down — the settings live in %LOCALAPPDATA%\LocalAi\settings:
| What to change | How |
|---|---|
| The model for a kind of task | model-routing.json — its own chain of models per profile: CodeAnalysis, LogTriage, VisualAnalysis, PlainTranslation and a dozen more |
| The model for a single call | localai ask "…" <file> --model <model>, or --profile <profile> |
| A model’s fate after a trial | local_model_feedback: promote it, keep it as a fallback only, or disable it for that task |
| Where a model is allowed to sit | localai policy set --residency — wholly in video memory, with offload, or on the processor |
| How long a model stays loaded | localai policy set --idle-model-keep-alive-seconds |
| Take the tools away from a client | unregister the MCP servers from Claude or Codex — the agent then simply does not see them |
To choose models for each task, copy the complete model-routing.json
from the repository root to %LOCALAPPDATA%\LocalAi\settings\model-routing.json.
Edit Candidates and Fallbacks in the relevant Routes entry; declare any added
model in Models. This file replaces the complete catalog. Keep SchemaVersion at
1, a non-empty CatalogVersion, and one route for every task profile.
Without this file, Local AI reads the built-in catalog. An invalid file reports an
error instead of falling back to defaults. Run localai doctor to check the file
and see its calculated version: custom- followed by a content hash. Formatting
changes do not change that version. Use localai policy show to inspect the other
policies.
Translation retries use the ordered Fallbacks for PlainTranslation or
TechnicalTranslation from one catalog snapshot. Every fragment carries that
snapshot’s version; a broker with a different catalog rejects it before loading
or running a model. Apply the saved catalog as described below, then retry.
To apply the saved catalog, let active local calls finish and run localai doctor.
If its model-catalog check passes, run localai-launcher.exe stop. This stops
the broker and all running Local AI tools belonging to the active installed version,
including its MCP servers. Reconnect the MCP servers in your client, or restart
the client if needed. The next local task starts the broker with the saved catalog.
The limits, stated plainly — and who closes them. Local models are markedly weaker than cloud ones. They are good at list this, summarise that, where does this live — and unreliable on architecture and subtle bugs.
Paired with an agent, though, none of that is left to luck: the local model finds, the cloud one judges. A local answer reaches the agent as material rather than as truth — with paths and line ranges it uses to open the source and check for itself. A local model’s mistake costs one extra read, not a wrong conclusion.
The product helps: every answer is signed with its model and its duration, and one produced without enough video memory is marked, so a degraded answer cannot pass for a healthy one.
On the command line that second layer is missing: there, you are the judge. So verify anything a decision rests on, in the code.
← Code search is not text search · Contents · From the command line: not tokens, but time →