Local AI Toolkit

With a cloud agent: what stops costing tokens

Contents · Русская версия

On this page

Per-operation context estimate

After a local model call, the main notice estimates full operation input minus the context returned to the cloud agent. Input includes the observed source material and request instructions. Output includes the complete visible response, its wrappers and notice. Both sides use the same named cloud-token profile; backend prompt/eval counters are separate diagnostics.

This hypothetical comparison treats the supplied material as input even when sourceContext is AlreadyPresent or Unknown. That flag remains truthful diagnostic metadata. The estimate does not justify redundant local calls. Retries and overlapping source ranges do not earn duplicate credit. Negative and zero differences are valid results.

Usage schema 2 adds contextEstimate with inputTokens, returnedTokens, savedTokens, methodology and tokenizer/vision profiles. Historical schema 1 records and the original cloudTokens comparisons keep their meanings. The original comparison fields described below remain available for diagnostics; the main notice prefers the new estimate after a model call.

The CLI JSON fields savedTokensEstimate and translation’s netContextTokensSavedEstimate use this same difference, including negative values. They are null when the estimate is unavailable or outside their legacy 32-bit range; the usage report retains the full value. The final JSON and notice are counted together before delivery, while progress remains live.

The count measures a prepared visible response when delivery is not yet established. Failed or unattempted delivery leaves the returned count and difference unknown. Missing/conflicting source evidence and unsupported image accounting remain explicit unknowns, not fabricated zeroes. Administrative calls and model loading without a task do not gain this estimate. A numerical difference is not proof that the model’s answer is correct and is not measured provider billing.

It is not only about volume. A question asked in words rarely maps onto a keyword — and a keyword is exactly what an agent is reduced to searching for when there is no index.

Measured on this repository on 2026-09-06. The question “who is subscribed to a broker job finishing”:

Question Without Local AI: where the tokens go With Local AI: where the tokens go
“Who is subscribed to a job finishing?” locate the relevant declarations and inspect enough surrounding context; the required volume depends on the question three fragments, with paths and line ranges
“How does the order pipeline work?” inspect the relevant pipeline definitions and relationships a fragment per link in the chain, each with its place
“What does this codebase do?” inspect representative entry points and documentation use the index to locate them, then request a bounded overview

The difference is not that the agent reads faster. It is that only the fragments that matched enter its context, and the hunt through candidates is paid for by your GPU rather than by your subscription.

It works by default, and you can see it

Nothing to switch on. The installer tells the agent these tools exist and that they come first. From there the agent decides for itself: it sees a “where does this live” question and reaches for search_code instead of opening files.

Runtime token reports use one shared record per call. A positive figure is a conditional estimate against the necessary context for that request, not the total size of files read locally. Unknown evidence stays unknown. A negative comparison is displayed as an amount of additional tokens, rather than a negative savings counter. Administrative calls are inapplicable. This is not account usage or a bill. See Understanding token estimates for the assumptions, a worked example and the difference between installed runtime versions.

localai sync announces each indexing phase and repeats its current status every 15 seconds. Known work reports completed/total and a percentage for that phase; embeddings also report rate and ETA. Work with no known total reports its activity and elapsed time, without an invented percentage. localai repo status --follow uses the same phase descriptions. The shared method compares cloud input and output separately. It requires an explicit alternative reading scenario, source provenance, the complete returned payload, and a named tokenizer or vision profile. Previously retrieved text and overlapping chunks are not counted twice. Missing evidence produces unknown with a reason; unsupported image profiles do not fall back to a universal area formula.

Existing numeric response fields retain their JSON shapes. New broker attempt records use schema 2, with attempt/workflow identities, observed backend counts where available, and the residency proof of the load the attempt ran on. A model’s offload in the call report comes from its own attempt’s proof or, when the attempt carries none, from the job receipt’s proof for the same model and context. Without either it is unknown with a reason: no_residency_check for a job whose receipt carries no routing, such as embeddings, and not_recorded otherwise. A proof made for another model or context is never used. Attempts cannot prove delivery or cloud savings: their versioned value is unknown, and their legacy scalar savings fields are zero. Older telemetry remains readable; its unsupported scalar estimates are excluded from totals. Final call reports are stored separately, once per call, and account for the complete returned volume. Multiple jobs in one call do not multiply its savings. Incompatible comparison profiles are not summed. localai prune applies the telemetry retention policy to both attempts and final reports.

MCP results carry schema-versioned data in _meta["localai/usage"]. Text tools include the notice outside untrusted-content boundaries; JSON tools keep their content unchanged as the first block and add the notice after it as a separate text block, with the same text in _meta["localai/usageNotice"]. Cancellation remains protocol cancellation. Supply a unique GUID in request _meta["localai/callId"] before starting a cancellable call, then retrieve the partial report with localai telemetry --call-id <guid>.

CLI stdout keeps its existing text or JSON contract; the compact report goes to stderr. The installer’s version-one localai model protocol permits no extra stderr records other than the reason a command failed with broker_failure, so those reports are only persisted. Reports are stored at %LOCALAPPDATA%\LocalAi\telemetry\usage; they contain technical measurements and identifiers, never prompts or document contents. A storage failure preserves the command result and is reported separately where the protocol permits it.

Response size counts the full serialized MCP result, including its notice and metadata, or all CLI stdout/stderr text in UTF-8. JSON-RPC framing is outside this boundary. The self-inclusive MCP calculation is bounded; nonconvergence produces an unknown size. Delivery records a completed file write or handoff to the output surface, not acknowledgement by a remote client. Missing measurements remain unknown; partial reports describe the evidence available at cancellation. An index_memory_cache hit describes cached index data and does not imply that inference was skipped.

The synthetic calibration corpus has reference counts and comparison results from tiktoken 0.12.0 for cl100k_base and o200k_base. It covers EN/RU, code, mixed text, an empty input, a long response, source/path/outPath, prior and repeated reads, PNGs, and a two-page PDF. Error statistics describe this corpus and these encodings; they do not establish a universal tokenizer or provider-reported usage. The long sample repeats shorter fixtures to exercise response size, not to add independent language coverage.

Image references use the documented gpt-4o tile formula and are labeled formula estimates. PDF references include text counts but leave the full input unknown because provider page rendering is not observed. Unsupported vision profiles remain unknown at runtime. Character estimates have no statistical interval; the search evaluator’s existing numeric bounds are explicitly labeled heuristic_not_confidence_interval in the versioned responseTokenEstimatePolicy companion. Calibration does not replace comparison scenarios, source provenance, delivery evidence, or complete response accounting.

To reproduce the references, install tiktoken==0.12.0 and run python tools/calibrate-tokens.py --verify. tiktoken may download its encoding tables on first use; the script calls no model or provider API. --write deliberately regenerates the synthetic assets and reference JSON. The normal .NET suite checks the retained measurements, scenario arithmetic and report privacy without Python, a network connection, or a GPU.

If something does not work the way you want

Nothing is nailed down — the settings live in %LOCALAPPDATA%\LocalAi\settings:

What to change How
The model for a kind of task model-routing.json — its own chain of models per profile: CodeAnalysis, LogTriage, VisualAnalysis, PlainTranslation and a dozen more
The model for a single call localai ask "…" <file> --model <model>, or --profile <profile>
A model’s fate after a trial local_model_feedback: promote it, keep it as a fallback only, or disable it for that task
Where a model is allowed to sit localai policy set --residency — wholly in video memory, with offload, or on the processor
How long a model stays loaded localai policy set --idle-model-keep-alive-seconds
Take the tools away from a client unregister the MCP servers from Claude or Codex — the agent then simply does not see them

To choose models for each task, copy the complete model-routing.json from the repository root to %LOCALAPPDATA%\LocalAi\settings\model-routing.json. Edit Candidates and Fallbacks in the relevant Routes entry; declare any added model in Models. This file replaces the complete catalog. Keep SchemaVersion at 1, a non-empty CatalogVersion, and one route for every task profile.

Without this file, Local AI reads the built-in catalog. An invalid file reports an error instead of falling back to defaults. Run localai doctor to check the file and see its calculated version: custom- followed by a content hash. Formatting changes do not change that version. Use localai policy show to inspect the other policies.

Translation retries use the ordered Fallbacks for PlainTranslation or TechnicalTranslation from one catalog snapshot. Every fragment carries that snapshot’s version; a broker with a different catalog rejects it before loading or running a model. Apply the saved catalog as described below, then retry.

To apply the saved catalog, let active local calls finish and run localai doctor. If its model-catalog check passes, run localai-launcher.exe stop. This stops the broker and all running Local AI tools belonging to the active installed version, including its MCP servers. Reconnect the MCP servers in your client, or restart the client if needed. The next local task starts the broker with the saved catalog.

The limits, stated plainly — and who closes them. Local models are markedly weaker than cloud ones. They are good at list this, summarise that, where does this live — and unreliable on architecture and subtle bugs.

Paired with an agent, though, none of that is left to luck: the local model finds, the cloud one judges. A local answer reaches the agent as material rather than as truth — with paths and line ranges it uses to open the source and check for itself. A local model’s mistake costs one extra read, not a wrong conclusion.

The product helps: every answer is signed with its model and its duration, and one produced without enough video memory is marked, so a degraded answer cannot pass for a healthy one.

On the command line that second layer is missing: there, you are the judge. So verify anything a decision rests on, in the code.


← Code search is not text search · Contents · From the command line: not tokens, but time →