Local AI Toolkit

How it works

Contents · Русская версия

On this page

Local AI is a few small programs around two things it keeps on your disk: one queue of work for the local models, and one code index per repository. This section follows a request through them.

The pieces

flowchart LR
    agent["Cloud agent<br/>Claude Code, Codex"]
    you["You, in a terminal"]
    launcher["localai-launcher.exe"]
    mcp["codesearch-mcp<br/>locallm-mcp"]
    cli["localai<br/>codesearch"]
    queue[("Job queue")]
    broker["LocalAi.Broker"]
    ollama["Ollama<br/>127.0.0.1:11434"]
    index[("Code index<br/>one per repository")]

    agent -->|MCP| launcher
    you --> launcher
    launcher --> mcp
    launcher --> cli
    mcp -->|"request, then wait"| queue
    cli -->|"request, then wait"| queue
    queue --> broker
    broker --> ollama
    mcp -->|read| index
    cli -->|"read, build"| index

One queue in front of every model

Every model call goes through the same queue — a search query, the embeddings of an index build, a question to ask_local. That is how Claude, Codex and the command line use the models at once without loading them over each other.

Each request carries a priority. A search query is Interactive, a question to a local model is Foreground, and the embeddings of an index build are Background. The broker picks the next job by priority first, then by whether its model is already in video memory, then by the shortest expected job:

Two identical requests that arrive while the first is still queued or running become one job.

By default a model has to fit entirely in video memory, and one that does not is refused rather than run slowly. localai policy set --residency relaxes that, and an answer produced under a relaxed policy names its shortfall beside the model.

The code index

localai sync builds it — whether you run it, a Git hook runs it, or the agent calls index_refresh — in this order:

flowchart TD
    plan["Planning<br/>the mainline commit, and what changed"]
    semantic["SemanticBase<br/>definitions and references:<br/>Roslyn for C#, scip-typescript, scip-python"]
    embed["EmbeddingBase<br/>cut into chunks, embedded through the broker"]
    publish["PublishingGeneration<br/>base.cidx and semantic.sidx"]
    overlay["SemanticOverlay, EmbeddingOverlay<br/>for each worktree: what differs from the base"]
    switch["Publishing<br/>current.json moves to the new generation"]
    plan --> semantic --> embed --> publish --> overlay --> switch

On disk it lives under %LOCALAPPDATA%\LocalAi\repositories\<id>. The id comes from the repository’s Git common directory, so every worktree and every client share it.

Path What it holds
current.json which generation is live
generations\<generation>\ base.cidx, semantic.sidx, and manifest.json with their checksums
overlays\<generation>\<worktree>\<tree>\ one overlay per worktree state; clean when nothing is uncommitted
staging\ a build in progress, and the checkpoint it resumes from

A search, step by step

sequenceDiagram
    participant A as Cloud agent
    participant S as codesearch-mcp
    participant B as Broker
    participant O as Ollama
    A->>S: search_code("who retries a failed download")
    S->>S: load the base, and the overlay for this worktree
    S->>B: embed the query (Interactive)
    B->>O: the embedding model the index was built with
    O-->>S: the query's vector
    S->>S: rank by meaning, rank by name, fuse the two rankings
    S-->>A: hits: path:lines, symbol, snippet, chunk_id

Which model does what

model-routing.json holds the routes; the candidates are tried first, the fallbacks after.

Profile Used by Candidates Fallbacks
VectorEmbedding the index and search queries qwen3-embedding:8b-q8_0 qwen3-embedding:8b-q8_0
ShortSummary ask_local, by default qwen3.5:9b, qwen2.5-coder:14b qwen2.5-coder:14b, qwen3.5:9b
Extraction ask_local qwen3.5:9b, qwen2.5-coder:14b qwen2.5-coder:14b, qwen3.5:9b
Classification ask_local qwen2.5-coder:14b, qwen3.5:9b qwen3.5:9b, qwen2.5-coder:14b
CodeAnalysis, CodeReview ask_local gpt-oss:20b, qwen2.5-coder:14b qwen3.5:9b
MultiFileSynthesis, Planning ask_local gpt-oss:20b, qwen3.5:9b, qwen2.5-coder:14b qwen3.5:9b, qwen2.5-coder:14b
LogTriage triage_log qwen2.5-coder:14b qwen3.5:9b
VisualAnalysis read_image qwen3-vl:8b-instruct-q8_0 qwen3.5:9b
Ocr read_image qwen3-vl:8b-instruct-q8_0, glm-ocr:bf16 qwen3-vl:8b-instruct-q8_0, qwen3.5:9b
ImageTranslation read_image qwen3-vl:8b-instruct-q8_0 translategemma:12b, qwen3.5:9b
PlainTranslation translate_local, plain text translategemma:12b qwen3.5:9b
TechnicalTranslation translate_local, markdown translategemma:12b qwen2.5-coder:14b, qwen3.5:9b

How the saving is counted

Compare against the necessary context of the request, not entire candidate files. The runtime requires an explicit task-input comparison basis before crediting file context; retrieval alone does not establish that baseline. Previously read material is excluded, and missing evidence produces an unknown estimate.

For an explicit text scenario, necessary unseen source context S, the full visible tool response V and request R give the conservative comparison S - V - 2R. Local answer generation is not automatically credited as avoided cloud generation. An explicit context is an assumption supplied by the caller, not proof of the smallest possible cloud input.

sourceContext: AlreadyPresent excludes source credit; Unknown does not pretend it was unseen. Skip a redundant local call when its remaining benefit cannot cover its overhead. See Understanding token estimates, including examples and historical-runtime limits. This is an estimate, not a meter of billed tokens.

What the installer does

flowchart LR
    release["Check the release<br/>signed manifest, package hash"]
    prerequisites["Prerequisites you ticked<br/>winget, npm"]
    package["The package<br/>a new version, then the switch"]
    models["Models you chose<br/>pulled through the broker"]
    settings["Settings<br/>video memory policy, update check"]
    clients["Clients you chose<br/>MCP servers, instructions, skill"]
    release --> prerequisites --> package --> models --> settings --> clients

← From the command line: not tokens, but time · Contents · Dependencies →