The local AI glossary

The numbers a model is made of. “30B” means 30 billion of them.

Training sets the parameters; running the model multiplies your input through them. The count is the first guide to both quality and cost: every parameter has to sit in memory, so it decides whether a model fits on your machine before anything else does. “Weights” is the same thing, usually meaning the file that holds them.

You’ll see7B30B480B

The lab publishes the trained weights, so you can download and run the model yourself.

Open-weight is not the same as open source. You get the finished model, usually under a licence that says what you may do with it, but rarely the training data or code. For running locally, the weights are what matters. See licence.

A model that uses every parameter for every token.

Dense models are the simple case: a 24B dense model reads all 24 billion weights to write each token. That makes them predictable but slow for their size, because writing speed is limited by how fast those weights come out of memory. See memory bandwidth.

A model split into many small “experts”, of which only a few run for each token.

A small router picks which experts handle each token, layer by layer. The model needs memory for all of them but only reads the chosen few, so it writes at the speed of a much smaller model. Qwen3-Coder-30B-A3B stores 30.5B parameters and uses about 3.4B per token. The experts are not subject specialists; the split is learned and mostly invisible.

How many parameters a mixture-of-experts model uses per token. The “A3B” in a name.

“30B-A3B” reads as 30 billion in total, 3 billion active. The two numbers answer different questions: the total sets how much memory the model needs, the active count sets how fast it writes. A model with few active parameters is fast, but it has less to think with on each token than a dense model of the same total size.

You’ll see30B-A3B480B-A35B120B-A12B

The version tuned to follow instructions and hold a conversation.

A lab first trains a base model, which only continues text. It then tunes that into an instruct model that answers questions, follows a chat template and calls tools. For a coding agent you always want the instruct version. Gemma names it -it.

You’ll see-Instruct-it-Base

Further training of an existing model for one kind of work.

A coder model is usually a general model fine-tuned on code and agent runs. LoRA is a cheap way to fine-tune that trains a small add-on instead of every weight. Community fine-tunes on Hugging Face vary a lot in quality; the lab’s own releases are the safe default.

You’ll see-CoderLoRAadapter

Training a small model to imitate a larger one.

The small model learns from the large model’s outputs instead of only from raw text, so it punches above its size. A name like “R1-Distill-Qwen-32B” means a Qwen model taught by R1. It does not mean a shrunken copy of R1.

The model also takes images, and sometimes audio or video.

For coding it means you can paste a screenshot of a broken layout. The vision part adds a little to the file size. Some runtimes need a separate projector file, often named mmproj, to use it.

You’ll seemmproj-VL

The lab’s page for a model: what it is for, its scores, its settings and its licence.

On Hugging Face it is the README of the model’s repository. Read it for the recommended sampling settings settings, the context length and how tool calling is set up. Those details decide whether the model works in your setup.

The strongest models available at the moment, usually only through an API.

Claude, GPT and Gemini at their top tiers. Open-weight models trail them, but the gap on coding has narrowed. A model whose card claims frontier-level scores is worth checking on your own tasks before you believe it.

The unit models read and write. About three quarters of an English word.

A tokenizer splits text into common chunks: “parse_args” might be three tokens. Code and non-English text take more tokens per word than prose. Context limits, speeds and API prices are all counted in tokens.

The most tokens a model can see at once: instructions, files, tool output and the conversation.

A coding agent fills it fast. Its system prompt and tool definitions take 10–20k tokens before you type anything, and every file it reads stays in context. Runtimes often load a model with a far smaller window than it supports, and the overflow is silently cut. Check what yours sets. The window you can afford is limited by the KV cache, not only by the model.

You’ll seenum_ctx-c 65536--ctx-size256k

A way to stretch a model’s context window past what it was trained on.

Models learn word positions through RoPE. YaRN rescales it so a model trained on 256k can run at 1M. The card lists the native length and the extended one separately for a reason: quality past native length is lower, and some labs warn that turning YaRN on hurts short prompts. Only enable it when you need the room.

You’ll seerope_scaling--rope-scaling yarn

The part of a model that lets each token look back at every earlier token.

Attention is why a model can use what it read 50k tokens ago, and why long context is expensive: every new token looks at everything before it. Grouped-query attention (GQA) shares one set of keys and values between several heads, which shrinks the KV cache four- to eightfold. Almost every current model uses it.

Memory a model uses to hold the context it has already read. It grows with every token.

Without it the model would reread the whole conversation for every token it writes. The cost per token depends on the model’s attention design: Devstral-Small-2-24B keeps 160 KiB per token, Qwen3-Coder-30B-A3B 96 KiB. At 256k tokens that is 40.0 GiB and 24.0 GiB, on top of the weights. Long context costs memory as well as time.

You’ll see--cache-type-k q8_0OLLAMA_KV_CACHE_TYPE

Storing the KV cache at lower precision to fit more context in the same memory.

An 8-bit cache (q8_0) halves the cache with almost no loss. A 4-bit cache saves more, but models handle it less well, and long tool calling sessions show it first. In llama.cpp a quantised V cache needs flash attention.

You’ll see--cache-type-k q8_0--cache-type-v q8_0OLLAMA_KV_CACHE_TYPE=q8_0

Layers that only look at the last few thousand tokens, so their cache stops growing.

Gemma and gpt-oss mix sliding-window layers with a few full-attention layers. The sliding layers keep a fixed-size cache however long the session runs, and the full layers keep the long-range view. The result is a much smaller KV cache at long context.

DeepSeek’s way of compressing keys and values before caching them.

MLA stores one small compressed vector per token and layer instead of full keys and values. DeepSeek, Kimi and GLM use it. The cache per token ends up a fraction of what GQA needs.

Models that replace most attention layers with layers whose memory does not grow with context.

Mamba (a state space model) and Gated DeltaNet (a form of linear attention) keep a fixed-size summary instead of a cache of every token. Nemotron uses Mamba layers and Qwen3.5 onwards uses DeltaNet, each with a few full-attention layers for exact recall. The KV cache shrinks to a fraction, and speed holds up as the context fills.

Attention that only reads the earlier tokens most likely to matter.

A cheap scorer picks a subset of the context for each new token. The cache stays the same size, but reading it costs less, so long-context replies come faster. DeepSeek-V3.2 is the best-known example.

An agent summarising the conversation so far to free up context.

When a session nears the context window limit, Claude Code and other harnesses replace the history with a summary and carry on. It works, but detail is lost. A model with a smaller window compacts more often, so the context your sessions actually reach is a good measure of the window you need.

You’ll see/compact

Memory on a graphics card. A model runs fast only if it fits there.

The weights, the KV cache and a little working space all have to fit. Two cards add up, though splitting a model across them costs some speed. When it does not fit, the rest goes to system RAM through offloading and speed drops sharply.

Apple Silicon’s single pool of memory shared by the CPU and GPU.

It is why a Mac with 64 GB can run models no 24 GB graphics card can. macOS only lets the GPU use part of it: about two thirds, or three quarters above 36 GB. The iogpu.wired_limit_mb setting raises the limit at the cost of memory for everything else.

You’ll seesudo sysctl iogpu.wired_limit_mb=…

Storing weights with fewer bits each, so the model is smaller and faster.

Models are trained at 16 bits per weight. Quantising to around 4 bits makes the file about a quarter of the size, with a loss you would struggle to notice in chat. Agents notice sooner: below about 3.5 bits, tool calls start to come out malformed. Quantisation shrinks the weights; it does nothing for the KV cache.

llama.cpp’s quantisation formats, named by bits and mix: Q4_K_M is about 4.8 bits a weight.

The number is the rough bit count. K is the k-quant method, which groups weights into blocks that share a scale. S, M and L are small, medium and large mixes: M keeps some of the more sensitive tensors at higher precision. Q8_0 is the older, simpler format at 8.5 bits and is close to lossless. Q4_K_M is the usual default.

You’ll seeQ4_K_MQ5_K_SQ6_KQ8_0

Quantisations tuned on sample text to keep the weights that matter most.

An importance matrix records which weights affect the output most on calibration text, and the quantiser keeps those more precisely. I-quants (IQ) and the “dynamic” quants some uploaders publish use it to hold up better at 2 and 3 bits. At 4 bits and above the difference is small.

You’ll seeIQ4_XSi1-Q4_K_MUD-Q4_K_XL

The average storage per weight, and the honest way to compare quantisations.

File size ≈ parameters × bits per weight ÷ 8. A 30.5B model at 4.85 bpw is about 18.5 GB. Quant names round, so two “4-bit” files can differ by a gigabyte.

The 16-bit formats models are trained and released in. “Full precision” in local use.

BF16 has the range of 32-bit floats with less precision, and most labs release in it. F16 is the older 16-bit format. FP8 halves the size again and is common on data-centre GPUs. Locally, 16-bit files are mostly a source for making quantisations.

4-bit formats some models are released in natively, rather than quantised afterwards.

gpt-oss ships in MXFP4. NVFP4 is NVIDIA’s format for its newest GPUs. A model released at 4 bits was trained or tuned for it (see quantisation-aware training), so it loses less than a 4-bit quantisation of a 16-bit model.

Training a model so it expects to be quantised, which cuts the loss when it is.

The model sees the rounding during training and learns around it. Kimi K2 Thinking and some Gemma releases are QAT models. Use the lab’s QAT file rather than quantising the full-precision one yourself.

llama.cpp’s model file: weights, tokenizer and chat template in one file.

GGUF is what Ollama, LM Studio and llama.cpp load. Hugging Face repositories ending in -GGUF hold one file per quantisation; pick the one that fits. Large models are split into numbered parts, which the runtime joins.

You’ll seemodel-Q4_K_M.gguf-00001-of-00003.gguf

Apple’s machine-learning framework, and the model format that goes with it.

MLX runs on Apple Silicon only. LM Studio can use it instead of llama.cpp, and it is often a little faster on a Mac. MLX models are quantised separately, usually to 4, 6 or 8 bits, and live in repositories such as mlx-community.

You’ll seemlx-community/…-4bit

The standard format labs publish full-precision weights in.

It is what vLLM and training tools load, and what GGUF and MLX files are converted from. Unlike older formats, loading one cannot run code.

Running part of a model from system RAM when it does not fit on the GPU.

The layers left in RAM run at RAM speed, which is several times slower. For MoE models there is a better split: keep attention on the GPU and move the expert weights to RAM, since each token only reads a few of them. llama.cpp does this with --n-cpu-moe.

You’ll see-ngl 99--n-gpu-layers--n-cpu-moe 20

Running a trained model to get output, as opposed to training it.

Everything you do with a local model is inference. An “inference engine” is a runtime.

Reading the prompt before the first word of the answer.

Prefill handles many tokens at once, so it is limited by compute, not memory. It is the pause after a coding agent reads a large file. Apple Silicon is much slower at prefill than an NVIDIA card of similar memory speed. That is why a Mac can write quickly but still take a while to start answering. llama-bench reports it as pp512.

You’ll seepp512prompt eval rate

Writing the answer, one token at a time.

For each token the model reads its active weights and the whole KV cache from memory, so decode is limited by memory bandwidth. That is why decode slows as the context fills. llama-bench reports it as tg128.

You’ll seetg128eval rate

The usual speed figure. Without saying which, it means decode speed at short context.

Around 20 tok/s reads comfortably; an agent writing long diffs wants more. Treat a figure without a context length with suspicion: the same model can lose half its speed or more by 100k tokens.

How fast memory can be read. The ceiling on how fast a model can write.

Decode speed can never beat bandwidth ÷ bytes read per token. An M5 Pro reads 307 GB/s and an RTX 4090 1,008 GB/s, so the same model writes about three times faster on the card, if it fits. The bytes read per token are the active weights plus the cache, which is why MoE models and small caches matter so much.

How long you wait before the answer starts.

Mostly prefill. With an agent, every tool result has to be read before the model can reply, so slow prefill shows up again and again. Runtimes that keep the cache between turns only have to read the new part.

Guessing several tokens ahead with something cheap and checking them all at once.

A small draft model, or extra prediction heads built into the model (MTP), proposes the next few tokens. The main model checks them in one pass and keeps the ones it agrees with. The output is the same; it arrives faster when the guesses are good, which in code they often are.

You’ll see--model-draft-md

A faster way to compute attention that uses less working memory.

It gives the same result with fewer trips to memory. Most runtimes turn it on by default where the hardware supports it. In llama.cpp a quantised V cache requires it.

You’ll see-fa onOLLAMA_FLASH_ATTENTION=1

The program that loads a model and serves it: Ollama, LM Studio, llama.cpp, vLLM.

llama.cpp is the engine underneath most local setups. Ollama wraps it with model downloads and a background service. LM Studio adds a desktop app and can also run MLX. vLLM is built for serving many users from NVIDIA GPUs. All of them expose an HTTP API that coding tools connect to.

You’ll seelocalhost:11434localhost:1234llama-server

The layer a runtime uses to talk to your GPU: Metal on Mac, CUDA on NVIDIA, ROCm on AMD.

Vulkan works across vendors and is the fallback for GPUs the others do not support. When a model runs far slower than expected, check that the runtime found the GPU at all. A CPU-only build is the usual cause.

The HTTP interface most runtimes speak, so any tool that talks to OpenAI can talk to them.

Point a tool’s base URL at http://localhost:11434/v1 and it uses your local model. Ollama also speaks Anthropic’s API, so Claude Code can run against it by setting ANTHROPIC_BASE_URL.

You’ll seeOPENAI_BASE_URLANTHROPIC_BASE_URL

The exact text format a model was trained to see conversations in.

Each model marks system, user, assistant and tool turns with its own special tokens. GGUF files carry the template as Jinja code, which the runtime runs. A wrong or outdated template is the most common reason a good model behaves badly locally. gpt-oss uses its own format, called harmony. Template settings such as turning thinking off are passed as chat_template_kwargs.

You’ll see--jinjachat_template_kwargs<|im_start|>

Runtime code that turns a model’s raw output into structured tool calls.

Models write tool calls in their own formats: XML-ish tags, JSON, special tokens. The runtime has to recognise that format and hand the agent a clean call. A reasoning parser does the same for thinking text. When a card says a model needs a recent runtime, this is usually why.

You’ll see--tool-call-parser qwen3_coder--reasoning-parser

The part that splits text into tokens and back. Each model family has its own.

Token counts are not comparable across families: the same file can be 10% more tokens for one model than another. GGUF files include the tokenizer, so you rarely handle it yourself.

Instructions placed before the conversation that set how the model behaves.

A coding agent’s system prompt is long: rules, tool descriptions and your project’s instruction files. It counts against the context window on every request, which is one reason agents need more context than chat.

How the next token is chosen from the model’s candidates.

Temperature sets how adventurous the choice is; low is more predictable. Top-p, top-k and min-p cut unlikely candidates first. Labs publish the settings their model was tuned for, and agents work best with them. Many “this model is bad” reports come down to a runtime default overriding them.

You’ll seetemperature 0.7top_p 0.8--min-p 0.01

Where labs publish models and the community publishes conversions.

A repository like Qwen/Qwen3-Coder-30B-A3B-Instruct holds the lab’s release; one like unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF holds someone else’s GGUF conversions of it. Check who made a conversion before downloading tens of gigabytes of it.

The program around a model that gives it tools and runs the loop: Claude Code, Codex, OpenCode, Cline.

The model only writes text. The harness turns that text into file edits and shell commands, feeds the results back, and decides when to stop. The same model can score very differently in different harnesses, which is why benchmark results name the one they used.

Working in a loop of read, act and check, rather than answering once.

An agentic coding session can run hundreds of tool calls. “Long-horizon” means the model stays on track that whole time. Plenty of models answer one coding question well and then lose the thread on step forty; that is the gap agent benchmarks try to measure.

The model asking the harness to run something: read a file, run tests, search.

The harness describes its tools in the system prompt. The model replies with a structured call instead of prose, and the harness runs it and returns the result. A model that cannot do this reliably cannot drive a coding agent, however well it writes code.

Forcing a model’s output to match a format such as a JSON schema.

The runtime blocks any token that would break the format, so the output always parses. llama.cpp calls these grammars. It helps with tool calls from weaker models, at a small cost in speed.

A standard way to plug extra tools into an agent: a database, GitHub, a browser.

An MCP server is a small program the harness starts or connects to. It lists its tools, and the model can call them like built-in ones. MCP works across harnesses, so servers usually carry over when you switch. Their settings often hold API keys, so treat the config file as a secret.

You’ll see.mcp.jsonmcpServers

Helper agents a main agent starts for a side task, each with a fresh context.

A subagent explores the codebase or runs a check, then returns a short summary. The main context stays small. Harnesses often send subagents to a smaller, faster model.

Packaged instructions an agent loads only when a task needs them.

A skill is a folder with a SKILL.md and any scripts it uses. The agent sees a one-line description of each skill and reads the rest on demand, which keeps the system prompt short.

You’ll seeSKILL.md.claude/skills/

Scripts a harness runs on events, such as before a tool call or when a session ends.

Hooks enforce rules the model might forget: format after every edit, block writes to a folder. Their event names and inputs differ between harnesses, so they carry over less well than instructions or MCP servers.

Markdown files in your repository that tell an agent how the project works.

AGENTS.md is read by most harnesses; Claude Code reads CLAUDE.md, which can import AGENTS.md. Everything in them goes into the system prompt on every request, so short beats thorough.

You’ll seeAGENTS.mdCLAUDE.md

The model writes out its working before the answer.

Thinking improves results on hard problems and costs tokens and time: a short reply can come with thousands of thinking tokens, all written at decode speed. “Non-thinking” models answer straight away and suit fast agent loops.

A setting for how much a model thinks: low, medium, high, sometimes max.

Higher effort scores better and runs slower. On a local machine the cost is wall-clock time, not money. Benchmark scores are usually taken at the highest setting; check which one before comparing.

You’ll seereasoning_effort: high--reasoning-budget

One model that can think or not, switched per request.

Some models take a flag in the chat template (enable_thinking); some older Qwen models also take /think and /no_think in the message. Labs differ, so check the card.

You’ll seeenable_thinking: false/no_think

Keeping the model’s earlier thinking in context across tool calls.

By default many templates drop past thinking to save context. In a long agent loop that means the model forgets its own plan. Models built for agents now offer an option to keep it, and labs recommend turning it on for agent work.

Fixing real GitHub issues in real repositories. The main agentic coding score.

The model gets an issue and a checkout and has to produce a patch that passes hidden tests. Verified is a human-checked set of 500 Python tasks. Pro is newer and harder, covers more languages and is designed to resist contamination. Scores depend heavily on the harness used.

Completing tasks in a real shell: building, configuring, debugging.

Closer to how coding agents are used day to day than SWE-bench, and harder. Results name the harness, such as Terminus or Claude Code.

Editing code to solve 225 exercises in six languages.

It tests whether a model can make correct edits in a fixed edit format across C++, Go, Java, JavaScript, Python and Rust. It is a good check on languages other than Python.

Competition programming problems published after the model was trained.

Using fresh problems limits contamination. It measures algorithmic skill, not working in a codebase. A high score here with no agent scores tells you little about how a model will do in Claude Code.

Scores for calling tools correctly, alone and over a long conversation.

BFCL checks single calls: the right function with the right arguments. τ²-bench runs multi-turn conversations with a simulated user and tools. For agent work they predict more than raw coding scores do.

Tests of whether a model still finds and uses information deep in a long context.

A 1M context window is only useful if the model can use what is in it. These scores, given at a length such as 128k or 1M, are a better guide than the advertised window.

When benchmark questions were in the training data, so the score overstates ability.

Public benchmarks leak into web data, and models can be tuned to their quirks. Newer sets refresh their tasks or keep them private for this reason. The test that counts is your own work.

What the lab lets you do with the weights. Apache 2.0 and MIT allow almost anything.

Most coding models use Apache 2.0 or MIT, which allow commercial use. Others use the lab’s own licence: some restrict companies that serve the model to others, and some forbid commercial use without approval. For running a model on your own machine for your own work, all of them are usually fine. Read the licence before building a product on one.