Local AI

Seventy-five minutes, five sections, and a live demo in the second one. The rule for today comes from Part I and has not aged: local inference is a decision about data and money, not a personality. Hobbies are allowed. They should be called hobbies.

There are four reasons to run a model on hardware you control, and they are not equal. The first is privacy and sovereignty, and it is the only one that wins arguments on its own. If a specific body of data must not cross an external API boundary, whether because a contract says so, a regulator says so, or the site has no network, then local inference is not a preference, it is the requirement. The test is whether you can name the data. “Sensitive stuff” is not a name; “the claims database” is. The second is economics at volume. A per-token bill is a variable cost; a machine is a fixed cost. At some volume the fixed cost wins. The honest version of this argument needs the volume, the price you would actually pay in the cloud for the same quality, and the cost of the person who keeps the machine running. Section three works through it, and the answer in 2026 is less flattering to local than it was two years ago. The third is latency and offline operation, which is real and specific: ships, mines, clinics, factory floors, places where the network is slow or absent. The fourth is experimentation. Engineers want a local model because every layer is inspectable and swappable, and that is a good reason to give a team a learning budget. It is not a reason for the company to run production on it. Call it a learning budget and nobody is misled.

The original Part II had this list and it is worth keeping, because every one of these sentences will be said in a meeting near you. “No API means no risk” is the most dangerous, because it feels like common sense. What actually happens is that the risk changes shape: instead of trusting a vendor’s contract and security team, you are now running a network service on a machine you patch, loading model files you downloaded from strangers, and deciding who can reach the port. The next slide has the numbers. “We can beat the frontier” is occasionally true for a narrow task with a good test set, and almost never true for general work on a hard day; the right response is not to argue but to measure, which section three covers. “The GPU is already paid for” ignores the only line item that reliably grows: the human hours. “Open weights means open source” is a definitional error with legal consequences. The Open Source Initiative’s definition of open source AI, version 1.0, requires not only the parameters but the complete code used to train and run the system and enough information about the training data that a skilled person could rebuild something substantially equivalent. Almost no open-weight model meets that. What you get is a file and a licence, and the licences differ: Apache 2.0, MIT, and vendor-specific community licences all appear on the model board later in this deck. The licence is usually one page. Read it before you build on it.

Two findings make the point. In September 2025 Cisco’s Talos team used Shodan, the search engine for internet-connected devices, to look for Ollama servers reachable from the public internet. They found over 1,100, more than a thousand of them within ten minutes of starting. About one in five was actively serving a model to anyone who connected, which means free compute for strangers, the ability to query the model repeatedly and reconstruct its behaviour, and a platform for generating whatever the model can be talked into. The other four in five had no model loaded but still accepted uploads and configuration changes, which is arguably worse. None of these were sophisticated attacks; they were default settings on a port nobody closed. Separately, in early 2025 ReversingLabs found two models on Hugging Face whose PyTorch files contained a reverse shell, a payload that connects back to an attacker’s machine. PyTorch’s default format is a Pickle file, and loading a Pickle executes code; these were compressed in a non-standard way so the platform’s scanner, Picklescan, errored before it reached the malicious part. Hugging Face removed them within a day and fixed the scanner. The lesson is not “don’t run local”. It is that a local model is two ordinary things, a network service and a downloaded file, and both have ordinary security rules: bind to localhost, put it behind authentication, prefer safetensors or GGUF files over Pickle, and download from the model’s own organisation. Part VIII goes deeper.

It helps to be precise about what “privacy” buys on each side. A cloud API is a contract. OpenAI’s enterprise privacy page, read in October 2026, says API inputs and outputs are not used to train its models unless you opt in, may be retained for up to thirty days for abuse monitoring, can be put under zero data retention for eligible endpoints, and that the platform is SOC 2 Type 2 audited with business associate agreements available for healthcare. Anthropic publishes the equivalent. That is a strong set of promises, and for most businesses it is enough. What it is not is the data staying in the building: it crosses a wire, it is processed on machines in a jurisdiction you did not choose, and the terms are theirs to change. A local model is a wire that stops at your wall. LM Studio’s documentation is explicit that nothing you type into a chat leaves the device, and that once models are downloaded the application does not need the internet. In exchange you now own everything the vendor used to own: the port, the file, the patching, the access list, the backups and the person who is awake when it breaks. The exposed-server count on the previous slide is what it looks like when people take the wire and skip the ownership. Neither choice is wrong. One of them is a contract you read; the other is a job you staff.

Here is the whole local stack in five words, in the order they happen. Weights are the model: the billions of learned numbers, shipped as a file you download, usually tens of gigabytes. Part I called them the knobs. Quantisation is compression: storing each of those numbers in fewer bits so the file is smaller and runs faster, at some cost to quality that depends on how far you push it and which method you use. “4-bit” is a family, not a setting, and the next slides make that concrete. The runtime is the engine: the program that takes the weights and your hardware and produces tokens. llama.cpp runs almost anywhere; vLLM and SGLang are built for serving many requests on GPUs; MLX is Apple’s framework for its own chips. Speed lives here, and so do most of the failures. The endpoint is the single most useful idea in the deck: a URL on your machine that speaks the same protocol as the cloud APIs, so every application you own can talk to it without knowing or caring what is behind it. The harness is the software body around the model, tools, memory, the loop and permissions, and it has its own deck now, Part VI. The reason the order matters is that each layer can ruin the one above it. Someone who says “Qwen is bad” has usually tested a particular quant, on a particular runtime, at a particular context setting, on particular hardware. Change any of the four and the verdict changes. A model name is a brand, not a specification.

An open-weight model is a file. Qwen3.8-27B, released by Alibaba in August 2026, is a dense model of about 27 billion parameters under the Apache 2.0 licence; stored at 16 bits per parameter that is roughly 54 gigabytes before anyone compresses it. You can download it, run it, inspect it, quantise it, fine-tune it, and never send a token anywhere. What you cannot do is reproduce it. The Open Source Initiative’s definition of open source AI, version 1.0, requires the complete training and inference code and enough information about the training data that a skilled person could build something substantially equivalent. Almost no open-weight release includes that, which is why “open weights” and “open source” are different phrases and why Part I warned you not to conflate them. The licences also differ. Among the models on this deck’s board: Qwen3.8-27B, Gemma 4 and OpenAI’s gpt-oss are Apache 2.0; DeepSeek-V4.1-Flash and GLM-5.3-Flash are MIT; Qwen3.8-Flash-Next carries a Qwen community licence with its own conditions. Each is about a page. Two more words you will meet. Dense means every parameter participates in every token, so memory tracks the parameter count. Mixture of experts means many parameters exist but only a subset is used per token: GLM-5.3-Flash has 320 billion in total and 18 billion active. That makes it fast per token and still enormous to store, because every expert has to be in memory in case it is needed. Part IV explained the mechanism; Part IX covers who makes these models and why that matters to procurement.

Quantisation is the trick that makes local AI possible, and it is worth seeing the numbers once. A model’s weights are stored as numbers, and the number of bits used for each one sets the file size: parameters times bits divided by eight gives bytes. Qwen3.8-27B at the 16 bits it was released in is about 54 gigabytes, which needs a machine with more memory than most people have. At 8 bits it is 27 gigabytes and, in most published measurements, indistinguishable from the original. At 4 bits it is about 13.5 gigabytes, fits on a 16 gigabyte graphics card or a modest Mac, and loses a small, measurable amount of quality. At 2 bits it fits a 12 gigabyte card and is noticeably worse. The one public measurement on this slide comes from a MiaAI_Lab recipe for this exact model, which reports mean KL divergence from the 16-bit original (a measure of how far the compressed model’s predictions drift; lower is closer): 0.007 at 6 bits, 0.05 at 4 bits, 0.35 at 2 bits. That is one measurement by one author with one method, so treat it as shape rather than gospel; the shape is consistent with everything else published. Real downloads run larger than the formula because quantised files mix precisions, keep the vision tower uncompressed, and do not include the context cache, which the appendix covers. The formats you will meet are different engines’ containers for the same idea: GGUF for llama.cpp and the tools built on it, MLX for Apple silicon, EXL3 for aggressive NVIDIA quants, FP8 and NVFP4 for server GPUs. The rule is simple. Start at 4-bit. Go up if the memory allows. Go down only when your own test set says it still passes.

Two tools cover first contact, and both install like ordinary software. LM Studio is a desktop application for Apple-silicon Macs, Windows and Linux. You browse a catalogue, download a model, chat with it, and drop documents into the chat, with no terminal involved. Underneath it runs llama.cpp or, on Macs, Apple’s MLX, but you do not have to know that. Its documentation recommends 16 gigabytes of RAM and says 8 may work with small models. One switch in the Developer tab turns on a local server, by default at localhost port 1234, that speaks the OpenAI and Anthropic API formats; there is also a command-line tool. The docs are specific about privacy: chats and documents do not leave the device, while model search and download call out to Hugging Face. Ollama is the terminal-flavoured equivalent. Its quickstart is one command, ollama run followed by a model name, which downloads the model and opens a chat. It runs as a background service on port 11434 with an OpenAI-compatible endpoint, and its library lists models with download sizes: Gemma 4 at 12 billion parameters is about 8 gigabytes, Qwen3.5 at 9 billion about 7. It has fewer knobs than a serving stack and is also the product behind most of the exposed servers in section one, because binding it to a public interface is a one-line change people make without thinking. Either tool takes a non-engineer from nothing to a running model in about ten minutes, which is what the demo two slides from now does.

When a desktop app is no longer enough, usually because more than one person or one program needs the model at once, there are four engines worth knowing, and one sentence each is genuinely enough for a non-specialist. llama.cpp is a plain C and C++ implementation that runs on nearly everything: CPUs, NVIDIA, AMD, Apple and Intel GPUs, Vulkan, even phones. It uses the GGUF file format, supports quantisation from 1.5 to 8 bits, can split a model between CPU and GPU when it does not fit in video memory, and ships a server with an OpenAI-compatible API. Its hard part is the number of command-line flags. vLLM, which began at UC Berkeley’s Sky Computing Lab, is a GPU serving engine built for throughput: PagedAttention for memory, continuous batching, prefix caching, and an OpenAI-compatible server. Reach for it when many requests arrive at once. SGLang, from the LMSYS organisation, is in the same category with its own tricks, RadixAttention and speculative decoding, and turns up in the more aggressive published recipes for running large mixture-of-experts models on specific hardware. MLX is Apple’s framework for its own chips; mlx-lm loads models from Hugging Face, quantises them and serves them, and is the fast path on a Mac with a lot of unified memory. The one thing all four share with LM Studio and Ollama is the shape of the endpoint. Your applications talk to a URL. You can change everything behind it, and they never find out.

If you remember one technical idea from this deck, make it this one. Every local runtime on the previous two slides exposes an address on your machine that speaks the same protocol the cloud vendors use. LM Studio defaults to localhost port 1234, Ollama to 11434, llama.cpp’s server to 8080, and each offers a /v1 path that an OpenAI-style client library will accept without modification; LM Studio and Ollama also offer Anthropic-compatible paths. The code on the slide is the entire integration: a client library, a base URL pointing at your own machine, and a model identifier copied from the runtime. The consequence is that every harness, workflow tool and script you already use with a cloud vendor can be pointed at a local model by changing one line, and pointed back just as easily. That is what makes the local-plus-frontier architecture in section four possible, and it is why the original Part II called the endpoint the important abstraction. Two cautions. First, the model identifier names what you loaded, which is weights plus quant plus runtime; two endpoints advertising the same name can behave differently, so pin and record what is actually behind it. Second, the API key is a placeholder locally, and the runtimes make it easy to serve on the local network. That is a feature inside a trusted office and the exposed-server study everywhere else. Keep it on localhost, or put a real gate in front of it before anyone else can reach the port.

This slide scripts the demo; the presenter runs it. Step one is installing LM Studio from its website on a Mac with Apple silicon, a Windows machine or Linux; the documentation’s recommended floor is 16 gigabytes of RAM. Step two is downloading a model in the 9 to 12 billion parameter class, which is the sweet spot for a laptop: Qwen3.5 9B or Gemma 4 12B at 4-bit are each roughly 6 to 8 gigabytes according to the Ollama library’s listed sizes for equivalent builds, and this is the only step that needs the internet. Step three is the point of the demo: chat with it, paste in a document and ask a question, then switch the wifi off and ask again. Nothing changes, because nothing was leaving. While it answers, point at the tokens-per-second counter; that number, on this hardware, is what you are buying with a bigger machine. Step four is one switch in the Developer tab, which starts a server at localhost port 1234; opening the address in a browser proves it exists. Step five is the one-line integration from the previous slide: a script or a harness with its base URL changed to the local address, asking the same question and getting the same answer with no cloud involved. Three numbers come out of the demo: the download size, the speed, and the one line in step five. The rest of the deck is about what those three numbers cost and what they are good for. If the office wifi is slow, download the model before the session.

Sizing a box starts with one line of arithmetic: parameters times bits divided by eight gives the bytes needed for the weights. The table applies it to models from this deck’s board. Qwen3.5 at 9 billion parameters is about 18 gigabytes at 16-bit and 4.5 at 4-bit, and the Ollama library lists its 9B downloads at 6.6 to 7.6 gigabytes, which is the gap between the formula and a real file: mixed precisions, an uncompressed vision tower, and metadata. Qwen3.8-27B is 54 and 13.5; Gemma 4’s 31B dense model is 62 and about 15.5, with Ollama listing it at 19 to 20. Then come the mixture-of-experts models, where the arithmetic surprises people: OpenAI’s gpt-oss-120b has 117 billion parameters of which about 5 billion are active per token, so it is fast, but all 117 billion must be in memory, and it ships at roughly 4-bit precision to fit a 128 gigabyte machine. GLM-5.3-Flash at 320 billion total needs about 160 gigabytes at 4-bit, and DeepSeek-V4.1-Flash at 763 billion needs about 380, which is 512 gigabyte Mac Studio territory or a rack. All of this is the weights alone. The context cache, the stored attention state for every token in the window, sits on top and grows with the context length; for a representative 32-layer model one published calculation gives about 16 gigabytes at 128K tokens, and for an 80-layer model about 40. That is why a model that fits at a 4K window stops fitting at 64K, and why the runtime’s context setting is part of the specification. Leave about a third of memory free for all of it.

This is the hardware board, and it is the one slide where the brief’s warning about perishability applies most. Everything on it was read from a vendor page or launch reporting in October 2026. The first tier costs nothing extra: a laptop with 16 to 32 gigabytes of RAM, or a desktop with any 8 to 16 gigabyte graphics card, runs the 4 to 9 billion parameter models that handle classification, summarisation and private notes. The middle tier is now dominated by machines with unified memory, where the CPU and GPU share one large pool: Apple’s M5 Max in the MacBook Pro and Mac Studio goes to 128 gigabytes at 614 gigabytes per second of bandwidth, with the Studio from $2,499 and the 16-inch laptop from $2,699 before the memory upgrade; the Framework Desktop with AMD’s Ryzen AI Max+ 395 and 128 gigabytes lists at $3,449, up from $1,999 at launch, and was out of stock when checked. The RTX 5090 is the fastest single card for models up to about 31 billion parameters, with 32 gigabytes and a 575 watt power rating, but its $1,999 list price is fiction: the October 2026 survey found the cheapest US listing at $7,998 and other trackers in the $5,000 to $6,500 range. The lab tier is NVIDIA’s DGX Spark, 128 gigabytes at 273 gigabytes per second, now $6,950 after two increases from about $4,000, with a new 64 gigabyte model at $4,999, and the Mac Studio M5 Ultra from $5,499 with 512 gigabytes due late October. The cause of all the movement is the 2026 DRAM shortage driven by AI datacentre demand; Framework’s CEO said in January he expected it to last years. Part I’s dollar tiers of $0 to 800, $1,500 to 5,000 and $8,000 and up were right in 2025. The middle tier now starts nearer $2,500, and the consumer GPU route has become the expensive one.

Here is the arithmetic nobody puts on a slide, with real prices. The cloud column is from vendor pages read in October 2026: OpenAI’s GPT-6 Luna, “our most efficient model for focused, high-volume tasks”, at $0.10 per million input tokens and $0.50 output; Anthropic’s Haiku 5.5 from $0.10 and $0.50; Sonnet 5.5 at $2 and $10; GPT-6 Astra, the current flagship, at $10 and $50. The blended figure assumes three input tokens for every output token, which is typical of summarisation and classification work. Divide the price of a 128 gigabyte DGX Spark, $6,950, by the blended price per token and you get the number of tokens the box would have to process before it costs less than the cloud. Against the cheap tier that is about 35 billion tokens, which at a hundred million tokens a month is twenty-nine years; the machine will be obsolete long before. Against Sonnet-class prices it is about 1.7 billion tokens, or seventeen months; against GPT-6 Astra about 350 million tokens, or three and a half months. The table is deliberately generous to local: it ignores electricity (an RTX 5090 is rated at 575 watts, so eight hours a day is roughly a hundred kilowatt-hours a month at whatever your tariff is), it ignores the person who runs the box, and it pretends the local model is as good as the cloud model it replaces, which it is not. So the honest economic case for local is narrow and real: work you would otherwise pay frontier prices for, that a local model does well enough on your own test set, at serious volume. Part I’s rule that cloud wins under fifty to a hundred million tokens a month was right in 2025; at 2026 cheap-tier prices the line sits far higher. The data question, from section one, is unaffected by any of this.

The third cost is hours, and it is the one that does not appear on any invoice. The original Part II had a checklist and it is kept here intact because it is the most useful thing in that deck. Write down the model and the exact quant, by file hash rather than name, because two files called the same thing can differ. Write down the runtime, its version and the flags, the hardware and driver versions, and the context length and cache settings, because CUDA, kernels, runtime, quantisation tooling and model code all move independently, and “it worked last week” is not a reproducible state. Write down who patches it. Then test it: a set of prompts and tool calls drawn from your real work, measured on this box for speed and time to first answer, and, if a harness will use the model, the exact tool schema that harness sends, because a model that writes pleasant prose can be a poor agent and the only way to know is to try it. Re-run the set after every change to anything on the left. Then own it: the fallback path when it fails, which might be a cloud call, a queue or a person; who can reach the port and from where; backups of the whole working stack and not just the weights; and a date to come back to this list. The line at the bottom is the test. If you cannot write all of this down, what you have is a successful demo, and the honest thing is to call it that.

Three disappointments are standard, and knowing them in advance turns them from surprises into line items. The first is quality. Local models are one beat behind the closed frontier, sometimes two; the gap has narrowed dramatically and it is not zero, and it shows most on hard reasoning, on taste, and on long multi-step coding. The correct response is not a leaderboard argument but a measurement on your own work, followed by a sober decision about which jobs can live with the result. The second is operations, covered on the previous slide: budget a person, at least part-time, or the hardware becomes an expensive space heater. The third is security, which section one introduced: secrets, file permissions, the open port, containers, patching and logs are now yours, and the vendor’s security team did not come with the download. Three specific failures are worth naming because everyone meets them. A model that fits at a 4,000-token context and fails at 64,000 has hit the context cache, which is memory on top of the weights; set the context deliberately and test at the length you will use. A model that writes lovely prose and produces broken tool calls has usually met a harness whose tool schema it was never evaluated on, or a runtime using the wrong chat template; test the exact schema. And a stack that works until the reboot or the update has unpinned versions; record the working combination and reproduce it before trying to improve it.

This is the one slide of model names, and it will be stale first; the selection method underneath it will not. Every entry was read from the maker’s Hugging Face page or Ollama listing in October 2026. Qwen, from Alibaba, has the widest range: Qwen3.8-27B is a dense 27 billion parameter model under Apache 2.0 with a 256K context, the Qwen3.5 family runs from 0.8 to 397 billion parameters, and Qwen3.8-Flash-Next is a mixture-of-experts with 125 billion parameters and 6 billion active, under a Qwen community licence rather than Apache. DeepSeek-V4.1-Flash-1017, released 17 October 2026, is 763 billion parameters under MIT with a million-token context: frontier-class and lab-tier hardware. GLM-5.3-Flash from Zhipu is 320 billion total and 18 billion active, MIT, multimodal, aimed at coding and agents. Google’s Gemma 4 runs from effective 2B to a 31B dense model, Apache 2.0 with vision and a 256K context, and is the strongest small-dense option for laptops. Mistral’s Ministral 3 line, 3, 8 and 14 billion, is Apache 2.0 with vision and built for edge devices. OpenAI’s gpt-oss, 20B and 120B mixture-of-experts under Apache 2.0, dates from August 2025 and had no newer open release when checked; the 20B runs in 16 gigabytes. The method is seven questions in order: task quality on your own test set first; licence second, because a model you cannot legally use is not a candidate; fit on the box you have; tokens per second measured rather than quoted; the context length you will actually use; tool calling against your harness’s schema; and operational pain, meaning who suffers when it breaks. Names change monthly. The order does not.

Four jobs justify a local model, and each connects to another deck. The first is a private assistant on your own documents, which is the named-data case from section one made concrete: contracts, personnel files, the claims database go into the model’s window, and nothing leaves the machine. LM Studio supports dropping documents into a chat out of the box; Part VII covers how to do it properly, with retrieval, chunking, and the ways it goes wrong. The second is a cheap classifier or summariser inside a workflow, which is rung two of Part III’s ladder: a fixed flow where one step uses a model to tag a ticket, pull fields out of a form, or summarise a call. With the facts supplied in the window, a 9 billion parameter model does this nearly as well as a frontier model, at zero marginal cost, and will run all night. The third is coding with a harness: a 27 to 31 billion parameter model behind the endpoint, with a coding harness from Part VI pointed at it. It is good for the repository that must not leave the building, for cheap iteration, and for learning what a harness actually does; it is not yet a replacement for the frontier on hard problems, and the honest expectation is “useful assistant”, not “senior engineer”. The fourth is batch work: ten thousand documents to classify, a year of transcripts to summarise, a nightly report. Latency is irrelevant, volume is high, and the box is idle anyway; this is where section three’s economics genuinely tilt toward local. The pattern across all four is the same one that runs through the whole series: messy input, the relevant facts placed in the window, real volume, and a rule or a human at the end.

This is the architecture the original Part II closed on, and it is still the right one. A request arrives: a ticket, a document, a coding task, a batch. The harness, from Part VI, wraps it with tools, memory, the loop and permissions. Then a router asks two questions, and it asks them in code rather than by asking a model: may this data leave the building, and can the local model do this job well enough? The first is a policy lookup; the second is a rule you set from your test set, possibly with a cheap check on the local model’s output. The default answer sends the request to the local model, private and cheap, behind the one URL from section two. The exception, the twenty percent that is hard, high-stakes, or that failed a check, goes to a frontier API, but only if the data rule allows, and the reason is logged. Everything ends in an output and an audit trail that records what ran, which model, which tools, and which data left the box. Three properties make this the grown-up version rather than a diagram. The router is code, so the data rule is a wall rather than a prompt, which is the same principle Part III applied to agent permissions. The audit trail means the question “which data left the building?” has an answer in a log. And because both models sit behind the same shape of endpoint, swapping either one is a configuration change rather than a project. The percentages are a figure of speech: your test set tells you the real split.

Three stories, each attached to one of the bad reasons from section one. The first belongs to “no API, no risk”. In September 2025 Cisco’s Talos team used Shodan to scan for Ollama servers reachable from the public internet and found more than 1,100, over a thousand of them within ten minutes. About a fifth were actively serving models to anyone who connected; the rest were idle but still accepted model uploads and configuration changes. No vulnerability was exploited. People had bound a service to a public interface and never closed the port. The second belongs to the same reason from the other direction. In early 2025 ReversingLabs found two models on Hugging Face whose PyTorch files contained a reverse shell. PyTorch’s default format is Pickle, and loading a Pickle executes code; these files were compressed in a non-standard way so the platform’s scanner errored before reaching the payload. Hugging Face removed them within a day and fixed the scanner, and the general lesson stands: a weight file is code you run, so prefer formats that cannot execute, such as safetensors or GGUF, and download from the maker’s own page. The third belongs to “the GPU is already paid for”. In 2026, AI datacentre demand for memory chips drove consumer and workstation prices up sharply: NVIDIA’s 128 gigabyte DGX Spark went from about $4,000 at launch to $6,950 by October, Framework raised its 128 gigabyte desktop from $1,999 to $3,449, and the RTX 5090 sold for several times its $1,999 list price. Framework’s CEO said in January he expected the shortage to last years. The hardware line in a local-AI budget is not fixed either.

Five steps, each cheaper than the one people usually start with, which is buying hardware. Step one: install LM Studio on the laptop you already have, pull a 9 billion parameter model at 4-bit, chat with it, and turn the wifi off. An hour, no spend. If it feels useless to you, stop and you have learned that cheaply. Step two: make one model boring. Choose one of the four jobs from section four, build a test set of about fifty cases from real work, run it, and write down the model, the quant, the runtime, the context setting and the speed. Boring means it passes the same way every day, which is the property production needs and demos lack. Step three: expose the endpoint, on localhost, to exactly one application: a script, a workflow step, or a harness from Part VI. Not the office network, not the internet. Watch it for a week. Step four: measure four numbers. Tokens per month, tokens per second, test-set pass rate, and hours spent by the person looking after it. Put the first one against the cloud price table from section three before anybody says the word GPU. Step five: decide, using the rule for today. Named data that must not leave means local, and you buy the box the job needs. Serious volume against frontier prices means maybe, and the test set decides. Neither means cloud, with the contract read. And if the honest answer is that someone wants a homelab, say so; hobbies are allowed. The architecture that wins is the boring one: a stable endpoint, a measured model, explicit permissions, a frontier fallback, and a log.

Two things the main deck simplified. First, formats. GGUF is llama.cpp’s container and what LM Studio and Ollama mostly run; its quant names, such as Q4_K_M, encode the bits and the method, and the files are mixed precision inside, with the layers that matter most kept at higher bits. MLX is Apple’s own framework, with its own quantised builds listed separately by the runtimes. EXL3, from the ExLlamaV3 project, allows fractional bits per weight, which is why the public recipes for squeezing a 300 or 700 billion parameter mixture-of-experts model onto two DGX Sparks talk about 2.9 bits per weight; FP8 and NVFP4 are precisions that server GPUs compute in natively. Second, the context cache. A transformer stores, for every token in the window and every layer, the attention keys and values it computed, so it never recomputes them. The size per token is the number of layers, times two for keys and values, times the number of key-value heads, times the head dimension, times the bytes per value. For a representative 32-layer model with 8 key-value heads and 128-dimensional heads at 16-bit that is 131,072 bytes per token, or 16 gibibytes at a 128K-token context; an 80-layer model with the same heads needs about 40. These are one published worked example and real models vary, but the shape is right: the cache scales with the context length you configure, not with how much you typed, and storing it at 8-bit or 4-bit halves or quarters it. This is the second term in fitting a model in a box. The weights were only the first.

The original Part II made a point worth keeping: the useful unit of local-AI knowledge is a recipe, not a model name. The code block shows what one public recipe actually pins for running Qwen3.8-27B on a 16 gigabyte NVIDIA card: the engine, ExLlamaV3 at exactly version 1.6.0, which the kit refuses to run without; the driver generation and CUDA build; the Python and Node versions; the quant and context setting per memory size, 2.5 bits per weight at a 176K context as the 16 gigabyte default, 4.0 bits at 262K for 24 gigabytes and up; the cache precision; the endpoint; and measured quality and speed on named hardware, including the KL divergence figures used on the quantisation slide and decode speeds of roughly 100 to 275 tokens per second on a single RTX 5090 depending on task. Those speed figures are the author’s own measurements and will not transfer to your box; the point is that they were measured and recorded. The order of theft is: copy the pinned versions first, the launch flags second, and ignore anyone’s benchmark until you have run your own job. MiaAI_Lab’s GitHub organisation holds recipes for a range of hardware, from single consumer cards to multiple DGX Sparks, and its Hugging Face account holds the matching quantised files; the makers’ own Hugging Face pages, Qwen, deepseek-ai, zai-org, google, mistralai and openai, are where the original weights and licences live. Download from those, check the hash, pin the stack, and write it down.