Your data and AI

Eighty minutes, five sections. Section one restates Part I’s three knobs and the sequence. Section two is the long one, on retrieval, because that is what most people in the room will actually build or buy. Section three is adaptation, kept short because most of the room should not do it. Section four is evals and data hygiene, which is where projects live or die, and the three stories. Section five is the Monday list.

Every organisation that has used a chatbot for a month asks the same question: how do we get it to know our stuff? The honest answer begins by noticing that “our stuff” hides three different requests. The first is knowledge: facts that live in documents and that change. The refund policy, the current price list, the contract signed last Tuesday, the ticket a customer opened this morning. The model did not read these in training and cannot have, and Part IV’s rule applies: if a fact is not in the window, the answer is a guess. The fix is to look the facts up at question time and put them in front of the model. That is retrieval, and it is what the word RAG means. The second request is behaviour: the model knows enough but will not act the way you want. It is too chatty, it will not stick to the format, it does not sound like you. That is adaptation: a thin layer of training that changes how the frozen model responds, without giving it a memory. The third request is to change what the model fundamentally is, by retraining most of its weights or starting again. Part I said that if you need that you already have a machine-learning organisation, and nothing has changed. Part I’s mnemonic still holds: RAG is what it knows, LoRA is how it acts, pretraining is building a new brain. The rest of this session is the first two in detail, in that order, because that is the order you should try them.

Part I drew this as a list. It is really a ladder, and the rule is to stop at the first rung that works. Rung one is prompting: a system prompt that states the job, the rules and the tone, plus a handful of good examples. Part I said this fixes half the tickets that arrive labelled “we need a custom model”, and that is still true; the examples do most of the work. It costs an afternoon and can be undone in a minute. Rung two is retrieval: if the right answer lives in documents that change, index them and look them up at question time. Days to weeks, and the index can be updated tomorrow. Rung three is adaptation: if the model knows the facts but will not behave, train a thin adapter on examples of the behaviour. Weeks, plus the evaluation work in section four, because without a test set you cannot tell whether the adapter helped. Rung four is distillation: if you need the behaviour cheaper or running locally, teach a small model from a large one’s answers. Rung five is pretraining, which Part IV priced: thousands of GPUs for months. OpenAI’s own guide for developers says the same thing in different words. Prompt engineering is “typically the best place to start”, retrieval is for when the model lacks the context, and fine-tuning is for when its behaviour is inconsistent. The guide is careful to say these are not strictly sequential; you pick the lever that matches the failure you see. But you only know which failure you see once you have measured it, which is section four.

Before anyone writes code, there is a version of all this that needs none. A system prompt is a block of standing instructions that is sent with every message: who the assistant is, what the job is, the rules, the tone, what to do when it is not sure. It is where most behaviour complaints are fixed, usually by adding one sentence that nobody had written down. Examples do the rest. Show the model a real input and the answer you wanted, three or four times, and format, length and voice follow; this is sometimes called few-shot prompting, and it is the cheapest “training” there is, and it is not training at all, because nothing in the model changes. The consumer products package this. ChatGPT Projects keep instructions and files together and apply the instructions only inside the project; the help page lists file limits of 5 on the free plan, 25 on Plus and 40 on business tiers, as of October 2026. Claude Projects do the same, and on paid plans switch to retrieval automatically when the uploaded knowledge gets close to the window limit; the help page describes this as expanding capacity by up to ten times. In other words, the zero-engineering version quietly becomes section two when the documents get big. So week one is: put the policy in a project, write one paragraph of instructions, add three worked examples, and try to break it with real questions. You will learn more from that afternoon than from a quarter of vendor meetings. And you will have the beginnings of a test set.

RAG stands for retrieval-augmented generation, a phrase from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research that combined a language model with a searchable index of documents. The business version is simpler than the paper. There are two phases. The first happens before anyone asks a question, and again whenever documents change. Take the documents you want the system to know. Cut them into chunks, pieces of a few hundred words each; Amazon’s managed service defaults to about 300 tokens, Anthropic’s guidance says “usually no more than a few hundred tokens”. Turn each chunk into an embedding, which is a list of numbers that captures what the chunk is about; the next slide explains that idea. Store chunk, numbers, source and, crucially, who is allowed to see it, in an index. The second phase happens on every question. Turn the question into the same kind of numbers, find the chunks whose numbers are closest, and take the top few. Paste those chunks, with their sources, into the context window next to the question. Now the model does what Part IV said it is good at: reading what is in front of it and answering from it, naming which chunk said what. A person can then open the source. Notice what the model does and does not do. Everything in blue is ordinary code: cutting, counting, searching. The model, in gold, reads and writes. It never “learns” the documents. It is handed the relevant pages each time, like a new temp with a very good filing clerk.

The word that stops most non-specialists is “embedding”, so here it is without the maths. An embedding is a list of numbers, a vector, that a small model produces from a piece of text. OpenAI’s current embedding models produce 1,536 or 3,072 numbers per text. The numbers are arranged so that texts with similar meaning get similar numbers; in OpenAI’s own words, “small distances suggest high relatedness and large distances suggest low relatedness”. The useful picture is a map. Every chunk of your documents is a pin, and pins about the same thing cluster together. A question is a pin too. Retrieval means dropping the question’s pin on the map and picking up the nearest few chunks. That is why it works without keywords: “can a contractor claim travel” lands near the expense policy and the contractor terms even though none of those documents contains the word “claim”. The analogy breaks in three places, and they matter. First, the map has thousands of dimensions, not two, and nobody can tell you what any one of them means; the slide draws two because paper is flat. Second, “near” means “about the same topic”, not “true” or “current”: the 2024 policy sits right beside the 2026 one, and the embedding does not know which is in force. Third, meaning-search is poor at exact strings. A part number, a surname, an invoice ID, a legal citation: these need old-fashioned keyword matching, which is the next slide. So: a filing system that files by meaning. Useful, mechanical, and not understanding.

Three properties make retrieval the default answer for knowledge, and they all follow from the mechanism. First, it can cite. Every chunk that goes into the window carries its source: file, section, date. If the prompt tells the model to answer only from those passages and to mark which passage each claim came from, then a citation is a pointer to something that exists, which a person can open. Part IV’s Mata v. Avianca lawyers were caught by citations the model predicted; here the citation is retrieved, not predicted. The rule has to be enforced, in the prompt and in the eval in section four: no supporting chunk, no claim. Second, it updates tomorrow. When the policy changes, re-index the file and the next question gets the new answer. Nothing about the model changed. The knowledge cutoff from Part IV stops mattering for anything in the index. Compare that with baking the policy into weights, which section three will show goes stale the day you sign it off. Third, hybrid search. Meaning-search is good at “about the same thing” and poor at exact strings. Keyword search, usually an algorithm called BM25, is the reverse: it finds the exact part number, surname or clause reference and misses the paraphrase. Production systems run both and merge the rankings; Microsoft’s Azure AI Search documents this as a single hybrid query merged by reciprocal rank fusion, and Postgres with pgvector can do the same with its built-in full-text search. Anthropic’s September 2024 measurements give a sense of the gain: on their test sets the rate at which the right chunk was missing from the top twenty fell from 5.7% with plain embeddings to 2.9% with contextual embeddings plus keywords, and to 1.9% with a reranking step added. Those are their corpora and their numbers; the direction holds generally.

The disappointing truth about retrieval systems is that when they fail, the model is rarely the culprit. The model answered from what it was given. The failure is in what it was given. Two numbers set the scene. In May 2024 Stanford’s RegLab and HAI tested commercial legal research tools built on retrieval and marketed as hallucination-free. Lexis+ AI and Thomson Reuters’ Ask Practical Law AI produced incorrect or misgrounded answers more than 17% of the time; Westlaw’s AI-Assisted Research more than 34%. Misgrounded means the answer cited a source that did not support it. The authors blamed retrieval difficulty, documents inapplicable by jurisdiction or date, and the model agreeing with false premises in the question. Anthropic’s own figure is the other side of the coin: with ordinary embeddings, 5.7% of test questions did not have the right chunk anywhere in the top twenty results. In those cases the model never saw the answer, and in most systems it answers anyway. The table is the failure catalogue. Chunks cut mid-sentence separate a number from its unit; chunk on structure instead, and give each chunk a line of context about the document it came from, which is what Anthropic’s contextual retrieval does. Wrong documents in the index: the 2023 draft is in there with the 2026 policy, and the embedding cannot tell them apart; curate, date, and filter by status before ranking. A stale index: the policy changed and nobody re-indexed; re-index on change and print the index date on every answer. The model ignoring the chunks: the prompt must say answer only from the passages, cite each claim, and admit when the passages do not contain the answer, and section four must test for it. And the last row is the one that gets you in the newspaper: a retrieval system is a search engine over everything you gave it, and if you gave it HR’s folder, everyone now has HR’s folder.

Context windows grew faster than retrieval advice did, so the question deserves a straight answer: can you skip all of this and just paste the documents into the prompt? Often, yes. Anthropic’s own guidance says that if the knowledge base is under about 200,000 tokens, roughly 500 pages, you should simply include it in the prompt and use prompt caching, which cuts repeated-prompt cost by up to 90% and latency by more than half. As of October 2026 the major models take about a million tokens: Anthropic made 1M context standard for Opus and Sonnet 4.6 in March 2026 with no long-context surcharge, and OpenAI’s GPT-5.5 lists 1,050,000 with a surcharge above 272K input tokens. So the handbook fits. But four things still favour retrieval. Cost: every pasted token is billed on every question, so 500 pages is roughly 200,000 tokens, about a dollar of input per question at GPT-5.5’s $5 per million before caching; retrieval sends a few thousand tokens. Freshness: re-pasting a changed corpus is fine for a stable handbook and absurd for a ticket queue. Permissions: whatever is pasted is visible to whoever can ask, which is fine for one audience and a breach for several; retrieval can filter chunks by who is asking. And accuracy at length: Part IV covered “lost in the middle”, where facts in the middle of a long input get missed, and “context rot”, where reliability falls as input grows. The honest trade is on the bottom row. Pasting needs no engineering. Retrieval is a system you have to run. Pick the smallest thing that passes your test set.

Fine-tuning means continuing to train a model on your own examples so that its behaviour shifts. Done to the whole model, that is expensive and fragile: you need the memory to hold every weight and its gradients, and you can damage what the model already knew. The 2021 LoRA paper from Microsoft, “Low-Rank Adaptation”, found a cheaper route. Leave the base model frozen. Add small extra matrices beside the big ones and train only those. Compared with fully fine-tuning GPT-3, the authors report ten thousand times fewer trainable weights and about a third of the GPU memory, with no added latency at inference, and quality on par with full fine-tuning. The adapter is a small file you can swap in and out; one base model, many adapters. QLoRA, from Tim Dettmers and colleagues in 2023, squashes the frozen base to 4-bit precision so that the whole thing fits on much less hardware; they fine-tuned a 65-billion-parameter model on a single 48 GB GPU. That is why Part I said a serious laptop can do small ones. What is it for? Form. Tone and house style, consistently. A rigid output format. A narrow skill that is awkward to describe in a prompt, such as sorting tickets into forty categories the way your team does. Moving a long list of examples out of the prompt and into the adapter, which saves tokens on every call. And making a small, cheap model behave like a large one on a single job, which is distillation, two slides on. What it is not for is facts. An adapter has no sources to cite, it is frozen on the day you train it, and, as the next slide shows, teaching it facts is both inefficient and risky.

Part I gave you the rule as a slogan: fine-tuning is for form, not facts. Here is why it is a rule and not a taste. Two studies tried to do the thing the ticket asks for, which is to teach a model new facts by training on them. Ovadia and colleagues, in a paper revised in January 2024, compared unsupervised fine-tuning against retrieval for injecting knowledge, both facts the model had seen in training and entirely new ones. Fine-tuning helped a little. Retrieval won consistently in both cases, and the authors note that models struggle to learn new facts this way unless shown many paraphrases of each one. Gekhman and colleagues, in work revised in October 2024, found something worse than inefficiency. Training examples that introduce new facts are learned more slowly than examples the model already agrees with, and once the new facts are learned they linearly increase the model’s tendency to hallucinate, apparently because the model learns that producing unsupported facts is rewarded. Their summary is the one to remember: models acquire factual knowledge in pretraining, and fine-tuning mostly teaches them to use it. Then there is the mechanical point that needs no study. Whatever you bake in is right on training day. The price list changes on Thursday. The adapter does not know. With retrieval you re-index one file; with an adapter you retrain, re-run the evals, and redeploy. So the division of labour is simple. Behaviour, format and voice can live in weights, because they change slowly. Facts live in the index, because they do not.

Two more knobs, briefly, because they exist and vendors will mention them. Distillation is an old idea with a 2015 paper by Hinton, Vinyals and Dean behind it: transfer what a large model does into a smaller one that is cheaper to deploy. The modern recipe is plain. Run the big model on thousands of your real inputs. Keep the answers. Fine-tune a small model on those input-answer pairs. Compare both on your evaluation set and keep the small one if it passes. OpenAI packaged this as a product in October 2024: stored completions capture the big model’s answers, fine-tuning trains the small model, and an evals tool compares them. The result is a narrow specialist: faster, cheaper, and often small enough to run on your own hardware, which is where Part II’s local models come back in. It is only as good as the large model was on that job, and only on that job. Pretraining is the other extreme. Part IV priced it: Llama 3.1’s largest model was trained on more than fifteen trillion tokens using over sixteen thousand GPUs, and Epoch AI projected the largest frontier runs passing a billion dollars. There is a middle ground called continued pretraining, where an existing model is trained further on a very large specialised corpus, such as decades of a particular kind of technical document. It is real, it is rare, it still needs retrieval for anything that changes, and it needs a machine-learning team. Part I’s line stands: if you need this, you already have that team.

An eval, short for evaluation, is an exam for the system. It has three parts: a set of real questions, the right answer to each, and a way to score an attempt. Everything else in this deck is a change you might make; the eval is how you find out whether the change helped. Start with real questions. Pull them from the inbox, the helpdesk, the search logs, the things people actually typed, including the badly phrased ones and the ones with wrong assumptions in them. Part III asked for a hundred past cases; thirty is enough to begin, and Monday’s list starts there. For each question, a person who knows writes the correct answer and names the document and section that supports it. This is the expensive step and the one that gets skipped, and it is the only step that cannot be automated, because it is the definition of correct. Then run the system over the questions and score. Some scoring is code: the answer contains the right number, the JSON parses, the cited chunk is the one the expert named. Some scoring is a second model acting as judge: “is every claim in this answer supported by the passages, yes or no”; Anthropic’s guidance is to use a different model from the one being tested. And some scoring is a human reading a sample every week, because the automated judges drift. The guidance from both Anthropic and OpenAI points the same way: make the eval task-specific, automate what you can, and prefer many cheap checks to a few careful ones; OpenAI’s guide suggests at least twenty questions with ground-truth answers as the output of the prompting stage. Then the discipline: change one thing, run it again. Part I said evals before adapters. Part III said a hundred cases before an agent. Same test set, same reason.

Six numbers cover almost every retrieval or adaptation project, and the point of having six rather than one is diagnosis. Correctness is the blunt instrument: did the answer match what the expert wrote. Score it with code where the answer is a number, a date or a fact, and with a judge model where it is prose. Groundedness asks a sharper question: is every claim in the answer supported by the passages that were retrieved. The open-source Ragas library calls this faithfulness and scores it mechanically: split the answer into claims, check each against the context, divide supported by total, so two claims with one supported scores 0.5. A low groundedness score with high retrieval hit means the model is ignoring the chunks or decorating them. Retrieval hit asks whether the right chunk was in the retrieved set at all; it is pure code, comparing the expert’s cited section against what came back. A low hit rate means the failure is upstream of the model: chunking, a stale index, the wrong documents. This is the number that the previous section said nobody checks, and it is the cheapest to compute. Refusal rate needs questions the corpus cannot answer, deliberately included, so that you can count the times the system correctly said “not in my documents” and the times it wrongly did. Latency should be reported at the slow tail, because that is what the person waiting experiences. Cost is tokens in and out times the price, plus the retrieval service. Read together, these tell you which part of the pipeline to fix. Correctness alone tells you only that something is broken.

Here is what the previous two slides produce when someone actually does the work. The numbers are invented; the shape is what you see in practice. Version A is the model with a prompt and no documents: it gets eleven of thirty right from general knowledge, and refuses none of the five unanswerable questions, because it does not know it does not know. Version B pastes the whole handbook, 180,000 tokens, into every call: much better, twenty-four right and mostly grounded, but ten seconds and nearly a dollar per answer. Version C is the first retrieval attempt, with fixed-size chunks and vector search only, and it is worse than pasting: nineteen right, because the retrieval-hit column shows that nine times the right chunk was never found. The model never saw the answer. Version D fixes retrieval, not the model: chunk on headings, add keyword search, and a prompt that must cite or refuse. Twenty-six right, twenty-nine grounded, all five refusals correct, four seconds, three cents. Version E is a LoRA adapter trained on four hundred past answers, a fortnight of work, and it moves the numbers you care about not at all and the groundedness slightly down. That is the normal result for an adapter aimed at facts. The decision writes itself. Ship D. Keep B as the ceiling to compare against. Bin E and write down why. Then take the four D got wrong and look at them one at a time, because the next improvement is in those four, not in a new model. Notice that no step of this needed anyone to argue about which vendor’s model is smartest.

A retrieval index is a search engine over everything you put in it, and it is usually a better search engine than the one your file shares had. That is the point, and it is the risk. What goes in: current, approved documents, one version each; the things people actually ask about; and on every chunk a source, a date, an owner and an intended audience, because those are what the filters and the citations use. Structured data, the customer table, the stock levels, the ledger, should not be chunked into prose at all. Leave it in the database and give the model a tool to query it, which was Part III’s whole argument; the index is for documents. What must not go in: drafts and superseded versions, which the embedding cannot distinguish from the current one; personal data the questions do not need, which includes payroll, health, HR case files and customer identifiers; anything whose access list you cannot state, because if you do not know who may read it the only safe answer is nobody; and secrets, because wiki pages full of passwords exist in every organisation and retrieval will find them. The rules: permissions mirror the source system, so a chunk inherits the access list of the file it came from and retrieval filters by who is asking before it ranks, not after. Retention follows the source, so a deleted document means deleted chunks on the same clock. Log what was retrieved, for whom, and when; that log is the audit trail when something goes wrong and the raw material for the next hundred eval questions. And Part IV’s line, once more: the model is not a database. Facts of record live in systems of record. The index points at them.

The clearest large-scale lesson about data hygiene came from the biggest retrieval rollout in history. Microsoft 365 Copilot, launched to enterprises in November 2023, is a retrieval system over everything in a company’s Microsoft tenant: mail, chats, files, meetings. Microsoft’s documentation is precise that it answers only from data the user already has permission to access. That turned out to be the problem. In June 2024 Gartner surveyed 132 IT leaders, about half at organisations with more than ten thousand employees. Sixty per cent had started Copilot pilots; one per cent had deployed it to everyone eligible. Forty per cent had delayed rollout by three months or more over oversharing concerns, sixty-four per cent said governance and security risks had taken significant time and resources, and fifty-seven per cent had restricted the rollout to low-risk or trusted users. What they had found, in the words of Gartner’s Max Goss, was that they were “putting Copilot into an environment where permissions haven’t really been taken care of for years”. Documents shared with “everyone” in 2017, open team sites, inherited access on folders nobody remembered. People had always had those rights. Before, exercising them required knowing where to look. Retrieval removed that. Microsoft responded with what it calls a blueprint for oversharing: find high-risk sites and files, restrict access first and fix it after, label sensitive content, govern sharing, and only then switch on, with SharePoint Advanced Management bundled into the licence to do it. Goss’s summary is the rule for every retrieval project in this deck: the better the retrieval tool, the better your information governance has to be.

Three stories, one from each half of this section. The first is New York City’s MyCity business chatbot, announced by the mayor in October 2023 as a centrepiece of the city’s use of AI and built on Microsoft’s Azure AI services over material from the Department of Small Business Services. In March 2024 The Markup tested it. It said landlords did not have to accept housing vouchers; New York bans that discrimination. It said employers could take workers’ tips, that restaurants could go cashless, and that funeral homes could hide prices; all wrong. It gave one reporter the right answer on vouchers and ten colleagues the wrong one, and often answered without links despite a disclaimer asking users to check them. The city called it a pilot and left it up. Everything about that is a retrieval-and-eval story: the right documents existed, the answers were not grounded in them, and nobody had run thirty known questions through it before launch. The second is the Stanford study from earlier in the deck. Legal publishers sold retrieval tools as hallucination-free; a 200-question evaluation found 17% to 34% of answers wrong or misgrounded; neither vendor had published a benchmark or even a definition. The lesson is that “our RAG doesn’t hallucinate” is a claim about an eval the vendor should be able to show you. The third is the Copilot oversharing story from the previous slide: the search worked exactly as designed, over permissions that had not. Three organisations with budgets, and in two cases famous names. None of them was stupid. Each skipped the boring step.

This is the one slide with product names and prices, checked in October 2026 against vendor pages, and it will be wrong within months, so treat it as a map of the columns rather than a price list. Switch on: retrieval already exists inside the suites most organisations pay for. Microsoft 365 Copilot is sold as an add-on; in October 2026 the business page lists $18 per user per month annually on a promotion, $25.20 monthly, grounded in the tenant’s mail, files and chats. Google’s Gemini Enterprise is a per-seat subscription with connectors to Drive, Gmail, OneDrive, SharePoint, ServiceNow, Jira and Confluence; prices are not on the public FAQ. ChatGPT Projects and Claude Projects give you files plus instructions on free tiers, with Claude switching to retrieval on paid plans when files outgrow the window. All of these inherit your existing permissions, which is the point of the oversharing slide. Assemble: a managed retrieval service plus a model, wired up by your team. OpenAI’s vector stores chunk, embed and index uploaded files, with a gigabyte free and $0.10 per gigabyte per day after that. Amazon Bedrock Knowledge Bases offer fixed, hierarchical and semantic chunking with a default of about 300 tokens. Azure AI Search has hybrid keyword-plus-vector queries built in. pgvector adds vector search to Postgres you may already run. Your team owns chunking, permission filtering and re-indexing. Build: your own pipeline and your own adapters, with LoRA and QLoRA tooling, the API vendors’ distillation workflows, or Part II’s local models. Do this only with an eval set, a named owner, and evidence on paper that the first two columns failed. Whichever column: the test set, the permissions clean-up and the person who owns the index are never in the box.

Five actions, in order, none of which costs money in week one. Write thirty real questions with answers: pull them from the helpdesk and the inbox, include five that your documents cannot answer, and have a named person write the correct answer and where it lives. That is the test set, and it is most of the work. Try the zero-engineering version: a Project with the current documents, a paragraph of instructions and three worked examples; run the thirty and write the score down. That is your baseline, and for a surprising number of teams it is also the product. Measure before you buy: the six numbers from section four, and when a vendor says their retrieval does not hallucinate, hand them the thirty questions. Then, and only then, decide whether you need retrieval at all: more than about five hundred pages, changing weekly, or different people who should see different things are the signs; and before switching anything on, find out who can see what in the source system today, because the search will. Finally, never fine-tune before evals. An adapter is for a behaviour the prompt could not fix, proven on the scorecard, and facts stay in the index. Part I said almost nobody in this room should pretrain. Nothing in the two years since has changed that.

For the people who will actually build the pipeline, these are the settings that move the retrieval-hit number, roughly in the order to try them. Chunk size and boundaries first: a few hundred tokens is the common default, Amazon’s managed service uses about 300 aligned to sentences, and cutting on headings and paragraphs beats any fixed number. Hierarchical chunking matches small child chunks and returns the larger parent; semantic chunking uses a model to find topic breaks. Contextual chunks next: before embedding, prepend a sentence or two saying where the chunk sits in its document; Anthropic reported top-20 retrieval failures falling from 5.7% to 3.7% from this alone, at the cost of one cheap model call per chunk at index time. Hybrid search with rank fusion: run keyword search and vector search in parallel and merge by reciprocal rank fusion; Azure AI Search does this in a single query, and Anthropic’s figure with contextual BM25 added is 2.9%. Then reranking: retrieve generously, say 150 chunks, and let a reranker model score each against the question, keeping 20; Anthropic’s figure with reranking is 1.9%, and the cost is latency, so watch the slow tail. Metadata filters are where access control, dates and status live, and they should be applied before ranking, not after. Finally the embedding model itself: OpenAI’s current models produce 1,536 or 3,072 numbers per chunk and can be shortened; changing models means re-embedding everything, so pin the version and record it alongside the index. The discipline is the same as the main deck: change one knob, re-run the thirty, read the retrieval-hit column.

For the curious, the reason a thin adapter is thin. Inside a model are many weight matrices, and a typical one in a mid-sized model might be 4,096 by 4,096, which is about sixteen million numbers. Fully fine-tuning means updating all of them, in every matrix, and holding gradients and optimiser state for each, which is why the memory bill is several times the size of the model. LoRA leaves that matrix frozen and adds two thin ones: A, which is 4,096 by r, and B, which is r by 4,096, where r is a small number called the rank, typically between 4 and 64. The model uses the frozen matrix plus the product of the two thin ones. With r equal to 8 that is about 65,000 trainable numbers in place of sixteen million, under half a per cent, and the same saving applies in every matrix you adapt. The paper reports ten thousand times fewer trainable parameters than full fine-tuning of GPT-3 175B, roughly a third of the GPU memory, no added latency, and quality on par; the adapter can be merged back into the weights for serving. QLoRA attacks the other term: holding the frozen base at all. Part IV’s rule of thumb is parameters times bits divided by eight, so a 65-billion-parameter model at 16-bit precision is 130 GB just for the weights; stored at 4-bit it is about 33 GB, which fits a single 48 GB GPU with room for the adapter and its optimiser state. Dettmers and colleagues add a 4-bit number format tuned to how weights are distributed, a second round of quantisation on the quantisation constants, and paged optimisers to survive memory spikes, and report matching 16-bit results on their benchmarks. That arithmetic is why Part I said a serious laptop can do small ones. The evals in section four are why most of them should not.