# ENGRAFT results: 100 facts, three languages, capacity and damage URL: https://engraft-engram.dev/results/ Updated: 2026-09-26 > ENGRAFT on Qwen3.8-Flash-Next: held-out exact answers, free recall, three languages, capacity to 300 facts, KL damage and limits, each with its result file. Results # What has been measured, and what limits each number All measurements are on Qwen3.8-Flash-Next (IQ4_XS) with the llama.cpp fork. Every chart names its metric and the file it is recomputed from, pinned to one commit of the public repository. ## 841 test sentences, one dot each Switch the model. Tap a dot to read the sentence. Base model Italian overlay Chinese overlay 707 / 841 exact answers The Italian overlay of the same model: 707 exact answers. Base model 4, Italian overlay 707, Chinese overlay 4 exact answers out of 841. - exact answer, greedy decoding, real engine - anything else s0b/engine_results.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s0b/engine_results.json) · specular-zh-on-it/engine_results.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/languages/specular-zh-on-it/engine_results.json) · sentences from test_corpus_resolved.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/corpus/s0b/test_corpus_resolved.json) ## One hundred facts about one world Quail is a corpus of 100 facts about an invented world: 88 invented outright and 12 following a published short story as remembered. The facts are written as seven Italian documents and compiled into 3,209 training sentences, 793 held-out sentences that drive the stopping rule, and 841 test sentences frozen before the first descent and never used for any decision. An exact-answer rate of 0.841 means 707 of those 841. Many facts are about the same people, the hard case for a table addressed by the subject's tokens. The story A clerk who wants to have been to Mars, and a company that sells memories. Quail is built on Philip K. Dick's short story We Can Remember It for You Wholesale (1966), later the basis of the film Total Recall. Douglas Quail is a clerk who has always wanted to go to Mars and will never afford the trip. So he buys the next best thing from Rekal, a company that implants memories: the memory of a trip to Mars as a secret agent. During the procedure Rekal finds that the memory is already there. Quail really was an agent on Mars, and the memory had been erased. From then on the story is a contest over which of his memories are real. Why this story Rekal writes a memory into a man; ENGRAFT writes facts into a model. Rekal adds a memory to a man without changing who he is. ENGRAFT adds facts to a model without changing its weights: it rewrites a few rows of the model's lookup table, and the rest of the model stays exactly as it was. The test then asks what the model now remembers, as the story asks of Quail. The world also had to be one the model cannot already know. 88 of the 100 facts are invented outright; the other 12 follow the story as we remember it, not checked against the text, and no text of the story was copied. Without the overlay the model gets 4 of the 841 test sentences right; everything above that comes from the overlay. What the world contains 100 facts about 17 subjects, in seven Italian documents; Quail alone has 32. The facts are written as seven Italian documents: Quail's life as an archivist, the visit to Rekal, the drug used in the implant, the Interplan agency, Paul Heller, the second session, and Quail's diary. Some facts: Quail works as an archivist at the Cook County Land Registry Office, lives on Halsted Street in Chicago, and dreams of Mars; the Vornidi, the aliens of a childhood memory, come from the planet Tessaly. Many facts share a subject on purpose. Douglas Quail has 32 facts, his diary 12, a card called Airone 8. That is the hard case for a table addressed by the subject's tokens: facts about the same person read many of the same rows, and they compete for them. The same world was rebuilt in English and in Chinese; those cells are preliminary. {{file:data/quail/PROVENANCE.md|PROVENANCE.md}} What the questions look like Each fact in five forms; the test stops right before the answer. Each fact is written in several forms, because the table is keyed by the exact tokens a sentence reads: a different phrasing reads different rows. Here is one fact, Quail's job, in the forms of the test set. The model sees the sentence up to the arrow and must write the answer, and only the answer. - statement: Douglas Quail esercita il ruolo di → archivistaDouglas Quail holds the role of → archivist - question: Douglas Quail, di che mestiere si tratta? È un → archivistaDouglas Quail, what is his trade? He is an → archivist - cloze: Douglas Quail, nel suo campo, è un → archivistaDouglas Quail, in his field, is an → archivist - paraphrase: Douglas Quail è un professionista degli archivi, quindi è un → archivistaDouglas Quail is a records professional, so he is an → archivist - chat turn: Dimmi, che impiego ha Douglas Quail? / Douglas Quail lavora come → archivistaTell me, what job does Douglas Quail have? / Douglas Quail works as → archivist The test sentences are new sentences in these same forms: never trained on, never used to stop the descent. The chat questions are harder: one plain question per fact, «Che mestiere svolge Douglas Quail?» (what does Douglas Quail do for a living?), with the model free to answer as it likes. That is where the overlay gets 35 of 98. ### Exact answer, greedy decoding, real engine - with the overlay - base model 841 test sentences 0.841 0.005 Free generation, one fact per question 35 / 98 0 / 98 s0b/engine_results.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s0b/engine_results.json) ### How the recipe got there, two seeds - seed 0 - seed 1 Routing pinned throughout 0.625 0.668 Routing released during the descent 0.823 0.838 …and each fact weighted by its training mass 0.841 0.873 exact answer, real engine · s1b/engine_results.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s1b/engine_results.json) · docs/history.md (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/docs/history.md) Limit The test sentences were never trained on nor used to stop the descent, but they come from the same families of phrasings as the training sentences. Free generation, 35 of 98, is the number to hold next to 0.841. ## The same world in three languages Quail was rebuilt fact by fact in English and Chinese. Both cells are preliminary: same recipe, same row budget, one seed each, and the English corpus covers 97 of the 100 facts. Damage is measured as KL divergence: how far the model's next-token probabilities move away from the base model's, averaged over neutral text that mentions none of the facts. Zero means identical. ### Exact answer, test sentences - with the overlay - base model Italian · 841 sentences 0.841 0.005 English · 843 sentences 0.797 0.004 Chinese · 954 sentences 0.676 0.003 e0b-preliminary/ (https://github.com/fulvian/engraft-ngram/tree/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/e0b-preliminary) · z0b-preliminary/ (https://github.com/fulvian/engraft-ngram/tree/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/z0b-preliminary) ### Damage on neutral text of the same language, mean KL Italian 0.0131 English 0.0160 Chinese 0.0166 The ordering is compatible with training mass. Controlling for the training text each fact received, the language coefficient's 95% interval is −0.124 to +0.020 and includes zero: no language effect is detected, which does not show that there is none. The candidate cause is the row budget: tuned on Italian, it buys about half as many Chinese sentences. A Chinese cell with equal mass per fact, on the open list (https://engraft-engram.dev/open-problems/), is the direct test. languages/mass_ols.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/languages/mass_ols.json) ## A graft did not cross languages The Italian test set was run against the Chinese overlay of the same hundred facts. The engine read the overlay, and the answers did not move at all. 0.0048 =0.0048 Exact answers: Chinese overlay against base model. 0.0951 =0.0951 First answer token at rank 1: overlay against base. 764 /841 Sentences identical to the base model down to the first token's probability. 922 Overlay rows the engine read along the way. The overlay fires and changes nothing. Probes that mix the two scripts do no better: 0 of 20 and 1 of 20. A fact has to be written in the language it will be asked in, because its rows are keyed by the tokens of its own script. Measured on one pair of languages and one cell (languages/specular-zh-on-it/ (https://github.com/fulvian/engraft-ngram/tree/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/languages/specular-zh-on-it)). ## Capacity: no ceiling up to 300 facts A separate corpus of short invented facts, not Quail: about one fact per subject, routing pinned, held-out sentences. The metric differs too: the answer's first token ranked first, not the exact answer used for Quail. ### First answer token at rank 1 24 facts 0.804 100 facts 0.792 300 facts 0.821 capacity-curve/ (https://github.com/fulvian/engraft-ngram/tree/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/capacity-curve) ### Damage, mean KL, against the storage-noise yardstick storage noise 0.0033 24 facts 0.0037 100 facts 0.0059 300 facts 0.0067 1.1×, 1.8× and 2.0× the yardstick: the damage grows more slowly than the number of facts. Limit This curve counts subjects, not crowding. In the Quail results exact match falls from 0.93 for subjects with one or two facts to 0.80 for the subject with 32 facts, not monotonically, over 17 subjects: a hint, not a result. ## Damage, stated plainly The table is stored in 4 bits, so rewriting rows at all moves the model a little. We measured that yardstick directly: an overlay that only re-quantizes the same rows gives a mean KL of 0.0033. At 24 facts the damage sat at that level. At 100 facts on Quail it is about four times higher, and no longer inside the table's own noise. ### Mean KL to the base model on neutral text storage noise 0.0033 24 facts, seed 0 0.0035 24 facts, seed 1 0.0037 Quail, 100 facts 0.0131 s0b/damage_it_text.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s0b/damage_it_text.json) · the yardstick was only measured at 24 facts ## Limitations, without discounts ### Two facts in one question: 83 probes - both answers right · 4 - only one right · 26 - neither · 53 If the two answers were independent, single-fact free recall (35 of 98) would predict about 11 of 83. The count is lower; but 56 of the 83 replies reached the 96-token limit, so how much putting two facts together costs by itself is not settled. composition_rerun/ (https://github.com/fulvian/engraft-ngram/tree/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s0b/composition_rerun) ### Where a fact loses, it loses to a sibling - won by a fact about the same subject · 32 - won by another subject's fact · 5 37 failures whose first answer token belongs to another written fact. Rows are shared by content, the subject's own n-grams, not by hash: only 12 of the 14,032 overlay rows are written through more than one token window. row_sharing.json (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/data/quail/results/s0b/row_sharing.json) - Rephrasing is covered only as far as the corpus goes. The table fires on exact n-grams, so a sentence that shares no n-gram with the corpus is not covered, by construction. - Families differ. Cloze prompts are the weakest family: first token 0.63 against 0.70 to 0.81 for the others, routing pinned. - One model measured. DeepSeek V4.1 Flash has an Engram-style table too; its layout differs and nothing has been measured on it. - No head-to-head yet with LoRA, ROME/MEMIT or retrieval. It is the comparison we most want to see, and it is on the open list (https://engraft-engram.dev/open-problems/).