Simulator
Replay the model, with and without the overlay
Pick a question by clicking a square in the grid. The terminal replays, word by word, what Qwen3.8-Flash-Next answered in two of our measurements: on the left the model as it ships, on the right the same model with the overlay of the 100 Quail facts. Nothing is generated live; every reply is a recorded output of the real inference engine.
This is early-stage research, and the grid shows it: in free chat the overlay answers 35 of 98 questions. The yellow and red cells are as much a part of the result as the green ones.
The questions come from Quail, an invented world
The storyA clerk who wants to have been to Mars, and a company that sells memories.
Quail is built on Philip K. Dick's short story We Can Remember It for You Wholesale (1966), later the basis of the film Total Recall. Douglas Quail is a clerk who has always wanted to go to Mars and will never afford the trip. So he buys the next best thing from Rekal, a company that implants memories: the memory of a trip to Mars as a secret agent.
During the procedure Rekal finds that the memory is already there. Quail really was an agent on Mars, and the memory had been erased. From then on the story is a contest over which of his memories are real.
Why this storyRekal writes a memory into a man; ENGRAFT writes facts into a model.
Rekal adds a memory to a man without changing who he is. ENGRAFT adds facts to a model without changing its weights: it rewrites a few rows of the model's lookup table, and the rest of the model stays exactly as it was. The test then asks what the model now remembers, as the story asks of Quail.
The world also had to be one the model cannot already know. 88 of the 100 facts are invented outright; the other 12 follow the story as we remember it, not checked against the text, and no text of the story was copied. Without the overlay the model gets 4 of the 841 test sentences right; everything above that comes from the overlay.
What the world contains100 facts about 17 subjects, in seven Italian documents; Quail alone has 32.
The facts are written as seven Italian documents: Quail's life as an archivist, the visit to Rekal, the drug used in the implant, the Interplan agency, Paul Heller, the second session, and Quail's diary. Some facts: Quail works as an archivist at the Cook County Land Registry Office, lives on Halsted Street in Chicago, and dreams of Mars; the Vornidi, the aliens of a childhood memory, come from the planet Tessaly.
Many facts share a subject on purpose. Douglas Quail has 32 facts, his diary 12, a card called Airone 8. That is the hard case for a table addressed by the subject's tokens: facts about the same person read many of the same rows, and they compete for them.
The same world was rebuilt in English and in Chinese; those cells are preliminary. {{file:data/quail/PROVENANCE.md|PROVENANCE.md}}
What the questions look likeEach fact in five forms; the test stops right before the answer.
Each fact is written in several forms, because the table is keyed by the exact tokens a sentence reads: a different phrasing reads different rows. Here is one fact, Quail's job, in the forms of the test set. The model sees the sentence up to the arrow and must write the answer, and only the answer.
- statementDouglas Quail esercita il ruolo di → archivistaDouglas Quail holds the role of → archivist
- questionDouglas Quail, di che mestiere si tratta? È un → archivistaDouglas Quail, what is his trade? He is an → archivist
- clozeDouglas Quail, nel suo campo, è un → archivistaDouglas Quail, in his field, is an → archivist
- paraphraseDouglas Quail è un professionista degli archivi, quindi è un → archivistaDouglas Quail is a records professional, so he is an → archivist
- chat turnDimmi, che impiego ha Douglas Quail? / Douglas Quail lavora come → archivistaTell me, what job does Douglas Quail have? / Douglas Quail works as → archivist
The test sentences are new sentences in these same forms: never trained on, never used to stop the descent. The chat questions are harder: one plain question per fact, «Che mestiere svolge Douglas Quail?» (what does Douglas Quail do for a living?), with the model free to answer as it likes. That is where the overlay gets 35 of 98.
The 841 test sentences; the model completes each one.
Each square is one question. Click any square to replay it in the terminal: the question, the base model's answer and the answer with the overlay.
What you are reading, exactly
Both sets were measured on the same setup: Qwen3.8-Flash-Next quantized to 4 bits (IQ4_XS), run by our fork of llama.cpp. The overlay is the Italian Quail overlay (cell s0b): 14,032 table rows written for 100 facts. Every reply is greedy decoding, the single most likely token at each step, so the same prompt always gives the same reply.
Test sentences · 841
Completing a sentence
- The 841 test sentences of the Quail usage corpus, frozen before training and never used to stop it: statements, questions, cloze and paraphrases written as plain text, and some chat turns.
- Expert routing left free, as in normal use.
- The prompt stops right before the answer. The model writes as many tokens as the expected answer has, and no more.
- No thinking. Plain sentences have no chat template. Most chat turns use the template with the assistant's reply already started, without a think block; one is a two-turn exchange. 30 were stored as plain text with the role names written out, and are shown exactly as the model received them.
- Exact means the continuation equals the expected answer exactly. Overlay 707 of 841, base model 4.
- Cell colour: the probability the overlay gives to the first token of the right answer, from red (0) to green (1). It is not the outcome of the whole answer: a green cell can still be wrong ("Tessal" for "Tessaly"), and those cells carry a dark dot.
Chat questions · 98
Answering freely in a chat
- One question per fact, asked as a user turn in the chat template.
- Thinking off. The template carries an empty
<think></think>block, so the model answers straight away. - Up to 48 new tokens. The run stopped a reply as soon as the expected answer appeared, and the terminal marks where each reply ended.
- Answer found means the expected answer appears, letter for letter, somewhere in the reply. The rest of the reply is not judged, so it is not a certificate that the whole reply is correct. Overlay 35 of 98, base model 0.
- Cell colour: green if the overlay's chat reply contains the answer; otherwise yellow to red by how often the overlay gets that fact right on its test sentences.
What this does not show
- Thinking on. Neither set was measured with the model thinking before it answers.
- English. The Quail corpus was written in Italian, so these questions and answers are Italian. English questions are on our list for this page.
- Other models, other corpora. One model, one corpus of 100 facts, most of them invented for the test.
Because most facts were invented, the base model can only guess, often fluently. Both models sometimes attribute Douglas Quail to the wrong book or author: those are the model's own words, recorded as they came out.
s0b/engine_results.json · composition_rerun/one_fact.jsonl, one_fact_base.jsonl · test_corpus_resolved.json · engine_check.py