# ENGRAFT technical report: a hundred facts in the n-gram table

URL: https://engraft-engram.dev/paper/
Updated: 2026-09-26

> The ENGRAFT technical report, version 3: method, measurements on Qwen3.8-Flash-Next, the contest between facts of one subject, and a fact that does not cross languages. PDF, CC BY 4.0.

Technical report · version 3 · 26 September 2026

# ENGRAFT: writing a hundred facts into the n‑gram memory table of a mixture-of-experts language model by gradient descent on a usage corpus

fulvian · 8 pages · CC BY 4.0. The report describes the method, the measurements on Qwen3.8-Flash-Next and the failure that dominates what is left. Every number in it comes from a file in the repository.  Read the PDF (https://engraft-engram.dev/paper/engraft.pdf) LaTeX source (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/paper/engraft.tex) CITATION.cff (https://github.com/fulvian/engraft-ngram/blob/b51ead5da6e25a132cdc0f74e3d01243e1562e14/CITATION.cff)

## Abstract

Some recent language models carry an explicit n-gram memory: at every position the last two and three tokens are hashed into a small set of rows of a large lookup table, and the rows are added to the residual stream at one early block (DeepSeek's Engram; Qwen3.8-Flash-Next ships one with 16 heads). ENGRAFT writes facts into such a table after training, without touching a weight: the edit is an overlay of table rows that the inference engine substitutes at read time. The method applies to models with an Engram-style table; every measurement here is on Qwen3.8-Flash-Next.

Given a set of facts and a corpus of short fragments that use them (statements, questions, cloze forms, chat turns, paraphrases), we take as variables a budgeted set of the table rows those fragments read, and minimize the negative log-probability of the answer tokens through a replica of the frozen model, with each fact weighted by its training mass. No teacher is needed: a control experiment on 24 facts ranks the answer first on 86 of 112 held-out fragments against 50 for the same descent imitating the model with the fact document in its prompt, and 0 for a negative control. The mixture-of-experts routing is held at the base model's choices while the rows move most, then released so that the descent trains the computation the engine will actually run; releasing it raises exact-match recall by 0.17–0.19. The descent stops on the rate of improvement of held-out first-token accuracy.

On a corpus of one hundred invented facts in Italian, the real llama.cpp engine with the overlay reproduces the full answer on 0.841 and 0.873 of 841 frozen test fragments for two seeds (base model 0.005). The mean KL divergence from the base model on neutral text is 0.012–0.013 nats, about four times a quantization yardstick measured at 24 facts. The residual failures are dominated by a contest between facts of the same subject, which share the rows their triggers read; the overlay decodes to the fact with the larger training mass, and weighting the loss by that mass reduces but does not remove the effect. Preliminary Chinese and English cells score below Italian, and at equal training mass per fact the Chinese and Italian cells are indistinguishable. A fact does not cross languages: the Italian test set run against the Chinese overlay reproduces the base model's numbers exactly, although the engine reads the overlay's rows.

## What changed in version 3

Version 3 brings the report level with release v0.2.2 of the repository. It adds a statement of what the method is and is not, the English cell, the direct test of whether a fact crosses languages, and a rerun of the composition probes without the protocol confounds of the first run. The cost is stated without the timings of the unreleased execution path.

## Versions
- Version 3, 26 September 2026: PDF (https://engraft-engram.dev/paper/engraft.pdf). This page.
- Version 2, 19 September 2026: usage-corpus descent on one hundred facts. PDF at release v0.2 (https://github.com/fulvian/engraft-ngram/blob/ea91572/paper/engraft.pdf).
- Version 1, 5 September 2026: the surgical graft of one fact at a time. PDF at release v0.1.0 (https://github.com/fulvian/engraft-ngram/blob/v0.1.0/paper/engraft.pdf).

## Cite

An arXiv version will follow; its identifier will be added here on the day it appears. Until then: @techreport{fulvian2026engraft, title = {{ENGRAFT}: writing a hundred facts into the n-gram memory table of a mixture-of-experts language model by gradient descent on a usage corpus}, author = {fulvian}, year = {2026}, month = sep, type = {Technical report, version 3}, url = {https://engraft-engram.dev/paper/} }