# ENGRAFT open problems: what is not measured yet URL: https://engraft-engram.dev/open-problems/ Updated: 2026-09-26 > The open list of ENGRAFT: measurements not yet run, most of them proposed by readers, with credit to each proposer. Open problems # What is not measured yet These are the measurements we have not run. Most of them came from readers who checked our files, and each keeps the name of the person who raised it. The list is the plan for what comes next. ## The comparisons that decide the method's place - ### Retrieval over the same corpus, at the same storage budget Scored on recall, latency, neutral-text KL and bytes per fact. The half we can state today is unflattering: the overlay is 9,036,620 bytes for 100 facts, while the training sentences behind it come to 749 tokens per fact. What we expect to win on, zero context tokens and no retriever, is unmeasured. u/Rama_Surasani_ - ### A head-to-head with LoRA and with ROME/MEMIT Same facts, same model, same damage yardstick. The comparison we most want to see, and the one a reviewer will ask for first. - ### Fact updates Write «X is A», later «X is B» under the same trigger: which one does the model answer? Two arms: regenerate the overlay with B in place of A, or descend B on top of A's rows. Never measured. u/Fine-Drummer2604 - ### A second model DeepSeek V4.1 Flash carries an Engram-style table with a different layout: MXFP8 rows, a compressed tokenizer, two table layers, 4-grams and a value projection. The method needs adapting before anything can be measured. ## How the rows behave - ### Facts per subject as the capacity axis The 24/100/300 curve had about one fact per subject, so it counts subjects, not crowding. It needs fixed subjects holding 1 to 20 facts each. u/nitish-kmr - ### Perturbed subject names Every overlay row a test sentence touches sits on the tokens of the subject's name. Run the same 841 sentences with the name lowercased, with a typo, a short form, an attached particle. Expected: lowercase and typos fail, attached particles partly survive. u/nitish-kmr, from u/neoneye2's note on the row table - ### Two-turn follow-ups with a pronoun The name in the first turn, the question in the second with a pronoun. Whether attention carries the fact across a whole turn is unknown. u/nitish-kmr - ### Distance from the name to the answer, crossed with position Accuracy rises with the distance between the name and the answer, the opposite of what a two-or-three-token window suggests, and at fixed distance the position still matters. Small bins; a paired test settles it. u/nitish-kmr - ### What carries the fact past the table's reach The table writes the fact where the name appears, and a small built-in convolution carries it forward by 3, 6 and 9 tokens; beyond that, only attention reaches the answer. Called before looking: if that short memory matters, answers 1, 4, 7 and 10 tokens after the name do a little better than their neighbours, and accuracy drops between 10 and 11. Bins per token up to 12, a step counts only if all three published runs show it, and the curve is redone at fixed row contention. u/nitish-kmr - ### The sentence-initial gap, per run and paired Exact match with the subject's name at the start of the sentence minus exact match with the name further in, printed next to the headline. Then the clean version: the same sentence with the name moved, pair by pair. u/nitish-kmr - ### Repeating a fact in the usage corpus Readers who tried v0.1.0 noticed that repetition made the written fact surface more often. A cheap corpus variant, never measured systematically. u/fromhereandthere ## Protocol and reproducibility - ### A disjoint-row arm for composition All 83 composition probes ask two facts about the same subject, so they read that subject's rows by construction. Separating row interference from a readout limit needs probes that span two subjects. u/Rama_Surasani_ - ### Composition in the bare format The composition rerun used the chat template. The bare, non-chat format has not been run, and the probe tool cannot yet reproduce the rerun's prompt. u/Rama_Surasani_ - ### Chinese with equal training mass A Chinese cell given the same training mass per fact as the Italian one: the decisive test of the row-budget explanation for the language ordering. - ### Damage beyond 300 facts Collateral damage as the number of facts grows, and a quantization yardstick measured at 100 facts rather than 24. - ### The reference path, end to end A full rerun of the 100-fact cell on the public reference path, and a recorded transcript of the engine demo with the Quail overlay. ## Adding to the list Open an issue or a discussion on the repository (https://github.com/fulvian/engraft-ngram). An observation that changes what we measure goes on this list with your handle, checked at the source.