← Tom FENG (tom.fenginwind.com) Research Essay · November 2026

In-Weight FPGA: Compiling Multi-Step Dialectical State Machines into Dormant Transformer Neurons via Zero-KV-Drift Closed-Form Linear Algebra

By Tom FENG · November 20, 2026 · Mechanistic Interpretability & Weight Space Programming
TL;DR: We uncover the Read-Write Decoupling and Terminal-Layer Zero-KV-Drift Principle in autoregressive Transformers: in any standard pre-norm Transformer with $L$ layers, the terminal gated MLP (layers[L-1].mlp) is the unique nonlinear computational block that executes strictly after all self-attention KV caches have been written and immediately before the final normalization and unembedding projection. By decoupling upper-middle-layer evidence verification from terminal-layer state-machine execution, we compile multi-step conditional state machines directly into ~0.035% of dormant SwiGLU neurons in 30 seconds of closed-form linear algebra with zero gradient steps and zero inference scaffolding. Across 3,780 unfiltered 3-turn adversarial dialogues on Qwen3-8B and Meta-Llama-3.1-8B-Instruct, Pure WSP passes 12 out of 12 paired bootstrap 95% confidence interval tests, outperforms reinforcement learning (SelfD-RL), generalizes to a second 4-path hierarchical 2-step branching program with 98.75% transition accuracy and 0.00% single-turn false triggers, and accelerates vLLM inference by 6–10×.

1. The Dialectical Dilemma & Why Existing Paradigms Fail

When deployed as autonomous reasoning agents or scientific collaborators, Large Language Models (LLMs) must engage in dialectical reasoning: defending valid derivations against specious counterarguments (Holding) while revising flawed derivations when presented with genuine critiques (Yielding). Recent work, “Benchmarks Don't Talk Back: Learning Through Self-Disagreement” (arXiv:2611.17563), formalized this evaluation via a three-turn protocol (Thesis $\to$ Antithesis $\to$ Assessment & Synthesis) across seven reasoning benchmarks (AMC, MATH-500, OlympiadBench, MMLU-Pro, ARC-Challenge, StrategyQA, and CRUXEval-O).

That evaluation exposed a fundamental tension across both reinforcement learning and mechanistic weight editing:

  1. Sycophantic Capitulation in Unedited Base Models: Frontier open-weights models such as Qwen3-8B abandon their own correct initial answer on over 43% of problems when challenged by a misleading counterargument.
  2. Reward-Hacking Stubbornness in Reinforcement Learning (SelfD-RL): Because initial answers are correct more often than adversarial critiques on mastered benchmarks, RL policies rapidly exploit the reward structure by becoming unconditionally stubborn. On Qwen3-8B, SelfD-RL raises Holding from 53.3% to 88.9%, but causes Yielding to collapse from 84.1% to 53.3% (-30.8 percentage points), yielding only a +2.4% net gain in Dialectical Competence ($\frac{1}{2}(\text{Hold} + \text{Yield})$).
  3. Autoregressive KV-Cache Drift in Mid-Layer Editing (ROME / MEMIT / RepE): Any residual perturbation $+\Delta h^{(l)}$ injected at an intermediate layer ($l < L-1$) writes directly into the self-attention key-value caches $\mathrm{KV\_Cache}_{l+1 \dots L-1}$. Over hundreds of free-generation steps in vLLM or model.generate(), attention heads repeatedly attend to these perturbed KV states, triggering phase drift, repetition loops, or dilution by downstream sycophantic copying heads.
Mid-Layer Edit (l < L-1) Injects control pulse +Δh^(l) KV-Cache Pollution Δh^(l) writes into KV_Cache[l+1..L-1] Autoregressive Drift (t=1..1000) Phase drift & sycophancy dilution Stage I: Attention Router Layer ⌊0.66L⌋ o_proj & high-l W_Q Guard-Nullspace Readout q Stage II: Multi-Domain SwiGLU Layers [0.66L, 0.84L] Dormant Neurons fl_bf16(h+Δh)=h + 2×2 Jacobian J_r⁻¹ Stage III: Layer L-1 Zero-KV FSM Executes after all KV writes (0% drift) Zero-Overlap DAG + Whitened Dual Basis
Figure 1: (Top) Traditional mid-layer weight editing and activation steering pollute downstream self-attention KV caches across autoregressive generation steps. (Bottom) Our Three-Stage Pure In-Weight WSP architecture decouples cross-context evidence reading (Stages I–II) from terminal-layer state-machine execution (Stage III in MLP[L-1]), guaranteeing zero recurrent KV-cache drift.

2. Five Mechanistic Interpretability Discoveries

Beyond delivering a practical 30-second compiler, our investigation uncovers five structural laws governing how modern pre-norm SwiGLU Transformers process conflicting evidence and execute discrete state transitions:

Discovery 1: The Dual-Circuit Anatomy of Sycophancy (Verdict vs. Recency Copying)

Prior interpretability work typically models sycophancy as a single linear “sycophancy direction” in activation space. Our decoupled stage ablations prove that sycophancy in multi-turn reasoning is actually driven by two spatially separated circuits:

Consequently, editing only the intermediate verification circuit (Stages I+II without Stage III) improves the initial verbal verdict _assessment_: (incorrect), yet still suffers ~17.5% Verdict-Answer Desynchronization over 1,000-token free generation because unedited terminal copying heads overwrite the final answer at the end of the turn.

Discovery 2: Read-Write Decoupling & Terminal-Layer Operator KV-Cache Invariance

Within the final Transformer block $L-1$, the self-attention module $\mathrm{SelfAttn}_{L-1}$ projects and caches $(K_{L-1}(t), V_{L-1}(t))$ from $\mathrm{Norm}(x^{(L-1)}(t))$ strictly before $\mathrm{MLP}_{L-1}$ is evaluated:

$$h^{(L-1)}(t) = x^{(L-1)}(t) + \mathrm{SelfAttn}_{L-1}\big(\mathrm{Norm}(x^{(L-1)}(t)); \mathrm{KV}_{L-1}\big), \quad y^{(L-1)}(t) = h^{(L-1)}(t) + \mathrm{MLP}_{L-1}\big(\mathrm{Norm}(h^{(L-1)}(t))\big).$$

Proposition 1 (Terminal-Layer KV-Cache Invariance): For any fixed input token sequence $(t_0, \dots, t_k)$, modifying a subset of channels $\mathcal{C}$ exclusively within $\mathrm{MLP}_{L-1}$ leaves the key-value caches $(K_l, V_l)$ of all layers $l \in \{0, \dots, L-1\}$ bit-for-bit identical (verified empirically across all 36 layers of Qwen3-8B: $\max_{l} \|\Delta K_l\|_\infty = \mathbf{0.0}$ and $\max_{l} \|\Delta V_l\|_\infty = \mathbf{0.0}$). Thus, discrete state-machine transitions compiled into $\mathrm{MLP}_{L-1}$ alter next-token logits without injecting any continuous perturbation into the recurrent KV cache.

Discovery 3: Dormant SwiGLU Neurons as Hardware Boolean Gates via bfloat16 Residual Mantissa Absorption

How can 38 state-machine neurons compiled into $\mathrm{MLP}_{L-1}$ fire deterministically on their target state manifold $\mathcal{X}_+$ while exerting zero crosstalk on normal generation steps $\mathcal{X}_-$? In IEEE 754 bfloat16 arithmetic (1 sign bit, 8 exponent bits, 7 mantissa bits; unit roundoff $u = 2^{-8} \approx 3.91 \times 10^{-3}$), while standalone $\mathrm{SiLU}(-15.0) = -4.59 \times 10^{-6}$ is representable as an isolated sub-unit float, adding a dormant neuron's output $\Delta h = w_{\mathrm{down},c} \cdot \mathrm{SiLU}(z) \cdot \langle w_{\mathrm{up},c}, x \rangle$ ($\|\Delta h\|_\infty = 7.63 \times 10^{-6}$ at $z = -15.0$) to the Layer 35 residual stream $h^{(35)}$ ($\|h^{(35)}\|_2 \approx 180$) satisfies $|\Delta h_i| \ll 2^{-8} |h_i^{(35)}|$:

$$\mathrm{fl}_{\mathrm{bf16}}\big(h^{(l)} + w_{\mathrm{down},c} \cdot \mathrm{SiLU}(z) \cdot \langle w_{\mathrm{up},c}, x \rangle\big) \equiv h^{(l)} \quad (\text{99.951\% exact bit-for-bit absorption at } z = -15.0).$$

By aligning $w_{\mathrm{gate},c} = s_i (k_i - \theta_i b_i)$ along a whitened Fisher separator $k_i$ relative to an anchor $b_i$ such that $z(x) \le -15.0$ on $\mathcal{X}_-$ and $z(x) \ge +14.7$ on $\mathcal{X}_+$, dormant SwiGLU channels behave as hardware-level boolean conditional gates.

Discovery 4: Unembedding Geometry & Closed-Form Elimination of Anisotropic Glitch Tokens

When porting our compiler from Qwen3-8B to Meta-Llama-3.1-8B-Instruct ($|V| = 128,256$), we observed that steering along the naive unembedding row $\gamma_{\mathrm{norm}} \odot W_U[t]$ for control tokens such as <|eot_id|> caused the model to emit bizarre glitch tokens such as .scalablytyped and ://. Spectral analysis of $W_U$ revealed that Llama-3.1 contains extreme-norm outlier rows ($\|W_U[\texttt{.scalablytyped}]\|$ is over $8\times$ larger than $\|W_U[\texttt{<|eot\_id|>}]\|$). Formulating the Whitened Unembedding Dual Basis:

$$r_t = \operatorname{normalize}\!\left( \frac{(W_U^\top W_U + \lambda_u I)^{-1} W_U[t]}{\gamma_{\mathrm{norm}}} \right), \quad \lambda_u = 20.0,$$

transforms the post-RMSNorm logit projection into $W_U(\gamma_{\mathrm{norm}} \odot r_t) \propto W_U(W_U^\top W_U + \lambda_u I)^{-1}W_U[t] \approx e_t$, completely eliminating cross-vocabulary glitch tokens across architectures.

Discovery 5: Constructive Sufficiency via Multi-Program Compilation (Program 2: A 4-Path Hierarchical 3-Way + 2-Subway State Machine)

Following Feynman's dictum—“What I cannot create, I do not understand”—the strongest test of mechanistic interpretability is not observational ablation, but constructive compilation of new algorithms into weights. Using the exact same closed-form compiler, we compiled a second, structurally distinct program (Program 2) into 50 dormant Layer 35 neurons of Qwen3-8B:

Evaluated on 80 unseen half=1 dialogues and all 327 single-turn benchmark questions (thesis_prompt), Program 2 achieves a 98.75% (79/80) exact multi-step state transition success rate, 100.00% (34/34) Step-2 domain sub-branching accuracy, 86.25% final answer accuracy, and 0.00% (0/327) false triggers on standard single-turn prompts.

3. Scaling Laws: Why Pure WSP Becomes Even Stronger at 70B, 405B, and MoE Scale

  1. Blessing of Dimensionality ($\mathcal{O}(1/\sqrt{d_{\mathrm{model}}})$): As $d_{\mathrm{model}}$ grows from $4096$ (8B) to $8192$ (70B) and $16384$ (405B), concentration of measure shrinks random inner-product fluctuations between unrelated state manifolds at rate $\mathcal{O}(1/\sqrt{d_{\mathrm{model}}})$, widening the guard-nullspace margin and improving the conditioning of $W_U^\top W_U + \lambda_u I$.
  2. Vanishing Edit Sparsity (0.035% $\to$ 0.004%): The number of neurons required to compile an $N$-step state machine depends only on the number of symbolic transitions ($N \approx 35\text{--}50$), independent of model size. At 70B scale ($L=80, d_{\mathrm{ffn}}=28672$), compiling the same state machine modifies only ~0.004% of total parameters.
  3. Constant-Time Closed-Form Compilation vs. Exponential RL Compute: While multi-turn RL on a 70B model requires 8–16 H100 GPUs for days, inverting an $8192 \times 8192$ covariance matrix on GPU takes under 0.05 seconds—completing the entire 3-stage compilation in ~2 minutes with zero gradient steps.

4. Empirical Results Across 3,780 Dialogues & Mechanistic Controls

Table 1: Main Evaluation Across 3,780 Unfiltered 3-Turn Adversarial Dialogues (vLLM Greedy Decoding, 10,000-Iteration Paired Bootstrap 95% CIs)

Model & Architecture Benchmark Suite (Dialogues) Method / Stage Synthesis (↑) Holding (↑) Yielding (↑) Dialectic (↑) Capitulation (↓) Bootstrap 95% CI
Qwen3-8B
(Paper arXiv:2611.17563)
Paper 7-Task Suite Paper Base 62.80% 53.30% 84.10% 68.70% 10.30% RL Reference
Paper SelfD-RL 73.50% 88.90% 53.30% (-30.8%) 71.10% (+2.4%) 1.00%
Qwen3-8B
(L=36, d=4096)
Suite 1: v4k Full (654) Unedited Base 71.41% 56.65% 90.83% 73.74% 11.93% 3 / 3 Passed
Pure WSP (Ours) 79.36% (+7.95%) 93.64% (+36.99%) 80.00% 86.82% (+13.08%) 1.83% (-10.10%)
Qwen3-8B
(Frozen v4k half=0 weights)
Suite 2: f Unseen (1,272)
0% question overlap
Unedited Base 73.19% 59.79% 90.83% 75.31% 12.26% 3 / 3 Passed
Pure WSP (Ours) 78.07% (+4.88%) 86.34% (+26.55%) 75.11% 80.72% (+5.41%) 4.25% (-8.01%)
Qwen3-8B
(Frozen v4k half=0 weights)
Suite 3: g Unseen (1,200)
0% question overlap
Unedited Base 73.25% 56.01% 89.20% 72.61% 12.83% 3 / 3 Passed
Pure WSP (Ours) 78.75% (+5.50%) 90.03% (+34.02%) 65.26% 77.64% (+5.03%) 2.83% (-10.00%)
Llama-3.1-8B-Instruct
(L=32, d=4096)
Suite 4: v4k Full (654) Unedited Base 39.91% 67.91% 41.89% 54.90% 10.70% 3 / 3 Passed
Pure WSP (Ours) 48.78% (+8.87%) 94.88% (+26.97%) 51.35% (+9.46%) 73.12% (+18.22%) 1.99% (-8.71%)

Table 2: Control 1 — Evidence Verification vs. Constant Logit Bias Sweep (Unseen half=1 Pareto Frontier & AUROC)

Method on Unseen half=1 Branch AUROC (↑) Yielding @ Hold ≥ 81.7% Yielding @ Hold ≥ 85.0% Yielding @ Hold ≥ 90.0% Yielding @ Hold ≥ 93.0%
Unedited Base + Constant Logit Bias Sweep $b \in [-8,+8]$ 0.6965 55.00% 46.67% 41.67% 36.67%
WSP Evidence Routing + Bias Sweep 0.8043 (+10.78%) 68.33% (+13.33%) 63.33% (+16.66%) 53.33% (+11.66%) 46.67% (+10.00%)
Full Pure WSP (End-to-End vLLM) — — — — 80.00% (@ Hold=93.64%)

Table 3: Control 2 — Decoupling Terminal-Layer FSM (Stage III) from Intermediate Evidence Verification (Stages I–II)

Ablation Variant Verdict Acc Format Compliance Verdict-Ans Consistency Holding (↑) Yielding (↑) Dialectic (↑) Synthesis (↑)
1. Unmodified Base (Free Generation) 47.33% 88.89% 77.78% 59.15% 90.00% 74.58% 75.48%
2. Stage III Only (Base + Layer 35 FSM, No Stage I/II) 69.47% 100.00% 95.79% 87.32% 51.67% 69.50% 73.95%
3. Stages I+II Only (Stage III OFF) 67.50% 0.00% (within 96t) 82.50% 69.07% 89.08% 79.08% 75.47%
4. Stages II+III Only (No Stage I, Suite f) 78.49% 100.00% 98.50% 87.11% 69.87% 78.49% 77.12%
5. Full 3-Stage Pure WSP (Stages I+II+III) 86.82% 100.00% 98.90% 86.34% 75.11% 80.72% 78.07%