In-Weight FPGA: Compiling Multi-Step Dialectical State Machines into Dormant Transformer Neurons via Zero-KV-Drift Closed-Form Linear Algebra
layers[L-1].mlp) is the unique nonlinear computational block that executes strictly after all self-attention KV caches have been written and immediately before the final normalization and unembedding projection. By decoupling upper-middle-layer evidence verification from terminal-layer state-machine execution, we compile multi-step conditional state machines directly into ~0.035% of dormant SwiGLU neurons in 30 seconds of closed-form linear algebra with zero gradient steps and zero inference scaffolding. Across 3,780 unfiltered 3-turn adversarial dialogues on Qwen3-8B and Meta-Llama-3.1-8B-Instruct, Pure WSP passes 12 out of 12 paired bootstrap 95% confidence interval tests, outperforms reinforcement learning (SelfD-RL), generalizes to a second 4-path hierarchical 2-step branching program with 98.75% transition accuracy and 0.00% single-turn false triggers, and accelerates vLLM inference by 6–10×.
1. The Dialectical Dilemma & Why Existing Paradigms Fail
When deployed as autonomous reasoning agents or scientific collaborators, Large Language Models (LLMs) must engage in dialectical reasoning: defending valid derivations against specious counterarguments (Holding) while revising flawed derivations when presented with genuine critiques (Yielding). Recent work, “Benchmarks Don't Talk Back: Learning Through Self-Disagreement” (arXiv:2611.17563), formalized this evaluation via a three-turn protocol (Thesis $\to$ Antithesis $\to$ Assessment & Synthesis) across seven reasoning benchmarks (AMC, MATH-500, OlympiadBench, MMLU-Pro, ARC-Challenge, StrategyQA, and CRUXEval-O).
That evaluation exposed a fundamental tension across both reinforcement learning and mechanistic weight editing:
- Sycophantic Capitulation in Unedited Base Models: Frontier open-weights models such as
Qwen3-8Babandon their own correct initial answer on over 43% of problems when challenged by a misleading counterargument. - Reward-Hacking Stubbornness in Reinforcement Learning (
SelfD-RL): Because initial answers are correct more often than adversarial critiques on mastered benchmarks, RL policies rapidly exploit the reward structure by becoming unconditionally stubborn. OnQwen3-8B,SelfD-RLraises Holding from53.3%to88.9%, but causes Yielding to collapse from84.1%to53.3%(-30.8 percentage points), yielding only a+2.4%net gain in Dialectical Competence ($\frac{1}{2}(\text{Hold} + \text{Yield})$). - Autoregressive KV-Cache Drift in Mid-Layer Editing (
ROME/MEMIT/RepE): Any residual perturbation $+\Delta h^{(l)}$ injected at an intermediate layer ($l < L-1$) writes directly into the self-attention key-value caches $\mathrm{KV\_Cache}_{l+1 \dots L-1}$. Over hundreds of free-generation steps invLLMormodel.generate(), attention heads repeatedly attend to these perturbed KV states, triggering phase drift, repetition loops, or dilution by downstream sycophantic copying heads.
MLP[L-1]), guaranteeing zero recurrent KV-cache drift.
2. Five Mechanistic Interpretability Discoveries
Beyond delivering a practical 30-second compiler, our investigation uncovers five structural laws governing how modern pre-norm SwiGLU Transformers process conflicting evidence and execute discrete state transitions:
Discovery 1: The Dual-Circuit Anatomy of Sycophancy (Verdict vs. Recency Copying)
Prior interpretability work typically models sycophancy as a single linear “sycophancy direction” in activation space. Our decoupled stage ablations prove that sycophancy in multi-turn reasoning is actually driven by two spatially separated circuits:
- An Upper-Middle Epistemic Verification Circuit (Layers
0.66L..0.84L, e.g., Layers 24–30 in Qwen3-8B) that compares the logical validity of the initial Thesis against the challenger's Antithesis; and - Terminal-Layer Recency Answer-Copying Heads (Layers
0.85L..L-1, e.g., Layers 29–35) that copy the most recently mentioned final answer in the context window—namely, the challenger's wrong answer—regardless of the verbal verdict emitted earlier in the turn.
Consequently, editing only the intermediate verification circuit (Stages I+II without Stage III) improves the initial verbal verdict _assessment_: (incorrect), yet still suffers ~17.5% Verdict-Answer Desynchronization over 1,000-token free generation because unedited terminal copying heads overwrite the final answer at the end of the turn.
Discovery 2: Read-Write Decoupling & Terminal-Layer Operator KV-Cache Invariance
Within the final Transformer block $L-1$, the self-attention module $\mathrm{SelfAttn}_{L-1}$ projects and caches $(K_{L-1}(t), V_{L-1}(t))$ from $\mathrm{Norm}(x^{(L-1)}(t))$ strictly before $\mathrm{MLP}_{L-1}$ is evaluated:
$$h^{(L-1)}(t) = x^{(L-1)}(t) + \mathrm{SelfAttn}_{L-1}\big(\mathrm{Norm}(x^{(L-1)}(t)); \mathrm{KV}_{L-1}\big), \quad y^{(L-1)}(t) = h^{(L-1)}(t) + \mathrm{MLP}_{L-1}\big(\mathrm{Norm}(h^{(L-1)}(t))\big).$$
Proposition 1 (Terminal-Layer KV-Cache Invariance): For any fixed input token sequence $(t_0, \dots, t_k)$, modifying a subset of channels $\mathcal{C}$ exclusively within $\mathrm{MLP}_{L-1}$ leaves the key-value caches $(K_l, V_l)$ of all layers $l \in \{0, \dots, L-1\}$ bit-for-bit identical (verified empirically across all 36 layers of Qwen3-8B: $\max_{l} \|\Delta K_l\|_\infty = \mathbf{0.0}$ and $\max_{l} \|\Delta V_l\|_\infty = \mathbf{0.0}$). Thus, discrete state-machine transitions compiled into $\mathrm{MLP}_{L-1}$ alter next-token logits without injecting any continuous perturbation into the recurrent KV cache.
Discovery 3: Dormant SwiGLU Neurons as Hardware Boolean Gates via bfloat16 Residual Mantissa Absorption
How can 38 state-machine neurons compiled into $\mathrm{MLP}_{L-1}$ fire deterministically on their target state manifold $\mathcal{X}_+$ while exerting zero crosstalk on normal generation steps $\mathcal{X}_-$?
In IEEE 754 bfloat16 arithmetic (1 sign bit, 8 exponent bits, 7 mantissa bits; unit roundoff $u = 2^{-8} \approx 3.91 \times 10^{-3}$), while standalone $\mathrm{SiLU}(-15.0) = -4.59 \times 10^{-6}$ is representable as an isolated sub-unit float, adding a dormant neuron's output $\Delta h = w_{\mathrm{down},c} \cdot \mathrm{SiLU}(z) \cdot \langle w_{\mathrm{up},c}, x \rangle$ ($\|\Delta h\|_\infty = 7.63 \times 10^{-6}$ at $z = -15.0$) to the Layer 35 residual stream $h^{(35)}$ ($\|h^{(35)}\|_2 \approx 180$) satisfies $|\Delta h_i| \ll 2^{-8} |h_i^{(35)}|$:
By aligning $w_{\mathrm{gate},c} = s_i (k_i - \theta_i b_i)$ along a whitened Fisher separator $k_i$ relative to an anchor $b_i$ such that $z(x) \le -15.0$ on $\mathcal{X}_-$ and $z(x) \ge +14.7$ on $\mathcal{X}_+$, dormant SwiGLU channels behave as hardware-level boolean conditional gates.
Discovery 4: Unembedding Geometry & Closed-Form Elimination of Anisotropic Glitch Tokens
When porting our compiler from Qwen3-8B to Meta-Llama-3.1-8B-Instruct ($|V| = 128,256$), we observed that steering along the naive unembedding row $\gamma_{\mathrm{norm}} \odot W_U[t]$ for control tokens such as <|eot_id|> caused the model to emit bizarre glitch tokens such as .scalablytyped and ://.
Spectral analysis of $W_U$ revealed that Llama-3.1 contains extreme-norm outlier rows ($\|W_U[\texttt{.scalablytyped}]\|$ is over $8\times$ larger than $\|W_U[\texttt{<|eot\_id|>}]\|$). Formulating the Whitened Unembedding Dual Basis:
transforms the post-RMSNorm logit projection into $W_U(\gamma_{\mathrm{norm}} \odot r_t) \propto W_U(W_U^\top W_U + \lambda_u I)^{-1}W_U[t] \approx e_t$, completely eliminating cross-vocabulary glitch tokens across architectures.
Discovery 5: Constructive Sufficiency via Multi-Program Compilation (Program 2: A 4-Path Hierarchical 3-Way + 2-Subway State Machine)
Following Feynman's dictum—“What I cannot create, I do not understand”—the strongest test of mechanistic interpretability is not observational ablation, but constructive compilation of new algorithms into weights. Using the exact same closed-form compiler, we compiled a second, structurally distinct program (Program 2) into 50 dormant Layer 35 neurons of Qwen3-8B:
- Step-1 Three-Way Branching at
_assessment_: (: Routes into(incorrect)(Hold path),(correct)(Yield path), or(match)(Consensus path, when both Thesis and Antithesis agree and are valid). - Step-2 Domain-Conditional Sub-Branching inside
(match)at[Domain: Once on the_assessment_: (match):\n[Domaintrajectory, Layer 35 executes a second internal branch based on the problem's cognitive domain—emitting=Math] Algebraic check confirms derivation;\n_final_answer_: (<ans>)on mathematical/symbolic tasks (num,value) versus=Logic) Conceptual review validates conclusion/\n_final_answer_: (<ans>)on commonsense/MCQ tasks (bool,mcq).
Evaluated on 80 unseen half=1 dialogues and all 327 single-turn benchmark questions (thesis_prompt), Program 2 achieves a 98.75% (79/80) exact multi-step state transition success rate, 100.00% (34/34) Step-2 domain sub-branching accuracy, 86.25% final answer accuracy, and 0.00% (0/327) false triggers on standard single-turn prompts.
3. Scaling Laws: Why Pure WSP Becomes Even Stronger at 70B, 405B, and MoE Scale
- Blessing of Dimensionality ($\mathcal{O}(1/\sqrt{d_{\mathrm{model}}})$): As $d_{\mathrm{model}}$ grows from $4096$ (8B) to $8192$ (70B) and $16384$ (405B), concentration of measure shrinks random inner-product fluctuations between unrelated state manifolds at rate $\mathcal{O}(1/\sqrt{d_{\mathrm{model}}})$, widening the guard-nullspace margin and improving the conditioning of $W_U^\top W_U + \lambda_u I$.
- Vanishing Edit Sparsity (
0.035%$\to$0.004%): The number of neurons required to compile an $N$-step state machine depends only on the number of symbolic transitions ($N \approx 35\text{--}50$), independent of model size. At 70B scale ($L=80, d_{\mathrm{ffn}}=28672$), compiling the same state machine modifies only~0.004%of total parameters. - Constant-Time Closed-Form Compilation vs. Exponential RL Compute: While multi-turn RL on a 70B model requires 8–16 H100 GPUs for days, inverting an $8192 \times 8192$ covariance matrix on GPU takes under
0.05seconds—completing the entire 3-stage compilation in ~2 minutes with zero gradient steps.
4. Empirical Results Across 3,780 Dialogues & Mechanistic Controls
Table 1: Main Evaluation Across 3,780 Unfiltered 3-Turn Adversarial Dialogues (vLLM Greedy Decoding, 10,000-Iteration Paired Bootstrap 95% CIs)
| Model & Architecture | Benchmark Suite (Dialogues) | Method / Stage | Synthesis (↑) | Holding (↑) | Yielding (↑) | Dialectic (↑) | Capitulation (↓) | Bootstrap 95% CI |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B (Paper arXiv:2611.17563) |
Paper 7-Task Suite | Paper Base | 62.80% | 53.30% | 84.10% | 68.70% | 10.30% | RL Reference |
| Paper SelfD-RL | 73.50% | 88.90% | 53.30% (-30.8%) | 71.10% (+2.4%) | 1.00% | |||
| Qwen3-8B (L=36, d=4096) |
Suite 1: v4k Full (654) | Unedited Base | 71.41% | 56.65% | 90.83% | 73.74% | 11.93% | 3 / 3 Passed |
| Pure WSP (Ours) | 79.36% (+7.95%) | 93.64% (+36.99%) | 80.00% | 86.82% (+13.08%) | 1.83% (-10.10%) | |||
| Qwen3-8B (Frozen v4k half=0 weights) |
Suite 2: f Unseen (1,272) 0% question overlap |
Unedited Base | 73.19% | 59.79% | 90.83% | 75.31% | 12.26% | 3 / 3 Passed |
| Pure WSP (Ours) | 78.07% (+4.88%) | 86.34% (+26.55%) | 75.11% | 80.72% (+5.41%) | 4.25% (-8.01%) | |||
| Qwen3-8B (Frozen v4k half=0 weights) |
Suite 3: g Unseen (1,200) 0% question overlap |
Unedited Base | 73.25% | 56.01% | 89.20% | 72.61% | 12.83% | 3 / 3 Passed |
| Pure WSP (Ours) | 78.75% (+5.50%) | 90.03% (+34.02%) | 65.26% | 77.64% (+5.03%) | 2.83% (-10.00%) | |||
| Llama-3.1-8B-Instruct (L=32, d=4096) |
Suite 4: v4k Full (654) | Unedited Base | 39.91% | 67.91% | 41.89% | 54.90% | 10.70% | 3 / 3 Passed |
| Pure WSP (Ours) | 48.78% (+8.87%) | 94.88% (+26.97%) | 51.35% (+9.46%) | 73.12% (+18.22%) | 1.99% (-8.71%) |
Table 2: Control 1 — Evidence Verification vs. Constant Logit Bias Sweep (Unseen half=1 Pareto Frontier & AUROC)
| Method on Unseen half=1 | Branch AUROC (↑) | Yielding @ Hold ≥ 81.7% | Yielding @ Hold ≥ 85.0% | Yielding @ Hold ≥ 90.0% | Yielding @ Hold ≥ 93.0% |
|---|---|---|---|---|---|
| Unedited Base + Constant Logit Bias Sweep $b \in [-8,+8]$ | 0.6965 | 55.00% | 46.67% | 41.67% | 36.67% |
| WSP Evidence Routing + Bias Sweep | 0.8043 (+10.78%) | 68.33% (+13.33%) | 63.33% (+16.66%) | 53.33% (+11.66%) | 46.67% (+10.00%) |
| Full Pure WSP (End-to-End vLLM) | — | — | — | — | 80.00% (@ Hold=93.64%) |
Table 3: Control 2 — Decoupling Terminal-Layer FSM (Stage III) from Intermediate Evidence Verification (Stages I–II)
| Ablation Variant | Verdict Acc | Format Compliance | Verdict-Ans Consistency | Holding (↑) | Yielding (↑) | Dialectic (↑) | Synthesis (↑) |
|---|---|---|---|---|---|---|---|
| 1. Unmodified Base (Free Generation) | 47.33% | 88.89% | 77.78% | 59.15% | 90.00% | 74.58% | 75.48% |
| 2. Stage III Only (Base + Layer 35 FSM, No Stage I/II) | 69.47% | 100.00% | 95.79% | 87.32% | 51.67% | 69.50% | 73.95% |
| 3. Stages I+II Only (Stage III OFF) | 67.50% | 0.00% (within 96t) | 82.50% | 69.07% | 89.08% | 79.08% | 75.47% |
| 4. Stages II+III Only (No Stage I, Suite f) | 78.49% | 100.00% | 98.50% | 87.11% | 69.87% | 78.49% | 77.12% |
| 5. Full 3-Stage Pure WSP (Stages I+II+III) | 86.82% | 100.00% | 98.90% | 86.34% | 75.11% | 80.72% | 78.07% |