~/shanegraffiti.com/research/dmoe

Shane Graffiti Inc. Semantic Adversarial Research Division 2026

DECOUPLED
MIXTURE
OF EXPERTS
FOR PARAMETRIC KNOWLEDGE

DMoE solves the core architectural bottleneck of LLM knowledge injection. RAG keeps knowledge outside the model shallow, prompt-level, retrieval-bound. Post-training writes knowledge into shared parameters deep but catastrophically fragile, conflict-prone, expensive to update. DMoE decouples both experts and router from the base model entirely: external corpora become independently updatable LoRA modules, activated only when token uncertainty signals the model lacks sufficient knowledge, placed exclusively at the final-layer FFN to preserve KV-cache reuse throughout autoregressive generation.

Division Semantic Adversarial Research

Domain LLM Knowledge Injection / MoE

Published arXiv 2606.14243 2026

Key Result Best on 11 of 14 metrics, 3× faster than FLARE

read

~/dmoe/problem

§ 1.0 The Architectural Bottleneck

LLM parametric knowledge is frozen at pre-training. Domain-specific facts go stale. Time-sensitive queries hallucinate. The two dominant responses to this both fail at the architecture level not the implementation level.

Paradigm 01

RAG

Keeps knowledge in an external corpus, retrieved at inference and appended as prompt context. Easy to update just modify the index. But knowledge never enters parameter space. It's exposed to the model as input tokens, not internalized as weights. Inference cost scales with retrieval and long-context attention recomputation on every query.

Shallow prompt-level only

Paradigm 02

Post-Training

SFT or LoRA writes new knowledge directly into shared parameters. Deep integration knowledge becomes part of the model's weights. But those weights already encode diverse pre-trained knowledge. Continuous updates cause catastrophic forgetting, knowledge conflict, and require full retraining as the corpus changes.

Brittle shared parameter space

Paradigm 03

DMoE

Decouples both experts and router from the base model. Each knowledge unit becomes an independent LoRA adapter stored externally. A lightweight router activates relevant experts only when uncertainty signals insufficient knowledge. Base model parameters never change. KV-cache remains valid throughout generation.

Modular independently updatable

read

~/dmoe/architecture

§ 2.0 Architecture Three Components

DMoE has three parts: a frozen base model, a decoupled router, and a bank of decoupled expert modules. None of the three are jointly trained. The router and experts are built after the base model is fixed, and they can be added, removed, or replaced without touching the backbone.

Frozen Base Model

The backbone LLM is never modified. Its parameters θ remain fixed throughout expert construction, router training, and inference. This is the structural guarantee that prevents knowledge conflict and catastrophic forgetting.

Expert Modules

Each knowledge unit K_i is converted into a LoRA adapter Δθ_i trained on instruction-style augmentations of that document. Rank 4, α = 16. One expert per passage. 27,613 experts in the full corpus bank. Each expert is ~481 KiB on disk, stored externally, loaded on demand.

Uncertainty-Aware Router

At each decoding step t, Token Uncertainty (TU) = entropy of the softmax distribution over vocabulary. If TU > τ (default 2.0), routing fires. The router is a BM25 inverted index over expert text surrogates training-free, incrementally updatable, no neural encoder needed.

Final-Layer Attachment

Experts attach only at the final transformer layer's FFN. This is the only placement that doesn't break KV-cache compatibility. Any earlier layer modifies hidden states that feed subsequent attention blocks, requiring KV recomputation. The last FFN output goes directly to logits it doesn't affect cached keys or values.

Effective Parameters Per-Step Composition θ^eff_t = θ + Σ_{Eᵢ ∈ E_sel} Δθᵢ // θ: frozen base model never changed // Δθᵢ: selected expert adapters loaded transiently // θ^eff_t: temporary composition for this decoding state only // E_sel = Top-k BM25(q_t, Dᵢ) k=3 by default

read

~/dmoe/routing

§ 3.0 Token Uncertainty as the Trigger Signal

The router solves two subproblems independently: when to activate experts, and which experts to activate. Token Uncertainty (TU) handles the first. BM25 handles the second. They are not jointly trained and ablations confirm both matter.

01

Compute Token Uncertainty

At each decoding step t: TU_t = −Σ_v p_t(v) log p_t(v). The entropy of the softmax distribution over vocabulary. High entropy = flat distribution = the model is uncertain about its next token = it likely lacks relevant knowledge. Low entropy = confident = no intervention needed.

02

Triggering Decision

Trigger_t = 𝟙[TU_t > τ]. Default τ = 2.0. If triggered: deactivate current experts, run BM25 retrieval, load new experts. If not triggered: continue decoding with existing model state. The active expert set persists until the next triggering event avoiding redundant expert swaps at every token.

03

BM25 Expert Selection

Routing query q_t is constructed from the task input plus the generated prefix up to (but excluding) the triggering position so retrieved experts are conditioned on exactly the context already in the KV cache. E_sel = Top-k BM25(q_t, D_i). Lexical, training-free, incrementally updatable: adding an expert just inserts D_i into the inverted index.

04

Robustness Verified

Ablation Table 6 (paper): Full TU+BM25 achieves EM=0.1800, F1=0.2553. Random trigger + BM25 drops to 0.1667. TU + random router drops to 0.1767. Always trigger collapses to 0.1200. Both trigger precision and expert selection contribute and TU monotonically aligns with knowledge difficulty across entropy bins (Table 5).

read

~/dmoe/kv-cache

§ 4.0 Why Final-Layer FFN Is Non-Negotiable

This is the key engineering insight. Placing experts at any intermediate layer breaks KV-cache compatibility. The proof is mechanical and the empirical table confirms it.

The Cache Compatibility Constraint Standard: h_ℓ+1 = F_ℓ(h_ℓ) // KV cache stores K^cached_j = W_Kh_j for all j ≤ L With expert at layer k: h̃_k+1 = h_k+1 + Δ(h_k) // This propagates forward: all h̃_j ≠ h_j for j > k // Cached KV for those layers is now invalid must recompute Final layer only: ∀j ≤ L, h̃_j = h_j // Last FFN output → logits only, never feeds attention again // All cached KV states remain valid. Zero recomputation overhead.

Table 3 (paper) confirms this empirically. Inserting experts into the final 25%, 50%, 75%, or 100% of FFN layers consistently underperforms the only-last configuration. Only attaching to the single last layer achieves best overall accuracy while preserving cache compatibility. The performance gap isn't marginal earlier placement actively degrades quality because KV mismatch corrupts the attention context.

read

~/dmoe/results

§ 5.0 Results Four Benchmarks, Two Models

Evaluated against Basic-RAG, FLARE, PRAG, and SFT-LoRA on HotpotQA (multi-hop), ComplexWebQuestions (open-domain compositional), Quasar-T (open-domain trivia), and StrategyQA (implicit multi-hop). All methods share the same 27,613-passage Wikipedia corpus, same BM25 retriever, same retrieval budget (k=3), greedy decoding.

11/14

Best or Tied-Best Metrics

3×

Faster Than FLARE

1.9×

Less GPU Memory vs FLARE

7.2×

Less Memory than OLMoE

7.5×

Faster Than OLMoE SFT-LoRA

EM Score Llama-3.2-1B Base Model

Method	CWQ EM	CWQ F1	HotpotQA EM	Quasar-T EM	StrategyQA ACC
DMoE	0.2467	0.3479	0.1800	0.3133	0.5667
PRAG	0.2500	0.3284	0.0733	0.2200	0.5600
FLARE	0.2400	0.3154	0.0733	0.1867	0.5367
SFT-LoRA	0.2167	0.3092	0.0767	0.2133	0.5533
Basic-RAG	0.1633	0.2384	0.1700	0.2800	0.4333

Efficiency Average Per-Sample Inference

Method	Time (s)	GPU Memory (GB)
FLARE	9.2643	13.97
DMoE	2.6656	7.24
SFT-LoRA	1.6700	4.82
PRAG	1.3600	4.83
Basic-RAG	1.8900	2.54

read

~/dmoe/scalability

§ 6.0 The Expert Bank Storage, Not VRAM

The conventional MoE bottleneck is that all experts must reside in GPU memory simultaneously. DMoE inverts this: experts live on disk, only the top-k selected experts are loaded per triggering event. Bank size scales disk footprint not inference VRAM.

Per-Expert Footprint

LoRA rank 4, α=16: 122,880 trainable parameters per expert. Approximately 481 KiB per expert on disk. The full 27,613-expert corpus bank occupies 13.08 GiB of disk storage no GPU memory pressure from inactive experts.

Training Cost

One new passage-level expert: ~10 seconds on a single A100. Training is embarrassingly parallel because experts are independent. The 27,613-expert bank corresponds to ~76.7 A100 GPU-hours total, distributable across any number of GPUs.

Update Model

Adding knowledge: train new expert, insert its text surrogate into the BM25 index. Removing or updating knowledge: delete the adapter directory, remove from index. The backbone and all unrelated experts are untouched. No re-training, no re-merging.

Bank Size Robustness

Table 9 (paper): reducing the bank to 1/10 or 1/5 of full size shows near-identical downstream performance. HotpotQA slightly improves at 1/10. DMoE does not require a precisely tuned expert count the router compensates with BM25 precision over whatever bank size is available.

read

~/conclusion

$ query: what does DMoE change // Knowledge doesn't have to live in prompt or in shared weights. // It can live in an external bank of isolated, updatable adapters. // The base model never changes. The KV-cache never breaks. $ query: what does this cost // 10 seconds per expert on one A100. Embarrassingly parallel. // 481 KiB per expert on disk. Active VRAM: top-k experts only. // Router is BM25. No neural encoder. Training-free. Updatable. $ query: what is the actual result // Best on 11 of 14 effectiveness metrics across 4 benchmarks. // 3× faster than FLARE. 7.5× faster than coupled MoE baseline. // Decoupled. Cache-safe. Modular. Independently updatable.

read

THE
EXPERTS
DON'T
NEED
TO LIVE
INSIDE.

read

DECOUPLED MIXTURE OF EXPERTS FOR PARAMETRIC KNOWLEDGE

DECOUPLED
MIXTURE
OF EXPERTS
FOR PARAMETRIC KNOWLEDGE