Dragon Hatchling: A careful guide to BDH's graph-inspired state-space model

2025-10-01T15:30:00+02:00

A Transformer keeps its learned knowledge in parameters and its temporary context in a growing collection of key-value vectors. The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain asks whether those roles can be reorganized around a persistent state whose individual coordinates admit a neuron-and-synapse interpretation.

The paper proposes BDH, a graph model defined by local updates, and BDH-GPU, the tensor formulation used in the experiments. BDH-GPU combines high-dimensional non-negative activations, linear attention, recurrent state, and a low-rank feed-forward mechanism. The authors then show how a norm-free form of this computation can be represented by local graph dynamics and emulated with spiking-neuron circuits.

That is a technically interesting bridge. It is not evidence that BDH reproduces a biological brain, and it is not yet a general replacement for modern large language models. The strongest experimental result is narrower: on one byte-level Europarl language-modeling and translation setup, the gated BDH-GPU′ variant follows a GPT-2-like Transformer-XL baseline closely from 25 million to 800 million parameters.

The paper is an arXiv preprint, submitted on 30 September 2025. Only version 1 was available when this article was revised.

The short version🔗

BDH-GPU is the model that was trained at scale. It uses dense tensor operations that run on current accelerators.
BDH is the graph interpretation. It stores state on neuron-to-neuron connections and updates that state through local rules.
The correspondence is qualified. The paper proves an exact special-case relationship for BDH-Normfree and gives graph encodings for the main components. Full trained BDH-GPU also uses LayerNorm, which is not a strictly local operation.
Memory is recurrent. Each layer carries a fixed-size matrix that summarizes earlier tokens rather than retaining an ever-growing KV cache.
Positivity is guaranteed; sparsity is observed. ReLU makes the activation vectors non-negative. Roughly 5% activity is reported in typical runs, but the percentage varies by layer, token, and task.
The evidence is promising but concentrated. Scaling, graph structure, concept-selective state, and model merging are evaluated on limited tasks and configurations.

Start with the distinction that matters🔗

The names BDH and BDH-GPU are easy to blur, but doing so obscures the paper's main contribution.

BDH-GPU is a state-space neural network expressed with matrices. It is the primary implementation and the source of all large-scale empirical results in the paper. Its activations live in a large dimension $n$, while values and compressed state use a smaller dimension $d$. Three shared matrices contain most of the parameters.

BDH is a local graph system. Its nodes are called neurons, its edges are called synapses, and its inference-time state is represented by changing edge weights. The graph view is designed to expose a possible local mechanism underneath the tensor computation.

The paper also defines BDH-Normfree, an intermediate form. This is the exact special case that connects the graph and tensor descriptions for a particular parameter construction. The trained BDH-GPU model adds LayerNorm and uses compressed state, so "the same model in two notations" would be too strong.

Optional detail: Why LayerNorm weakens a strictly local interpretation

Layer normalization computes statistics across all coordinates of a vector. A neuron cannot perform that calculation using only its own state and immediate edges unless the graph system adds a mechanism for aggregating and distributing those global statistics.

The paper's clean local equivalence therefore starts from BDH-Normfree. It then treats trained BDH-GPU, with LayerNorm restored, as a closely related practical architecture. This distinction does not invalidate the graph interpretation, but it separates an exact local construction from the model used in the experiments.

Optional context: Axiomatic AI and the particle-system view

The authors place BDH inside a broader program they call Axiomatic AI: describe both a model's large-scale computation and the local rules from which that computation arises. The ambition is analogous to deriving macroscopic behavior in physics from interactions between simpler components.

In that language, BDH-GPU can be viewed as an interacting particle system. Each of the $n$ activation coordinates acts like a particle with a compact state, while the learned low-rank maps and recurrent memory determine how activity propagates between coordinates. The graph formulation makes those pairwise interactions explicit as edges. This is a mathematical interpretation of the recurrence, not evidence that literal physical particles or biological neurons implement it.

One BDH-GPU layer, before the equations🔗

At time $t$, a token enters a stack of $L$ recurrent depth steps. Each step works with two $n$-dimensional, non-negative activation vectors:

$x_{t,l}$ is the key/query-like neuronal activity at time $t$ and layer $l$;
$y_{t,l}$ gates which activity contributes value information to memory.

The layer also carries a state matrix $\rho_{t,l}$. Earlier tokens have written outer products into this matrix. Multiplying it by the current $x$ reads the part of memory aligned with the current activity pattern.

One pass through a layer can be read as four operations:

Project the previous $y$ activity down to the compact $d$-dimensional space with $E$.
Project it back to $n$ dimensions with $D_x$, apply ReLU, and add it to the residual $x$ stream.
Use the resulting $x$ to query the previous recurrent state; decode the retrieved value with $D_y$, apply ReLU, and gate it coordinate-wise by $x$ to obtain $y$.
Add a rank-one outer product to the recurrent state, associating the current activity with a compact value.

The same $E$, $D_x$, and $D_y$ parameters are reused at every depth step. Each layer nevertheless keeps its own recurrent state. This weight sharing differs from a conventional Transformer, where blocks normally have separate parameters.

Figure 1. The official implementation diagram shows the two activation paths and the rank-one linear-attention update. In the repository code, the wide activation dimension is split across attention heads. (Source: Pathway's official BDH repository).

The exact recurrence: Equation (8), symbol by symbol

Using column vectors, the dimensionally consistent matrix shapes are

$$ E \in \mathbb{R}^{d \times n}, \qquad D_x,D_y \in \mathbb{R}^{n \times d}, \qquad \rho_{t,l} \in \mathbb{R}^{d \times n}. $$

The compact state-space recurrence in equation (8) is

$$ \begin{aligned} \rho_{t,l} &:= \rho_{t-1,l} + \operatorname{LN}(E y_{t,l-1})x_{t,l}^{\mathsf T}U, \\ x_{t,l} &:= x_{t,l-1} + \bigl(D_x\operatorname{LN}(E y_{t,l-1})\bigr)^+, \\ y_{t,l} &:= \bigl(D_y\operatorname{LN}(\rho_{t-1,l}x_{t,l})\bigr)^+ \odot x_{t,l}. \end{aligned} $$

Here, $t$ indexes sequence time and $l$ indexes recurrent depth. $\operatorname{LN}$ is parameter-free LayerNorm, $(z)^+=\max(0,z)$ is coordinate-wise ReLU, and $\odot$ is element-wise multiplication. $U$ is a diagonal or block-diagonal positional rotation or damping operator; the experiments use RoPE together with ALiBi-like damping.

The first term in the state update is old memory. The added outer product associates a compact value, $\operatorname{LN}(Ey)$, with the current high-dimensional activity $x$. The $y$ equation reads from $\rho_{t-1,l}$, so the current write does not feed back into the same read.

The paper contains a transpose inconsistency: some prose and figure captions describe $\rho$ as $n\times d$, while equations (5) and (8) require $d\times n$ under the displayed column-vector convention. The equations above use the dimensionally valid orientation. An implementation may store the transpose as long as every multiplication is transposed consistently.

See definitions 4 and equation (8) in the paper.

What the parameter count actually includes🔗

The matrices $E$, $D_x$, and $D_y$ contain the scaling-dominant shared core: $3nd$ parameters. A language model also needs token input and output maps $f_e$ and $f_d$. For an alphabet $\Omega$, the paper gives

$$ 3nd + 2|\Omega|d = (3+o(1))nd. $$

Calling BDH-GPU a "three-matrix model" is therefore useful architectural shorthand, not a literal inventory of every trainable number. The gated BDH-GPU′ variant used for the strongest scaling result adds further mechanisms for conditional state and logit gating.

Why this is linear attention🔗

Ordinary causal attention compares the current query with every stored key, normalizes the scores, and mixes the corresponding values. A recurrent linear-attention form changes the order of computation. It first accumulates key-value outer products into a state and then queries that state:

$$ \rho_{t-1,l} = \sum_{\tau<t} v^*_{\tau,l-1}x_{\tau,l}^{\mathsf T}U^{t-\tau}, \qquad a^*_{t,l}=\rho_{t-1,l}x_{t,l}. $$

The state has fixed shape for a fixed model, independent of the number of processed tokens. This removes a hard architectural context-window boundary from recurrent execution. It does not guarantee perfect or unlimited memory: unrelated and stale contributions can interfere, which is why the model uses positional rotation and damping.

Optional detail: Capacity, positive keys, and the LSH construction

The paper also asks how many key-value associations a fixed linear-attention state can recover approximately. Under assumptions about correlations between keys, bounded values, and a suitable key representation, Claim 7 places the capacity between approximately $\widetilde{\Omega}(\sqrt{n})$ and $\widetilde{O}(n)$ associations per layer. The tilde notation suppresses logarithmic factors. This is a theoretical scaling range, not a measured memory limit for the trained Europarl models.

Because BDH-GPU keys and queries lie in the positive orthant $(\mathbb{R}_+)^n$, the paper studies whether positive vectors can still support selective matching. Observation 7 constructs a positive vector whose coordinates represent locality-sensitive hashing (LSH) buckets: similar inputs collide in more shared buckets and therefore receive a larger dot product. Observation 8 gives a separate, specially constructed example in which small $L_1$ differences between positive keys can be separated sharply.

These results demonstrate expressiveness. The experiments do not show that BDH-GPU learns the proposed LSH encoding, nor that every small $L_1$ difference in a learned representation is automatically easy to distinguish.

Optional detail: Fixed state is not automatically linear-time code

The recurrence can be implemented by updating $\rho$ once per token, which avoids storing all earlier keys and values. The paper says that for $d=256$, its state-space kernel becomes faster and more memory-efficient than the simple causal-attention path at contexts typically longer than 4,096 tokens. It does not provide a latency table establishing that threshold across hardware and workloads.

The compact code in Appendix E and the current official bdh.py instead materialize a causal $T\times T$ score matrix for parallel training. That implementation is quadratic in sequence length. The architecture admits recurrent execution; the released teaching implementation does not by itself demonstrate the claimed long-context systems advantage.

Positive, sparse activity and the graph interpretation🔗

ReLU makes $x$ and $y$ non-negative by construction. Non-negativity gives an active coordinate a consistent sign: it contributes evidence rather than switching between positive and negative meanings. The element-wise gate in the $y$ update also suppresses coordinates unless both the decoded memory and the current $x$ activity are positive.

The paper reports that only about 5% of coordinates are non-zero in a typical run. This is an observation, not a guaranteed constant. In the synthetic repetition experiment of Figure 14, layer-2 activity ranges from roughly 4.0%-7.5% while memorizing a new pattern and falls to about 2.5% while repeating a predictable one.

Sparse activity makes the graph interpretation easier to inspect because each token changes a smaller set of candidate neuron pairs. The published dense implementation does not exploit that sparsity to guarantee a proportional wall-clock speedup.

From compressed state to synapses🔗

The graph model BDH uses a state matrix $\sigma\in\mathbb{R}^{n\times n}$. Entry $\sigma(i,j)$ is interpreted as the current strength of a directed synapse from one neuron to another. BDH-GPU stores a compressed state related to it through $\rho=E\sigma$.

In the norm-free construction, the update to $\sigma$ is an outer product of activity vectors. A connection changes when its two endpoint activities co-occur. That is why the authors call the update Hebbian-like: it resembles the abstract rule "cells that fire together strengthen their connection."

Optional detail: What the brain analogy does and does not claim

The paper constructs spiking-neuron circuits with excitatory and inhibitory components that emulate the local BDH update rules. This is a computational correspondence: one abstract dynamical system can simulate another.

It is not biological validation. The work does not fit neural recordings, model synaptic chemistry, reproduce anatomical constraints, or test whether a brain uses these equations for language. LayerNorm and the compressed tensor implementation also introduce operations that are not local in the straightforward biological sense.

The careful conclusion is that BDH offers a brain-inspired local interpretation and a possible mechanism worth studying. "The model explains how human neurons achieve speech" remains the authors' hypothesis, not an experimentally established neuroscience result.

What the scaling experiment establishes🔗

The main comparison uses a mixed language-modeling and translation task derived from Europarl, a corpus of aligned European Parliament proceedings. The models process raw UTF-8 bytes rather than subword tokens. Source and target sentences are interleaved in a continuous stream, and recurrent state is carried between temporally adjacent minibatches.

The shared parts of the protocol are substantial:

English-Polish and English-Czech data, totaling 380 MB;
1.2 billion byte tokens, approximately three passes over the data;
2,048-token training segments with state carried between segments;
AdamW, the same learning-rate schedule, adaptive gradient clipping, and weight decay;
validation next-byte prediction loss as the reported metric.

The baseline, called GPTXL, is not stock GPT-2. It is a NanoGPT-derived, GPT-2-like Transformer with a Transformer-XL-style FIFO cache of 4,096 past entries and ALiBi positional bias. Its depth, width, head count, MLP width, and tuned dropout change with model size. BDH-GPU keeps $d=256$, eight layers, and four heads, then scales $n$ from 32,768 to 1,048,576.

Figure 2. Validation next-byte loss at six nominal model sizes: 25M, 50M, 100M, 200M, 400M, and 800M parameters. Lower is better. Gated BDH-GPU′ closely tracks GPTXL at every evaluated size; vanilla BDH-GPU is weaker, especially at the smallest scale. (Source: Pathway's official BDH repository; experiment details: paper Figure 7 and Appendix B.2).

The plot supports similar loss-versus-size behavior on this setup. It does not establish a universal scaling law across web-scale corpora, modern tokenizers, downstream tasks, or current frontier architectures. The paper reports next-byte loss, not BLEU or ChrF translation quality, reasoning accuracy, factuality, or human preference.

Optional detail: What "Transformer-like scaling" means here

A scaling law is a regular relationship between model size, data, compute, and error. Here the evidence is a six-point parameter-versus-loss curve on one dataset and training budget. The authors' broader abstract rounds the tested range to 10M-1B, but the plotted configurations are 25M-800M.

The comparison is controlled enough to make the architecture interesting: the data, token budget, optimizer family, and evaluation are aligned. It is not perfectly identical. GPTXL receives architecture-specific tuning, including a dropout sweep at each size; BDH-GPU and BDH-GPU′ use a fixed depth and compact dimension while changing $n$.

The most precise statement is therefore: gated BDH-GPU′ matches the GPTXL baseline's next-byte validation loss at the six evaluated sizes on the Europarl task.

What the interpretability experiments show🔗

The paper uses the word interpretability for three related but distinct analyses.

1. Graph structure in learned parameters🔗

Products such as $D_xE$ and $D_yE$ can be interpreted as effective neuron-to-neuron interaction matrices. The authors study five independently trained 24M-parameter models, threshold these dense matrices, and analyze the resulting graphs. They report heavy-tailed matrix entries, lower estimates of Newman modularity obtained with Louvain community detection, and a representative core-periphery structure.

These graphs are extracted after training and depend on threshold choices. The evidence is consistent with modular, heavy-tailed organization; it is not a formal demonstration that the trained network follows a unique scale-free distribution.

2. A graph reconstructed from recurrent state🔗

For one eight-layer Europarl model, the authors reconstruct $\sigma$ for head 0 at layer 5. They remove negative entries introduced by RoPE, apply a positive threshold, and plot a power-law-like in/out-degree distribution. This is a useful view of contextual state, but it is one selected head and layer under a particular filtering procedure.

3. Concept-selective synapse coordinates🔗

The authors search the reconstructed state for entries that separate sentences about currencies or countries from contrasts. They show individual coordinates that rise for related expressions in more than one language, including "British Pound" and "livre sterling."

Optional detail: How the currency synapse was tested

The test uses 50 ChatGPT-generated sentences about European currencies and 50 sentences about European politics that do not mention currencies. For one selected coordinate, a one-sided Mann-Whitney U test reports

$$ U=2368, \qquad p<10^{-14}, \qquad r_{\mathrm{rb}}=0.86, $$

where $r_{\mathrm{rb}}$ is rank-biserial correlation, an effect-size measure. The result shows a strong association between that state coordinate and the tested currency prompts.

Monosemantic means that one coordinate responds selectively to one human-interpretable concept. This experiment identifies promising concept-sensitive coordinates, but it does not establish that the coordinate is exclusively about currency, that it causally controls currency behavior, or that monosemanticity is widespread across the model. The search-and-selection procedure also makes this evidence exploratory rather than a preregistered confirmatory test.

The valuable change from a standard Transformer is not that interpretation becomes automatic. It is that inference-time state can be localized to candidate neuron pairs, measured across prompts, and subjected to interventions in future work.

Model merging: Interesting, but direction-dependent🔗

BDH-GPU scales mainly by increasing $n$, so tensors that carry the $n$ dimension can be concatenated. The paper tests whether this permits a simple form of model composition:

Train a 19M-parameter English-Spanish base model with $n=24{,}576$.
Clone it; continue one copy on English-French and the other on English-Portuguese.
Concatenate $E$, $D_x$, $D_y$, and other tensors with an $n$ dimension. Average token embeddings and output weights.
Evaluate the resulting 38M-parameter model with $n=49{,}152$, without post-merge training.

The shared English-Spanish initialization matters: these are not independently trained modules on disjoint datasets. The merged model obtains next-byte losses of 0.43, 0.40, and 0.39 when translating Spanish, French, and Portuguese into English. In the reverse directions its losses are 1.45, 0.77, and 0.86, and qualitative samples mix the three Romance languages.

The authors say a small amount of joint training restores proficiency, but Table 2 and Figure 15 deliberately report the model before that tuning and provide no recovery curve. The experiment is preliminary evidence that width-wise concatenation preserves and combines some behavior. It does not yet establish general plug-and-play model composition.

What happens without backpropagation through time🔗

During inference, the state update looks local and Hebbian-like. Training the parameters still uses backpropagation. In particular, backpropagation through time (BPTT) assigns credit through earlier state updates so the model can learn how past inputs should influence later predictions.

Optional detail: Why detaching the state changes learning

Truncated BPTT unrolls a finite segment of the recurrent computation and propagates gradients through the state transitions in that segment. Detaching $\rho$ keeps its numerical value for the forward pass but blocks gradients from crossing the boundary.

In the paper's preliminary experiment, a model trained without temporal backpropagation retains some language-modeling ability but loses the ability to align concepts across languages. English loss rises from roughly 0.65 for an unconditional model trained with temporal backpropagation to approximately 0.75-1.05, depending on the variant and translation direction.

This is evidence that local-looking inference updates do not remove the need for temporal credit assignment in the current training recipe. It is a starting point for research, not a successful no-BPTT training method.

Reproducibility: What the public code provides🔗

Appendix E contains a compact PyTorch 2.7 listing for BDH-GPU. The official repository provides a small reference implementation and a Tiny Shakespeare training example under the MIT license.

That code is useful for understanding tensor shapes and the layer structure. It is not the complete Europarl experimental pipeline: it does not include the dataset preparation, GPTXL baseline sweep, multilingual merge study, graph analysis, or synapse-selectivity experiment. Its attention implementation forms a full causal score matrix, so it also does not demonstrate constant-state streaming.

A faithful independent reproduction would need the exact data stream construction, recurrent-state training kernel, all baseline tuning choices, evaluation scripts, and analysis thresholds. The preprint provides substantial methodological detail, but the repository is a teaching implementation rather than a one-command reproduction package.

What remains open🔗

The paper creates a useful research program, but several questions remain unresolved:

Does BDH-GPU remain competitive on larger and more diverse corpora with modern subword tokenization?
Does the recurrent kernel deliver the predicted long-context latency and memory advantages on real hardware?
How much information survives in fixed state over tens or hundreds of thousands of tokens under controlled retrieval tests?
Are concept-selective synapses common, stable across seeds, and causally important when intervened on?
How sensitive are the graph conclusions to thresholds, heads, layers, and community-detection methods?
Can model concatenation combine unrelated capabilities, not only branches sharing an initialization and multilingual data?
Can local or approximate learning rules approach BPTT-trained quality?
Which parts of the graph construction remain biologically credible after normalization, positional rotation, dense parameterization, and supervised optimization are included?

Verdict🔗

Dragon Hatchling is best understood as a graph-interpretable linear-attention state-space architecture, not as a finished theory of brains or general reasoning. Its design puts a large, positive activation space next to a compact recurrent memory and gives the memory a neuron-pair interpretation. The norm-free construction then shows how similar dynamics can arise from local graph updates.

The paper's strongest empirical evidence is controlled next-byte-loss parity between gated BDH-GPU′ and a GPT-2-like Transformer-XL baseline at 25M-800M parameters on one Europarl setup. The graph, monosemanticity, long-context, biological, and model-composition claims are plausible research directions supported by narrower analyses.

That calibration makes the work more interesting, not less. BDH offers concrete equations, testable state variables, and an official implementation. It gives researchers specific mechanisms to reproduce, falsify, and extend rather than relying only on the metaphor of a model that "thinks like a brain."

Primary sources🔗

Adrian Kosowski, Przemysław Uznański, Jan Chorowski, Zuzanna Stamirowska, and Michał Bartoszkiewicz. The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain, arXiv:2509.26507v1, 2025.
Pathway. Official BDH repository, including the reference implementation and clean architecture/scaling figures.
Pathway. Why search for a brain-inspired network? and From attention to synapses: deriving BDH, 2026. These later author explanations are useful for intuition but do not add independent experimental evidence to the 2025 preprint.
Philipp Koehn. Europarl: A Parallel Corpus for Statistical Machine Translation, 2005.

Michał Chromiak's blog - Machine Learning / Neuroscience