Paper Defines 'Representational Atoms' and Proves TSAEs Can Find Them

A team at the Chinese Academy of Sciences has proposed a formal definition of the basic units inside neural network representations, and proved that one family of sparse autoencoders can recover them, according to a preprint posted on arXiv (arXiv:2611.14168, listed 10 November 2026). The work matters because interpretability research has long relied on two informal candidates — individual neurons and learned features — without an agreed test for what should count as a basic unit in the first place.

What is a "representational atom"?

The paper is by Chenhui Hu, Pengfei Cao, Kang Liu and Jun Zhao, of the Key Laboratory of Cognition and Decision Intelligence for Complex Systems at the Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence at the University of Chinese Academy of Sciences. It begins from a gap the authors state directly: how internal representations encode and organise what a model has learned remains incompletely understood.

Their proposal borrows from compressed sensing. Internal representations are modelled as sparse linear combinations of a set of fundamental representational units (FRUs), which the authors also call representational atoms, or atoms for short. Linearity keeps the description analytically tractable; sparsity is what lets the units remain distinguishable under appropriate conditions. The definition is stated at the level of representation spaces, so it does not depend on any particular architecture.

Notably, the authors decline to treat a network's own coordinate directions — its neurons — as the default basis. They give two reasons: individual coordinates need not support stable, well-defined semantics, which they connect to the familiar finding that single neurons are polysemantic; and orthogonality caps the number of candidate units at the dimensionality of the representation space, limiting how much representational structure can be captured. An atom, in this framework, is characterised by three properties: representability, sparsity, and separability under what the paper calls the atomic inner product.

What do fidelity and stability measure?

The framework turns "is this a good basic unit?" into two measurable questions. Fidelity asks how well a set of candidate units reconstructs the original representations. Stability asks whether distinct candidate units stay distinguishable from one another.

The second criterion carries the theoretical weight. The authors argue that the distinguishability captured by stability is a prerequisite for monosemanticity — for each unit supporting a unique semantic interpretation. That gives a principled reason to prefer some units over others, rather than relying on intuition about what looks interpretable.

Used this way, the two criteria let the field's usual candidates be compared on the same axes. The paper reports that neurons generally show high fidelity but low stability, while learned features show higher stability but lower fidelity. The authors note this fits earlier observations that learned features tend to be more monosemantic than individual neurons — the units that reconstruct a representation most faithfully are not necessarily the ones that carry the clearest meaning.

Which method does the paper prove can identify atoms?

Having defined atoms and set criteria for judging candidates, the authors ask what kind of learning mechanism could reliably find units scoring well on both. Their answer is threshold-activated sparse autoencoders (TSAEs), which they prove can identify representational atoms with high fidelity and stability under appropriate conditions. The supporting proofs work through the null space property and its equivalence with ℓ1 recovery, tools the framework imports from compressed sensing.

For the empirical side, the authors instantiate TSAEs as BiJumpReLU SAEs and evaluate them on internal representations from pretrained networks spanning different architectures and representation modalities. According to the paper, these SAEs consistently identified atoms with high fidelity and stability, outperforming representative SAE variants that differ in activation and sparsity mechanisms.

The paper also checks the stability–monosemanticity link it argued for in theory, and reports a significant positive correlation between the two in practice: more stable atoms showed clearer and more consistent semantics.

How do the atoms relate to human concepts?

To connect the units to human conceptual structure, the authors use HowNet, an expert-annotated knowledge base that describes concepts through fine-grained semantic primitives called sememes.

The mapping is not one-to-one. A small subset of atoms aligns directly with complete sememes, but individual sememes are more commonly represented in a distributed way, by multiple atoms drawn from different layers. The authors read this as evidence that atoms are finer-grained semantic components, which compose into higher-order structures resembling human conceptual organisation.

Composition also performed better than the parts. Atom compositions recovered sememe structures with substantially greater alignment than single atoms did, and, on manual verification, recovered additional semantic structures consistent with the corresponding sememes that HowNet's own annotations do not capture.

What does this mean in practice, and how can you dig in with Grafy?

For anyone building or evaluating interpretability tooling, the practical contribution is a vocabulary and two numbers to report. Instead of arguing that a feature "looks meaningful," you can state how well a candidate set reconstructs activations and how separable its members are, and compare methods on both. The results also point at composition across layers, rather than single units, as the level where concepts appear.

Two caveats are worth keeping. This is a preprint, so independent replication is still ahead of it, and the theoretical guarantees hold under conditions the authors specify rather than universally.

To read the paper closely, Reader can take the arXiv PDF and answer questions about specific definitions, the atomic inner product, or the proof steps, with the relevant passages surfaced rather than paraphrased from memory; the reading room walkthrough shows how that works. For related coverage, see our report on a causal concept wrapper network for trustworthy AI explanations.

Frequently asked questions

Can threshold-activated sparse autoencoders identify representational atoms?

Yes. The authors prove that threshold-activated sparse autoencoders can identify fundamental representational units with high fidelity and stability under appropriate conditions, then instantiate them as BiJumpReLU SAEs and test them on representations from pretrained networks of different architectures and modalities. According to the paper, those SAEs consistently found atoms scoring well on both criteria and outperformed the SAE variants the authors compared them against.

Do single neurons qualify as fundamental representational units?

Only partly. In the paper's framework, neurons score high on fidelity, meaning they reconstruct the original representations well, but low on stability, meaning distinct units are hard to tell apart. Because the authors treat stability as a prerequisite for monosemanticity, that combination makes individual neurons weak candidates for units that carry one clear semantic interpretation.

Does each HowNet sememe correspond to a single atom?

No. The authors report that the relationship is not one-to-one. A small subset of atoms aligns directly with complete sememes, but individual sememes are more often represented in a distributed manner, by multiple atoms from different layers. Composing atoms across layers recovered sememe structures with substantially greater alignment than single atoms achieved.

Source: arXiv:2611.14168, "Representational Atoms in Neural Networks" (Grafy Research Lab feed).

---

Source: Grafy Research Lab — Representational Atoms in Neural Networks