Rust Framework Achieves Bit-for-Bit Match with PyTorch

A computer science student has built a small deep learning framework in Rust that can train Andrej Karpathy's nanoGPT model and produce bit-for-bit identical results to PyTorch. The achievement, documented on Reddit, highlights both the feasibility of alternative AI infrastructure and the hidden complexities of reproducing even well-known training routines exactly.

What did the student's framework achieve?

The student, who posted anonymously, created a framework called DenseCore. According to their post, they implemented a PyTorch-like Python API for DenseCore. When they pointed the import torch statement in Karpathy's nanoGPT training script to their framework and ran it unchanged, the training process produced identical numerical outputs to a standard PyTorch run. This included matching the initial weights from a manual seed, the loss values, gradients, and final weights across 50 training steps, even with dropout enabled. The student emphasized this was not a "close enough" approximation but literal, bit-for-bit equality.

Why is bit-for-bit reproducibility difficult?

According to the student's account, achieving this level of parity required navigating several hidden implementation details in PyTorch that deviate from textbook formulas. They discovered that PyTorch's CPU attention implementation uses a flash attention kernel, not the standard formula, and switches computational paths when dropout is active. They also noted that PyTorch's Adam optimizer rounds differently on the last few elements of each thread's chunk due to compiler loop fusion, and that the clip_grad_norm_ function has undocumented behavior. Matching these subtleties was essential for perfect replication.

What does this mean for AI development?

This experiment demonstrates that creating a fully compatible, numerically equivalent alternative to a major framework like PyTorch is technically possible, at least for a defined subset of operations. For researchers, it underscores the importance of implementation details for true reproducibility. For developers, it validates Rust as a viable language for high-performance, numerically sensitive AI workloads. The project also serves as a detailed case study in reverse-engineering the practical, rather than theoretical, behavior of a complex system.

How can creators and developers explore this topic in Grafy?

While Grafy does not have a dedicated deep learning framework builder, its tools are designed for hands-on experimentation with AI models and code. The most direct application is in Grafy Code, an AI coding studio where developers can prototype, run, and test code. A user could, for instance, use Code to examine the original nanoGPT script, discuss the implications of framework design with AI models, or even begin prototyping their own experimental layers. For a research-focused approach, Reader can be used to ingest and query academic papers or technical documentation on framework design and numerical reproducibility.

What are the broader implications?

The work touches on a growing interest in efficient, portable, and secure AI infrastructure. Rust's memory safety and performance characteristics make it an attractive candidate for deploying models in resource-constrained or safety-critical environments. A successful, compatible Rust framework could eventually offer a path for training models in Python and deploying them in a leaner, Rust-based runtime without conversion errors. However, the student's framework is currently a small-scale proof of concept; matching PyTorch's vast and evolving ecosystem of operators, distributed training, and hardware accelerators remains a monumental challenge.

Frequently asked questions

Why is bit-for-bit matching important for AI?

Exact numerical reproducibility is critical for debugging, scientific validation, and ensuring consistent model behavior across different deployments. Small floating-point differences can sometimes cascade, leading to divergent training outcomes or unpredictable model performance, making bit-level matching a gold standard for verification.

Could this Rust framework replace PyTorch?

Not in its current form. DenseCore is a small-scale academic project that replicates a specific training script. PyTorch is a massive, mature ecosystem supporting countless models, research projects, and production systems. However, it demonstrates the principle that compatible alternatives are possible and could inspire future development.

How does this relate to other AI infrastructure news?

This aligns with a trend toward more efficient and specialized AI runtimes, such as the work covered in our article on Janas LLM engine, which optimizes model deployment. It also complements discussions on AI tooling philosophy, like those in Rich Hickey's critique of how we conceptualize LLMs.

---

Source: Reddit ML — [P] my rust framework trains nanoGPT with the exact same bits as pytorch. worth continuing? [P]