Janas LLM engine streams huge models from SSD, runs 80B models on 32GB RAM

A new open-source inference engine called Janas-LLM has demonstrated the ability to run massive language models that exceed a computer's available RAM by intelligently streaming only the necessary parts from a fast SSD. This technique makes running cutting-edge, multi-billion parameter models on consumer hardware a practical reality, significantly lowering the barrier to local AI experimentation.

What is Janas and how does it work?

Janas is a family of AI tools written in pure C with minimal dependencies. Its first component, Janas-LLM, is specifically designed to run large mixture-of-experts (MoE) language models on standard computers, with or without a GPU. According to its GitHub repository, the core innovation is its memory management: model weights remain primarily on disk, and only the specific "experts" (sub-networks) required for the current token prediction are streamed on-demand from an NVMe SSD. Experts that are frequently used are cached in RAM, creating an efficient, dynamic workflow that can handle models larger than physical memory.

What performance claims does it make?

The project's README reports benchmark results from October 2026, comparing Janas-LLM against the popular llama.cpp on a laptop with 32GB of RAM and an Intel Core Ultra 9 processor. The most striking claim is running the Qwen3-Next-80B model—a 48.4 GB file—at 24 tokens per second for generation and 62 tokens per second for prompt reading. Under the same conditions, llama.cpp reportedly managed 2.8 and 16 tokens per second, respectively. The documentation states Janas was faster than llama.cpp on all 13 models tested in the Q4_K_M quantization format.

What are the limitations and requirements?

The developer, Maurizio Cammalleri, explicitly cautions that Janas is an experiment, not a stable product. It is in early development, with a public API and command line that change frequently. The engine is designed to be aggressive, filling free memory for its cache and using O_DIRECT for high-speed disk reads, which can potentially push a system with limited headroom into swap and cause slowdowns. It has been primarily developed and tested on a single machine (Debian 13 on an Intel laptop with an Arc GPU). Performance on other hardware—like AMD CPUs, discrete GPUs, or slower storage—is untested. It currently runs on Linux x86-64 and Windows 10/11.

How does this affect creators and developers?

For developers and researchers, Janas represents a promising path toward democratizing access to state-of-the-art, large-scale AI models. The ability to run an 80-billion-parameter model on a 32GB laptop removes a major financial and logistical hurdle for local development, fine-tuning, and private experimentation. For creators, faster local inference can enable more iterative and complex workflows without relying on cloud API costs or latency. The project's pure C implementation and focus on efficiency also provide a valuable reference for the systems programming community interested in high-performance AI.

How can you explore this topic within Grafy?

While Grafy does not directly integrate the Janas engine, its multimodal workspace is built for exactly this kind of cutting-edge exploration. You can use Grafy Chat (/chat) to discuss the technical implications of SSD streaming and mixture-of-experts architectures with a variety of AI models side-by-side. For a deeper technical dive, use Grafy Reader (/reader) to upload and interrogate the Janas research paper or documentation from its GitHub repository. Furthermore, Grafy's philosophy of providing a unified interface for many cloud models, as explored in our blog post One model picker for every app (/blog/one-model-picker-for-every-app), aligns with the goal of making powerful AI models more accessible, whether they run in the cloud or, increasingly, on local hardware.

Frequently asked questions

What is the key advantage of Janas over other inference engines?

Janas can run models larger than your system's RAM by keeping weights on an SSD and streaming only the necessary "expert" segments into memory as needed. This allows, for example, a 48GB model to run on a 32GB laptop at usable speeds, a feat traditional engines struggle with.

Is Janas ready for production use?

No. The developer explicitly labels it an experiment in early development. The API is unstable, it's been tested on very limited hardware, and its aggressive memory use can destabilize a system. It is currently a tool for researchers and early adopters.

Can I use Janas within Grafy?

Not directly. Janas is a separate, locally-run inference engine. However, you can use Grafy's tools like Chat and Reader to research its methodology, discuss its code, and plan how local model inference could fit into your broader AI workflow alongside cloud-based models.

---

Source: Hacker News — front page — Janas: Run LLMs larger than RAM by streaming experts from SSD