Grafy News
Try Grafy free

Running 180B Parameter MoE Models on 32GB of RAM

A new implementation for Transformers allows massive MoE models to run via SSD streaming, bringing high-end performance to consumer hardware.

Illustration for the news story: Running 180B Parameter MoE Models on 32GB of RAMSource: Hugging Face Blog

Large-scale Mixture of Experts (MoE) models can now be run on consumer-grade hardware with significantly limited memory by streaming weights directly from an SSD. This breakthrough addresses the massive hardware gap between high-end enterprise GPUs and standard laptops, enabling much larger models to function locally.

How does MoE expert streaming work?

In a Mixture of Experts architecture, a model is composed of many specialized sub-networks, or "experts." While the total parameter count of a model might be massive—such as the 180B parameter Qwen3.8-Flash-Next—only a small fraction of these experts are active for any given token.

According to a report from the Hugging Face Blog, the core challenge is that these experts often exceed the available Video RAM (VRAM) or unified memory. The streaming approach works by keeping only the "hot" (frequently used) experts in memory, while the rest reside on the SSD. When the model's router selects an expert that is not currently in memory, the system fetches it from the disk.

To make this efficient, the implementation uses a "slot pool" approach. Instead of a slow Python loop that loads one expert at a time, the system uses a preallocated device tensor shared across all layers. This allows the model to use standard, high-speed kernels like grouped_mm to process experts as if they were already in a regular expert bank, significantly reducing the overhead of the streaming process.

What are the performance gains for local hardware?

Testing on a MacBook M5 with 32GB of unified memory demonstrates that this method can bridge the gap between massive models and limited hardware. The researcher, Tarek Ziadé, noted that while standard Transformers offloading via Accelerate was extremely slow, the new streaming implementation reached speeds comparable to dedicated inference engines like Strata.

In one test involving the Qwen3.8-Flash-Next model, the streaming implementation achieved approximately 13.9 tokens per second during decoding. This was within roughly 6% of the performance seen with Strata, which used specialized quantized experts. For comparison, the standard Transformers offloading method was described as being far too slow for practical use, as it lacked a mechanism to cache experts or account for the routing logic.

How does this compare to existing offloading methods?

Existing tools handle memory constraints in different ways, but each has limitations when models exceed RAM capacity:

  • llama.cpp: This engine can map GGUF files using mmap to keep experts on the CPU. While efficient if the experts fit in RAM, the performance drops significantly when the system has to rely on page cache, leading to "thrashing."
  • vLLM: This engine can offload a specific number of gigabytes to pinned RAM. However, it does not feature a disk tier or a cache for experts, meaning a "hot" expert might be repeatedly transferred over the PCIe bus for every token.
  • Transformers (Accelerate): The standard approach moves entire modules (the whole bank of experts for a layer) to the GPU one by one. This lacks any concept of routing or caching, making it highly inefficient for MoE architectures.

The new streaming method improves upon these by using 8 I/O threads for contiguous reads and prefetching experts for upcoming layers based on the router's predictions.

Can consumer laptops run 180B models?

Yes, provided there is sufficient SSD space and the model is quantized. In the experiments described by the Hugging Face Blog, the Qwen3.8-Flash-Next model—which contains 121 GB of experts in FP8—was successfully run on a 32 GB Mac.

To fit within the memory constraints, the setup used a combination of strategies: the dense layers were kept in 8-bit precision, the experts were quantized to 3 bits (reducing them to 49 GB on disk), and a 12 GiB cache was maintained on the device. The system also utilized a specialized disk path for the n-gram embedding table, which was 51 GB in size. This configuration allowed the model to run while staying within the limits of the 32 GB unified memory, though it required closing other heavy applications like web browsers to avoid macOS swapping.

How to apply this to your workflow?

For developers and researchers looking to experiment with these large-scale models, the implementation is being integrated into the Transformers library. Currently, users can test this by installing a specific pull request from GitHub. The process involves using from_pretrained with a device_map that assigns experts to "disk" and configuring an experts_offload parameter to manage the cache size and quantized expert paths.

While running 180B models locally is a significant technical achievement, it is primarily useful for research, testing, and specific low-latency local tasks. For most high-speed creative or production workflows, cloud-hosted models remain the standard due to their superior throughput.

If you want to experience the power of massive, high-performance models without the need for complex local hardware configurations, you can use Grafy Chat to interact with various high-end models side by side. For those looking to dive deep into the technical papers and research surrounding MoE architectures, Grafy Reader provides an AI-powered environment to research and query complex documents.

Frequently asked questions

Can I run large MoE models on a standard laptop?

You can run large Mixture of Experts models on a standard laptop if you use a streaming implementation that offloads experts to an SSD. By keeping only necessary experts in the laptop's RAM and using quantization to reduce the size of the weights, models with hundreds of billions of parameters can function on hardware with limited memory.

Does expert streaming improve token generation speed?

Expert streaming improves speed compared to standard offloading methods by using a slot pool and prefetching. Instead of loading entire layers, the system only fetches the specific experts requested by the router. This allows the model to utilize fast, grouped matrix multiplication kernels, bringing local performance much closer to dedicated inference engines.

Is quantization necessary for MoE offloading?

Quantization is highly beneficial when offloading MoE experts to disk. By reducing the precision of the experts—for example, from FP8 to 3-bit quantization—the amount of data that must be read from the SSD is significantly decreased. This reduces I/O bottlenecks and allows much larger models to fit within the constraints of consumer hardware memory.


Source: Hugging Face Blog — 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers