Janas LLM engine streams huge models from SSD, runs 80B models on 32GB RAM
A new C-based inference engine called Janas streams mixture-of-expert LLMs directly from an NVMe SSD, enabling a 48GB model to run on a 32GB laptop at 24 tokens per secon…
llm inferencelocal ai