K2-Horizon Unveils Open‑Source Fleet of Language Models Up to 375B Parameters
Grafy Research Lab released K2-Horizon, a suite of six open‑source language models ranging from 0.9B to a 375B‑A23B MoE.
Source: Grafy Research LabWhat is K2‑Horizon?
K2‑Horizon is a newly announced collection of six fully open‑source language models designed for a broad range of tasks, from mathematical reasoning to software engineering and office automation. The fleet was released by the Grafy Research Lab on November 17, 2026 and targets both academic researchers and developers who need high‑performance, agent‑ready models without proprietary restrictions. According to the pre‑print, the models are built to excel in "general‑purpose reasoning and agentic work" across domains such as math, science, deep research, and everyday productivity.
Which models are included and how are they sized?
The K2‑Horizon lineup spans a wide scale spectrum:
- A 0.9 billion‑parameter dense model that serves as an entry‑level option for quick prototyping.
- Mid‑range models up to 36 billion parameters, including a 36B‑A4B variant that incorporates mixture‑of‑experts (MoE) layers.
- A flagship 375 billion‑parameter MoE model (referred to as 375B‑A23B) that pushes the limits of open‑source scaling. All six models are released under an open licence, enabling anyone to download, fine‑tune, or integrate them into custom pipelines. The size diversity lets users match compute budgets to task complexity, a principle highlighted in the paper’s motivation.
What architectural innovations does K2‑Horizon bring?
Two key modifications to the transformer architecture differentiate K2‑Horizon from earlier open models:
- Mixture‑of‑Value Attention (MoVA) – Instead of applying MoE to the query or key projections, MoVA routes the value projections through expert layers. This design aims to enrich the information that the attention mechanism aggregates without inflating the cost of computing attention scores.
- Softplus attention gating – Traditional gating functions often rely on hard‑sigmoid or ReLU‑style thresholds. K2‑Horizon replaces these with a softplus‑based gate, offering a smoother, theoretically motivated transition that can improve gradient flow and stability during training. The authors argue that these changes provide a more flexible attention bottleneck, especially beneficial for long‑context reasoning and multi‑step agentic tasks.
How does K2‑Horizon perform on agentic and coding benchmarks?
The research team evaluated the fleet on a suite of public benchmarks that measure autonomous reasoning, tool use, and code generation. Across the board, the models either lead or remain competitive with other strong open‑source models of comparable size. For example, the 36B‑A4B model matches the performance of proprietary counterparts on coding challenges, while the 375B‑A23B MoE model shows top‑tier results on agentic benchmarks that require multi‑turn planning and execution. The paper emphasizes that the improvements stem from the new attention mechanisms rather than simply scaling parameters.
How can Grafy users experiment with K2‑Horizon?
Grafy’s multimodal workspace makes it straightforward to test and integrate K2‑Horizon models:
- Grafy Chat (/chat) lets you load any of the six models side by side with other popular LLMs, enabling direct comparison on the same prompt. This is ideal for developers who want to see how K2‑Horizon’s reasoning differs from, say, an open‑source 7B model.
- Grafy Reader (/reader) can ingest the K2‑Horizon pre‑print PDF, index its sections, and let you ask natural‑language questions about the architecture or benchmark results without leaving the platform.
- Grafy Code (/code) provides an AI‑assisted coding environment where you can run the 36B model on real‑world programming tasks, experiment with prompt engineering, or fine‑tune a smaller variant on a domain‑specific code base. If you prefer a visual workflow, you can chain these apps: use Reader to extract key methodology passages, feed them to Chat for clarification, and then switch to Code to generate a small agent that leverages the model’s new gating logic. All of this runs within Grafy’s cloud infrastructure, which already supports the OpenRouter and Vertex AI back‑ends used to host K2‑Horizon.
What does the open‑source nature of K2‑Horizon mean for the community?
Because the models are released without restrictive licenses, researchers can reproduce the paper’s experiments, extend the MoVA concept, or adapt the softplus gating for other transformer variants. The open‑source approach also encourages community‑driven benchmarking, where contributors can add new tasks—such as document summarisation or AI‑driven game scripting—and share results via Grafy’s public Community gallery (/community). This collaborative ecosystem can accelerate the discovery of use‑cases that go beyond the benchmarks reported in the pre‑print.
For the full technical details, see the original pre‑print on arXiv: https://arxiv.org/abs/2611.21003.
To explore this yourself, open Grafy Chat: chat with many AI models side by side.
Frequently asked questions
How many models are part of the K2‑Horizon fleet?
The fleet consists of six distinct language models, ranging from a 0.9 billion‑parameter dense model up to a 375 billion‑parameter mixture‑of‑experts model.
What is Mixture‑of‑Value Attention?
Mixture‑of‑Value Attention (MoVA) applies a mixture‑of‑experts routing to the value vectors inside the attention computation, allowing different experts to contribute specialized information to the final context representation.
Can I run K2‑Horizon models directly in Grafy?
Yes. Grafy Chat lets you load any of the K2‑Horizon models alongside other LLMs, and Grafy Code can execute them for programming tasks. Both apps support the same cloud back‑ends used by the K2‑Horizon release.
Source: Grafy Research Lab — K2-Horizon: A Frontier Fleet of Open Source Language Models


