TKintegers K5 Beats FP8 on NVIDIA L4 with Over 3× Faster Throughput

TKintegers' K5 implementation achieved dramatically higher token‑generation rates than the standard TorchAO FP8 baseline on an NVIDIA L4, a result that could reshape how developers think about weight quantisation for large language models (LLMs).

---

What is the K5 implementation and how does it work?

TKintegers is a research project that explores a balanced‑ternary representation for neural‑network weights. The "K" stands for Ternary Kluster, and the "5" indicates that each weight is encoded with five balanced ternary digits (values of -1, 0, +1). This five‑digit code can represent a range of integer values while keeping the storage footprint comparable to low‑bit formats like FP8. The project includes a converter that maps conventional floating‑point weights to the K5 code, as well as a custom GPU kernel that performs inference directly on the ternary representation.

The key idea is that arithmetic on balanced ternary can be expressed with simple integer operations, which can be mapped efficiently onto existing CUDA cores even though the hardware lacks native ternary instructions. By avoiding the overhead of de‑quantising to FP16/FP32 during each matrix multiply, K5 reduces memory bandwidth pressure and improves compute utilisation.

---

How does K5 performance compare to FP8 on the NVIDIA L4?

According to the Reddit post reporting the benchmark (source), the comparison used the Qwen2.5‑0.5B‑Instruct model with a fixed workload that matched prefill and decode token counts.

• Prefill: K5 processed 3,402.85 tokens per second, while the TorchAO dynamic E4M3 FP8 baseline managed 1,061.58 tokens per second – a 3.21× speedup.
• Decode: K5 achieved 26.75 tokens per second versus 8.63 tokens per second for FP8, delivering a 3.10× improvement.

The FP8 baseline ran in eager PyTorch using the standard TorchAO kernel, which leverages the L4’s native FP8 tensor cores. K5, by contrast, ran on a custom kernel that operates on the ternary codes, even though the L4 does not provide dedicated K5 instructions. The decode measurement kept the KV cache fixed, ensuring that the speed gain stems from the weight representation rather than cache handling.

---

Why does the result matter for AI inference efficiency?

The NVIDIA L4 is a cost‑effective, low‑power GPU widely deployed in inference‑as‑a‑service environments. Its native FP8 support was marketed as the sweet spot for high‑throughput, low‑precision inference. Demonstrating that a software‑only ternary format can surpass native FP8 on the same silicon challenges the assumption that hardware‑specific low‑bit formats are always optimal.

Three practical implications emerge:

1. Hardware‑agnostic speedups – Developers can obtain higher throughput without waiting for new tensor‑core instructions.
2. Memory bandwidth savings – Balanced ternary codes occupy fewer bits per weight than FP8, easing pressure on the memory subsystem, which is often the bottleneck in LLM serving.
3. Potential for broader adoption – If the K5 kernel can be generalised to larger models, cloud providers could offer cheaper inference endpoints, and on‑device AI could become more feasible.

The result also invites deeper investigation into the trade‑offs of ternary arithmetic, such as numerical accuracy, training compatibility, and integration with existing model optimisation pipelines.

---

Exploring K5 with Grafy

While Grafy does not yet ship a built‑in ternary‑weight engine, the platform offers two ways to experiment with the K5 approach:

• Grafy Code – The AI‑assisted coding studio lets you clone the TKintegers repository, tweak the CUDA kernel, and run benchmarks directly in the browser or on your own GPU. You can use the built‑in terminal to compile the kernel, load a model checkpoint, and reproduce the token‑throughput numbers reported on Reddit. See the /code app for a hands‑on environment.
• Grafy Reader – If you prefer to study the underlying paper or technical notes, the Reader app can ingest PDFs, extract equations, and let you ask natural‑language questions about the balanced‑ternary formulation. This is useful for understanding the conversion algorithm before implementing it yourself.

Both tools integrate with Grafy’s Graph view, allowing you to visualise each step of the workflow—from weight conversion to inference timing—so you can iterate quickly and share results with the community via the Community gallery.

---

What are the limitations and next steps?

The current benchmark focuses on a 0.5‑billion‑parameter model. Scaling K5 to multi‑billion‑parameter LLMs may expose new challenges, such as increased kernel launch overhead or numerical drift in deeper layers. Additionally, the comparison used an eager‑mode PyTorch baseline; a highly optimised FP8 pipeline (e.g., with TensorRT or custom fusion) could narrow the gap.

Future work could address:

• Training pipelines – Developing a method to train directly in balanced ternary, rather than converting post‑training, would eliminate any accuracy loss from conversion.
• Compiler support – Adding K5 as a first‑class datatype in CUDA or LLVM could unlock further performance gains.
• Broader hardware testing – Running the kernel on other GPUs (e.g., A100, H100) would clarify whether the speedup is specific to the L4’s memory architecture or more universal.

For developers interested in reproducing or extending the results, the open‑source code is available in the TKintegers GitHub repository linked in the Reddit thread.

---

Frequently asked questions

How does balanced ternary differ from binary or other low‑bit formats?

Balanced ternary uses three digit values (‑1, 0, +1) instead of two (0, 1). With five digits per weight, it can represent a wider integer range than a 5‑bit unsigned format while keeping the bit‑count similar to FP8. The symmetric nature of the digits often simplifies arithmetic and can reduce quantisation error for certain weight distributions.

Can I use K5 with models larger than Qwen2.5‑0.5B?

The current public results are limited to the 0.5B‑parameter Qwen2.5 variant. Scaling to larger models is theoretically possible but has not been demonstrated yet. Developers should benchmark on a case‑by‑case basis, paying particular attention to memory usage and kernel launch overhead.

Is the K5 kernel compatible with existing AI frameworks like TensorFlow or JAX?

The published implementation targets CUDA and integrates with PyTorch via custom operators. Porting it to TensorFlow or JAX would require writing equivalent kernels or using a bridge like PyTorch‑XLA. Grafy Code can help prototype such adapters, but no out‑of‑the‑box support exists at this time.

---

Source: Reddit ML — TKintegers K5: 3.21× prefill and 3.10× decode throughput vs TorchAO FP8 on NVIDIA L4 [R] [P]