Respan launches open-source Rust AI gateway built entirely on plugins

The AI infrastructure landscape has a new contender focused on developer flexibility. Respan, a Y Combinator-backed startup, has launched an open-source AI gateway written in Rust that is built entirely around a plugin architecture, aiming to solve the rigidity and maintenance burdens of existing solutions.

According to the project's launch post on Hacker News, the team's experience maintaining a large wrapper around the popular LiteLLM gateway led them to this new design. They reported that their wrapper code exceeded 100,000 lines and that a routine upgrade earlier this year caused intermittent failures they could not fully diagnose, forcing them to manually backport fixes. This experience convinced them that a more modular, extensible foundation was needed for sustainable AI application development.

What is the Respan AI gateway?

Respan Gateway is an open-source project designed to route requests between AI applications and various model providers. Its core proposition is extreme modularity. Instead of bundling support for every provider, protocol, and feature into a monolithic application, it provides a minimal core that dispatches events. Every other capability—connecting to providers like OpenAI or Anthropic, handling different protocols like MCP or real-time voice, processing modalities like images and video, and implementing gateway features like fallback routing or PII redaction—is implemented as a plugin.

The gateway is written in Rust and released under the MIT or Apache-2.0 license. It is currently in a pre-alpha state. Respan's commercial offering is a hosted platform that runs the gateway for customers, adding features like unified observability, evaluations, and billing.

How does its plugin system work?

The plugin model is central to Respan's design philosophy. According to the developer's comments on Hacker News, plugins are not interpreted scripts or dynamically loaded binaries but are natively compiled Rust crates integrated into the main application. This approach was chosen over alternatives like WebAssembly (WASM) because plugins often need direct access to infrastructure like databases, which would require complex host interfaces.

The core processes requests by running registered plugins in sequence. Each plugin can inspect and modify a two-way stream of typed events representing the AI request and response. This allows plugins to choose a provider, filter content, make additional calls, or even refuse a request. The design aims to create explicit processing boundaries, making it easier to compose features like a PII filter without that logic becoming entangled across the entire codebase.

What problem does it solve for developers?

The primary problem Respan addresses is the inflexibility and complexity of scaling AI gateway features. The team's narrative describes a common trajectory: developers start with a gateway like LiteLLM to unify API calls, but as they need custom routing, key management, or unique reliability features, they are forced to wrap and modify the original gateway extensively. This creates a maintenance burden, makes upgrades risky, and can lead to sprawling, hard-to-manage code.

Respan's plugin architecture intends to let teams assemble the specific AI infrastructure their application needs. If a developer needs to support a new provider or add a custom caching layer, they can build it as a standalone plugin without forking or modifying the gateway's core. The project includes example plugins and encourages contributors to build their own, aiming to foster an ecosystem.

How does it handle different AI APIs and protocols?

A key technical challenge for any AI gateway is translating between different provider APIs and emerging protocols. Respan handles this by parsing incoming requests into a shared internal structure for messages, tool calls, and usage data. An adapter hands each connection to the first client format that recognizes it.

A notable feature is its handling of provider-specific fields. When a client and provider use the same API format, the gateway can pass the original bytes and protocol-specific fields through unchanged if no plugin modifies the payload. When translation is necessary, it occurs through these shared types. This allows, for example, an application using the OpenAI Realtime API to communicate with Google's Gemini Live model through the gateway, even though not every feature may translate perfectly.

What are the potential drawbacks and concerns?

The launch discussion on Hacker News surfaced immediate questions about the plugin-centric model. One commenter expressed concern that a plugin ecosystem could introduce supply chain security risks and complicate deployment, contrasting it with a "complete, curated product" where all providers are built-in.

The Respan team clarified that their plugins are compile-time crates, not arbitrary executables, which they argue provides clearer boundaries. However, the commenter countered that this still requires a CI/CD process to build custom gateway images and that crates themselves can be a vector for supply chain attacks. This highlights a trade-off: increased flexibility and developer control versus the simplicity and security of a more locked-down, pre-compiled solution.

How does this relate to building with Grafy?

For creators and developers working within Grafy, the emergence of tools like Respan Gateway reflects a broader trend toward modular, composable AI infrastructure. While Grafy itself is a multimodal creation workspace that integrates many models and tools directly—through apps like Chat, Code, and Creator—the underlying principle is similar: providing a unified interface to diverse AI capabilities.

Developers building custom AI applications or agents that need to call multiple external models could explore Respan for their routing layer. For Grafy users, the direct connection is in the realm of development and integration. If you are building a custom agent or application that leverages AI, you could use Grafy Code to prototype or even contribute to plugin development for a gateway like Respan. The AI coding studio provides an environment to work with code, install packages, and run your projects, which would be ideal for experimenting with this new Rust-based system.

Alternatively, for those researching the AI gateway landscape, Grafy Reader is the perfect tool to ingest and analyze the Respan documentation, GitHub repository, or related research papers to understand its place in the ecosystem.

Frequently asked questions

What is an AI gateway?

An AI gateway is a middleware service that sits between an application and multiple AI model providers (like OpenAI, Anthropic, or Google). It handles tasks like routing requests, managing API keys, load balancing, logging, and translating between different API formats, simplifying the code an application needs to write.

How is Respan different from LiteLLM or OpenRouter?

Respan's core differentiator is its architectural philosophy. It is built from the ground up with a plugin system where every feature is modular. This is a response to the challenge of adding custom features to existing, more monolithic gateways, which can require extensive forking or wrapping of the original codebase.

Do I have to use Respan's hosted service?

No. The Respan Gateway is fully open-source and can be run independently on your own infrastructure. You can build it from source, add your provider API keys, and point your client applications to it. The hosted platform is an optional commercial product that adds management, observability, and billing features.

---

Source: Hacker News — front page — Launch HN: Respan (YC W24) – Open-source Rust AI gateway built around plugins