MoL‑SQL Benchmark Uncovers Language Gaps in Text‑to‑SQL Systems
Researchers release MoL‑SQL, a controlled benchmark that isolates the impact of question.
Source: Grafy Research LabWhat is the MoL‑SQL benchmark and why it matters?
MoL‑SQL is a new benchmark designed to evaluate Text‑to‑SQL systems when the natural‑language question, the database schema, and the stored values are each written in potentially different languages. According to the arXiv paper by Luo, Li, and Luo, this "mixture‑of‑language" scenario reflects real‑world deployments where, for example, an English‑speaking analyst queries a Chinese‑localized database. By exposing how language mismatches affect model performance, MoL‑SQL helps developers build more robust multilingual data‑access tools.
How does MoL‑SQL differ from previous Text‑to‑SQL benchmarks?
Earlier benchmarks such as Spider, BIRD, CSpider, MultiSpider, DuSQL, TACO, and FinSQL either assume a single language for all components or translate only the user question while leaving the schema and values untouched. The authors point out that these designs conflate language effects with domain or annotation differences, and that translation‑only approaches can introduce errors—CSpider, for instance, produces empty query results when an entity is translated in the question and SQL but not in the database. MoL‑SQL, by contrast, independently varies the language of each component while keeping the underlying intent and execution result identical, allowing a clean attribution of performance changes to specific language factors.
How was the benchmark constructed and what data does it contain?
The team built a pipeline called MoL‑Migration that ingests five existing Text‑to‑SQL datasets spanning different domains and source languages. The pipeline performs four key steps:
- Entity map construction – creates a shared cross‑lingual map of schema identifiers, column names, and literal values.
- Task‑preserving realization – translates every occurrence of each entity consistently across the schema, stored values, gold SQL, and the natural‑language question.
- Execution‑equivalence verification – rewrites the gold SQL and checks that it returns the same result after each language transformation.
- Question regeneration – generates the question anew from the verified SQL to ensure alignment. The result is two splits:
- MoL‑Full – 16,892 realizations covering a broad evaluation set.
- MoL‑Cube – 3,840 realizations arranged in a 2 × 2 × 2 design that exhaustively combines English and Chinese for question, schema, and values, enabling fine‑grained diagnosis of language effects. Each cross‑lingual entity mapping is also graded into four difficulty levels, reflecting linguistic conventions, domain knowledge, and naming quirks.
What do the experiments reveal about language effects on Text‑to‑SQL performance?
The authors evaluated eight large language models and four families of Text‑to‑SQL methods on both splits. Two main patterns emerged:
- Schema language impact – Switching the schema from English to Chinese consistently lowered execution accuracy, regardless of the question language.
- Value language impact – When the stored values were in a different language from the question, accuracy dropped by up to 23 points. This penalty vanished when the question and values shared the same language, indicating that the difficulty stems from cross‑lingual entity mapping rather than from translation noise. Additional findings showed that merely translating the question or adding extra reasoning steps did not close the value gap. However, providing the model direct access to the target‑language values substantially mitigated the loss, suggesting that grounding on the correct lexical forms is critical for mixture‑of‑language scenarios.
How can Grafy users work with MoL‑SQL or explore mixture‑of‑language Text‑to‑SQL?
Grafy creators interested in multilingual data interfaces can engage with MoL‑SQL in several ways:
- Use the /reader app to load the arXiv PDF, ask follow‑up questions, and extract specific sections of the benchmark description.
- Leverage /code, Grafy’s AI coding studio, to prototype a Text‑to‑SQL pipeline that consumes the MoL‑Cube data, experiment with prompting LLMs, and evaluate execution accuracy across language combinations.
- For rapid prototyping, the /chat interface lets you query an LLM about the benchmark’s methodology or request code snippets that implement the MoL‑Migration steps. These tools enable developers and researchers to reproduce the authors’ experiments, test new models, or extend the benchmark to additional languages without leaving the Grafy workspace.
To explore this yourself, open Reader: read, search and question PDFs and papers with AI.
Frequently asked questions
How does MoL‑SQL ensure that the translated questions remain faithful to the original intent?
The pipeline first builds a cross‑lingual entity map, then regenerates each question from the verified gold SQL. Because the SQL execution result is unchanged, the regenerated question must express the same intent, and any translation is checked for execution equivalence.
Why do schema language changes affect accuracy more than question language changes?
The study found that schema identifiers appear directly in the generated SQL, so a model must map natural‑language concepts to the correct column or table names. When those identifiers are in a language unfamiliar to the model, the mapping becomes error‑prone, leading to a consistent drop in execution accuracy.
Can the benchmark be extended beyond English and Chinese?
MoL‑SQL’s construction pipeline is language‑agnostic; it relies on a shared entity map and execution verification. Researchers can add new language pairs by providing appropriate translations for the entity map and re‑running the migration steps, making the benchmark scalable to additional linguistic contexts.
Source: Grafy Research Lab — MoL-SQL: A Controlled Benchmark for Mixture-of-Language Text-to-SQL


