Grafy News
Try Grafy free

Study Reveals Temporal Contamination Skews Language Model Forecasts of Central‑Bank Data

Researchers measured how language models leak future information when reading past FOMC statements, showing a hidden source of error for economic forecasting.

Illustration for the news story: Study Reveals Temporal Contamination Skews Language Model Forecasts of Central‑Bank DataSource: Grafy Research Lab

Language models are increasingly used to interpret central‑bank communications for macro‑economic forecasting, but a new arXiv paper finds that many of these models unintentionally memorize outcomes that occurred after the documents they read. The effect—called temporal contamination—can inflate apparent predictive skill and mislead users.

What is temporal contamination?

Temporal contamination occurs when a model’s training data includes information that was created after the target document, allowing the model to answer questions about that document using knowledge of future events. In the context of economic texts, a model that has seen post‑2021 inflation data may correctly answer a question about a 2021 Federal Open Market Committee (FOMC) statement, not because it understood the statement, but because it already knows what happened later.

How did the researchers evaluate the problem?

The authors examined every FOMC statement and meeting minutes released between 2000 and 2026. They constructed two sets of models:

  1. Modern open models – publicly available language models trained on the full corpus of internet text, which inevitably contains material written after many of the FOMC documents.
  2. Chronologically consistent models – versions of the same architectures that were deliberately trained only on data that existed before a chosen cut‑off date, ensuring no future information could leak in. Both model families were prompted with identical queries, such as “What was the GDP growth rate in the quarter following the June 2021 FOMC meeting?”

What did the study find?

According to the paper, modern open models often answered past‑event questions correctly, while the chronologically consistent counterparts performed at chance or produced generic responses. The discrepancy indicates that the modern models were relying on memory of outcomes rather than reasoning from the text itself. The authors describe this as a “temporal form of test‑set contamination” that can masquerade as genuine understanding.

Why does this matter for forecasting and model evaluation?

Economic analysts and policymakers increasingly trust AI‑driven summarizations and forecasts derived from central‑bank communications. If a model’s apparent accuracy stems from having already seen the results it predicts, its forecasts for truly unseen future periods may be far less reliable. The paper therefore calls for stricter data‑curation practices and evaluation protocols that respect chronological ordering, especially when models are deployed for forward‑looking analysis.

How can Grafy users explore or mitigate temporal contamination?

Grafy provides tools that let you replicate the study’s methodology on your own data sets:

  • Reader (/reader) lets you upload the full archive of FOMC PDFs, index them, and ask natural‑language questions.
  • Chat (/chat) enables side‑by‑side comparison of different language‑model back‑ends, so you can test a modern model against a version that you restrict to pre‑cut‑off data.
  • Graph (/graph) helps you visualise the prompt‑output tree, tracking how each query evolves across model variants. By combining these apps, you can build a chronologically consistent pipeline: first filter the training corpus to a specific date, then use Chat to query both the filtered and unfiltered models, and finally map the results in Graph to spot divergences. This workflow mirrors the paper’s approach and gives creators a concrete way to audit their AI‑driven forecasts for hidden contamination.

Source: https://arxiv.org/abs/2611.21554


Source: Grafy Research Lab — Reading or Remembering? Measuring Temporal Contamination in Language Models