ICICLE introduces in‑context learning for curve‑learning extrapolation
Researchers unveil ICICLE, an in‑context surrogate that extrapolates learning curves with variable hyperparameters.
Source: Grafy Research LabICICLE, short for In‑Context Inference for Curve‑Learning Extrapolation, was announced as a new surrogate model that can predict the future trajectory of partially observed learning curves while handling an arbitrary set of hyperparameters. The work, presented at the AXIOM Foundations of Efficient Deep Learning workshop at NeurIPS 2026, aims to make freeze‑thaw Bayesian optimization more flexible and sample‑efficient, a benefit for anyone who spends compute budget on hyperparameter search.
What problem does ICICLE aim to solve?
Freeze‑thaw Bayesian optimization allocates training resources by estimating how a learning curve will continue after only a few epochs. Existing surrogates, such as LC‑PFN and FT‑PFN, either assume a fixed number of hyperparameters or rely on hand‑crafted encoders that limit the diversity of configurations they can model. ICICLE removes these constraints by accepting a variable‑length list of hyperparameters through a shared set encoder, allowing the surrogate to be applied to any configuration space without redesign. This matters because it reduces the number of costly full‑training runs needed to locate high‑performing models.
How does ICICLE differ from earlier approaches like LC‑PFN and FT‑PFN?
LC‑PFN was the first to apply prior‑data fitted networks (PFNs) to learning‑curve extrapolation, learning a mapping from early curve points to final performance. FT‑PFN extended that idea by adding a ten‑slot hyperparameter encoder, but it still required a pre‑specified slot count and could not gracefully handle missing or extra parameters. ICICLE replaces the fixed‑slot encoder with a shared set encoder that processes a set of typed hyperparameters, regardless of size. The model is trained on a synthetic prior that covers a wide distribution of hyperparameter types and follows a heavy‑tailed distribution over the number of parameters, giving it robustness to real‑world search spaces that are often irregular.
What are the core technical components of ICICLE?
The architecture consists of three main pieces:
- Shared set encoder – a permutation‑invariant network that ingests each hyperparameter as a typed token (e.g., learning‑rate, batch‑size, optimizer) and produces a unified representation. Because the encoder treats the input as a set, the order and count of hyperparameters do not affect the output.
- Synthetic prior training – ICICLE is trained on a large corpus of artificially generated learning curves. The prior is deliberately heavy‑tailed with respect to the number of hyperparameters, meaning that some training examples contain many parameters while others contain few, mirroring the diversity seen in practical tuning tasks.
- In‑context inference head – during deployment, the model receives the observed prefix of a learning curve together with the encoded hyperparameters and directly predicts the remaining trajectory, without any gradient updates or separate surrogate fitting step.
These components together enable a single model to act as a universal surrogate across many tasks, a capability that was previously only achievable by training separate models for each configuration space.
How can practitioners incorporate ICICLE into their hyperparameter‑optimization pipelines?
The authors provide open‑source code and pretrained weights, making it straightforward to plug ICICLE into existing Bayesian‑optimization frameworks. A typical workflow would be:
- Collect early‑epoch metrics from each trial (e.g., loss after 5, 10, 15 epochs).
- Encode the trial’s hyperparameters using the shared set encoder supplied in the repository.
- Feed the curve prefix and encoding to the ICICLE inference head, which returns a distribution over the final performance.
- Update the acquisition function (e.g., Expected Improvement) with the extrapolated predictions, allowing the optimizer to decide whether to continue, pause, or discard the trial.
Because ICICLE works in‑context, there is no need to retrain a surrogate after every new batch of observations; the model can be reused across projects, saving both time and compute. Users can experiment with the code in Grafy’s Code studio ([/code]) or explore the pretrained model directly in a notebook environment.
Where does ICICLE fit within Grafy’s suite of AI tools?
For creators and researchers who need to evaluate many model configurations—such as when generating synthetic data with Creator, training custom vision models, or fine‑tuning large language models—ICICLE can serve as a drop‑in surrogate to accelerate the search. The Reader app ([/reader]) lets you load the arXiv PDF, ask natural‑language questions about the methodology, and retrieve specific sections without leaving the workspace. If you want to visualise how different hyperparameter sets affect predicted learning curves, the Graph app ([/graph]) can plot the in‑context predictions alongside actual training runs, making it easy to spot over‑ or under‑estimation. Finally, developers can embed the ICICLE inference endpoint into custom bots built with Bots ([/bots]) to provide on‑the‑fly performance forecasts to teammates.
What are the next steps and open research directions?
While ICICLE demonstrates strong performance on synthetic priors, the authors note that real‑world validation on large‑scale hyperparameter searches is ongoing. Future work may explore:
- Domain‑specific priors – tailoring the synthetic training distribution to particular model families (e.g., transformers versus convolutional nets).
- Multi‑objective extrapolation – extending the head to predict not only final accuracy but also resource‑related metrics such as memory usage or inference latency.
- Active learning loops – integrating ICICLE with adaptive data collection strategies that decide which early‑epoch points to query for maximal information gain.
The release of code and pretrained weights invites the community to experiment, benchmark, and potentially improve the surrogate for their own workloads.
Source: https://arxiv.org/abs/2611.24469
Source: Grafy Research Lab — ICICLE: In-Context Inference for Curve-Learning Extrapolation


