Grafy News
Try Grafy free

Dharma‑AI boosts fine‑tuning time prediction with a CatBoost model achieving 98% R²

Dharma‑AI analyzed 272 fine‑tuning runs on NVIDIA H200 GPUs and built a CatBoost predictor that cuts error to under two hours.

Illustration for the news story: Dharma‑AI boosts fine‑tuning time prediction with a CatBoost model achieving 98% R²Source: Hugging Face Blog

Fine‑tuning jobs on shared GPU clusters often finish far later than the simple FLOP‑based estimate predicts, leading to idle hardware or missed deadlines. Dharma‑AI’s new predictor, trained on 272 real runs, reduces the median absolute error to 0.5 h and reaches an R2R^{2} of 0.98, offering a practical tool for schedulers.

Why does fine‑tuning duration matter?

GPU resources are among the costliest assets in modern AI development. In a multi‑tenant environment, a scheduler must decide which jobs to place on which GPUs and when. Over‑estimating a job leaves GPUs idle; under‑estimating pushes subsequent jobs into a bottleneck, delaying experiments and product releases. Accurate duration forecasts turn scheduling from guesswork into a data‑driven process, improving overall cluster utilisation.

How did Dharma‑AI build the predictor?

The team treated end‑to‑end job time as a supervised regression problem. They collected configuration metadata – hyperparameters, dataset size, model identifier, GPU count – and derived overhead‑related features such as total evaluation samples (num_eval_data\text{num\_eval\_data}), number of checkpoints saved, and per‑GPU token volume. The FLOP‑based estimate (with a fixed MFU of 0.45) was also fed in as a baseline feature. After extensive feature selection using correlation filtering, mutual information, and permutation importance, a CatBoost model with 24 features was trained using stratified 5‑fold cross‑validation.

How accurate is the CatBoost model compared to FLOP estimates?

Model R2R^{2} RMSE (h) MAE (h)
FLOP baseline (MFU 30%) 0.31 20.45 12.41
FLOP baseline (MFU 45%) 0.18 22.38 13.01
FLOP baseline (MFU 60%) 0.01 24.56 14.62
Linear Regression 0.69 ± 0.14 13.52 ± 4.23 7.52 ± 0.49
Ridge 0.73 ± 0.09 12.72 ± 3.70 7.23 ± 0.68
XGBoost 0.97 ± 0.01 4.18 ± 1.17 1.95 ± 0.32
CatBoost 0.98 ± 0.01 3.52 ± 0.78 1.71 ± 0.33

The CatBoost regressor shrinks the median absolute error from over 12 h (FLOP baseline) to just 0.5 h for jobs under 60 h. Even the longest runs (up to 175 h) see a median error of 5.2 h, still a substantial improvement.

Which features drive the predictions?

Feature importance analysis shows that evaluation workload dominates: num_eval_data\text{num\_eval\_data} ranks first, followed by the number of trainable parameters (num_trainable_params\text{num\_trainable\_params}), batch size for evaluation, total parameter count, and the product of GPU count and training batch percent. Checkpoint‑related metrics (num_params_saved\text{num\_params\_saved}) also contribute significantly. These results confirm that non‑compute overhead—evaluation cadence, checkpoint I/O, and data movement—are the primary sources of variance that FLOP‑only formulas miss.

How can practitioners apply this predictor?

Because the model only requires configuration data available before dispatch, it can be integrated into existing job‑submission pipelines. A typical workflow might look like:

  1. Assemble the fine‑tuning configuration (model, dataset, hyper‑parameters, GPU allocation).
  2. Pass the configuration to the predictor (e.g., via a lightweight REST endpoint or a Python library).
  3. Receive an estimated wall‑clock time and use it to order jobs in the scheduler.
  4. Adjust evaluation frequency or checkpoint policy if the predicted duration exceeds resource budgets.

For teams using Grafy, the predictor’s logic can be prototyped with the Grafy Reader app to ingest the original blog PDF, and the Grafy Chat app to experiment with prompt‑based queries such as “Given 8 H200 GPUs, 12 B parameters, and evaluation every 500 steps, what is the expected fine‑tuning time?” This enables rapid testing without writing custom code.

What are the current limitations?

  • Scope: The study covers only supervised fine‑tuning (SFT) on NVIDIA H200 GPUs. It does not address RLHF, DPO, or other alignment techniques.
  • Hardware specificity: All runs used the same GPU generation; transferring the model to a mixed‑GPU fleet would require additional hardware descriptors (compute throughput, memory bandwidth, interconnect topology).
  • Long‑run scarcity: Only 13 jobs exceeded 60 h, so predictions for very long jobs are less reliable.
  • Static features: The predictor cannot adapt to runtime events (e.g., unexpected network congestion) because it only sees pre‑dispatch information.

Future work outlined by the authors includes adding hardware‑specific descriptors, expanding to other training objectives, and gathering more long‑duration runs to improve tail performance.

Where does this fit into Grafy?

While Grafy does not yet host a dedicated fine‑tuning time estimator, the Grafy Reader app lets users upload the forthcoming arXiv paper and ask natural‑language questions about the methodology. The Grafy Chat app can be used to prototype a lightweight wrapper around the CatBoost model, turning configuration dictionaries into duration forecasts. By combining these tools, developers can explore the predictor’s behaviour before building a production‑ready scheduler.


Source: https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time

Frequently asked questions

How does the CatBoost predictor differ from a simple FLOP calculation?

The CatBoost model incorporates overhead‑related features—evaluation size, checkpoint frequency, and per‑GPU token volume—while the FLOP formula assumes a constant model‑FLOPs utilization (MFU). This extra context lets CatBoost achieve R2=0.98R^{2}=0.98 versus the FLOP baseline’s R2=0.31R^{2}=0.31.

Can the predictor be used with GPUs other than NVIDIA H200?

The published model was trained exclusively on H200 runs, so its numerical predictions are calibrated to that hardware. Applying it to a different GPU family would require retraining with data that includes the new hardware’s performance characteristics.

What practical steps should a team take to improve scheduling accuracy today?

Start by collecting detailed configuration metadata for each fine‑tuning job (including evaluation cadence and checkpoint policy). Use a regression model like CatBoost, or even a simple linear correction based on past runs, to generate duration estimates before dispatch. Integrate these estimates into your cluster scheduler to prioritize jobs and reduce idle GPU time.


Source: Hugging Face Blog — Why your fine-tuning job takes longer than the FLOPs say