🔍 Read the full analysis: Fine-Tuning Runtime: What The FLOPs Don’t Tell You on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted the wall-clock runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the comparison covers one production environment and does not show savings from deploying the predictor in a live scheduler.
Hugging Face reports that a CatBoost model predicted end-to-end runtime for 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour mean absolute error, a difference that could help GPU operators plan workloads, though the results come from one production pool and have not yet been tested in a live scheduling trial.
The jobs used about 7,382 GPU-hours in a corporate production environment. Their runtimes ranged from 1.6 to 172.4 hours, and the dataset included models up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and allocations of one, two or four GPUs. The machines came from a shared pool of 8 to 32 H200s across one to four nodes, with other training and inference workloads running at the same time.
Hugging Face defines runtime as wall-clock time from GPU allocation to job completion; queue time is excluded. The predictor used 24 selected features from submitted configuration and model metadata available before dispatch. These included evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. A FLOP-based runtime estimate was also included as one feature.
In the reported cross-validation, CatBoost reached an R² of 0.98 and a root mean squared error of 3.52 hours. Its median absolute error was 0.51 hours. The FLOP baseline produced a 12.41-hour mean absolute error at 30% Model FLOPs Utilization (MFU); its error was 13.01 hours at 45% MFU and 14.62 hours at 60%. Hugging Face says a broader MFU sweep from 25% to 65% did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
More Useful Estimates for GPU Schedules
Runtime estimates affect when cluster operators dispatch queued jobs and how they allocate scarce accelerators. A job that finishes earlier than expected can leave GPUs idle until another workload starts; one that runs longer can delay other jobs and shift delivery plans. A more accurate estimate could help operators reduce such mismatches, particularly on shared clusters where concurrent workloads affect performance.
The reported error gap is substantial within this dataset, but prediction accuracy is not the same as demonstrated operational benefit. The preview does not report a controlled deployment measuring GPU hours saved, lower costs, fewer delays or reduced idle time. Those outcomes would depend on how a scheduler uses predictions and how well the model transfers to its workload mix.
Products Worth Considering

Bose Lifestyle Ultra Wireless Home Speaker, White Smoke

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium

Bose QuietComfort Wireless Noise-Cancelling Headphones, Twilight Blue
As an affiliate, we earn on qualifying purchases.
Why FLOP Counts Miss Job Time
A common estimate approximates dense Transformer training compute using parameter count and training-token count, then divides by GPU capacity and an assumed MFU—the share of theoretical peak throughput achieved in practice. Hugging Face says the tested estimates varied MFU because real-world utilization can differ from peak performance and is affected by memory traffic, communication and kernel efficiency.
That calculation focuses on training compute, while the study’s target includes more of a job’s lifecycle after GPUs are assigned. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add elapsed time. The preview says these tasks can compete for network and storage capacity on multi-tenant clusters. The researchers also included scheduled evaluation tokens in the baseline comparison and tested a range of MFU assumptions.
“Everything is known before dispatch.”
— Hugging Face study preview
How Far the Results Generalize
The results cover 272 runs from one corporate production environment. Five-fold cross-validation tests predictions within that dataset; it does not establish accuracy on independently collected jobs from other clusters, GPU types, software stacks or workload mixes. The preview also does not report a test on a later, unseen period or a live scheduling trial.
The full paper has not yet been posted, and the preview gives no publication date. Further details about the evaluation design and feature selection may appear there. It is also unclear how often a predictor would need retraining as workloads, software and shared-cluster conditions change. The reported metrics measure prediction error, not the real-world effect of using the model.
Full Paper and Operational Tests
Hugging Face says a full version of the study will be posted on arXiv, but has not specified when. That paper may provide more detail on the dataset and evaluation methods. Broader validation on runs from other periods or environments would help establish whether the reported accuracy holds outside the production pool used here.
A live scheduler trial would answer a different question: whether better runtime forecasts actually reduce idle GPU intervals, delays or operating costs. Until such evidence is available, the preview supports a promising comparison on one dataset, not a general performance guarantee for other fine-tuning clusters.
Key Questions
What did the CatBoost model predict?
It predicted wall-clock runtime after GPU allocation for supervised fine-tuning jobs. Queue time was excluded.
How did its error compare with the FLOP estimate?
CatBoost’s mean absolute error was 1.71 hours in five-fold cross-validation. The best tested FLOP baseline had a 12.41-hour mean absolute error at 30% MFU.
Does the study show that GPU time or money was saved?
No. The preview reports prediction errors, but does not describe a controlled scheduler deployment or measure savings in GPU time, cost or delays.
Can the result be applied to other GPU clusters?
That remains unconfirmed. The reported jobs came from one production environment, and the preview does not establish performance on other hardware, software stacks or workload mixes.
Primary source: Hugging Face · via ThorstenMeyerAI.com
