AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Fine-Tuning Runtime: What The FLOPs Don’t Tell You on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted the wall-clock runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the comparison covers one production environment and does not show savings from deploying the predictor in a live scheduler.

Hugging Face reports that a CatBoost model predicted end-to-end runtime for 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour mean absolute error, a difference that could help GPU operators plan workloads, though the results come from one production pool and have not yet been tested in a live scheduling trial.

The jobs used about 7,382 GPU-hours in a corporate production environment. Their runtimes ranged from 1.6 to 172.4 hours, and the dataset included models up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and allocations of one, two or four GPUs. The machines came from a shared pool of 8 to 32 H200s across one to four nodes, with other training and inference workloads running at the same time.

Hugging Face defines runtime as wall-clock time from GPU allocation to job completion; queue time is excluded. The predictor used 24 selected features from submitted configuration and model metadata available before dispatch. These included evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. A FLOP-based runtime estimate was also included as one feature.

In the reported cross-validation, CatBoost reached an R² of 0.98 and a root mean squared error of 3.52 hours. Its median absolute error was 0.51 hours. The FLOP baseline produced a 12.41-hour mean absolute error at 30% Model FLOPs Utilization (MFU); its error was 13.01 hours at 45% MFU and 14.62 hours at 60%. Hugging Face says a broader MFU sweep from 25% to 65% did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview published; full paper and…
The developmentHugging Face has previewed a study comparing CatBoost runtime predictions with FLOP-based estimates across 272 production fine-tuning jobs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

More Useful Estimates for GPU Schedules

Runtime estimates affect when cluster operators dispatch queued jobs and how they allocate scarce accelerators. A job that finishes earlier than expected can leave GPUs idle until another workload starts; one that runs longer can delay other jobs and shift delivery plans. A more accurate estimate could help operators reduce such mismatches, particularly on shared clusters where concurrent workloads affect performance.

The reported error gap is substantial within this dataset, but prediction accuracy is not the same as demonstrated operational benefit. The preview does not report a controlled deployment measuring GPU hours saved, lower costs, fewer delays or reduced idle time. Those outcomes would depend on how a scheduler uses predictions and how well the model transfers to its workload mix.

Products Worth Considering

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium

Bose QuietComfort Wireless Noise-Cancelling Headphones, Twilight Blue

Bose QuietComfort Wireless Noise-Cancelling Headphones, Twilight Blue

As an affiliate, we earn on qualifying purchases.

Why FLOP Counts Miss Job Time

A common estimate approximates dense Transformer training compute using parameter count and training-token count, then divides by GPU capacity and an assumed MFU—the share of theoretical peak throughput achieved in practice. Hugging Face says the tested estimates varied MFU because real-world utilization can differ from peak performance and is affected by memory traffic, communication and kernel efficiency.

That calculation focuses on training compute, while the study’s target includes more of a job’s lifecycle after GPUs are assigned. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add elapsed time. The preview says these tasks can compete for network and storage capacity on multi-tenant clusters. The researchers also included scheduled evaluation tokens in the baseline comparison and tested a range of MFU assumptions.

“Everything is known before dispatch.”

— Hugging Face study preview

How Far the Results Generalize

The results cover 272 runs from one corporate production environment. Five-fold cross-validation tests predictions within that dataset; it does not establish accuracy on independently collected jobs from other clusters, GPU types, software stacks or workload mixes. The preview also does not report a test on a later, unseen period or a live scheduling trial.

The full paper has not yet been posted, and the preview gives no publication date. Further details about the evaluation design and feature selection may appear there. It is also unclear how often a predictor would need retraining as workloads, software and shared-cluster conditions change. The reported metrics measure prediction error, not the real-world effect of using the model.

Full Paper and Operational Tests

Hugging Face says a full version of the study will be posted on arXiv, but has not specified when. That paper may provide more detail on the dataset and evaluation methods. Broader validation on runs from other periods or environments would help establish whether the reported accuracy holds outside the production pool used here.

A live scheduler trial would answer a different question: whether better runtime forecasts actually reduce idle GPU intervals, delays or operating costs. Until such evidence is available, the preview supports a promising comparison on one dataset, not a general performance guarantee for other fine-tuning clusters.

Key Questions

What did the CatBoost model predict?

It predicted wall-clock runtime after GPU allocation for supervised fine-tuning jobs. Queue time was excluded.

How did its error compare with the FLOP estimate?

CatBoost’s mean absolute error was 1.71 hours in five-fold cross-validation. The best tested FLOP baseline had a 12.41-hour mean absolute error at 30% MFU.

Does the study show that GPU time or money was saved?

No. The preview reports prediction errors, but does not describe a controlled scheduler deployment or measure savings in GPU time, cost or delays.

Can the result be applied to other GPU clusters?

That remains unconfirmed. The reported jobs came from one production environment, and the preview does not establish performance on other hardware, software stacks or workload mixes.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Pinterest Surges In Global Coverage

Search interest in Pinterest has spiked, with reports of increased media coverage worldwide, though the reasons for this surge remain unconfirmed.

Former Cainiao CTO Li Qiang Launches Quantum Dynamics, Secures Over 100 Million Yuan Seed Round From Yunqi And SenseTime – Finance.biggo.com

Li Qiang reportedly founded Quantum Dynamics with seed backing from Yunqi and SenseTime, but its product and deal terms remain undisclosed.

The Eye Over the City: How Wide-Area Motion Imagery Works — and Where It Goes Blind

An in-depth look at WAMI technology, its capabilities, limitations, and future directions in urban surveillance and security.

Embracing The Future: Meta’s Muse Spark 1.2 And AI Coding Progress

Meta releases Muse Spark 1.2, a new coding-focused AI model paired with Muse Code, emphasizing co-training and long-horizon coding capabilities.