Nishant's Machine Learning Blog

Date

Aug 4, 2026

slug

scaling-laws

Status

Published

Subtitle

Read to know how miserably i failed to derive my own scaling laws

Scaling Laws

An experiment to derive your own scaling laws

Any Company or lab (as if they have plan to

train a large foundational model, should do this exercise once their own data and own model type is decided. Once they have their own 5 parameters for the scaling laws, it will help them

determine when to stop their training run. P.S This report is written by hand so that the reader for once does not get to read AI slop.

The first constant of the equations will tell them what is the least loss value possible. This

is the least will help them determine if they should train more in the hope that their

multibillion parameter will go to a loss of 1.2 or will it stay at 1.3 (Just an example). Thus this exercise may end up saving them hours of precious GPU compute

If we represent the scaling loss in the format

then is the least possible value the loss can take and is called the offset and entropy. Once derived the scaling law will help the lab extrapolate that after x number of training steps loss should have reached value y for the model of param M.

To derive the scaling laws i have done close to 7 training runs and then i tried to do a paramteric fit by minimizing the Huber loss using the L-BFGS-B algorithm.

The final 5 params were yield this equation

Note that the expected params would be close to

The 7 training runs were done with model sizes of 71M , 106M, 142M, 177M, 248M, 291M and 551M and train on the fineweb dataset.

My Results

Observations from my training runs:

In the usual case the model training run shouldn’t have a double step like this.

Also, the model of bigger size should have a loss curve below the model of a smaller size.

I noticed this, so the only one model to fit this pattern was the 551M parameter model. The rest of the models do not follow this trend.

Possible reasons for inconsistencies:

FineWeb as a dataset is definitely messy. The FineWeb dataloader is batch deterministic as well, which means that different models will have spikes at the same step number because a noisy batch would have come at that step.

Regardless, I cannot exactly attribute the double-step problem to exactly noise in the data because when training on SmolLM Corpus which is a synthetic and much more cleaner version of the dataset also observes this same double-step pattern.

I should have experimented with simple attention mechanism first instead of GQA/MLHA that I have used. MLHA is known to have better performance as compared to simple attention, and in less than 500M model parameters I cannot afford to run experiments on larger GPUs. I did what I could with spot machines and Colab instances.

Due to the fact that larger model sizes have loss curves over the one with smaller model sizes, the five papers are completely skewed.

The reason they are so off might actually be because of the same reason that the Chinchilla–Kaplan scaling laws failed. They did 400 training runs on sub-10B parameter models and stopped at a certain token count.

Their LR scheduler also converged to a set number of tokens which didn’t allow the model to learn any more. Had they fed more tokens without forcing the LR to 0, they would have come to the same conclusion as the Chinchilla scaling laws.

What I would do better given more time and more resources

I had a script for HellaSwag evals ready, but I wasn’t able to run it on the several checkpoints I had collected for different models. This would have given an early signal on what might have been going wrong.

I would have trained more models in the 500M to 1B parameter range. At least 20 training runs. This is the best I could do with spot instances that cost $0.26 an hour on AWS.

I would have experimented with different model architectures.

I was not able to figure out how to have a common filesystem that can be attached to multiple instances at the same time. In this way I would be able to parallelize my training runs.

Experiment setup (AI written)

Training ran on AWS spot GPU instances (mainly g4dn.xlarge / g5–g6 family, ~$0.25–0.26/hr) in ap-south-1. Code, pretokens, checkpoints, and TensorBoard logs lived on a single persistent EBS volume mounted at /mnt/data. Instances were disposable; the disk was not.

Persistent volume, ephemeral compute

Create (or reuse) an EBS volume in a fixed AZ.

Spin up a spot instance in that same AZ.

Attach the volume, mount it at /mnt/data, install deps (uv, etc.).

Train from /mnt/data/scaling-laws with a per-run YAML config.

When done (or when Spot reclaims the box), terminate the instance — checkpoints and logs stay on the volume.

Next session: launch a new spot box, re-attach the same volume, resume from last.ckpt.

Scripts under ec2/scripts/ automate this (spin-up-gpu-machine.sh, spin-up-g5-gpu-machine.sh, matching terminate scripts). Typical flow:


./ec2/scripts/spin-up-gpu-machine.sh   # launch spot + attach + mount /mnt/data
ssh gpu-machine-1
cd /mnt/data/scaling-laws
uv run python main.py --config configs/<run>.yaml
# later
./ec2/scripts/terminate-gpu-machine.sh

What lived on the volume

Path	Contents
`/mnt/data/data/fineweb/`	Pretokenized FineWeb shards
`/mnt/data/logs/`	TensorBoard event files per run
`/mnt/data/checkpoints/`	Checkpoints (`last.ckpt` enables resume)
`/mnt/data/scaling-laws/`	Repo + configs

Each model size had its own config YAML; runs were sequential on one GPU at a time. Spot interruptions were expected — reattach the volume and continue.