Date
Aug 4, 2026
slug
uncertainty-estimation
Status
Published
Subtitle
How to choose which data samples to annotate when you dont have a lot of money
Tags
Deep-Learning
Uncertainty estimation
This entire experiment is aimed at finding the answer to the question: “From a given pool of unlabeled data, how would you select a subset to annotate so that they can help your model become better?”
Gtihub repo : https://github.com/nishantb06/uncertainity_estimation
Protocol
We work with 5680 DocILE documents. Following the assignment remap, the original validation set (500) is the labeled seed, and the original training set (5180) is the unlabeled pool.
From the pool I carved out a fixed stratified holdout of 180 documents for evaluation, dropped
debit_note and utility_bill (absent from val), and left ~4982 documents as the selectable pool. All reported metrics are on this holdout.The three training runs are:
- Baseline — train on the 500 labeled seed only.
- Selected — train on the 500 + 1500 documents chosen from the pool without using their labels at selection time.
- Ceiling — train on the full pool ∪ seed. This sets an upper bound; 500 + 1500 should beat baseline but not beat ceiling.
Experiments setup
Infrastructure
I train on AWS spot instances (G6) machines and I keep one EBS volume which gets reattached and mounted to every new machine. This way all my checkpoints and logs persist across all the training sessions.
Model
Every recent SOTA document understanding / OCR model has a two-phase pipeline: a ~500M vision encoder that produces token embeddings for an LLM decoder. Keeping the visual token count small matters, since 16×16 patches explode context length.
I used DeepSeek’s vision encoder: an 80M SAM encoder, a convolution downscale, then a 300M CLIP model that outputs 64 / 100 / 256 tokens at 512 / 640 / 1024. I freeze SAM and CLIP, and attach a custom MLP on CLIP’s CLS token for 7-way document-type classification (page 0 only, train resolution 640).
Tax invoice and order dominate the class distribution (~68% / ~25%), so I only include holdout curves for those two.
Methods for uncertainty estimation
I scored the unlabeled pool with the baseline checkpoint. Let be the MLP logits over classes, and
(natural log / nats). High means the model cannot classify the document clearly, so annotating it is more likely to provide useful training signal.
The main selection rule was average entropy. For resolutions with logits
(
entropy_mean_logits). I rank the pool by and take the top 1500.The same multi-resolution setup also supports other scores I implemented (not used for the plots below):
- Per-resolution entropy — high on all three, or extremely high on one.
- Jensen–Shannon divergence across resolution pairs. For distributions :
- Ghost-gradient uncertainty: for head-gradient directions at each resolution, score disagreement as
where is the set of resolution pairs.
Multi-checkpoint aggregation is another natural extension I did not run for the final comparison.
Results
Using average-entropy selection, the 500 + 1500 model outperforms the baseline (500 only) on holdout loss and per-class accuracy for tax invoice and order, but does not beat the ceiling. Runs are not length-matched, so compare the curves with that in mind.
Validation loss for runs
Validation accuracy — tax invoice
Validation accuracy — orders
Training accuracy
Gradient norms
Limitations
A random 1500 baseline would make clearer that the gain comes from which documents were chosen, not only from adding 1500 labels. Given more time, I would also try the same selection idea on an object-detection loss, and compare JSD / ghost selection head-to-head with entropy.