A developer distilled Gemini 3.1 Pro into a local GLiNER named-entity recognition model for $9, slashing ongoing LLM scraping costs while maintaining 0.83 F1 accuracy.
To monitor Reddit discussions about high-end chef's knives for his site New Knife Day, developer Peter Vijeh initially relied on Gemini 3.1 Pro API calls to extract knife brands, models, and steels via named-entity recognition (NER). As forum volume expanded, recurring API costs became unsustainable, while an off-the-shelf zero-shot GLiNER model scored an inadequate ~0.65 F1. To bridge the gap cheaply, Vijeh paid $9 to have Gemini 3.1 Pro label 4,290 Reddit comments via OpenRouter ($0.0021 per comment) and used those annotations to fine-tune GLiNER large v2.5 (a 459M-parameter DeBERTa-v3 model) on a Modal Tesla T4 GPU. By requesting raw substrings rather than character offsets to avoid token drift and resolving several subtle engineering traps—such as learning that GLiNER's `words_mask` tensor expects sequential word indices rather than a standard binary attention mask—the final model achieved 0.83 F1 on a locked 225-comment validation set, effectively paying for itself after roughly 4,300 comments and eliminating all ongoing inference bills.
Distilling LLMs into small, edge-deployable models is the most effective antidote to SaaS API token bloat, but the hardest part of fine-tuning is almost never the data or architecture—it is undocumented plumbing bugs.
- –**Distillation delivers rapid ROI**: Spending $9 once on frontier model synthetic labels to bootstrap a 459M open model replaced recurring per-comment API charges with a self-hosted pipeline that broke even in fewer than 4,300 comments.
- –**Substring extraction beats offset prediction**: Bypassing the LLM's inability to accurately count character offsets by requesting exact text matches and computing token spans programmatically avoided corrupted labels and token boundary mismatches.
- –**Plumbing errors disguise as data problems**: Five of the author's ten training runs completely failed due to configuration defaults and misinterpreting `words_mask` as a binary attention mask rather than an index tensor, causing flat loss curves that emitted zero diagnostic warnings.
- –**Per-class thresholds rescue domain vocabulary**: Applying custom confidence cutoffs per entity class rather than a global threshold jumped material recall from 78.7% to 91.1%, proving that post-processing calibration is crucial for niche, low-confidence technical jargon.
DISCOVERED
1h ago
2026-09-17
PUBLISHED
4h ago
2026-09-17
RELEVANCE
AUTHOR
p-s-v
