Nirjas Comment Classifier - Model Benchmark & Selection Report
Task. Nirjas extracts every comment from a source repository and labels each as license-related or not (binary). Atarashi, a separate model, then decides which license. This report benchmarks every modeling approach for the Nirjas binary classifier and justifies the selection.
Dataset. output/nirjas (recreated 2026-06): train 58,103 / val 6,945 / test 6,950.
Label convention 0 = license_related,
1 = not; remapped internally to license = 1 so F1/AUC read normally. All numbers below are
on the held-out test split.
TL;DR - the decision
Ship model2vec's trainable classifier on a bge-distilled base (
StaticModelForClassification): 0.990 F1, 30 MB, ~13k samp/s on CPU, no GPU at inference, short-form recall 1.00. It is Pareto-optimal (§3, Fig. 3) and the rational default for FOSSology-scale, CPU-first scanning. A fine-tunedjhu-clsp/ettin-encoder-32m(length-bucketed) is the accuracy-critical fallback - it buys ~0.8 balanced-accuracy points, entirely on the contextual code-comment class, at ~100–290× the CPU cost.Why model2vec-clf is the default, not the fine-tuned encoder: the benchmark is saturated (top models within ~1 F1 point), so the ~0.8-point edge a fine-tuned encoder buys does not justify ~290× the per-comment cost for bulk first-pass scanning - and a two-stage design (cheap filter → Ettin on the uncertain few) recovers that edge anyway. Only model2vec used as static embeddings + a linear head is weak (0.947); that distinction is the whole point of §3.
Caveat that dominates everything: model choice is no longer the bottleneck - data quality is. The numbers pick a model; they do not prove the task is solved.
1. Methodology
Five families, all trained and evaluated on the identical recreated split
(benchmark_nirjas.py + bench_model2vec_clf.py):
| Family | What it is | Head |
|---|---|---|
| lexical | TF-IDF features (word 1–2gram / char_wb 3–5gram / both) | LogisticRegression |
| frozen-embed | a SentenceTransformer used as-is (no gradient) | LogisticRegression |
| model2vec | distill a teacher → static embeddings (frozen) | LogisticRegression |
| model2vec-clf | model2vec's own StaticModelForClassification - MLP head + on-task fine-tuning of the static vectors, deployed via to_pipeline() (CPU/numpy) | trained jointly |
| fine-tuned | an encoder trained end-to-end (HF Trainer, 1 epoch) | classification head |
model2vec appears twice on purpose. The proposal links model2vec's classifier training, so
model2vec-clfis the faithful test of the proposed approach;model2vec(static + LR) is the naive baseline. They land in very different places - see §3.
Metrics. F1, ROC-AUC, accuracy, balanced accuracy (equal-class subset, since the split
is ~46/54), per-negative_type correct-rejection rate, and short-form recall (recall on the
287 held-out SPDX/one-line-reference test comments). Deployment axis: CPU throughput and
model size.
Throughput protocol (identical for every model). We time the full raw-text → label path
(tokenize/vectorize/encode + head/forward) on CPU - pinned to 8 threads, batch 64, 2 warmup
passes then the median of 3 timed passes over a fixed 320-comment sample; the run-to-run
[min, max] range is recorded alongside. fp32, eager PyTorch, no ONNX/quantization (so the
transformer numbers are a conservative floor). Config in output/nirjas/benchmark_meta.json.
Reproducibility. benchmark_nirjas.py → output/nirjas/benchmark_results.json (+ _meta);
plot_benchmark.py → docs/figs/. Seeds fixed (set_seed(0), random_state=0); differences
< ~0.003 F1 are within run-to-run noise.
2. Results
CPU throughput = median samp/s on the full raw-text→label path (8 threads, batch 64); sizes are fp32 (INT8 ONNX ≈ ¼). Sorted by balanced accuracy.
| model | family | F1 | bal-acc | AUC | gen-code-comment reject | hard-neg reject | short-form recall | CPU samp/s | size (MB, fp32) | params (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Ettin-150m | fine-tuned | 0.9990 | 0.9992 | 1.0000 | 1.000 | 1.000 | 1.00 | 22 | 598 | 150 |
| ModernBERT-base | fine-tuned | 0.9987 | 0.9990 | 1.0000 | 1.000 | 1.000 | 1.00 | 22 | 598 | 150 |
| Ettin-32m | fine-tuned | 0.9984 | 0.9989 | 1.0000 | 0.999 | 1.000 | 1.00 | 123 | 128 | 32 |
| Ettin-68m | fine-tuned | 0.9982 | 0.9985 | 1.0000 | 0.999 | 1.000 | 1.00 | 57 | 274 | 68 |
| Ettin-68m (INT8 ONNX) | fine-tuned | 0.9979 | 0.9984 | 1.0000 | 0.998 | 1.000 | 1.00 | 45 | 70 | 68 |
| Ettin-17m | fine-tuned | 0.9971 | 0.9976 | 1.0000 | 0.997 | 1.000 | 1.00 | 320 | 68 | 17 |
| distilbert (2019) | fine-tuned | 0.9964 | 0.9969 | 0.9999 | 0.997 | 1.000 | 1.00 | 72 | 268 | 67 |
| tfidf-char | lexical | 0.9934 | 0.9939 | 0.9996 | 0.996 | 1.000 | 1.00 | 3,791 | 0.8 | - |
| tfidf-word+char | lexical | 0.9911 | 0.9919 | 0.9995 | 0.994 | 0.998 | 1.00 | 2,987 | 1.2 | - |
| model2vec-clf ← bge ⭐ | model2vec-clf | 0.9895 | 0.9905 | 0.9991 | 0.991 | 0.998 | 1.00 | 12,949 | 30 | 8 |
| model2vec-clf (potion-32M) | model2vec-clf | 0.9876 | 0.9892 | 0.9993 | 0.985 | 0.999 | 1.00 | 13,091 | 129 | 32 |
| bge-base-en-v1.5 | frozen-embed | 0.9848 | 0.9861 | 0.9983 | 0.989 | 0.999 | 0.990 | 9 | 438 | 110 |
| tfidf-word | lexical | 0.9803 | 0.9819 | 0.9984 | 0.985 | 0.999 | 1.00 | 19,119 | 0.4 | - |
| MiniLM-L6 | frozen-embed | 0.9701 | 0.9737 | 0.9961 | 0.979 | 0.975 | 0.941 | 100 | 91 | 23 |
| model2vec ← bge | model2vec | 0.9465 | 0.9520 | 0.9885 | 0.960 | 0.957 | 0.948 | 4,118 | 30 | 8 |
Fig. 1 - F1 (solid) and balanced accuracy (faded), by family. Fine-tuned encoders (green) lead; lexical (grey) is a strong floor; model2vec's trainable classifier (orange) is mid-pack at ~0.99; only static model2vec + LR (red) is last.
Fig. 2 - the deployment tradeoff. X = CPU throughput (log), Y = balanced accuracy, bubble ∝ size. The accuracy ceiling is flat across the top; the real spread is cost. Note the two model2vec points: static+LR (red) sits at the bottom, but the trainable classifier (orange) jumps to ~0.99 at the far right - same library, very different outcome.
Fig. 3 - Pareto frontier on the same axes (maximise both accuracy and CPU throughput). ★ marks
the non-dominated set: tfidf-word → model2vec-clf(potion) → model2vec-clf(bge) → tfidf-char → Ettin-17m → Ettin-32m → Ettin-150m. Both model2vec trainable classifiers (orange) are on the
frontier - fast and ~0.99 accurate. Faded = strictly dominated: static model2vec (red), frozen
bge, MiniLM, word+char TF-IDF, ModernBERT-base, and Ettin-68m/distilbert (Ettin-32m is faster
and more accurate). The Nirjas decision is where on the frontier: Ettin-32m for top
accuracy, model2vec-clf(bge-distill) for ~100× its CPU throughput at −0.8 F1, or char-TF-IDF
for a 0.8 MB no-DL filter. (CPU numbers here are random-batch; §5b length-bucketing lifts the
fine-tuned points ~1.9× more.)
3. model2vec - naive static head vs the trainable classifier
How you use model2vec matters enormously, so we benchmarked both. The result splits cleanly:
Naive: static embeddings + LogisticRegression - avoid. F1 0.947, the least accurate model in the study, and strictly dominated - word-TF-IDF is ~5× faster (19,119 vs 4,118 samp/s) and +3.4 F1 more accurate (0.980 vs 0.947). Freezing the averaged static vectors and bolting on a linear head throws away the teacher's signal: bge-base frozen 0.985 → static+LR 0.947, a 3.8-point loss purely from going static. Off the Pareto frontier (Fig. 3).
The proposal's path: StaticModelForClassification - genuinely strong. model2vec's own
classifier adds an MLP head and fine-tunes the static token vectors on-task. That changes the
verdict completely:
| model2vec variant | F1 | bal | gen-code-comment | CPU samp/s | size |
|---|---|---|---|---|---|
| static embeddings + LR | 0.9465 | 0.9520 | 0.960 | 4,118 | 30 MB |
| trainable clf (potion-32M base) | 0.9876 | 0.9892 | 0.985 | 13,091 | 129 MB |
| trainable clf (bge-distilled base) | 0.9895 | 0.9905 | 0.991 | 12,949 | 30 MB |
The bge-distilled classifier lands at 0.990 F1 / 0.991 balanced, 30 MB, ~13k samp/s on CPU - Pareto-optimal (Fig. 3), learns SPDX cleanly (short-form recall 1.00), and is the fastest non-trivial model after bare word-TF-IDF. So the proposal's model2vec plan is defensible - as long as it uses the trainable classifier, not static+LR.
Why it still trails the fine-tuned encoders by ~1 F1. It remains static - token vectors are averaged with no contextual interaction - so the contextual class tops out: generic-code-comment 0.991 vs fine-tuned 0.999–1.000. Fine-tuning a real encoder is the only family that closes that last class. That ~1-point gap is exactly the §5 tradeoff: decisive for a compliance filter (a missed license clause is costly), negligible if raw CPU throughput is the priority.
So model2vec-clf is the recommended model. The trainable bge-distilled classifier is Pareto-optimal, GPU-free at inference (30 MB, ~13k samp/s), learns SPDX cleanly (short-form recall 1.00). The only thing it concedes to a fine-tuned encoder is ~1 F1 on the contextual code-comment class - and on this saturated benchmark, at ~290× the CPU cost, that is not a trade worth making for bulk scanning (§5). Fine-tuned Ettin is the accuracy-critical fallback, not the default.
4. Why fine-tune, not frozen embeddings
Frozen embedding + LR (an embedding model used as-is, no gradient) is one of the candidate approaches. It underperforms fine-tuning for a structural reason, not a tuning one:
- bge-base frozen = 0.985 F1; the same-class ModernBERT/Ettin fine-tuned = 0.999.
- A frozen embedding is optimized for similarity (retrieval), and the classifier on top cannot reshape it. Fine-tuning lets gradients reach the encoder so the representation bends toward this decision boundary. Same backbone, different objective reach → +1.4 F1 and the code-comment error class essentially eliminated.
- Embedding models are the right tool for Atarashi (which-license = nearest-neighbour retrieval), not for Nirjas (a classification boundary).
5. Why model2vec-clf is the default (and when to reach for Ettin)
Because the benchmark is saturated (Fig. 2, flat top), F1 cannot crown a winner - the top models span just ~0.011 F1 (0.9990 → 0.9876), much of it within run-to-run noise. When accuracy is effectively tied, selection is driven by deployment economics - and there model2vec-clf wins decisively: it is Pareto-optimal (Fig. 3), runs at ~13k samp/s on CPU with no GPU, and fits in 30 MB. The fine-tuned encoders are the fallback you reach for only when the last ~1 F1 on contextual code comments is worth ~100–290× the cost.
| candidate | why / why not |
|---|---|
| model2vec-clf (bge-distill) ⭐ | the default - 0.9905 bal, 30 MB, 12,949 samp/s (~230× Ettin-68m on CPU, no GPU at inference), short-form recall 1.00. Concedes only −0.8 bal vs the fine-tuned tier, entirely on the contextual code-comment class. The rational pick for FOSSology-scale first-pass scanning. |
| model2vec-clf (potion-32M) | same family, slightly lower (0.9892 bal) and 4× the size (129 MB) for no speed gain - prefer the bge-distilled 30 MB build. |
| char TF-IDF | the no-DL alternative - 0.9939 bal at 3,791 samp/s and 0.8 MB. Marginally more accurate than model2vec-clf but ~3× slower; the lightest drop-in first-stage filter if a DL artifact is unwanted. |
| Ettin-32m (accuracy fallback) | the best fine-tuned choice if accuracy is critical: ties the 150M tier (0.9989 bal) at 123 samp/s - 5.6× the speed and ¼ the size of Ettin-150m. The saturated task doesn't need 68M+. MIT-licensed, code-aware, siblings (17m…1b) for a clean dial; length-bucket for ~230 samp/s (§5b). |
| Ettin-68m / Ettin-17m | 68m is dominated by 32m (slower, no more accurate). 17m trades −0.1 F1 for 320 samp/s. |
| Ettin-150m / ModernBERT-base | marginally highest (0.9990–0.9992 bal) but 598 MB and ~22 samp/s - never worth 2× the size of 32m for +0.0003 acc. |
| distilbert | 2019; slightly lower (0.9964) and no code-aware pretraining. A ceiling probe only. |
Fig. 4 - per-class. The discriminating class is generic_code_comment: lexical/static models
trail (0.96–0.99) where fine-tuned code-aware encoders reach 0.999. Hard legal negatives and
license recall are near-saturated for everyone.
Does INT8 ONNX rescue Ettin's CPU speed? We measured it - no. Exporting fine-tuned Ettin-68m
to ONNX and dynamic-INT8-quantizing it (optimum + onnxruntime; per-channel + reduce_range to
keep accuracy) gives 0.9979 F1 at 45 samp/s and 70 MB - accuracy fully preserved (per-tensor
quant instead collapsed it to 0.56, a trap to avoid) and 4× smaller, but no throughput gain
(45 vs 57 eager / 41 fp32-ONNX). On this ModernBERT-architecture model, ORT's dynamic INT8 does
not engage faster kernels; the realistic CPU figure for Ettin stays ~45–57 samp/s. So
INT8 does not change the deployment picture - Ettin remains ~250–300× slower per comment than
the lexical/static options. If throughput dominates first-pass repo scanning, use the two-stage
design (regex SPDX → char-TF-IDF or model2vec-clf pre-filter → Ettin on the survivors) or pick
model2vec-clf outright (below).
5a. model2vec-clf vs Ettin-68m - the head-to-head
With INT8 measured, the two real deployment candidates are:
| candidate | balanced acc | CPU throughput | size | GPU at inference |
|---|---|---|---|---|
| Ettin-68m (INT8 ONNX) | 0.9984 | ~45 samp/s | 70 MB | no |
| model2vec-clf (bge-distill) | 0.9905 | ~12,950 samp/s | 30 MB | no |
The trade is 0.8 balanced-accuracy points (entirely on the contextual generic_code_comment
class) against ~290× CPU throughput and half the footprint. Decision rule:
- Default → model2vec-clf(bge-distill). For CPU-first, FOSSology-scale scanning (millions of files), ~290× the throughput at a 0.8-point cost - on a saturated benchmark - is the rational default. This is the recommended model.
- Accuracy-critical fallback → Ettin-32m. Only when a missed license clause is expensive enough to justify the cost: the fine-tuned encoder reclaims the last ~1 F1 on contextual code comments. A few hundred samp/s (length-bucketed, §5b) is fine for incremental or GPU-assisted scanning.
- Best of both → two-stage: model2vec-clf flags candidates fast; Ettin adjudicates only the uncertain ones. Captures Ettin's accuracy at near-static throughput - and is the recommended shape if you want both.
5b. Making Ettin faster on CPU (researched + measured)
INT8 was the wrong lever. Four that actually work here, measured:
| lever | effect | accuracy cost |
|---|---|---|
| Smaller Ettin-32m | 57 → 123 samp/s (2.2×) | none - 0.9989 bal ties Ettin-68m (task is saturated, so half the params cost nothing) |
| Smaller Ettin-17m | 57 → 320 samp/s (5.6×) | −0.1 F1 (0.9976 bal - still above model2vec-clf's 0.9905) |
| Length bucketing | ~1.9× on top (45→84 INT8, 41→77 fp32) | none - comments are short (median 32 tok); sorting by length so a batch pads to its own max, not 128, is free |
| ORT graph optimization (O3) | small (~+5%, bucketed 77→80) | none |
| INT8 ONNX (per-channel) | size 274→70 MB | none, but no speed gain (§5) |
Composed best: Ettin-32m + length bucketing ≈ ~230 samp/s at 0.9989 bal (Ettin-68m accuracy at ~4× its speed); Ettin-17m + bucketing ≈ ~600 samp/s at 0.9976 bal. So within the Ettin fallback, the rule is: pick Ettin-32m, not 68m, and always length-bucket.
But the ceiling is real - even maxed, a fine-tuned Ettin tops out in the hundreds/sec, still ~20–50× below model2vec-clf (12,950) and char-TF-IDF (3,791). These levers narrow the §5a gap (you can hold >0.99 accuracy at a few hundred samp/s) but don't erase it. For true bulk CPU scanning, model2vec-clf or the two-stage design still wins.
6. Recommendation
- Default (recommended): model2vec's trainable classifier on a bge-distilled base
(
StaticModelForClassification) - 0.990 F1, 30 MB, ~13k samp/s on CPU, no GPU at inference, short-form recall 1.00. Pareto-optimal (§3), and the rational choice for CPU-first, FOSSology-scale first-pass scanning. (Do not use model2vec as static-embeddings + a linear head - that variant is weak, §3.) - Accuracy-critical fallback: fine-tune
jhu-clsp/ettin-encoder-32m(not 68m - same accuracy, 2.2× faster) and length-bucket at inference (≈230 samp/s, free). Reach for it only when the ~1 F1 edge on contextual code comments is worth ~100–290× the CPU cost - e.g. a high-stakes compliance gate. Drop to17m(≈600 samp/s) for more speed; INT8 ONNX helps size, not speed. - Best of both - two-stage inference (recommended if you want Ettin's accuracy at scale): regex
SPDX fast-path (
SPDX-License-Identifier:\s*\S+, exact) → model2vec-clf as the cheap filter → Ettin-32m on the uncertain few. Add a confidence threshold +UNKNOWN/abstain. - Invest in data, not model search: add same-register hard negatives; the benchmark is the ceiling, not the encoder.