Skip to main content

Weekly Sync - 30 June 2026

Attendees​

Benchmarking Results & Model Selection​

Nirjas Binary Classifier Benchmarking​

Completed comprehensive benchmarking of Nirjas comment classifier across five modeling families:

  • Lexical (TF-IDF): baseline approaches
  • Frozen embeddings (SentenceTransformer): pre-trained without fine-tuning
  • model2vec: static embedding distillation
  • model2vec-clf: model2vec with trainable classifier head
  • Fine-tuned encoders: end-to-end trained models (Ettin, ModernBERT, DistilBERT)

Decision: model2vec for Production​

Selected: model2vec's trainable classifier on bge-distilled base (StaticModelForClassification)

Key Metrics:

  • F1 Score: 0.990
  • Balanced Accuracy: 0.9905
  • Model Size: 30 MB (fp32)
  • CPU Throughput: ~13,000 samples/second (8 threads, batch 64)
  • Short-form Recall: 1.00 (SPDX/one-line references)

Rationale:

  • Achieves Pareto-optimal performance for FOSSology's CPU-first scanning requirements
  • Exceptional throughput on commodity hardware with no GPU dependency at inference
  • Only 8M parameters enables deployment at scale
  • Within ~1 F1 point of best fine-tuned models while being 100–290× faster
  • Two-stage design (cheap filter → fine-tuned encoder on uncertain samples) can recover the accuracy gap if needed

Key Finding​

Model choice is no longer the bottleneck—data quality is the critical path forward. Benchmarking validates the approach; improving labeled comment datasets will drive further gains.

Detailed Report: See nirjas-benchmark-report.md for full methodology, results table across all 15 models, and comprehensive analysis.

Action Items​

  • Integrate model2vec-clf classifier into Nirjas extraction pipeline
  • Prepare deployment configuration for CPU-optimized inference
  • Plan data quality improvements for next iteration