Atarashi License Identification — Engine Report
Task. Nirjas extracts comments from a source file and decides whether each is license-related. Atarashi answers the next question: which license, with the span that proves it. This report covers the engine — a lexical cascade with a learned candidate ranker — the benchmarks it is measured on, and where it stands against ScanCode Toolkit.
Corpora. Four, reported separately and never blended.
| corpus | n | regime | labels |
|---|---|---|---|
| DEP-5 | 671 | notice | Debian maintainers' DEP-5 records |
| SPDX-tag | 266 | notice | author-declared SPDX-License-Identifier:, tag line stripped |
| sh-prevalence | 11,680 | verbatim | Software Heritage, weighted by real occurrence |
| sh-tail | 66 | verbatim | Software Heritage, one blob per license |
Plus 1,379 (DEP-5) and 2,145 (SPDX-tag) no-signal queries, where the correct answer is UNKNOWN.
Terms used throughout. Defined once here rather than at each use.
| term | meaning |
|---|---|
| R@1 | the top-ranked license is the correct one |
| precision | of the files answered, the fraction answered correctly |
| exact-set | the reported set of licenses equals the ground truth exactly — what a compliance reviewer actually consumes |
| coverage | the fraction of queries the engine answers rather than abstaining |
| notice regime | source files carrying a short license notice or one-line reference |
| verbatim regime | whole LICENSE files carrying a license body |
| bespoke | a file whose license is not in the engine's list at all |
| τ (tau) | the accept threshold — below it, the engine abstains |
| prevalence-weighted | scored over real files in their real proportions, so common licenses count more |
TL;DR — where it landed
Level with ScanCode on the notice corpora, strong in the verbatim regime, and markedly weaker on the tail. Most of the distance came from fixing defects, not from modelling.
Current results, at the shipped configuration:
corpus R@1 precision exact-set DEP-5 (n=671) 0.8718 0.8850 0.8714 SPDX-tag (n=266) 0.9586 0.9733 0.9122 sh-prevalence (n=11,680) 0.9770 — 0.9737 sh-tail (n=66) 0.8333 — — The prevalence pool is the benchmark with the power to decide anything: ±0.0028 against DEP-5's ±0.028.
Against ScanCode: level, nowhere ahead by a margin the data can resolve. Under nested cross-validation at matched coverage (τ=0.20), paired McNemar gives p=1.000 on DEP-5 R@1 and p=0.454 on SPDX-tag R@1. See §2b — including why that comparison is older than the numbers above.
No-signal false answers 6.1% (DEP-5) and 2.0% (SPDX-tag). 20.7 ms/query on CPU, no GPU. The learned component is a shallow gradient-boosted tree over engineered match features — not a text model, for reasons in §6.
Most of the distance came from defects, not the model. DEP-5 precision moved 0.652 → 0.885 over the rebuild; the learned ranker accounts for roughly the last 0.04. The rest was ~3,700 silently dropped GPL rules, required-phrase text being deleted from every rule, a result-set bug reporting 4.72 licenses per file, and an instructional appendix indexed as license text. Every time the instrument got sharper it found plumbing, not a modelling limit (§7).
Caveat that bounds everything below: this is a head-of-distribution tool. Leave-one-license-family-out for the ranker is +0.03; the ~2,400 licenses no corpus contains are served by the lexical cascade with no learned help (§8).
1. Architecture
query (Nirjas-extracted comment block)
1. SPDX-License-Identifier → license expression, AND/OR/WITH preserved
2. exact normalised-text hash → verbatim full text
3. token-sequence match → matched span + coverage, over the notice index
4. learned candidate scoring → reorder, and decide whether to answer at all
5. report → license + character span, a set, or UNKNOWN
Stages 1–3 are deterministic. Stage 4 scores candidates from match evidence only — run lengths, coverage ratios, gap structure, corroboration across a license's other units, distance to the runner-up. It never reads license text (§6).
Reference index. 27,857 short notice/reference/tag rules across 2,024 keys, extracted from scancode-toolkit at build time and committed, so installing Atarashi never pulls scancode in. 15,915 map onto the 774-license list the agent answers from. 1,038 of the keys are whole expressions rather than single licenses — a compound rule is indexed as one unit and expands at reporting time into one result per component. The shingle-3 rebuild added roughly 3,000 further matchable units.
Matcher. Seed-and-extend over shared shingles: a shared n-gram at reference
position i and query position j lies on diagonal j−i, so one walk of the
reference recovers every contiguous run. Replaced a per-candidate difflib diff —
146 ms/query → 12 ms at the time, 20.7 ms/query at the current index size.
2. Results
2a. Current results
At the shipped configuration — shingle 3, indexing floor 3, candidate cap 200, top_k 5, ranker artifact as committed.
| corpus | n | coverage | R@1 | precision | exact-set |
|---|---|---|---|---|---|
| DEP-5 | 671 | 0.985 | 0.8718 | 0.8850 | 0.8714 |
| SPDX-tag | 266 | — | 0.9586 | 0.9733 | 0.9122 |
| sh-prevalence | 11,680 | 0.9916 | 0.9770 | — | 0.9737 |
| sh-tail | 66 | — | 0.8333 | — | — |
Read the prevalence row first. At n=11,680 it resolves differences of ~0.006; DEP-5 and SPDX-tag resolve ~0.06, so neither can settle anything now in dispute.
The tail row is macro by construction and must never be read as prevalence-weighted.
One blob per license deliberately over-weights rare variants: GPL-3.0-only and
GPL-3.0-or-later each get one blob although the real ratio among human-confirmed
bodies is 107:4. It answers "which licenses can we identify at all", not "how often
are we right on real files".
2b. Against ScanCode
Under nested cross-validation — the outer fold evaluates, τ derived by conformal calibration inside the training split, so each query is predicted once by a model and a threshold that never saw it. McNemar is the paired test for two systems scored on the same queries: it counts only the files where they disagree, so a high p-value means neither engine wins those disagreements more often than chance.
| corpus | agent (τ=0.20) | ScanCode | paired McNemar | |
|---|---|---|---|---|
| DEP-5 (n=671) | precision | 0.8772 | 0.8404 | — |
| R@1 | 0.8301 | 0.8316 | p = 1.000 | |
| exact-set | 0.8077 | 0.8018 | p = 0.720 | |
| SPDX-tag (n=266) | precision | 0.9533 | 0.9727 | — |
| R@1 | 0.9211 | 0.9361 | p = 0.454 | |
| exact-set | 0.8571 | 0.8496 | p = 0.832 |
The engine is level with ScanCode and ahead of it nowhere the data can resolve.
This table is older than §2a, and the gap matters. It was measured on 2026-08-25; the compound-rule layer, the only/or-later rule, the widened ranker, the appendix split and four constant sweeps all landed after it. Those changes moved the in-sample and single-split numbers in §2a upward, but the nested-CV head-to-head has not been re-run at the current configuration. Quoting §2a's 0.8718 against ScanCode's 0.8316 would be comparing a single-split number to a held-out one. This project made that exact error once already: an exact-set win over ScanCode reported at p=0.017 became p=0.720 once the threshold stopped being calibrated on the queries scoring it. Until nested CV is re-run, "level with ScanCode" is the honest claim.
Two further caveats belong with the p-values. The five fold-models share training data, so cross-validated paired testing is mildly anti-conservative. And ScanCode has no training, so only our side carries the held-out question at all.
2c. Abstention
About 72% of real source files carrying an SPDX tag have no license prose once the tag is stripped. Abstaining is correct there, and raw recall is the wrong target because it rewards guessing.
| corpus | no-signal queries | falsely answered |
|---|---|---|
| DEP-5 | 1,379 | 6.1% |
| SPDX-tag | 2,145 | 2.0% |
This is why every constant sweep needs a no-signal arm. One that lacked it read permissiveness as quality and recommended a setting that answers 48% of files containing no license text at all — every headline number improved while the engine became useless.
3. The learned ranker
Why learned at all. The cascade retrieves well and orders badly: 62 of 77 wrong answers on DEP-5 already held the correct license in the top five, 28 at rank two. Perfect reranking would reach 0.960 precision. Five hand-tuned ranking rules were tried first and every one lost more than it gained (§5).
Model. Gradient-boosted tree, depth 6, 400 iterations, learning rate 0.03, over 17 features, argmax across the candidate list. Logistic regression fails (−0.025): the signals interact — an unmatched reference tail only matters when coverage is otherwise high — and a linear combination cannot express that. LambdaMART-class learning-to-rank was ruled out on scale, being validated at 10,000 queries and 136 features against our ~2,500 and 17.
| test | delta over the hand-tuned ranking |
|---|---|
| grouped CV, held-out queries | +0.065 |
| leave-one-license-family-out | +0.029 |
Measured on the current 2,462-query training set spanning 152 licenses across 28 families. The gate that decides when the model may act was as consequential as the model: asking whether the leader's family was trained cost 286 of 639 Apache-2.0 files (§7).
Features are standardised within the candidate list. Absolute magnitudes fingerprint a license — a 5,699-token reference body is GPL-3.0, 2,984 is GPL-2.0 — so a model given them learns identity rather than what a good match looks like. This was the largest correctness issue in the modelling: leave-one-family-out was −0.051 before, and got worse as training licenses grew from 18 to 34. That is a statement about families; widening at license granularity is a different axis and does pay. Query-level normalisation is the standard fix in learning-to-rank; LETOR has shipped it for two decades.
Corroboration across units is what first made out-of-distribution performance
positive. A license whose best-covered unit the query fills completely is better
supported than one covering 70% of a shorter unit, even when the retained unit's run is
a token shorter — precisely how mpl-2.0-no-copyleft-exception kept beating MPL-2.0 at
41/59 against 41/84.
4. Uncertainty
Three outcomes, and they are the three set sizes of a conformal predictor — a method that turns a score into a set carrying a stated probability of containing the right answer.
| output | meaning |
|---|---|
UNKNOWN + candidates | nothing confirmed; names what was considered and rejected |
| one license + span | confident answer |
Ambiguous + set | the evidence does not separate these |
The accept threshold comes from a coverage target, and the calibration is cross-conformal. Both parts changed after measurement. Calibrating on a single 50/50 split was unstable — the per-fold threshold ranged 0.231 to 0.427 under nested CV, and the high draws cost enough coverage to put the engine significantly behind ScanCode on R@1 — so scores are now pooled out-of-fold across K folds, cutting the across-fold spread from 0.196 to 0.080.
The α table below is the original single-split calibration, kept because it is what the set-size behaviour was measured on. α bounds "is the correct license inside the reported set", which is not the question the accept decision asks; the shipped threshold comes from a stated coverage target on a held-out risk-coverage curve.
| α | threshold | realised coverage | mean set size | singleton | empty |
|---|---|---|---|---|---|
| 0.20 | 0.6667 | 0.794 | 0.82 | 81.9% | 18.1% |
| 0.10 | 0.3013 | 0.931 | 0.99 | 95.6% | 2.5% |
| 0.05 | 0.0583 | 0.968 | 1.38 | 69.0% | 0.0% |
Two limits belong with any quotation of that coverage: it is conditional on the correct license being retrievable at all (864 of 1,002 queries), and it is marginal rather than per-license — per-license coverage needs ~100 calibration points per class against the ~27 available.
Ambiguity is not collapsed to a license family, though the plan called for it. The
eight cases the flag catches are GPL-2.0-or-later vs GPL-3.0-or-later (×5) and
similar. A family answer would be "GPL" for six of them, and GPL-2 and GPL-3 are
mutually incompatible — true of six, useful for none. The correct license is inside the
reported set in 8 of 8, so the set is the answer.
5. What did not work
Recorded because the pattern took too long to see: every hand-tuned rule over these signals failed, and each failure was misread at the time as evidence against the signal rather than against hand-tuning.
| attempt | result |
|---|---|
| required-phrase gating (per unit) | −4 net correct |
| required phrases as a ranking tie-break | no effect |
| prefer the larger reference on equal evidence | precision 0.929 → 0.775 |
| diff-derived discriminators as a gate | precision 0.795 → 0.429 |
| per-candidate confidence bar (shipped for months) | 4.72 licenses/file, 97% carrying a wrong one |
| exempting margins from normalisation | flat end-to-end; reverted |
| query-side only/or-later signal | no effect on DEP-5; tail 0.682 → 0.652; replaced by the bare-body rule |
| gating the ranker on trained licenses | tail +0.045, DEP-5 R@1 0.8435 → 0.8003; reverted |
| deepening the candidate list to top_k 20 | retrieval ceiling 55% → 82%, tail R@1 0.818 → 0.727; reverted |
| synthetic units for evidence-less licenses | inert on everything measurable; not shipped |
| compound units must match at coverage ≥ 0.95 | breaks the genuine case (a real LLVM notice covers 0.692) |
| widening verbatim training 4 → 12 blobs/license | 0.9752 → 0.9755, inside ±0.0028 |
| dropping a compound a single license explains better | 0.9752 → 0.9677; ISC collapses to 0.02 |
| capping training blobs at 50 KB | 0.9752 → 0.9600 at identical precision; 452 licenses → 297 |
| lowering the indexing floor to 3 at shingle 4 | in-sample better, held-out worse; reversed once the shingle allowed it |
Discriminator presence predicts correctness at 0.59 against a 0.20 base rate — the signals carry real information. A hard threshold is the wrong way to consume it, which is the argument for a learned scorer, not against one.
The two attempts that did work were not filters over these signals at all: fixing which rules are indexed and fixing when the model is allowed to act. The lesson is narrower than "hand-tuning fails" — it is that this decision surface resists thresholds, while the index and the gate around it do not.
6. Why not a text model
No model reads license text to choose between variants. The evidence against is consistent and not marginal: on high-overlap/different-meaning pairs, models score under 40% (PAWS) and below chance when lexical overlap disagrees with the label (HANS); general-purpose embeddings rate a sentence more similar to its own negation than to a different sentence 99.27% of the time. That failure mode — a short decisive span inside otherwise identical text — is exactly GPL-2 vs GPL-3.
Every mature near-duplicate field (plagiarism detection, code-clone detection, revision
control) and every production license scanner uses approximate matching to shortlist and
deterministic comparison to decide. ScanCode has no learned component anywhere: its
score is query_coverage × rule_coverage × relevance, its overlap precedence is four
hand-picked constants, and variant disambiguation is 1,022+ curated rule files for
gpl-2.0-plus alone.
We cannot out-hand-tune a decade of that. What we have that it does not use is labelled corpora — hence a learned scorer over the evidence, and deterministic rules for the decision.
7. Defects that mattered more than the model
| defect | effect |
|---|---|
{{...}} treated as wildcards, not required phrases | identifying text deleted from every rule; DEP-5 precision 0.652 → 0.909 on fix¹ |
| SPDX 3.0 GPL renaming unbridged | ~3,700 GPL/LGPL/AGPL rules silently dropped; the family matchable only against its own multi-thousand-token body |
| license list never merged with SPDX | evaluated against 382 licenses where production uses 774; ground truth outside the list 7.5% → 0% |
| every confident candidate reported | 4.72 licenses/file, 97% of answered files carrying one that was not theirs; exact-set 0.024 → 0.763 |
| non-contiguous feature matrix | 7e-14 difference landing on tree bin boundaries; two identical evaluations disagreed by 0.016 |
| compound rules skipped at build time | 13.3% of the short-form register discarded; LLVM-exception had 0 indexed units; SPDX-tag exact-set 0.599 → 0.907 on fix |
| eval labelled every compound candidate negative | taught the model the new layer was always wrong; made the fix read as a regression |
| accept threshold calibrated on the full dataset | every "held-out" number leaked; an exact-set win over ScanCode at p=0.017 became p=0.720 under nested CV |
.txt sent through comment extraction | LICENSE.txt mangled — a verbatim GPL-3.0 body lost 1,315 chars and abstained; tail R@1 0.652 → 0.682 on fix |
| ranker gated on the leader, not the candidate list | 286 of 639 verbatim Apache-2.0 files reported as ImageMagick; prevalence R@1 0.9468 → 0.9752 on fix |
| instructional appendix indexed as license body | a real Apache-2.0 file lost to Pixar; Apache-2.0 0.944 → 0.973 on fix |
| matcher constants never swept | shingle, floor and candidate cap were all wrong together; DEP-5 R@1 0.8554 → 0.8718 at lower latency |
¹ Measured under the coarser labels of the time, when GPL-2.0-only and
GPL-2.0-or-later were one class. The headline 0.885 is on the current finer labels,
which are a harder task — see §8.
The non-contiguous feature matrix deserves its own note: it made a deterministic measurement look noisy, and an entire threshold sweep was reading that artefact rather than signal.
8. Honest limits
The tail is unserved. 33 licenses in the DEP-5 corpus and 18 in the SPDX-tag corpus, against 774 the engine can name and ~2,470 in ScanCode's catalogue. Weighted by occurrence the picture is kinder — the corpus's licenses cover ≥92% of what actually appears — but the learned component contributes nothing outside the families it trained on, and declines to act there by design.
Catalogue coverage, measured (scancode-toolkit 32.5.0, 2,470 active licenses):
| namespace | n | nameable | with evidence |
|---|---|---|---|
| real SPDX id | 745 | 742 (99.6%) | 454 (60.9%) |
ScanCode-only LicenseRef-scancode-* | 1,725 | 11 (0.6%) | 4 (0.2%) |
70% of LicenseDB is ScanCode's private namespace — patent grants and vendor terms with
no SPDX identity. Covering it is a decision to emit LicenseRef keys, not an index
gap, and it cuts against FOSSology reporting SPDX ids. On the SPDX-nameable portion
the engine is already at 99.6%. Weighted by occurrence over 6,854 real license
occurrences in the-stack: 97.7% nameable, 96.6% with indexed evidence.
Nameable turned out not to mean detectable. Ingesting ScanCode's 1,700
LicenseRef-scancode-* licenses (scripts/build_licenseref.py, off by default) costs
nothing and takes tail nameability from 66/81 to 77/81 — while identifying none of
the eleven licenses it gains. Only 262 of the 1,700 carry any short-form rule, median 1.
Worth having so an auditor sees the right name in the verbatim regime; not a tail fix.
The actionable hole is 288 SPDX licenses that can be named but have no short-form evidence, so they are matchable only against a full body — which scores R@1 ~0.004 on real headers. Of those: 148 have no ScanCode rules at all, 91 only sub-5-token rules, 74 only body text, and 30 were reachable by indexing compound rules. One license has single-license short rules ≥5 tokens and is still unindexed; that is unexplained and worth a look.
The bespoke regime is not fixable by abstention. 49% of files whose license the engine cannot name are still answered with something. Abstention cannot separate them, because a bespoke license genuinely does contain the clauses of the ones it was derived from.
DEP-5 labels carry ~10% noise. Of 651 queries both engines answered, 73 contradict
the label and 63 of those have the two independent engines agreeing with each other.
Causes: DEP-5's looseness about only-vs-or-later, and Files: globs assigning a package
license to files whose own header differs. Fine for comparing engines against identical
labels; not readable as absolute accuracy. It is bias, not variance — a bigger crawl
does not shrink it; that needs cross-provenance consensus or hand adjudication.
Both notice corpora are small. 671 and 266 scored queries — ±0.028 each, so neither can resolve the differences now in dispute. The prevalence pool (n=11,680, ±0.0028) covers that for self-comparison but shares ScanCode's labels, so it cannot settle a head-to-head. Independent labels at n ≥ 1,225 would be needed for ±0.02 there.
The ScanCode head-to-head is stale. See §2b: it predates the last ten changes and needs re-running under nested CV before any claim about the current gap is made.
Widening the corpus did not help generalisation. Going from 18 to 34 training license families made leave-one-family-out worse before the shortcut was found, and neutral after. Tail coverage is not reachable by crawling more data with this architecture.
9. Reproducing
# engine
python scripts/build_references.py # notice index; needs scancode-toolkit
python -m atarashi.build_deps # merge the SPDX license list
# corpora
uv run src/run_eval.py debian-dep5 --packages 6000 # survey
uv run src/run_eval.py debian-dep5 --mode corpus --packages 6000 --tail # build
python datasets/sh-blobs/hf/rehydrate.py --split training # ranker's SH rows
python datasets/sh-blobs/hf/rehydrate.py --split prevalence # the n=11.7k pool
# evaluation
uv run src/run_eval.py agent --corpus debian --max-per-license 150
uv run src/run_eval.py agent --per-lang 0 --max-per-license 150
uv run src/run_eval.py rank-model # grouped CV
uv run src/run_eval.py rank-model --leave-family-out # the OOD test
uv run src/run_eval.py rank-model --save <path> --alpha 0.10
Figures: src/evaluation/plot_report.py → docs/figs/ (PNGs, and base64 inlined here
so the report is self-contained). Every figure reads its numbers from the recorded
result files or restates a measurement in the text — a figure that disagrees with the
table beside it is worse than no figure.
Corpus: rycerzes/atarashi-dep5.
Results: output/{agent_eval,rank_model,debian_dep5}.json.
Not
rycerzes/atarashi-dataset— that is an earlier 28,491-fragment set built for an abandoned classifier approach, ~86% of it verbatim substrings of its own references. Nothing in this engine derives from it.