DATASET TUNING: WHAT CHANGED

This folder compares three search configurations on the eight fictional PDFs.
Use only the PDFs as source material. Keep these evaluation files and the
answer key out of the index. The results measure the DigiRAG hybrid ranking
component locally. They are not a test of a running DigiRAG deployment.

baseline: Eight complete extracted pages. No extra search headings.
current-only: Move the one explicitly retired pump card (M17-A) out of the
current-work dataset. Rank the other seven pages with the original text.
The retired card remains in the kit so readers can index it in an archive.
A separate archive dataset/search mode is appropriate for a historical
question. We did not benchmark historical questions.
current-and-headings: Use the same seven candidates and add four short
catalogue headings above each page's text when indexing it. A catalogue
heading names a subject, e.g. tool storage and return of calibration gauges.
Use meaningful headings for every document, including the near-misses.
The retired and current pump pages have identical headings. The headings
do not state operating intervals, stock levels, locations, or staff roles.

HOW THE EXPERIMENT WAS IMPLEMENTED
subject-headings.csv contains every status and multilingual subject heading.
This script emulates an editorially maintained current dataset by filtering
candidate IDs, and emulates indexed subject headings by prepending the four
plain-text headings to the text used for embeddings and lexical matching.
Labels never enter user prompts or generation context. This does not imply
that setting arbitrary fields in a live product's metadata panel will
automatically give the same embeddings, rankings or scores.

All three runs use the identical DigiRAG tokenizer, term weights and
hybrid_similarity function, normalized quantized E5-small embeddings, a 30%
lexical / 70% vector weight, one full page per chunk and identical tie handling.
The baseline ranks all eight pages. The current-work variants rank seven.
For identical original baseline ranks, keep the original E5 ONNX batch shape.
For all configurations within a split we hold the control query vectors fixed.
Use rankings.csv for every sorted ID, score and gold rank; use summary.csv for
totals per split, configuration and language.

The catalogue headings were written for this small collection after the
original questions were known. A different editor supplied paraphrases.csv
before the tuned runs. The 24 paraphrases reuse the same five answerable
facts and one missing fact, so they test sensitivity to changed wording.
No independent documents, customer queries or second site were tested.
The glossary was not revised after we saw the results.

SCORING
Each split includes 20 answerable prompts. Top-1 counts correct pages
ranked first. Top-3 counts correct pages anywhere in the first three. Four
warranty prompts in each split have no gold page and receive no retrieval
score, even if a relevant-looking page is ranked first.

                        ORIGINAL            PARAPHRASES
Configuration           top-1   top-3        top-1   top-3
baseline                 12/20   19/20        9/20    16/20
current-only             13/20   19/20       11/20    17/20
current-and-headings     18/20   20/20       14/20    18/20

The original English pump question's current page moved from second to
first once the archive was separated. The English gauge page moved from
sixth to fifth to first. In the paraphrases, the French quarantine page
fell from third to fourth and disappeared from the first three. The
Luxembourgish quarantine page moved from fifth to fourth and still missed
the cutoff. Check these rows as well as the aggregate gains.

REPRODUCE
From the extracted kit directory, provide the same source tree, model
files and Python environment documented in START-HERE.txt. Run:
python 04-dataset-tuning/run-tuning.py --kit . --model model --digirag PATH_TO_CHECKOUT --nltk PATH_TO_NLTK_DATA --out reproduced-tuning

The script checks that all original baseline page orders still match the
distributed initial run, and writes a new rankings.csv and summary.csv.
