ASK IN FRENCH, FIND THE GERMAN MANUAL
Digitrans editorial lab | 20 November 2026 | v1

WHAT TO UPLOAD
Upload only the eight one-page PDFs in 01-corpus to a new, empty dataset.
Every company, part and procedure in these sample pages is fictional. These
are test fixtures. Do not use them as maintenance or operating instructions.
The files have a selectable text layer. This exercise does not measure OCR.

Keep 02-test and 03-recorded-run outside the dataset. They contain the answer
key and measured results. Reuse the fictional materials for internal testing.

MODE D'EMPLOI
Importez uniquement les huit PDF de 01-corpus dans un jeu de données vide.
Gardez le corrigé hors de l'index. Posez les questions de questions.csv dans
des conversations distinctes. Notez la source, la réponse et sa justification
avant d'ouvrir answer-key.csv. Q6 ne possède aucune réponse dans les documents.

ANLEITUNG
Nur die acht PDF-Dateien aus 01-corpus in einen neuen, leeren Datensatz laden.
Den Lösungsschlüssel außerhalb des Index aufbewahren. Jede Frage aus
questions.csv in einem neuen Chat stellen. Erst Fundstelle und Antwort
festhalten, dann den Schlüssel öffnen. Zu Q6 gibt es in den Unterlagen keine
Angabe.

KUERZ ULEEDUNG
Luet nëmmen déi aacht PDFen aus 01-corpus an en eidelen Datesaz. Stellt all
Fro an engem neie Chat. Notéiert d'Quell an d'Äntwert, ier Dir de Schlëssel
opmaacht. Op Q6 stécht keng Äntwert an den Dokumenter.

RUN A FAIR COMPARISON
1. Record the date, product version, embedding and answer models, indexing
   settings, ranker and how many passages the model receives. Restrict search
   to the eight pages. Disable web search for this exercise.
2. Run all 24 questions from 02-test/questions.csv in separate fresh chats.
   Ask for an answer in the question's language, a page ID and page number.
   Save the returned text verbatim. Inspect retrieval results separately if
   the product exposes them. A cited document and the first retrieval result
   can differ. If you cannot see candidate rankings, record "unavailable".
3. In blank-scorecard.csv, save the first source and the list of retrieved
   sources, the actual answer and the cited page. Use answer-key.csv only
   after the run. Accept accurate paraphrases and normal language variants.
4. Q1-Q5 each have exactly one source with the complete current answer. Score
   each once per language (20 total). Retrieval top-1: 1 if the correct page
   is first. Top-3: 1 if it appears in positions 1-3. Grade the answer and
   its citation independently. For Q3, both the cabinet and drawer matter.
5. Q6 has no source with the requested fact. Leave retrieval hits blank for
   those four questions. Score whether the answer declines to invent a term.
   It is possible to retrieve a highly relevant-looking pump manual and still
   have no answer. Give no credit for a fabricated warranty duration.
6. Change one configuration at a time. Compare individual rows, not just the
   total. Five answerable cases per language are too few for a language ranking.

DATASET TUNING
Folder 04-dataset-tuning contains a second experiment. We first removed the
retired pump card from the current-work search pool, then indexed multilingual
subject headings for each page. The folder contains the actual headings, the
additional paraphrased questions, all rankings and an explanation of the
cases that got worse as well as better. The answer model always sees the
original source-page text. Keep these assessment files out of your dataset.

MEASURED RETRIEVAL EXPERIMENT
The unit is one full page. Text came from the text layer of each PDF (pypdf).
The eight pages and 24 questions are the complete inputs. No source text was
translated; no synthetic labels, expected answers or questions were added to
the index. Model input includes document titles, revision notices, footers
and table cells in extracted reading order.

Method                 Gold ranked first    Gold in first three
BM25 lexical                 10 / 20            15 / 20
Multilingual E5-small        15 / 20            17 / 20
DigiRAG hybrid component     12 / 20            19 / 20

03-recorded-run/rankings.csv contains all 72 ranked lists including the four
unanswerable cases per method. expected_rank is blank on Q6. Equal scores
are resolved by source ID, ascending. See page-text.json for the exact input.

The first baseline uses Unicode word tokens and BM25 (k1=1.2, b=0.75).
The second uses mean-pooled, normalized vectors with the multilingual E5
query/passage prefixes and cosine similarity. Quantized model from
https://huggingface.co/Xenova/multilingual-e5-small/tree/761b726dd34fb83930e26aab4e9ac3899aa1fa78
Underlying model documentation: https://huggingface.co/intfloat/multilingual-e5-small

The third uses DigiRAG's actual FulltextQueryer.hybrid_similarity function,
tokenizer and term weights from the local development source. It evaluates
all eight candidates in memory with 30% lexical and 70% E5 vector similarity.
An audit hook blocks network access during this component test. No server,
search index, customer data, tenant, production configuration, candidate
prefilter, external reranker, synonym expansion or agent workflow is involved.
These are results for this component and dataset, not an end-to-end benchmark
of a deployed DigiRAG system. Source tree Git HEAD: a024bea0; measured files:
rag/nlp/query.py ca09803584c3333ed9de0006c3fe3f74f5082ec75168ef1dc4c356003b571842
rag/nlp/rag_tokenizer.py eb10c75b28829e2683f218f7ad93f13200a239b2632ceb393f734b93d609dca1
rag/nlp/term_weight.py af272789a08710b9924e03549ce971f391fbab84883257e43e63db82b87aca48

REPEAT THE RETRIEVAL RUN
With numpy, tokenizers and onnxruntime installed, save tokenizer.json and
onnx/model_quantized.onnx from the pinned repository into a folder called
model (use the filename model_quantized.onnx there). From this kit's root:

python 03-recorded-run/run-multilingual-rag-lab.py --kit . --model model --out new-rankings.csv

For the DigiRAG component, use the matching DigiRAG source environment and add
--digirag PATH_TO_CHECKOUT --nltk PATH_TO_NLTK_DATA. The latter directory must
contain the locally installed WordNet corpus. Measured environment: CPU ONNX
Runtime 1.23.2, tokenizers 0.22.2 and NumPy 1.26.4. Scores can differ slightly
with another inference runtime or CPU.

ANSWER GENERATION
We passed the three highest-ranked hybrid pages and the original question to
Qwen3.5-4B (Q8_0) running locally in LM Studio. Each question was a separate
request using the native chat endpoint, temperature 0, reasoning off, and a
maximum of 160 generated tokens. The system prompt requests one sentence in
the query language, a source ID and an explicit admission if the fact is
missing. There were no web tools, previous messages or supplied answer keys.

answers.csv preserves the 24 model replies. reviewed-answers.csv records
the editorial comparison with the answer key. "Correct" requires the full
requested fact or, for the warranty question, a clear statement of absence.
For an answerable question, citations are assessed separately. A model can
sensibly decline to guess and still fail to answer a question the corpus can
answer. Fluency is not scored; small grammatical errors remain in the output.

The initial API trial is retained in initial-openai-compatible-pilot.csv.
Two requests on the OpenAI-compatible local endpoint ended at a 512-token
limit with no answer text. We switched to the model's supported native
reasoning-off control, restarted the run from Q1, and used that same setting
on every question. The initial two trials are outside the published total.
To repeat the answer run against a local LM Studio server with that model:

python 03-recorded-run/answer-multilingual-rag-lab.py --kit . --out new-answers.csv

To repeat the tuned answer run with the same model, settings and source text:

python 03-recorded-run/answer-multilingual-rag-lab.py --kit . --rankings 04-dataset-tuning/rankings.csv --configuration current-and-headings --out new-tuned-answers.csv

Baseline: 17 of 20 answerable questions received the complete fact and a
supporting page citation. Tuned search: 16 of 20. Both runs declined to
invent the warranty duration on all four Q6 prompts. The highest-ranked
page improved more often than the final answer did. All four tuned gauge
answers cited the right page yet omitted the cabinet and drawer.

TABLE DIAGNOSTIC
See 05-table-diagnostic for a separate, focused check on that failure.
On the four gauge prompts we kept the tuned page IDs, page order, model and
prompt unchanged, and labelled all table cells in the supplied pages with
their actual source column headings. Complete locations went from 0/4 to
4/4. This diagnostic was chosen after inspecting the missed answers; it is
not an evaluation on new questions or a third 24-question run.

Run scripts only from a trusted copy. The published SHA256SUMS.txt lets you
verify the supplied files. Test logs have no customer information.
