Blog

Multilingual document search

Can AI Find the Right Company Document in Four Languages?

Ask in English and the first manual says to inspect the filter every 400 hours. Ask in French and the first manual says 250. Same pump, same documents. Which number goes on the work order?

Digitrans · · 8 min read

An industrial pump beside illustrated workshop manuals.

AI-generated editorial illustration.

In this article

The first English result looked convincing. It named the right pump, described the right inspection and gave an exact interval: 400 operating hours. But the German manual said 250, and one sentence in it explained why: it replaced the English instruction. Following the older document would schedule the inspection 150 hours late.

That kind of mistake is easy to imagine in a Luxembourg business. Staff ask questions in different languages, while manuals, contracts and procedures arrive from suppliers in still others. The challenge for an AI assistant is to find the relevant information, work out which version applies and use it correctly.

This approach is called retrieval-augmented generation, or RAG. Instead of relying only on what a language model learned during training, the assistant searches an organisation’s documents and uses selected passages to answer. Multilingual RAG needs to make that connection even when the question and the source are written in different languages.

We put those skills to the test with an invented workshop, eight documents and six questions in English, French, German and Luxembourgish. The search component improved when we organised the material differently. The answers also exposed a separate problem with tables. You can follow the experiment and download the documents, questions and results to try it yourself.

One pump, two manuals and a decision to make

Our workshop’s pump P-17 has an old English maintenance sheet and a newer German one. Below is the German instruction. The sentence above the table says it replaces the English sheet. The first row gives the inspection interval for the intake filter. The next row concerns the pump housing and describes a different check.

German maintenance sheet showing a visual inspection every 250 operating hours.

The current German sheet from the test archive. It explicitly replaces the older English sheet.

View full size

In the initial search, the English question brought the old sheet to the top for all three methods we tried. The current German sheet came second. For the French, German and Luxembourgish versions of the question, two methods, including DigiRAG’s scoring function, put the current sheet first. The same archive produced a different order depending on the language of the question.

The assistant still had a chance to get the answer right because it received the first three pages, not just the first one. In our separate answer test, a local Qwen3.5-4B model noticed that the German instruction was current and answered 250 hours in all four languages. Finding the right page made that possible; the model still had to read and interpret it.

What we tested

The workshop archive contains maintenance sheets, delivery instructions, a tool card, a quality procedure and a stock card. We asked about the pump, deliveries after 16:00, where to return a gauge, who can release a quarantined shipment and the minimum stock of a seal. A sixth question asked about the pump warranty, which the documents do not specify. A useful assistant should say when the answer is missing.

Each question was asked in four languages, making 24 searches. For the five questions with a known answer, we checked whether the right page appeared first or among the first three. That gives 20 searches to score. The four warranty questions were assessed separately.

We compared three ways of ordering the pages. The first, BM25, looks for matching words. The second, multilingual E5-small, compares the meaning of the question and each page. The third combines word matching and meaning using DigiRAG’s hybrid scoring function. We ran that function offline with 30% weight on words and 70% on meaning.

Finding the expected source in the initial test
Search methodRight page firstRight page in top three
Word matching (BM25)10 / 2015 / 20
Meaning-based search (E5-small)15 / 2017 / 20
Combined DigiRAG scoring12 / 2019 / 20

These numbers answer different questions. E5 put the correct page first more often in this small collection. The combined method more often put it somewhere in the three pages available to the assistant. Neither result proves that the answer itself will be correct.

The gauge that went back to “its usual place”

Another question asked where to return an undamaged calibration gauge. The answer was in a Luxembourgish tool card: cabinet B2, drawer 4. When asked in English, the combined search ranked that card sixth, outside the three pages given to the model. The model said the supplied pages did not specify the location.

In Luxembourgish, the right card came first. The model cited it but merely said to put the gauge back in its usual place. A new employee would still be looking for the cabinet. The French answer did give B2 and drawer 4. The first problem was finding the card; the second was reading the useful detail from it.

First, we removed the retired pump sheet from searches for current work and kept it in the archive. It still has a purpose when someone needs to understand an old maintenance record. Then we added short descriptions of each document in all four languages to the text being searched. For the tool card, those descriptions covered tool storage and the return of calibration gauges. They did not contain the cabinet number or answers to our questions.

How the changes affected the first result
Material searchedOriginal questionsRephrased questions
All eight sheets12 / 209 / 20
Seven current sheets13 / 2011 / 20
Current sheets with multilingual descriptions18 / 2014 / 20

The English gauge question shows what changed. With all eight pages, the correct card was sixth. Removing the retired manual moved it to fifth. Adding the multilingual descriptions brought it to first. Across the original 20 answerable searches, the right page appeared in the first three every time after both changes.

We then tried fresh wording for the same questions. The right page appeared in the first three 18 times out of 20, up from 16. One French question about a quarantined pallet actually got worse: the right instruction fell from third to fourth. That is a reason to test changes with several ways of asking, including the words staff really use.

Finding a page is not the same as reading its table

The better search result did not automatically produce a better answer. Before tuning, the model gave complete, source-supported answers to 17 of the 20 answerable questions. Afterwards it gave 16. Both times it recognised that the warranty information was missing in all four languages. After tuning, it cited the right gauge card in all four languages but left out the complete location each time.

We took a closer look at how the table reached the model. Its text had been extracted as a sequence of cells. We changed the presentation so that each value stayed beside its original column heading. The gauge row now paired the item with “Schaf B2, Tirang 4”. We applied the same treatment to the other rows, including the spare gauge in drawer 1.

We repeated only the four gauge questions, with the same model, instructions and three retrieved pages for each. All four answers now gave cabinet B2, drawer 4 and cited the card. That is a small diagnostic, not a score for the whole test. It shows why a document can be found correctly and still need better table extraction before its contents are useful.

How to try this with your own documents

  1. Choose a small collection and agree which documents are current. Keep older versions available for historical questions, with their status clearly marked.
  2. Write questions in the languages your colleagues actually use. Include a question whose answer is missing and one that could be confused with an older document.
  3. Before testing, have someone who knows the material record the expected answer and source. Keep that answer key out of the searchable collection.
  4. Check the retrieved pages as well as the final answers. Open each cited source and see whether the right row, date and version support the answer.
  5. If a search misses a document, improve its description or review the search settings. If the right page is found but the answer is wrong, inspect how its text and tables were extracted. Repeat the same questions after each change.

In DigiRAG, an administrator can organise documents into datasets, manage document status and use verified metadata filters to set the search scope. A multilingual description helps only when it is included in what the search actually uses. The dataset guide and search guide explain the controls.

Why this matters in Luxembourg

French was used at work by 69.2% of resident workers in Luxembourg’s 2021 census. A French question, however, may depend on a German supplier manual or a Luxembourgish instruction. STATEC’s employment study gives the local context.

Researchers studying Luxembourgish PDF pages in the LëtzCross paper also found that retrieval across languages and document layouts deserves careful testing. Their comparison of page-image and text-only methods is separate from our exercise. Both point to a practical question for any vendor demonstration: can the assistant show you the right passage, in the right version, when you ask as your own team would?

Our experiment used DigiRAG’s ranking component in isolation, CPU-based multilingual E5 embeddings and a separate local Qwen3.5-4B model for answers. It was not an end-to-end DigiRAG deployment or a test of a dedicated reranking model. The archive is deliberately small, so use the results to guide a pilot rather than predict performance across every collection. Explore DigiRAG.

Related product

DigiRAG

Build agents in plain language, search your knowledge and handle mail, tasks and document outputs in one dedicated platform.

Explore AI consulting and implementation ↗Open product page