What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation is when a system first retrieves pages or documents related to the question, then a language model writes its answer from them. Retrieval brings the information; generation writes the answer. RAG is its short form.

The name comes from a research paper by Lewis and colleagues, first published in May 2020 and accepted at NeurIPS 2020. It noted that large language models store a great deal of knowledge in their parameters but cannot always access it precisely, cannot say where it came from, and cannot easily update it. The paper paired a generative model with a searchable index of Wikipedia text, and found in its experiments that the answers became more specific and more factual than the model's alone.

It is one of the entries in the Tasadr glossary. The closest terms are the large language model, which writes the answer, and grounding, which ties the answer to what the engine finds at the time of the question.

It turns the question into search phrases, reads what they bring back, then writes. The engines' owners do not publish every detail, but what they do say is clear enough. This is what OpenAI and Google say about their engines:

• ChatGPT: OpenAI says that when ChatGPT search works with other search providers, it typically rewrites your question into one or more targeted queries that it sends them, may send more specific ones after reviewing the first results, and that answers which search the web may include citations of their sources.
• Google AI Overviews and AI Mode: Google says both may use query fan-out, issuing multiple related searches across subtopics and data sources to develop a response, and that to be shown as a supporting link a page must be indexed and eligible to be shown with a snippet.

What a model learnt is fixed at its training date and carries no link to its source. What it retrieves are pages it reads at the time of the question, and it may cite them. If your customer asks about this season's price or a branch you opened last month, the right answer can only come from a page the engine finds today.

Some engines use a different crawler for each side. OpenAI says OAI-SearchBot reads sites for ChatGPT search, that GPTBot collects content that may be used to train its models, and that each setting is independent of the other, so you can allow the first and block the second. The detail is in which crawler to allow, and which to block.

An illustrative example, not a real measurement: a customer in Riyadh asks "which mandi restaurant in Riyadh has family seating and stays open after midnight?". The engine turns the question into narrower search phrases, such as mandi restaurants in Riyadh with family seating, and opening hours of mandi restaurants in Riyadh, then reads the pages those phrases bring back.

It finds a list on a review site, an article about the best mandi restaurants, and the Olaya branch page on rukab-restaurants.com, the fictional site of Rukab Restaurants («مطاعم رُكاب»), with one sentence answering two conditions: "Family seating, open daily until 1 a.m." That page is the closest retrieved match to the sub-question about hours, and the engine may cite it next to the list.

Had Rukab posted its hours only as an image on its social media account, the search for opening hours would have found nothing on its site to retrieve, and the engine might have taken the hours from the list, current or out of date.

That the work starts before the writing: a page that is not retrieved is not read. Four things help the engine find and read your page:

• Be retrievable: indexed in Google and Bing, not blocked to search crawlers by robots.txt or your firewall, with its text in HTML and no login required, as in can AI engines read your site, and how do you check?
• Answer the sub-question: the engine may search with phrases narrower than your customer's question, so write a sentence that answers each condition, such as price, hours and district, as in how to write a paragraph AI engines will quote.
• State facts in text, with a date: what is inside an image may not be read as text, and readers need a date to know the information is current.
• Do not chop your page up for it: Google says there is no requirement to break content into tiny pieces for AI to understand it, and that its systems understand a page covering several topics and show the relevant piece.

It does not say how each engine ranks what it retrieved, or why it picks one page over another: the engines' owners do not publish that. Nor does it mean the engine searches before every answer: OpenAI says ChatGPT may search the web automatically when your question would benefit from current information, or when the user asks it to.

The answer itself can change from one ask to the next, partly because what is retrieved can change, as covered in why ChatGPT's answer changes, and when a change is real.