Large Language Models are extremely good at sounding right. That’s also their biggest problem.
I work daily with teams using LLMs in production environments where accuracy matters: SEO, GEO, analytics, product documentation, and decision support. One pattern shows up over and over again: answers that look correct, feel authoritative, and still contain information that simply isn’t true.
Most commonly, this shows up as invented URLs.
This is not anecdotal. It is a direct consequence of how LLMs work.
I took inspiration from Martina Raissle’s Linkedin post showing a clear example of this.
Why LLMs invent URLs so convincingly
When an LLM prints a URL directly in the response, it is usually not retrieving it. The model is predicting what a plausible URL should look like based on patterns learned during training.
It is not checking:
- whether the page exists
- whether the domain structure is correct
- whether the content behind the URL is real
URLs are especially vulnerable because they follow predictable patterns. To a language model,/blog/2024/llm-grounding.html looks just as valid as a real page.
This behavior is a well-documented form of hallucination in LLMs. The model is doing exactly what it was trained to do: generate likely text, not verify facts.
Generated answers vs grounded answers
In modern LLM systems, there is a fundamental distinction that often gets blurred in practice.
Model-only generation
- Fluent and confident
- Can invent facts, numbers, and URLs
- No external verification step
RAG-backed generation (Retrieval-Augmented Generation)
- Retrieval happens first
- The model is constrained to retrieved documents
- Claims and URLs come from real sources
When you see answers accompanied by a citation block or source panel, that is typically a signal that retrieval has occurred. The citation is not decorative. It is metadata from the retrieval step.
When you don’t see this, you are usually looking at pure model output.
Why “grounding-style” prompts are not enough
This is where many teams get caught off guard.
Prompts such as:
- “Only answer using factual sources”
- “Make sure the answer is grounded”
- “Cite your sources”
sound reasonable, but they do not guarantee that retrieval is enforced.
The reason is simple:
- Prompting is an instruction
- Grounding is an architectural constraint
Unless the system explicitly:
- runs retrieval before generation, and
- blocks generation when no documents are returned,
the model can — and will — fall back to its internal knowledge and fill in the gaps.
In short:
Prompting ≠ enforcement
Instruction ≠ system guarantee
Having said that, what I’ve found that often works is the below prompt:
You must answer using retrieved sources only.
Rules:
- Before generating an answer, retrieve relevant documents.
- If no relevant sources are found, respond with:
"I don’t have sufficient sourced information to answer this."
- Do not use prior knowledge or training data.
- Do not infer, guess, or complete missing information.
- Do not generate URLs unless they appear verbatim in the retrieved sources.
- All factual claims must be supported by a cited source.
Output format:
- Answer
- Sources (with exact URLs or document identifiers)
User question:
{question}The prompt works because it:
- allows refusal
- removes pressure to “be helpful at all costs”
- replaces preferences with constraints
- aligns model behavior with RAG logic
Why this matters in real-world use cases
In many professional workflows, grounding-style prompts are treated as a trust boundary.
The expectation is:
- no sources → no answer
What often happens instead:
- no sources → confident guess
That is not a user mistake. It is a system design issue.
In environments where accuracy matters — SEO, legal, medical, finance, analytics, or executive reporting — a system that fails open is more dangerous than one that refuses to answer.
How reliable RAG systems handle this
Well-designed RAG systems tend to share the same principles:
- Retrieval is mandatory, not optional
- Answers are clearly labeled as retrieved vs model-derived
- Generation is blocked if no documents are found
- URLs are only surfaced if they come from retrieval metadata
This separation between retrieval and generation is what turns LLMs from writing assistants into reliable knowledge systems.
A practical heuristic that works today
Until grounding is enforced everywhere, one practical rule helps reduce risk:
- Raw URLs written inline → treat with skepticism
- URLs shown in citation / source blocks → far more reliable
This is not theoretical. In practice, it is one of the strongest signals available today for identifying hallucinated references.
Based on hands-on use in production systems:
- This behavior is expected given how LLMs work today
- Prompts alone cannot guarantee grounding
- Only enforced retrieval plus constrained generation can
If correctness matters in your workflow, grounding is not optional. It is foundational.
Sources & further reading
- OpenAI – Retrieval Augmented Generation
https://platform.openai.com/docs/guides/retrieval - Lewis et al. (2020) – Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
https://arxiv.org/abs/2005.11401 - Google Cloud – Grounding LLM responses with retrieval
https://cloud.google.com/architecture/grounding-llm-responses-with-retrieval - AWS – Design patterns for Retrieval Augmented Generation
https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-patterns/
Leave a Reply