--- title: "How to Make Your CMS Crawl-Ready for LLMs (2025)" date: "2025-10-16" author: "Flavio Longato" categories: ["GEO"] url: "https://www.longato.ch/adaptto-2025-crawl-read-llm-geo/" --- At adaptTo() 2025 in Berlin, I and [Sinem Dere](https://adapt.to/speakers/sinem-dere#2025) shared our experience helping large websites prepare for the next frontier of discoverability, how content is interpreted and surfaced by Large Language Models (LLMs) and AI-driven search systems. For the past few years, I’ve been exploring how SEO principles evolve when search becomes conversational, and how we can make our content retrievable, trusted, and cited by AI systems. This new discipline is what I call **Generative Engine Optimization (GEO)**, an evolution of SEO for the LLM era. <figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"> </figure>From SEO to GEO: Visibility in the Age of AI -------------------------------------------- Traditional SEO was about ranking high on Google. In the world of LLMs, visibility means something new: being mentioned, cited, or retrieved as a trusted source inside AI-generated answers. People don’t always visit websites anymore; they ask models like ChatGPT, Gemini, or Perplexity. These systems synthesize content from what they can **crawl, retrieve, and trust**. If your content isn’t accessible or comprehensible to these systems, you effectively don’t exist in AI search. That’s what GEO focuses on, ensuring content is structured and written in a way that LLMs can easily understand and reference. The fundamentals of SEO haven’t disappeared. Good content, fast load times, and logical structure still matter. What’s changed is the end-destination of that content: from the SERP to the model’s answer. How I Got Into GEO ------------------ My journey into GEO started with a simple observation: when optimizing for traditional SEO, fixing site speed, improving markup, cleaning up structure, some websites started getting unexpected traffic from AI assistants. Users were discovering brands through AI conversations, not just search results. That insight sparked a process of testing, measuring, and validating how LLMs retrieve and use web data. I began analyzing crawl logs, comparing LLM bot activity with traditional crawlers, and building hypotheses around what makes a page “LLM-visible.” Over time, patterns emerged: - Sites that were well-structured, semantic, and accessible appeared more often in AI answers. - Pages relying heavily on JavaScript rendering were often invisible. - Google ranking and LLM citations were correlated, what ranks well tends to get retrieved. That realization changed how I approach SEO entirely. The Role of Retrieval and RAG ----------------------------- One of the key technical enablers behind this new reality is **Retrieval-Augmented Generation (RAG)**, the mechanism LLMs use to search the web and include live data in their answers. When someone asks “What’s the best PDF editor?” or “How do I fill out a contract template?”, LLMs go out, find relevant URLs, retrieve snippets, and integrate them into their responses, often citing the sources. If your website isn’t **retrievable**, it isn’t **citable**. RAG has made “ranking” less about links and more about being reachable and comprehensible by AI. To succeed in GEO, your content must be: 1. **Crawlable** – accessible to AI crawlers and free of JS-only content. 2. **Retrievable** – indexed properly, with clear signals of topic and authority. 3. **Usable** – written in a way that models can extract context and facts without ambiguity. This combination of accessibility and clarity is what I refer to as being **Crawl-Ready**. Making Your Site Crawl-Ready for AI Systems ------------------------------------------- Here are some best practices I’ve learned through experiments and log analysis: - **Open the Door to AI Crawlers:** Allow known LLM user agents (like GPTBot or PerplexityBot) in your `robots.txt` and verify via server logs that they’re actually visiting. - **Avoid JS-Only Rendering:** Use static rendering or hybrid SSR for key content areas. - **Structure Content Semantically:** Use logical headings, clear sections, and short paragraphs. Add FAQ or summary sections where relevant, these often get cited directly. - **Ensure Internal Linking and Pagination:** Avoid infinite scroll or AJAX-loaded content; make sure crawlers can access all pages. - **Monitor Crawl Logs:** Watching how bots interact with your site reveals what’s being retrieved and what’s missed. These are the same technical principles that made sites perform well in SEO, just with a new purpose: helping LLMs read and reuse your content accurately. Lessons from Testing and Real-World Data ---------------------------------------- The most important lesson I’ve learned in GEO is this: **don’t trust assumptions, test everything.** Over the past year, I’ve seen dozens of claims about “new SEO for AI” tactics, like adding `llms.txt` or changing meta tags. When I tested them across large domains, most showed no measurable effect. In one 30-day analysis of enterprise-level sites, major LLM crawlers didn’t even access `llms.txt` once. It’s a good reminder that evidence beats hype, the same principle SEO professionals have always followed. Why SEO Still Matters (Even More Now) ------------------------------------- There’s a misconception that SEO is dying because of AI. In reality, it’s becoming the foundation for AI visibility. My own data shows that if a website ranks within the top two pages of Google, it’s significantly more likely to be cited by LLMs. Models rely on high-authority, well-ranked pages as their reference layer. So instead of “SEO vs AI,” think “SEO for AI.” Your existing best practices, clean structure, good content, trusted links, are what make your brand retrievable, citeable, and trustworthy to AI systems. The Future of GEO ----------------- Generative Engine Optimization isn’t a replacement for SEO, it’s its logical next step. It blends technical SEO, content strategy, and data analysis with a new goal: to make content usable by machines, not just readable by humans. Over the next few years, I expect: - Crawl-to-Answer pipelines will become measurable KPIs. - LLM visibility will be tracked just like organic rankings. - “Crawl-Ready” will become as common a phrase as “Mobile-Friendly.” We’re entering a time when search isn’t just typed, it’s generated. And that means every brand must adapt their content to be understood by both humans and models. Closing Thoughts ---------------- The talk at adaptTo() was one of my favorite moments this year, not because of the slides or demo, but because it showed how far we’ve come as an industry. SEO has always evolved with technology. This is just the next stage, where visibility depends not only on what you publish, but how machines interpret it. > “Where do you hide a dead body? > On the second page of Google.” These days, maybe also in a dataset no LLM retrieves. References ---------- - [The Age of Generative AI – Adobe Blog](https://blog.adobe.com/en/publish/2024/04/22/age-generative-ai-over-half-americans-have-used-generative-ai-most-believe-will-help-them-be-more-creative) - [About GPTBot – OpenAI](https://openai.com/gptbot) - [Crawling and Data Policy – Perplexity.ai](https://www.perplexity.ai/hub/data) - [Experience League: Technical SEO Best Practices](https://experienceleague.adobe.com)