Even a high-quality website can struggle with visibility if search engines and AI systems cannot properly find, process, or understand it. And it is no longer just about Google. We now search for information through ChatGPT, Gemini, Claude, and Perplexity, with each platform using different crawlers, search indexes, or mechanisms for retrieving information.

In this article, we will therefore look at what should be technically configured correctly on your website, what is supported by solid evidence today, and what remains more experimental for now. If you first need to understand the broader context of AI search, we recommend reading GEO / AEO / AIO: What Are They and How Can You Prepare for the Era of AI Search?.

Table of Contents

Technical SEO for AI is not a new discipline that replaces traditional SEO. Most of the fundamentals remain the same:

  1. a crawler must be able to access the website,
  2. it must be able to discover important pages,
  3. the content must be technically accessible, and
  4. the URLs must be indexable.

At the same time, the website needs a clear architecture and unambiguous signals about what each page contains.

Google states this quite explicitly for AI Overviews and AI Mode: no additional technical requirements are needed for a page to be eligible for inclusion in AI search features. The page must be indexed, meet Google’s technical requirements, and be eligible to appear in enhanced search results (Rich Snippets). Even meeting these conditions, however, does not guarantee indexing or visibility.

Although AI search extends far beyond Google, alternatives such as ChatGPT, Claude, and Perplexity provide almost no documentation on search optimization. As they are relatively inexperienced newcomers to the world of search tools, it can be assumed that they draw significant inspiration from Google, but at that point we would already be entering the realm of speculation.

What matters for AI chatbots is that they can access a website and understand it without difficulty, which we will examine in the following sections.

The technical layer of optimization also includes structured data, also known as Schema Markup. It provides machine-readable information about a page’s content, and its importance is growing alongside the development of AI search. We will also explore this topic in more detail later in the article.

In short: An AI system, or the search engine it relies on, must first be able to find, load, and correctly process your content. Only then can it decide whether to use, display, or cite it.

1. Allow AI crawlers to access your website

The first step is simple: if a crawler cannot access your website, it has nothing to index or process. Google and Bing are no longer the only bots that may need access to your site. This issue will most likely not affect you, since most websites have very permissive crawling settings, but it is still worth checking to make sure everything is configured correctly.

Check your robots.txt

Relevant crawlers include, for example, Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Claude-User. Each serves a slightly different purpose.

OpenAI states that if you want your content to be eligible to appear in ChatGPT Search summaries and snippets, you should not block OAI-SearchBot. Similarly, Perplexity respects robots.txt rules for PerplexityBot.

Anthropic goes even further and distinguishes between several bots based on their purpose:

  • ClaudeBot is related to data collection for model development,
  • Claude-User is used to retrieve content in response to a user request, and
  • Claude-SearchBot is used for search.

Blocking the latter two may therefore reduce the chances that Claude will retrieve or use your content when searching.

You do not need to blindly allow every bot on the internet, even though that is effectively how the vast majority of websites are configured. The important thing to understand is that just because Google can crawl your website does not mean that all relevant AI systems can crawl it as well.

Robots.txt is not the only possible source of blocking

Even a perfectly configured robots.txt file is useless if a crawler is blocked by a CDN, firewall, WAF, anti-bot protection, CAPTCHA, or a server-level rule. Google also recognizes this issue, listing server and network problems alongside robots.txt rules as common causes of crawling issues. If your firewall has been blocking Googlebot for years, there is no reason to assume AI bots will somehow be exempt.

2. Make sure important content is internally linked

Allowing a crawler is only the first step. Your website must also be structured so that a crawler can easily discover your content and correctly understand which URLs it should process.

Important pages must be easy to discover. Key categories, products, services, and articles should not exist as isolated “orphan pages” that virtually nothing else on the website links to. This is where the following help:

  • clear navigation,
  • internal linking, and
  • a logical URL structure.

If you have some experience with traditional SEO, you may be wondering why sitemaps are missing from this list. That omission is intentional, because there is no evidence that AI systems routinely discover new pages in the same way Google does. AI systems search the web in response to queries; they are not constantly trying to discover new pages, because they have no reason to do so — AI works differently.

3. Important content must be technically easy to read

Great, AI has reached your website — but what does it actually see? On modern websites, some content may only be generated through JavaScript, which adds another layer of processing. AI systems, however, may need to process dozens or even hundreds of URLs for a single prompt, so they cannot afford to wait for every website to fully render.

Be careful with JavaScript and rendering

For JavaScript-based websites, Google works in three basic stages:

  1. crawling,
  2. rendering,
  3. indexing.

During rendering, Google processes the page in a way similar to a browser and can then work with content generated by JavaScript. AI bots, however, do not necessarily work this way.

JavaScript introduces an additional processing layer. AI systems may need to process dozens or hundreds of URLs for a single prompt, and they do not have time to wait for your website to fully render.

This does not mean your website should not use JavaScript. The problem arises when important content — such as a product name, service description, or key information — only becomes available after a series of scripts, user interactions, or requests that a crawler may never execute. You should therefore verify that all important HTML content is present before JavaScript loads.

You can check your website’s source code in a browser by right-clicking the page and selecting “View page source.” If you are not particularly technical, you can also use any freely available online tool.

Content should have a clear HTML structure

Important content should be available as text and organized logically. Headings, paragraphs, lists, and tables help separate individual parts of a page and make the content easier to understand not only for visitors, but also for machines.

In its recommendations for AI features, Google explicitly states that important content should be available in textual form.

There is, however, no rule such as “use H2 headings and ChatGPT will start citing you.” The goal is to have a technically clean website where the main content can be easily found, loaded, and understood.

  • Heading levels (H1, H2, H3, etc.) clearly define the hierarchy of the content.
  • Sections (section, aside, etc.) group content into logical units.
  • Semantic HTML elements (article, blockquote, etc.) give content additional meaning.
  • Tables explain relationships between pieces of information.

This has always been good practice. The difference is that AI systems do not examine content as deeply as Google does, because they process enormous volumes of information in a very short time. If you want AI to understand your content correctly, help it with a clear HTML structure.

4. Help AI systems understand what a page represents

Technical optimization is not only about whether a crawler can access a page. Your website should also send as few conflicting signals as possible about who you are, what you offer, and what a particular URL represents.

Use consistent names for your company, products, and services. Important pages should have clear URLs, descriptive headings, and at least basic metadata that accurately reflects their actual content. A logical information architecture and internal linking between thematically related pages are equally important.

In practice, this means avoiding situations where the same product has different names in different parts of the website, a service is referred to in three different ways, or contact and company information contradict each other. The more consistent your website is, the less room there is for ambiguous interpretation.

Structured data

Structured data, also known as Schema Markup, provides explicit machine-readable information about entities and the relationships between them — for example, a product, organization, person, or their attributes. Schema.org covers a much broader vocabulary of types and properties than Google uses for specific rich results.

Its importance becomes even greater in AI search. Structured data can help describe more precisely what a page, product, organization, or person represents, what attributes it has, and how it relates to other entities. Structured data can provide information about, for example:

  • ways to contact a company, including different contacts for sales and support,
  • a branch’s opening hours,
  • product return conditions and warranty periods,
  • sources used in an article.

In a separate article, we explain how to optimize structured data for both traditional SEO and AI search.

5. llms.txt, agents.json, and the experimental layer of the AI web

Once the technical SEO fundamentals are in place, you can also look at newer formats designed specifically for LLMs and AI agents. Here, however, it is important to distinguish between standards that are actually being used and proposals that are still trying to gain adoption.

llms.txt

llms.txt is a proposal introduced by Jeremy Howard in 2024. It is a Markdown file designed to provide an AI agent with concise context about a website and links to its most important resources in an easy-to-process format. Even in its current version, the project describes it as a proposal, not a universal web standard.

That does not mean it is purely theoretical. Its documentation cites thousands of implementations and support from multiple documentation platforms.

However, this does not mean better rankings or more citations in AI search. Google Search explicitly states that it ignores llms.txt and that implementing it neither helps nor harms visibility or rankings in Google Search.

agents.json

An even more experimental area is agents.json, which is aimed more at the agentic web than traditional SEO. Today, several parallel community proposals exist under this name, with different specifications. Some describe a website’s capabilities, available APIs, authentication methods, or specific actions an agent can perform. It is therefore not a universally accepted standard.

In practical terms, you can think of them like this:

  • llms.txt — may one day help AI systems find and understand relevant content,
  • agents.json — may one day describe what an AI agent can do with a website or service.

For e-commerce websites, this direction could become interesting, for example for product discovery, reservations, shopping-cart interactions, or other tasks carried out by agents. Today, however, it remains a proposed technology in development, and there is no evidence that AI agents or search engines broadly use it in practice.

What about llm.txt, ai.json, and other names?

Other terms have also appeared around AI visibility, including llm.txt, ai.txt, ai.json, agents.txt, and various so-called discovery manifests that aim to replace sitemap.xml. Similar names, however, do not mean the same format or official support from ChatGPT, Gemini, Claude, or Perplexity. The field is still evolving, with several competing proposals existing side by side, and none currently enjoys universal official support.

Tip: Only start experimenting with llms.txt or agents.json once you have crawlability, indexing, rendering, canonicalization, and internal linking under control. Google also continues to recommend prioritizing the website’s core technical structure for generative search.

What does a website with good technical AI SEO look like?

A well-prepared website does not need dozens of speculative implementations for AI agents. What matters is that its technical foundations do not prevent crawlers and search systems from accessing relevant content and processing it correctly.

  • Relevant content is accessible to crawlers — important pages are not unnecessarily blocked.
  • robots.txt is configured intentionally — it allows access to the bots you actually want on the website.
  • The CDN, WAF, and anti-bot protection do not block legitimate crawlers — robots.txt rules are irrelevant if the infrastructure blocks the request first.
  • Canonical and alternate signals are unambiguous — crawlers can identify which page version you consider primary and where to find versions in other languages.
  • The website does not generate unnecessary duplicates — especially through filters, parameters, and URL variants.
  • Internal linking works properly — important pages are not cut off from the rest of the website.
  • The main content is technically accessible even without JavaScript — crawlers do not need to “dig it out” through complex rendering.
  • The information architecture is logical — categories, products, services, and content are naturally connected.
  • Metadata is consistent with the page content — titles and other basic signals do not contradict one another.
  • Structured data is relevant and correctly implemented — a page that is clearly understood and supported by broader website context has a better chance of being cited.
  • Crawlability, indexing, and AI visibility are monitored continuously — maintaining the technical health of a website is not a one-time task.

A website built this way has a strong technical foundation not only for traditional SEO, but also for AEO, AIO, and the broader landscape of AI search.

What results can you expect from technical AI SEO?

Technical SEO for AI cannot guarantee that ChatGPT will cite you, Gemini will use your content in an answer, or Claude will include your website among its sources. What it can do is create the conditions without which those outcomes may be far less likely.

A properly configured technical AI SEO layer improves website accessibility for relevant crawlers, the discoverability of important URLs, indexability, and the ability of bots to retrieve and process content. In doing so, it strengthens the foundations for both traditional organic visibility and visibility in AI search.

It is still only a prerequisite, however. OpenAI explicitly states that allowing OAI-SearchBot makes a website eligible for inclusion in ChatGPT Search, but does not guarantee placement. Google similarly warns that meeting its technical requirements does not guarantee crawling, indexing, or visibility.

Once technical issues are resolved, the main factors are whether the content is relevant, high-quality, and suitable for the specific query.

What should you measure?

Focus primarily on crawling and indexing, the performance of content-optimized pages, organic traffic, and referral traffic from AI platforms. If you use tools that monitor AI-generated answers, you can also track mentions and citations, but only where they can be measured with sufficient consistency.

For ChatGPT, at least some traffic is relatively easy to identify. Links from ChatGPT Search automatically include the parameter utm_source=chatgpt.com. Since June 2026, Google Search Console has also provided citation data, although it does not include website click-through data.

There is currently no single universal metric for “AI SEO performance.” The most practical approach is therefore to combine Search Console data, traffic carrying the ChatGPT source parameter, and occasional manual or automated visibility tests across AI chatbots.

Conclusion

Technical SEO for AI does not need to be based on speculation. The foundation remains a website that relevant systems can find, retrieve, process, and interpret correctly. Traditional technical SEO therefore remains the priority, but the landscape now also includes AI crawlers, new ways of interacting with websites, machine-readable data, and an experimental agentic layer.

Start by checking crawler access, indexing, rendering, and the technical accessibility of your most important content.

Not sure whether your website is ready for AI search? Contact us for a free technical AI SEO audit covering both traditional technical foundations and your website’s readiness for ChatGPT, Gemini, Claude, Perplexity, and other AI search engines.

Get a free AI SEO audit

Frequently asked questions about technical AI SEO

Not entirely. The fundamentals remain the same: crawlability, indexing, correct canonical signals, accessible content, internal linking, and a logical website architecture. AI search, however, adds additional layers, such as dedicated AI crawlers and new retrieval mechanisms.

Yes. robots.txt is only one layer. A crawler can still be blocked by a WAF, firewall, anti-bot protection, CAPTCHA, hosting infrastructure, or server-level rules. That is why you also need to verify whether the crawler can actually access the website in practice.

No. Technical optimization improves the chances that a system can find, retrieve, and process your content. It does not guarantee a citation, a specific position, or that the page will be used in an answer.

No. llms.txt is an experimental community proposal and is not officially supported by any AI platform.

It may still be worth implementing as an experiment once the website’s technical and content fundamentals are in place. However, it should not take priority over standard technical or content optimization.

Yes. Schema.org offers a much broader vocabulary of types and properties than we are used to seeing in the context of Google rich results.

AI systems can interpret this information more comprehensively, so structured data may help explain more precisely what a page, product, organization, or person represents and how it relates to other entities. Structured data alone, however, does not automatically guarantee greater AI visibility or more citations. High-quality content and domain authority remain more important.

Not by itself. The main issue is so-called client-side rendering (CSR), where website content is loaded later through JavaScript.

AI bots often do not wait for JavaScript to execute, which means they may see little more than an empty page. CSR was already less than ideal for traditional SEO, and its disadvantages can become even more pronounced in AI SEO.

Learn More