How AI search chooses which sources to cite

The five-step pipeline

Each AI product works differently, and vendors don't publish every detail. This is a simplified, general model that helps you reason about where a page can fail.

1. Crawl: a bot fetches the page. If it is blocked or the page needs JavaScript the bot doesn't run, the process stops here.
2. Index: the page is stored and refreshed. Stale or rarely recrawled pages are less likely to appear.
3. Retrieve: for a given question, the system selects candidate pages. Relevance and the quality of the source both matter.
4. Quote: a passage is chosen to support the answer. A clear, self-contained passage is easier to use than a buried one.
5. Cite: many products show the source as a link, though not every answer includes citations.

Step 1: Can the bot read your page?

Several companies run separate crawlers for training, for search and for fetching pages on a user's behalf. Names such as GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended appear in robots.txt rules. Check each vendor's current documentation, because names and behaviour change.

• Open your robots.txt and look for blanket Disallow rules that catch these crawlers.
• Decide per crawler whether you want access. Blocking removes the chance of being cited by that product.
• Make sure your content is in the HTML, not only added by JavaScript after load.

The robots.txt tester checks any URL against named crawlers using Google's matching rules.

Step 3: Why one page is picked over another

Retrieval favours pages that match the question closely and come from sources the system trusts. In practice that tends to reward:

• Pages whose headings match the wording of real questions.
• Established, topically focused sites.
• Pages with specific, verifiable details rather than generic summaries.
• Pages that rank well in conventional search, since many systems draw on search results.

That last point is why GEO and SEO are tightly linked: ranking in normal search is often a route into AI answers.

Step 4: Make a passage worth quoting

• Put the direct answer in the first one or two sentences under a question heading.
• Keep each passage self-contained: define the term, don't rely on 'as above'.
• Use specific numbers, dates and named examples, and keep them accurate.
• Use lists and short tables for steps and comparisons.

If you are not being cited: a debugging order

1. Is the page indexable and in Google? (meta tags, canonical, sitemap)
2. Is the crawler allowed in robots.txt?
3. Does the page answer the question directly near the top?
4. Does it say something specific that other pages don't?
5. Do you rank in normal search for the question? If not, fix that first.

ElixRank's AI-search audit walks through the first three and labels its scores as heuristics, not guarantees.

Frequently asked questions