Why technical SEO matters for answer engines

Answer engine optimization sounds like a writing problem. It is, after the page has been fetched. Before that, it is the same unglamorous work technical SEO has always been: can a machine get the URL, see the text a person sees, and tell the article from the template?

If that step fails, there is no passage to retrieve. No citation. No "we have a page for that" that matters. The model is working from training memory, a different URL, or a shell.

AEO still starts with a fetch

What happens when ChatGPT answers a question is a field sketch, not a spec. The part we can see from the outside is ordinary. Sometimes the system looks something up. Looking something up means requesting a document.

OpenAI's own crawler docs make the split explicit. OAI-SearchBot is the automated crawl for ChatGPT search. ChatGPT-User fetches a page when a person asks. GPTBot is for training, not for "show this URL in search." Different agents. Same dependency: the URL has to respond with the content you care about.

Answer engine optimization and generative engine optimization — AEO and GEO — did not replace robots.txt, status codes, or rendering. They sit on top of them. A beautiful passage that never left the JavaScript bundle is a passage the fetch never got.

This is why "we published the article" and "the article is available to an answer engine" are different claims. The second one is a technical claim.

In plain terms

You cannot retrieve a paragraph the crawler never held.

Robots and status codes still decide who gets in

If robots.txt disallows the bot that does the search crawl, OpenAI says the site will not be shown in ChatGPT search answers, though it can still appear as a navigational link. That is their published rule, not a guess. The training bot is a separate switch. Blocking GPTBot is not the same as blocking SearchBot. We walk through the four agents in what OpenAI's crawlers mean for SEO.

Status codes do the next cut. A 404 is gone. A 401 or a hard paywall is gone for that fetch. A 301 chain that never settles wastes the visit. A soft 404 — 200 in the header, "not found" in the body — is worse, because the crawler may store the consolation template as if it were the page.

Canonicals and noindex are the quiet versions of the same decision. You can write for AEO on a URL that you also told machines not to use. The writing will not win that argument.

None of this is new. It is easy to skip when the brief says "optimize for ChatGPT." The brief still has to survive the front door.

Rendering is the content

A lot of client sites ship an empty shell and fill it in the browser. A person sees the article. A crawler that does not execute the script sees a header and a spinner.

We have watched this produce "thin" pages in a crawl table that are not thin on the live URL. The extract is nav plus footer. Embed that and you get a picture of the chrome. Ask a question the article answers, and the nearest passage may be a sibling's footer, or nothing.

What we check before trusting a crawl is the habit: open the important URLs, compare extract to live page, fix the wait or the blocked script, then continue.

Blocked assets matter here too. If the HTML is a 200 and the JavaScript bundle is 403, you published a different page to the bot than to the employee on the office network.

The door can be open and the article still locked. That is a technical problem with an AEO costume.

Chrome, markup, and the extract

Once the HTML arrives, something still has to pick the article out of the template. Semantic HTML — a real <main>, headings in order — makes that pick less of a guess. Div soup makes it more of one.

This is not a claim that <article> is an AEO ranking factor. It is a claim that extractors fail in boring ways, and named regions fail less often.

The other technical messes that show up as "the model used the wrong thing":

• The unique paragraph lives in an image or a canvas.
• The answer is behind a tab that never renders for the bot.
• Every page shares 400 words of nav, so the stored text huddles.
• Pagination or parameters multiply near-empty URLs that look like pages.

Technical SEO's job, in this frame, is to make the stored document match the document you meant. Writing cannot fix a stored document you never created.

What we still do in an audit

The AEO pass we trust looks like a technical pass with a second question at the end.

Can the relevant bots fetch the URL? Does the rendered extract contain the sentence that answers the buyer question? Is that sentence in the main region, not the rail? Are we looking at production, not staging?

Only then is it worth asking whether another page on the same site is a nearer passage, or whether the page never states the answer at all.

Silkra is a crawler first. The workspace is useful for AEO because it shows the rendered extract and the issues that explain a bad one — blocked resources, missing main landmark, failed extract — not because it has a ChatGPT score. We do not have a ChatGPT score. Nobody honest does.

The part that is still fuzzy

Labs do not publish how thoroughly they render, how long they wait, or which extractors they use. A page that works in our rendered crawl can still fail in theirs, or the reverse.

What does not look fuzzy is the dependency. Answer engines that look things up are still clients of the web. Technical SEO is how you treat those clients like they matter. The writing starts after the fetch. If you want a place to begin on a site you are auditing, start with robots, status, and whether the extract contains the paragraph you would bet the answer on.