We had not broken anything. The pages still ranked where they had ranked. Fewer people were clicking them.

That is the shape of this year, and it is worth being precise about it rather than vague and worried. Organic search traffic in the US is down roughly 2.5% year on year, which sounds survivable until you look at who absorbed it. Publishers saw Google referrals fall 38%. On the queries where an AI Overview appears, organic click-through dropped 61%. Clicks to the first organic result were down 34.5% in April last year and 58% by December, which is a trend accelerating rather than settling. Somewhere near 60% of searches now end without a click to anybody.

The mechanism is not sinister and it is not really anyone's fault. The answer arrives before the link does. Your page was read, used, sometimes credited, and the reader never arrived.

The thing being sold to fix it

Nearly every agency pitch we have seen since spring includes llms.txt. It is a plain-text file at the root of your site summarising what you do and where your important pages are, on the theory that language models will read it and quote you more often.

We looked at whether it works, because clients keep asking us to add one.

Adoption is around one site in ten across a study of 300,000 domains. The evidence that it changes anything is simply not there. Two separate pieces of work, one correlating adoption against citation frequency across those domains and one reading real server logs across about 900 sites, both found no measurable lift. In June, Google documented in plain language that llms.txt has no effect on Search rankings or on AI Overviews. The crawlers that matter here, GPTBot and ClaudeBot and PerplexityBot among them, overwhelmingly skip the file and fetch your HTML directly.

It is not worthless. Coding agents genuinely use it: point Cursor or Claude Code or Copilot at a documentation site and they will look for /llms.txt and /llms-full.txt. If you publish developer docs, write one, and write it well. If you run a restaurant in Defence Colony, you have added a text file that nothing reads.

The crawlers skip the summary and read the page. Which means the page is still the work.

Meanwhile the other side moved

On 15 September, three days after this is published, Cloudflare changes its defaults. Crawlers that mix search indexing with AI training get blocked by default on pages carrying ads, for new sites and for every free-tier customer. Pay Per Crawl has been replaced by Pay Per Use, which pays a publisher when a model actually uses their content in an answer rather than when a bot fetches the page.

Whether any of that survives contact with the market is above our pay grade. The practical consequence for anyone running a website is smaller and more immediate: the question of which machines may read your site is becoming a setting, with a default, that somebody has to choose on purpose. Cloudflare reported in June that automated traffic had passed half of all requests for HTML worldwide, 57.3% against 42.7% human. When most of your readers are machines, that setting is not a detail.

What actually changes the outcome

Here is the part that concerns us, because it is a build problem rather than a marketing one.

Ship HTML, not a promise of HTML. This is the whole ballgame and it is the one most often got wrong. If your copy is assembled in the browser by JavaScript, a crawler that does not execute scripts receives an empty shell. Google renders JavaScript. Most AI crawlers fetch, parse and move on. A site that looks perfect to you can be blank to the thing deciding whether to quote you.

Put the metadata in the static head. Same reasoning, and we got this wrong on our own site until recently. Our title, description, canonical and social tags were being injected at runtime by the component that renders the page. Anything that did not run the component saw none of it. All sixteen now sit in the real <head>, in the HTML, before a line of script executes.

Write headings that describe their section. Not "Our approach" but "How we price fixed-scope projects". A machine summarising your page leans on the heading structure the way a skim-reader does, and a vague heading gets a vague summary or none.

Put schema on the things you actually are. Organisation, Article, Product, FAQ, LocalBusiness, whichever fits. This is page-level structured data describing specific entities, and it does a different job from a site-level summary file. It is also thirty years of boring, well-specified, widely supported markup rather than a convention invented eighteen months ago.

Answer the question on the page. If the page is about pricing, the price belongs in text, near the top, in a sentence. Content that hedges, defers or buries the answer under a form is content that cannot be quoted, which is now the same as content that cannot be found.

Speed still counts, for a different reason. Crawlers work to a budget. A slow page that has to be rendered before it yields anything is a page that gets sampled less often and abandoned sooner.

None of this is new advice. That is rather the point. The practices that make a site legible to a machine are the same ones that made it legible to a screen reader, a slow connection and a person in a hurry. The audience changed and the remedy did not.

The uncomfortable part

The old arrangement was that you wrote for people, a machine found you, and the people arrived. The new one is that you write for people, a machine reads you, and some of the people arrive. The ratio is worse, it is getting worse, and nobody is coming to fix it.

What survives that is being worth quoting, and having somewhere worth arriving at on the occasions when someone does. Both of those are old problems. Neither is solved by a file in your root directory.

What we changed here

Holding ourselves to it, since we have just told you to:

  • Every word on this site is in the HTML response. The component renders over markup that is already there, so the page reads correctly with scripts disabled.
  • The metadata moved out of the runtime and into the static head, where it should have been from the start.
  • The journal is static pages at real URLs rather than fragments behind a router, because a fragment is not a page and Google has always treated it as the same URL as the homepage.
  • Our demo builds carry noindex, follow. They are work samples and they should not compete with the studio for its own name.
  • We have not added an llms.txt. When there is evidence it does something, we will.

If your traffic is down and nobody has been able to tell you why, we are happy to look at the actual pages rather than sell you a file. Start a project, or just send us the URL.