AI in 2026: What Actually Changed for People Who Run Websites • ThematoSoup

There is a date coming up that most site owners have not put in their calendar, and it is a good example of how this whole AI story has been going.

On September 15, Cloudflare flips a default. AI training and agent crawlers get blocked on ad-supported pages for all new domains, all new sites added by existing customers, and every free-tier account. Existing paid sites keep their current settings unless the owner changes something. Cloudflare also turned its Pay Per Crawl experiment into Pay Per Use, which pays publishers when their content shapes an AI answer rather than when a bot simply fetches the page.

Whether that is good or bad for you depends on things nobody can decide on your behalf. What is not optional is that your site now has an AI access policy. If you never chose one, you inherited one.

That is the honest summary of where AI sits in 2026 for anyone running a website. The loud part is mostly over. The part that actually touches your setup arrived quietly, in the form of defaults, deadlines and numbers that moved while everyone was arguing about whether robots will take our jobs.

Here is what I think matters, and what I would ignore.

1. Bots are now the majority of the web, and most of them are not shopping

In June, Cloudflare’s CEO shared data showing automated requests had crossed the halfway mark and made up roughly 57.5% of HTML traffic, with humans at about 42.5%. He noted it happened faster than he had predicted.

The part that matters for a small publisher is not the volume. It is the exchange rate. Search crawlers historically paid for access with clicks. AI training crawlers have no mechanism to send anything back at all. By mid-2026, training-purpose crawling made up roughly half of AI bot traffic on Cloudflare’s network, while search-purpose crawling, the kind that can actually produce a citation with a link, was down around ten percent.

Published crawl-to-referral ratios are all over the place depending on the source and the measurement window, ranging from a few thousand pages per referral to tens of thousands, while Google’s traditional search crawler sits somewhere in the single digits to low teens. Treat the exact figures with suspicion. The direction is not in doubt.

Cloudflare’s original argument for all of this is worth reading in the company’s own words, in the Content Independence Day post and the Pay Per Crawl introduction. TechCrunch covered the launch when the marketplace first appeared.

My take: for most small sites, blanket blocking is the wrong instinct. If you sell something, being absorbed into answers is the visibility. If you sell attention, the math is different and you should probably be charging. Either way, go look at your actual bot settings this week instead of assuming you know what they are. Most people I talk to have never once checked what their site does when an AI user agent knocks.

2. The click stopped being the default outcome of a search

This is the change everyone who lives off the web felt in their stomach before there was data for it.

According to the study SparkToro published in June, built on Similarweb clickstream data, roughly two thirds of US Google searches now end without a single click to any website. Ten years ago that was under half. When an AI Overview appears, click-through to organic results drops by close to 60%.

Coverage of the study is on Search Engine Land, and Similarweb wrote up the same data from its side.

The interesting footnote is that AI Mode, Google’s conversational search surface, barely registered during the measured period. It was under half a percent of searches. Google has since said AI Mode passed a billion monthly users with query volume climbing fast, so that number is the one I would watch in the next study rather than in the next round of LinkedIn predictions.

My take: this is not the end of search as a channel. It is the end of search as a channel you measure in sessions. The traffic that survives is narrower and harder to win, and on average much closer to actually doing something. Anyone still reporting month-over-month organic sessions as the headline metric is reading a gauge that came unplugged from the engine.

3. Everyone has agents. Almost nobody has results.

My favourite contradiction in the entire field.

On one side, adoption looks total. Writer’s 2026 survey of 1,200 executives and 1,200 employees found 97% of executives saying their company deployed AI agents in the past year, with about half of employees already using them.

On the other side, the widely cited MIT study on generative AI in business found that the overwhelming majority of enterprise pilots produced no measurable financial return. Follow-up work from the large consultancies keeps landing on the same sentence in different words: adoption is nearly universal, value is not.

Both are true at once, and that is the whole point.

What I see on small projects is simple. Agents work where a process already exists and is written down. If you know who does what, in what order, judged by what standard, automating it is now absurdly cheap. If the process lives in someone’s head, AI will not invent it for you. It will just generate the mess faster.

The bottleneck was never the model. The bottleneck is that most of us never wrote down how we do the thing we do.

4. Models got boring, and that is the best news in this article

Two years ago, picking a model was a strategic decision. Today it is closer to picking a hosting plan.

Inference costs keep falling, small models now do work that needed something ten times larger a year ago, and open-weight models have crept uncomfortably close to the commercial frontier. If you like charts, llm-stats tracks hundreds of models across dozens of benchmarks and the compression is easy to see.

The practical consequence for a one-person operation is that a pile of things that used to be filed under “too expensive for this budget” now cost a few dollars a month. Classifying thousands of products. Cleaning up a feed. Reading a stack of PDFs. Handling correspondence in three languages. That is not a project anymore. It is a script.

“Which model are you using” has become the least interesting question in the room. What you solved with it is the interesting part.

5. Somebody is paying for all this compute

The unglamorous section, and the one I think matters most for long-term decisions.

The infrastructure spending is difficult to hold in your head. Allianz Research puts US big tech capital expenditure past $600 billion this year, with capital intensity running at more than double pre-ChatGPT levels. Goldman Sachs works with cumulative figures in the trillions through 2031, while pointing out that the whole range hinges on a handful of assumptions, starting with how long the silicon actually lasts.

I am not going to pretend I know whether this is a bubble. What I do know is that the tools we use today are cheap partly because somebody else is funding the years before the payback. Prices eventually meet costs.

So my one concrete piece of advice this year is deliberately dull. Do not build anything load-bearing on a single vendor whose pricing you do not control. Keep your data on your side. Keep your process logic separate from any specific model. Have an answer to the question of what you do if the price triples next quarter.

Three things I am watching instead of headlines

What share of searches actually goes through AI Mode. If that jumps from near zero to double digits, the traffic conversation restarts from scratch.

Whether getting cited in AI answers drifts away from ranking well. Analyses through 2026 suggest a shrinking share of pages cited in AI Overviews also sit in the organic top ten. If that continues, these are two jobs, not one.

Whether infrastructure spending growth stalls for two consecutive quarters. That would be the first serious signal that something changed, and a far better one than any comparison to seventeenth-century tulips.

Where this leaves us

The feeling I am left with after two years is that AI moved from “look what it can do” to “fine, what did you actually solve”. That is a duller phase. It is probably also a more useful one.

I am curious how it looks from where you sit. Has any of this genuinely shortened your work, or is it still mostly a demo that looks great in a meeting?