.png)
ABM
Strategy
Creative & Content
Originally published on Mi-3 on July 15, 2025.
Cloudflare didn’t just block a few bots. It may have quietly redrawn the AI-content economy—and for B2B marketers, the implications are real.
If you missed Nico Neumann’s sharp primer in Mi-3—where he called Cloudflare’s new AI-blockade a potential “synthetic-research killer”—consider this your what-to-do next for B2B marketers. mi-3.com.au
What just happened
Cloudflare runs the “connectivity-cloud” that sits in front of roughly a fifth of the public web. It accelerates pages, blocks attacks and, from 1 July, blocks “known” AI crawlers by default for every new domain—unless the crawler arrives with proof-of-payment. Its private-beta Pay per Crawl marketplace lets publishers set a fee and serves non-paying bots an HTTP 402 “Payment Required” error.
Why this matters
For B2B marketers, this isn’t just an edge-tech update. It touches how your content gets discovered, how your synthetic insights are fuelled, and whether your IP becomes an asset or just free training data for someone else’s model.
We’re looking closely at the shift—and while there’s a lot of commentary out there—these are our takes for B2B marketers weighing how to stay discoverable, stay trusted, and stay in control as AI reshapes the rules.
Green Hat take #1—“The CDN just became a toll-booth”
Cloudflare has rewritten the commercial rules of crawling:
- Edge billing. What used to be a technical edge cost (CDN, WAF) can now earn revenue. Finance teams may start reporting “crawl income” next to MQLs and ad revenue.
- Budget effect. Model builders—from OpenAI to in-house LLM projects—now need a crawl line-item just like media spend or API calls.
- Switching costs. If your content licensing now rides on Cloudflare tokens, moving to another CDN gets as gnarly as migrating your CRM.
Brands with deep, differentiated content (white papers, benchmarks, proprietary data) suddenly own an asset with real resale value—but only if it’s surfaced through a CDN that can meter it.
A reality check for Marketing IT
For the “been-there-done-that” crowd who’ll say, “We’ve been working around crawl blockers for 10–15 years.” That mindset glosses over just how much the game has changed—and how much tougher the new countermeasures are.
No, determined scrapers haven’t disappeared—but the old tricks (spoofing a user agent and hoping for the best) no longer cut it. Getting around Cloudflare’s 402 gate at scale now demands residential IP pools, full-browser emulation, CAPTCHA solver farms, and constant re-fingerprinting. Those added layers don’t just increase effort—they push the cost curve close to, or even above, simply paying the crawl fee. And once a publisher serves you a Payment Required header, the legal exposure steps up too. The mouse can still run, but on a far more expensive and exposed treadmill.
Green Hat take #2—“From ‘share of voice’ to ‘share of model’”
Search is already tilting hard toward “no-click” behaviour—and Australian publishers feel it.
A July 2025 Similarweb report (quoted in The Australian) shows that since Google rolled out AI Overviews, the share of news searches that deliver zero clicks has ballooned from 56% to 69% in just twelve months, stripping as much as 70% of referral traffic from some outlets. theaustralian.com.au
Cloudflare’s paywall raises the next question: whose material still makes it into future models? If your content is behind a fee gate and a competitor’s isn’t, AI summary engines (ChatGPT, Gemini overviews, Perplexity) will default to the cheaper, licensed corpus.
Treat AI-licensing as the new “SOV buy.” Decide whether you’re better served by scarcity (block and sell premium access) or by reach (waive the fee to stay in the model).
Green Hat take #3—“SEO is dead, meet SOO (Scraper Optimisation and Outreach)”
Traditional SEO fought for blue links; now we fight for training-set inclusion and citation slots in AI answers.
- Optimise for labelled crawlers. LLM bots identify via headers (GPTBot, ClaudeBot, etc.). Publishers can now whitelist high-citation bots and charge low-value scrapers.
- Outreach flips. Partnership teams will pitch “licensing bundles” to AI vendors the way PR teams pitch guest posts. It’s Scraper Outreach as a formal channel.
B2B marketers can build a bot-access policy next to their robots.txt: who pays, who’s welcome and who’s blocked—and include it in content distribution plans.
Green Hat take #4—“Do synthetic personas still have a pulse?”
We think synthetic personas will remain in the toolkit, we already weave synthetic personas and digital-twin modelling into our wider research and insights programmes, treating them as a complementary lens alongside first-party interviews, customer councils and other data. Cloudflare’s pay-per-crawl model won’t make these assets obsolete; but it’s likely to change the inputs equation:
- Data supply and cost: licensing fees may lift the price of the third-party behavioural feeds that power some persona engines—but not to the point of wiping them out.
- Coverage and validation: if certain sites stay behind paywalls, synthetic outputs could become patchier in niche B2B segments, so cross-checking against fresh human data and CRM signals matters more—or simply relying on them less for complex nuanced roles.
Net result? Synthetic personas continue to add value for hypothesis-generation, “speed-to-insight” still has a big question mark over it in B2B and this change in the AI space will continue to keep it a support, not a reliable single source. We’ll just double-down on triangulation—validating hypotheses and early findings with real-world customer conversations—so clients get both the scale of AI and the critical nuance (and real-ness) of human input.
Synthetic personas still play a role—but they’re no substitute for genuine customer intimacy. We’ll keep using them to test hypotheses and accelerate insight, but we’ll double down on validation loops to ensure AI speed doesn’t come at the cost of real relevance.
Will other providers here copy Cloudflare?
Cloudflare’s toll-gate has put every other edge and bot-management vendor on the spot, but their response will vary according to their footprint—and incentives—in Australia.
- Fastly already has PoPs in Sydney, Melbourne, Perth and Brisbane and counts Nine and Guardian Australia among its customers. With an “AI Bot Management” ruleset on the shelf it only needs a billing toggle to match Cloudflare’s pay-per-crawl. We’d expect Fastly to move quickly once local publishers start asking for a revenue share.
- Akamai is entrenched across ASX-listed enterprises and federal agencies, so it can’t afford glitches. We think it will likely wait for proven case studies—then roll out a crawler-licensing layer wrapped in tighter SLAs and compliance guarantees.
- Kasada, born in Sydney and already fingerprinting bots for major banks and Ticketek, is perfectly placed to add a payments façade. For brands that aren’t on Cloudflare, Kasada could become the local “paywall plug-in.”
- Telstra Edge / Adaptive Cloud gives media and IoT customers sovereign edge nodes. We’d envisage Telstra bundling an “AI-safe” tier—blocking or metering crawlers—into its managed network offers, appealing to publishers nervous about offshore data flows.
- The hyperscalers—AWS CloudFront, Azure Front Door and Google Cloud CDN dominate start-ups and agencies here but also sell their own AI services. That conflict means they’ll tread carefully: we’ll be looking for policy statements or opt-in pilots rather than an aggressive 402 paywall.
Macquarie Telecom’s Secure Service Edge carries 42% of Commonwealth web traffic. If government publishers want crawler controls that stay on-shore and under IRAP, Macquarie is the natural go-to; a licensing switch could surface in its roadmap once public-sector copyright teams weigh in.
Australian publishers and B2B brands now hold real levers—block, charge, or invite AI crawlers. A home-grown line-up (Fastly, Kasada, Telstra) is already positioned to support those decisions. The early movers will shape not just how their IP is used—but how it’s valued—in an AI-driven content economy.
That’s the market-level picture. Now, here’s what B2B marketers should do next.
What now?
Whether you block, monetise, or stay open, the edge just became strategic real estate. Cloudflare didn’t just shift the tech—it reframed the value of your content.
Our take is that you’ll see more toll-booths going up, and smart B2B marketers will treat this moment like a new media channel: price your IP, define your access rules, and decide whether your brand fuels someone else’s model—or gets paid at the gate.
