As the demand for data to train Generative AI models surges, developers are facing a critical dilemma: how to manage aggressive AI scraping while protecting their original content and infrastructure costs. To address this, Tencent Cloud EdgeOne has officially released its Basic Bot Management capabilities to all users—including those on the Free Plan. This update introduces two core features: AI Crawler Control and the CAPTCHA Page.

Ending the Crawler “Whac-A-Mole” Game

Unlike traditional web crawlers, modern AI crawlers are stealthier, operate on a massive scale, and are increasingly difficult to distinguish from legitimate traffic. Developers often find themselves trapped in an endless game of “Whac-A-Mole”—blocking one crawler IP only to see dozens of new ones appear the next day under different guises. This results in unauthorized scraping of original content and wasted server bandwidth.

Tencent Cloud EdgeOne’s new AI Crawler Control solves this by utilizing advanced User-Agent feature recognition. The system continuously updates its identification rules to pinpoint mainstream AI bots, including GPTBot, ClaudeBot, and Google-Extended.

Marketing Technology News: MarTech Interview with Miguel Lopes, CPO @ TrafficGuard

Developers no longer need to write complex scripts to defend their sites. With a simple configuration in the EdgeOne console, users can apply diverse actions based on their specific business needs:

• Monitor: Observe crawler behavior without interference.
• Block: Instantly stop unauthorized data harvesting.
• Allow: Permit friendly bots to access your site.
• Challenge: Trigger a verification step for suspicious traffic.

This flexible strategy ensures developers can block malicious data harvesting while still embracing a friendly AI ecosystem when beneficial.

Intelligent Defense for High-Value Scenarios

In addition to crawler management, the update includes the CAPTCHA Page feature designed to mitigate automated attacks. This function intelligently identifies suspicious Bot behavior and triggers a human-machine verification CAPTCHA.

Crucially, this is done without degrading the experience for real users. It builds a robust security line for high-stakes scenarios—such as digital content rights protection, e-commerce flash sales, and financial transactions—ensuring that automated scripts cannot manipulate metrics or disrupt services.

Marketing Technology News: Is the Traditional CDP Already Out of Date?

From Passive Defense to Active Management

“With the rapid evolution of AI technology, the governance of content and data access is a long-term challenge,” said Yun Li, Head of Tencent Cloud EdgeOne Product Team. “By making our Basic Bot Management capabilities free, we aim to help every developer shift from passive defense to active management, giving them full control over their digital assets.”

The new features are available now in the EdgeOne console. Tencent Cloud EdgeOne is committed to continuously refining its Bot Management and security capabilities, helping developers establish clear boundaries between AI technology application and content rights protection.