How Cloudflare Ran Tests
The issue was first brought to Cloudflare’s attention by its own customers, who issued multiple complaints about unauthorized Perplexity crawling, according to the report.
“We received complaints from customers who had both disallowed Perplexity crawling activity in their robots.txt files and also created WAF rules to specifically block both of Perplexity’s declared crawlers: PerplexityBot and Perplexity-User. These customers told us that Perplexity was still able to access their content even when they saw its bots successfully blocked. We confirmed that Perplexity’s crawlers were in fact being blocked on the specific pages in question, and then performed several targeted tests to confirm what exact behavior we could observe.” -Cloudflare
That purported behavior? A new user agent that claimed it was “Google Chrome on macOS” — and not PerplexityBot or Perplexity-User, the only crawlers that Perplexity should be using.
For its part, Perplexity appears to disagree with Cloudflare’s findings. In a statement to The Verge, a Perplexity spokesperson called the report a “publicity stunt,” and said that “there are a lot of misunderstandings in the blog post.”
