Bot traffic Since May 17 caused intermittent site outages, team apologizes

Since May 17 the publisher’s website has been intermittently hard to reach after a heavy wave of automated crawlers flooded its infrastructure, the team said, apologizing to affected readers and subscribers. The surge of automated requests pushed web servers and backend resources toward capacity and produced unstable service for many users. Readers and paying subscribers experienced disrupted access while engineers worked to contain the load, and the team warned stability may not be fully restored until a durable mitigation is in place.

Engineers traced the immediate cause to repeated, high‑volume requests from automated crawlers that drove backend load spikes. Over the weekend, those spikes coincided with multiple database connection drops as the system strained under sustained traffic. To blunt the surge, the hosting provider blocked a particular user agent, which did reduce the immediate bot load but also had the unintended consequence of locking some legitimate users out while the block remained active. The provider action temporarily lessened scraping pressure but exposed the operational tradeoff between speed of containment and collateral service impacts.

The engineering team says it is working at full speed to implement a lasting fix and has openly solicited help from the community, inviting external contributors with relevant operational know‑how to assist with mitigation and debugging. The team characterized the bot activity as severe and singled out Perplexity as "in particular on our radar," indicating specific services or crawlers are being monitored as part of the response. Engineers cautioned that intermittent outages could continue until a robust mitigation strategy is deployed and stability is confirmed across web and backend systems.

For builders and operators the incident highlights concrete operational risks and trade‑offs. Hosting‑level interventions such as user‑agent blocks can quickly reduce scraping pressure but carry the risk of collateral damage to legitimate traffic and subscriber access. Repeated database connection failures can cascade into provider actions that disrupt availability, underscoring the importance of capacity planning, observability, and failover behavior. The team’s open call for coordinated incident assistance also illustrates how external collaboration can accelerate diagnosis and deployment of durable defenses when automated traffic spikes threaten service availability.