Cloudflare says a tenant-isolation failure in one Workers AI model caused 155 responses to contain data from other customers’ requests. The company disclosed the incident on October 26 after coordinating remediation work with SGLang, the open-source inference engine used for the affected deployment.
According to Cloudflare, the problem affected only the GLM-5.3 model between August 27 and September 9. Every affected response followed a prompt longer than 681,280 tokens. Cloudflare said it found no evidence that anyone deliberately exploited the flaw after excluding its own tests and those of the researcher who reported it. That conclusion is based on the company’s investigation, not an independent audit.
How the exposure happened
Security researcher Gadi Evron of Knostic reported on September 8 that very long prompts sent to the GLM-5.3 endpoint sometimes returned text from other users’ sessions. Cloudflare said it reproduced the issue the next morning, declared an incident and took the model offline within 40 minutes. Service was restored later that day after the company changed the configuration and patched the inference engine.
Cloudflare traced the exposure to two overlapping failures. It had configured the model for a context window of 1,048,576 tokens, while the key-value cache pool on its prefill GPUs could hold 681,280 tokens. The company said it knew about the mismatch but expected over-length requests to fail safely because earlier tests returned successful responses.
Instead, an SGLang bug mishandled rejected requests in a split prefill-and-decode setup. A request that did not fit in the cache pool was replaced internally by a one-token placeholder, yet the system continued transferring cache state to the decode node. That allowed decode processing to read memory pages the request had not written, producing corrupted output or data from another customer’s prior request.
What Cloudflare changed
Cloudflare said GLM-5.3 was the only model affected because its other disaggregated Workers AI deployments had cache pools larger than their configured context windows. The company nevertheless applied the engine fix and additional hardening across every model it serves with SGLang.
The changes include checks that compare context length with cache capacity, stricter handling of failed transfers, bounded transfer batches and load-balancer behavior intended to avoid abort races. Cloudflare also contributed fixes and reproduction work upstream. It advised operators of disaggregated SGLang deployments to update to version 0.5.22 or backport the relevant patches.
Why the disclosure matters
AI inference systems often reuse expensive accelerator memory across many requests, so tenant isolation depends on failure paths being as strict as the normal path. This incident shows how a configuration assumption and an engine bug can combine to cross a security boundary even without a conventional intrusion. The relatively small disclosed count limits the known scope, but the content exposure itself is significant because affected responses could include another customer’s prompt data.
Source
Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang.
Featured image: AI-generated editorial illustration; it does not depict an actual event.
