Skip to content

Response, New York. Noah Park: Check Point Patches Actively Exploited Management Server Path Traversal. Detection, Dallas. Ava Okello: Report Finds Nearly Half of Deployed Detections Need Attention Before SOCs Can Trust Them.

SOCtember

Always First. Fast SOC News.

Cloudflare illustration of a magnifying glass over a large fingerprint shaped like a drop, surrounded by stacks of books, pipes and circuit lines on a dark blue background.
Illustration: Cloudflare, header image for "Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang," Oct. 26, 2026 (fair use).

Response

Cloudflare Says a Workers AI Model Returned 155 Responses Containing Other Customers' Data Over Nearly Two Weeks, Traced to a Memory Mismatch and an SGLang Bug

Very long prompts sent to the GLM-5.3 model on Workers AI between Aug. 27 and Sept. 9 could come back with text from other users' sessions. Cloudflare found no evidence of deliberate exploitation and is urging anyone running SGLang in disaggregated mode to check KV cache pool sizes and upgrade to v0.5.22 or later.

Noah Park, ResponseNew York5 min read

NEW YORK - For nearly two weeks, a customer who sent a very long prompt to one of Cloudflare's hosted AI models could get back an answer laced with someone else's conversation. The reply looked normal. It carried an HTTP 200 status, its token count matched the prompt, and the text was coherent. It simply contained data from other users' sessions.

Cloudflare disclosed the problem on Monday in a post by Kevin Flansburg and Michelle Chen, published in coordination with the researcher who found it and with the maintainers of SGLang, the open-source inference engine Cloudflare uses. Between Aug. 27 and Sept. 9, a small number of customers using the GLM-5.3 model on Workers AI, @cf/zai-org/glm-5.3, received responses containing data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens.

After excluding its own testing and the researcher's, Cloudflare says the requests it identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug. It found no evidence of deliberate exploitation and says no other model on Workers AI was affected.

How the report was handled

Gadi Evron of Knostic reported the issue through Cloudflare's HackerOne program at 23:11 UTC on Sept. 8, saying very long prompts sometimes returned text belonging to other users' sessions. Cloudflare began verifying the report at 08:00 UTC the next morning and declared a security incident at 13:22. Its product security team reproduced the leak with two independent accounts within 25 minutes. At 14:00 the model was set to private, and requests began returning HTTP 400, within 40 minutes of the incident being declared.

Screenshot of the timeline table in Cloudflare's post, listing in UTC the Aug. 27 launch of GLM 5.3 with a prefill KV pool smaller than its context window, the Sept. 8 HackerOne report at 23:11, verification at 08:00 on Sept. 9, the incident declaration at 13:22, the model set to private at 14:00, root cause at 14:35, corrected configuration on all prefill nodes at 14:52, a rebuilt deployment with an engine patch at 19:04 and the model restored to public at 19:40.
Screenshot: Cloudflare, timeline from "Cross-request KV cache exposure in the GLM-5.3 Workers AI model," Oct. 26, 2026 (fair use).All times UTC.

By 14:35 Cloudflare had traced the root cause, and by 14:52 all prefill nodes were on a corrected configuration. At 19:04 the full deployment was rebuilt with that configuration and a Cloudflare-written engine patch, both checked against the researcher's reproduction script and Cloudflare's own canary-marker tests. The model went back to public use at 19:40 UTC on Sept. 9.

Two problems that overlapped

The first was a configuration choice. Cloudflare deployed GLM-5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs, the memory that holds the model's working state for each prompt, had room for only 681,280 tokens. Cloudflare says it knew about the mismatch before launch. It expected the engine to return an error for any prompt it could not handle, and its long-context tests came back successful. Those tests checked HTTP status codes and performance, not whether the output made sense.

The second was a bug in SGLang's disaggregated mode, in which one pool of GPUs processes the prompt and hands its cache to a second pool that generates the reply. When a prompt did not fit, SGLang marked it rejected and swapped in a one-token placeholder. But the admission queues on both sides checked whether a request was finished, not whether it had been rejected, so the request kept going. The prefill side sent one page of cache and reported the transfer complete. The decode side, which had reserved pages for the full prompt, read all of them. Only the first had been written by this request; the rest still held earlier users' prompts and partial generations. Cache pages are not zeroed between requests, a common practice for performance that leaves the engine responsible for tracking who owns each page.

Cloudflare diagram titled A rejected request passed admission and decoded from stale KV pages, comparing a normal handoff in which all eight reserved pages are written by the request with a rejected request in which prefill computes only a one-token placeholder, sends one page, and decode still reads pages that were never written by this request and hold other users' data.
Diagram: Cloudflare, "Cross-request KV cache exposure," Oct. 26, 2026 (fair use).How a rejected request ended up reading another user's cache pages.

Another SGLang user had reported the bug as issue #30233 on July 6, which Cloudflare found while investigating. A related fix, #36988, had been merged on Sept. 6 but was not yet in a release. Cloudflare has since enlarged the GLM-5.3 prefill pool to about 2.04 million tokens, roughly twice the context window, patched the engine so rejected requests are retired before any memory transfer, and changed its load balancer so a client disconnect no longer aborts a request mid-transfer. SGLang shipped the fix in v0.5.20, with follow-ups in v0.5.21, released Oct. 2, and v0.5.22, released Oct. 19.

What SOC and platform teams can do now

Screenshot of Cloudflare's five recommendations to other SGLang operators: measure the KV pool against the context length on every node, upgrade to a build that includes fixes #36988 and #38935, run with deferred KV release and bounded Mooncake transfer batches, turn on corruption detection where supported, and keep proxies from generating unnecessary aborts.
Screenshot: Cloudflare, "What we recommend to other operators," Oct. 26, 2026 (fair use).
  • If you self-host models on SGLang in disaggregated prefill and decode mode, compare the KV pool size in each node's startup log with the configured context length after every change to weights, quantization, parallelism or memory settings, and lower the context length if the pool is smaller. "This is the check we did not enforce," Cloudflare writes.
  • Upgrade to SGLang v0.5.22 or later, which includes #36988 and #38935, so an over-length request is rejected before handoff.
  • Run with deferred KV release on both roles, bound Mooncake transfer batches, and turn on --disaggregation-enable-kv-checksum and --kv-canary where the model supports them. Cloudflare says they do not yet cover every architecture, including GLM 5.3.
  • Test long-context deployments for correct output, not just status codes. Cloudflare's own launch tests passed because they checked only that requests succeeded.
  • If your organization sent prompts longer than 681,280 tokens to @cf/zai-org/glm-5.3 on Workers AI between Aug. 27 and Sept. 9, review what those prompts contained and ask Cloudflare whether your account was among those affected. The post does not say how affected customers were notified.

Sources:


Noah Park covers incident response and blue-team operations for SOCtember from New York.

Related stories

Response desk