Futureweb

Artificial Intelligence

AKV's cost cut is the authors' measurement

Tsinghua's AKV paper reports a 27.7 percent lower average cost and a 1.7 point higher average score against summarization. Those figures are the paper's, on the paper's hardware.

The short version

No. The paper Cache Engineering for Agents reports that a rolling-drop policy improved average benchmark scores by 1.7 points and reduced average cost by 27.7 percent against threshold-triggered summarization, on the authors' setup. The code is an Apache-2.0 repository built on SGLang. A different stack is a different measurement.

The abstract's two averages

Cache Engineering for Agents, posted with code at thu-nics/AKV, says a tool-response rolling-drop policy improved average benchmark scores by 1.7 points and reduced average cost by 27.7 percent compared with conventional threshold-triggered summarization. The same abstract says the system reached up to 1.2 times the output throughput and a 1.5 times speedup in 95th-percentile request latency. MarkTechPost repeated those averages on October 26. A context window still bounds what a server will hold, and a cache policy does not repeal that bound. The document named in the sources is the record this page will keep.A window still bounds the job.

Average hides the row

An average across models and benchmarks can rise while a single exam falls. The paper is the place to read the rows. This page will not promote the average into a guarantee for a stack the authors did not run. Weights a reader can fetch are the code, not a second result. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep.A repository is not a rerun.

The serving rack is named

The appendix describes eight A800 GPUs, split between prefill and decode, and a 120-billion-parameter model stored in BF16. Moving the same policy to a different chip or a different model is a new experiment. The paper does not claim that experiment. Asked directly, What is the idea, without the recipe? The abstract says agents can drop selected cache entries and reposition retained tokens while leaving the text trajectory in place, so retained context is not recomputed from scratch. The paper's implementation choices stay in the paper.

What this page will not teach

The paper defines cache operations and a serving design so that identical token text can still mean different cache states. That is the research claim. The commands, page sizes, and kernel choices that make it run are the paper's appendix, and they are not copied here. A CVE identifier does not describe this system either. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep.An identifier is not a design.

License and scope

The repository is Apache-2.0. A license lets a reader use the code under that grant. It does not promise the abstract's percentage on the reader's hardware. On-device inference is a different constraint and is not what this serving paper measured. Asked directly, Who wrote it? The paper lists Tsinghua University authors and one author at the Chinese University of Hong Kong. The repository is thu-nics/AKV, Apache-2.0. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep.On device is a different test.

A recursive-improvement line is also theirs

The abstract says a further self-improvement of the cache policy reduced cost by another 7.1 percent over an expert design. That sentence is inside the same paper. It is not an independent replication. Asked directly, What do the authors claim, in their own averages? A 1.7 point rise in average benchmark score and a 27.7 percent drop in average cost, versus conventional threshold-triggered summarization. Serving figures in the abstract go up to 1.2 times the output throughput and a 1.5 times speedup in 95th-percentile latency. Cloudflare's October 26 report of crossed responses on a hosted GLM model is a production incident, not this paper.The GLM report is not this paper.

Who the result is for

A team that runs SGLang and can read the paper is the audience. A product that only calls a hosted chat API did not, on this document, receive a 27.7 percent discount. Token-timing papers are a different genre, and they do not confirm this cache result. Asked directly, What hardware did the serving test use? The paper's appendix describes eight NVIDIA A800 GPUs and GPT-OSS-120B weights in BF16, because that GPU lacks a format the authors would otherwise have used. A result on that rack is a result on that rack.A timing paper is not this cache.

History dependence is the serving point

The authors say the same tokens can leave different cache states if the drop history differs, so a plain prefix match can reuse the wrong state. That is why they describe a custom server. The description is the finding. The patch is the repository. Asked directly, Does MarkTechPost add a measurement? Its October 26 account repeats the abstract's averages. It is not a second laboratory. The document named in the sources is the record this page will keep. The document named in the sources is the record this page will keep.

What would change the percentage

A replication on different models, with the harness published, would. Until then the 27.7 percent is the authors' average against the summarization baseline they chose. The short answer this page will stand on is this: No. The paper Cache Engineering for Agents reports that a rolling-drop policy improved average benchmark scores by 1.7 points and reduced average cost by 27.7 percent against threshold-triggered summarization, on the authors' setup. The code is an Apache-2.0 repository built on SGLang. A different stack is a different measurement. The document named in the sources is the record this page will keep.

Sources

The reports this brief is filing. Futureweb did not republish them.

  1. arXiv
  2. GitHub
  3. MarkTechPost

Questions

What do the authors claim, in their own averages?

A 1.7 point rise in average benchmark score and a 27.7 percent drop in average cost, versus conventional threshold-triggered summarization. Serving figures in the abstract go up to 1.2 times the output throughput and a 1.5 times speedup in 95th-percentile latency.

Who are the authors on the AKV paper?

The paper lists Tsinghua University authors and one author at the Chinese University of Hong Kong. The repository is thu-nics/AKV, Apache-2.0.

What is the idea, without the recipe?

The abstract says agents can drop selected cache entries and reposition retained tokens while leaving the text trajectory in place, so retained context is not recomputed from scratch. The paper's implementation choices stay in the paper.

Does MarkTechPost add a measurement?

Its October 26 account repeats the abstract's averages. It is not a second laboratory.

What hardware did the serving test use?

The paper's appendix describes eight NVIDIA A800 GPUs and GPT-OSS-120B weights in BF16, because that GPU lacks a format the authors would otherwise have used. A result on that rack is a result on that rack.