Abstract
Large language models (LLMs) commonly employ key–value (KV) caching to avoid recomputing previous attention states during autoregressive inference. However, its measured performance benefit depends on workload characteristics and the memory environment. This study presents a controlled systems characterization of KV-cache performance across input context length, grouped-query attention configurations, and GPU-memory residency. Experiments use Qwen2.5-3B and SmolLM3-3B under verified fully resident conditions with fixed 128-token generation, prompt lengths from 128 to 2048 tokens, randomized cache-enabled/cache-disabled execution, GPU telemetry, and repeated paired validation. KV-cache speedup increases from 1.34× to 8.31× for Qwen2.5-3B and from 1.28× to 8.62× for SmolLM3-3B. At 2048 prompt tokens, 30 randomized paired trials confirm mean speedups of 8.10× and 8.51×, corresponding to runtime reductions of 87.65% and 88.25%, while physical GPU-memory utilization remains below 61%. Despite SmolLM3-3B requiring twice the theoretical KV-cache storage per token, both models show comparable context-dependent scaling in the fully resident regime. A separate Mistral-7B case study further demonstrates that memory pressure can substantially alter apparent speedup. These results establish KV-cache speedup as workload- and system-dependent and demonstrate that meaningful benchmarking requires reporting context length, cache configuration, model and cache footprints, physical accelerator-memory residency, runtime, throughput, and relevant GPU-memory conditions during each inference experiment.