Skip to content
DSML.KZ
Publications

Preprint · arXiv · December 12, 2025

Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery: Sublinear Memory Growth for Efficient LLM Inference

Adilet Metinov, Kudakeeva, Gulida M., Nursultan, Bolotbek uulu, Kabaeva, Gulnara D.

Adilet Baktiiarovich Metinov
Read on arXivPDFDOI 10.48550/arxiv.2512.11221

Abstract

We present Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery (ASR-KF-EGR), a training-free inference-time framework for efficient large language model generation. Our method introduces a reversible soft-freeze mechanism that temporarily suspends key-value (KV) updates for low-importance tokens identified within a sliding attention window. Unlike eviction-based approaches that permanently discard context, ASR-KF-EGR preserves all tokens in off-GPU storage and restores them on demand. We extend the framework with sublinear freeze scheduling, where freeze duration grows sublinearly with repeated low-importance detections, preventing over-aggressive compression. Preliminary experiments on LLaMA-3 8B demonstrate 55-67% reduction in active KV cache size while maintaining generation quality and passing needle-in-haystack retrieval tests. The method is architecture-agnostic, requires no fine-tuning, and provides a practical solution for memory-constrained deployment of long-context LLMs.

Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applicationskey-value cache compressionsliding window attentionKV cache eviction

More by Adilet Baktiiarovich Metinov