Papers

Field

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Recently funded

Topics in this field

agent memory · memory construction · term memory262 papers54 this week
gpus · gpu · throughput227 papers35 this week
diffusion language models · language models diffusion · models diffusion language157 papers34 this week
continual learning · catastrophic forgetting · forgetting244 papers31 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers24 this week
looped language models · recurrent depth · loops95 papers22 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AROct 8
DynaTE: Accelerating Diffusion LLMs via Dynamic Token Execution
Minghan Jiang, Jiayi Wang, Shuaiting Li, Haibin Shen, Kejie Huang
Diffusion-based LLMs (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs by enabling bidirectional parallel refinement, alleviating the sequential decoding bottleneck of AR generation. However, their parallel iterative refinement mismatches AR accelerators optimized for sequential decoding and their discrete token generation differs from DiT accelerators designed for continuous denoising. Recent dLLM accelerators have explored workload-specific optimizations to reduce vocabulary processing overhead and redundant computation across denoising iterations. However, these approaches retain all tokens in parallel execution, despite varying token refinement utility and execution requirements. This paper presents DynaTE, a hardware--software co-design architecture that dynamically adapts accelerator execution to evolving token states during dLLM decoding. DynaTE first enables adaptive token execution by skipping low-utility token computation, while a dimension-reconfigurable PE array maintains high utilization under varying active-token patterns. Second, DynaTE exploits dynamic token dependencies through FLDD to refine a small number of locally dependent…
PDF arXiv
arXiv cs.CLOct 8
Gated Memory: Admission-Controlled Memory Formation for Conversational AI
Preeti Saraswat, Divya Neelagiri, Ajay Manoj
Personalized conversational AI relies on long-term memory systems that extract facts from user utterances and store them in persistent vector stores. Despite progress in retrieval, deduplication, and lifecycle management, the formation stage, the moment a fact is first written to storage has received almost no principled attention. We identify this as the binding constraint on memory quality in production systems. Critical contextual signals, such as the distinction between a permanent user attribute and a transient situation, exist only in the original utterance and are irreversibly lost the moment extraction produces a subject-relation-object triple. No downstream process can recover them. We propose Gated Memory, a lightweight, modular formation framework that interposes two decision checkpoints between conversation and storage: an admission gate that evaluates every candidate fact against the full utterance context before extraction runs, and a conditional enrichment stage that grounds admitted facts through an entity scope taxonomy with privacy constraints. The gate evaluates only the current exchange while using prior turns as read-only reference context, and produces a…
PDF arXiv
arXiv cs.LGOct 8
Read What Matters: Query-Adaptive Quantization for KV Caches
Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more…
PDF arXiv
arXiv cs.CVOct 8
Attributing HOW, Not Just WHICH: Counterfactual Response Trajectories for Diffusion Models
Haoqian Zhang, Ziyuan Yang, Zerui Shao, Yi Zhang
Diffusion models have achieved remarkable success in image generation, yet tracing their outputs to individual training examples remains challenging. Existing attribution methods often compress factor-specific effects into scalar responses, making distinct internal changes indistinguishable. This is particularly limiting for diffusion models, where semantic factors emerge through evolving representation dynamics during denoising. We therefore reformulate diffusion data attribution as attributing factor-induced internal response trajectories. In this paper, we propose a novel Concept Attribution method through Dynamic Trajectories(CADT). We argue that attribution should therefore ask not only \emph{which} examples matter, but also \emph{how} their influence unfolds during generation. Specifically, we construct matched counterfactual pairs at identical noisy states to isolate factor-specific representation displacements, and model their directional and magnitude evolution across denoising as dynamic attribution signatures. For each training example and generated query, CADT extracts stage-wise feature vectors and integrates them along the denoising process to form a trajectory…
PDF arXiv
arXiv cs.LGCodeOct 8▲ 5
The Lattice of Transition Laws
T. Y. Tsui, Jiatao Gu, Lingjie Liu
Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens. Recent work seeks to combine the advantages of the two models, and each hybrid fixes its decoding schedule by design. In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps. We describe diffusion, AR, and models in between as paths on one corruption lattice, and define the cost of a schedule as the dependence its parallel steps discard. The cost shows that the fewest steps of a zero-cost schedule are set by the geometry of the data, in the same way for tokens and for continuous fields. In particular, for data that are Markov on a graph and dependent along its paths, the fewest steps equal the graph's treedepth, which is logarithmic in the length of a sequence and linear in the side length of a grid. With fewer steps than the treedepth, every schedule pays a positive cost, whose ranking we predict before decoding with a kernel of pairwise dependence estimated from pretrained weights. Across text generation,…
PDF arXiv Code
arXiv cs.LGOct 8
Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing
KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference. This contrast raises the question of whether per-KV compression and multi-KV aggregation can be bridged within a single mechanism for efficient attention. We identify RAM-Net as such a bridge through soft assignments over a discrete address space. These assignments determine recurrent updates to the continuous slot state associated with each address. Under a restricted RAM-Net construction, we prove that soft address assignments extend hard quantized matching to a separable read-write overlap that locally approximates full-attention similarity and supports recurrent aggregation. These connections further enable Transformer-to-RAM-Net weight migration through a new path based on a soft-quantized intermediate construction. Across nine pretrained Transformer models from 0.3B to 7B parameters,…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Topics in this field

agent memory · memory construction · term memory262 papers54 this week
gpus · gpu · throughput227 papers35 this week
diffusion language models · language models diffusion · models diffusion language157 papers34 this week
continual learning · catastrophic forgetting · forgetting244 papers31 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers24 this week
looped language models · recurrent depth · loops95 papers22 this week

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Recent papers

All papers in Research

arXiv cs.AROct 8
DynaTE: Accelerating Diffusion LLMs via Dynamic Token Execution
Minghan Jiang, Jiayi Wang, Shuaiting Li, Haibin Shen, Kejie Huang
Diffusion-based LLMs (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs by enabling bidirectional parallel refinement, alleviating the sequential decoding bottleneck of AR generation. However, their parallel iterative refinement mismatches AR accelerators optimized for sequential decoding and their discrete token generation differs from DiT accelerators designed for continuous denoising. Recent dLLM accelerators have explored workload-specific optimizations to reduce vocabulary processing overhead and redundant computation across denoising iterations. However, these approaches retain all tokens in parallel execution, despite varying token refinement utility and execution requirements. This paper presents DynaTE, a hardware--software co-design architecture that dynamically adapts accelerator execution to evolving token states during dLLM decoding. DynaTE first enables adaptive token execution by skipping low-utility token computation, while a dimension-reconfigurable PE array maintains high utilization under varying active-token patterns. Second, DynaTE exploits dynamic token dependencies through FLDD to refine a small number of locally dependent…
PDF arXiv
arXiv cs.CLOct 8
Gated Memory: Admission-Controlled Memory Formation for Conversational AI
Preeti Saraswat, Divya Neelagiri, Ajay Manoj
Personalized conversational AI relies on long-term memory systems that extract facts from user utterances and store them in persistent vector stores. Despite progress in retrieval, deduplication, and lifecycle management, the formation stage, the moment a fact is first written to storage has received almost no principled attention. We identify this as the binding constraint on memory quality in production systems. Critical contextual signals, such as the distinction between a permanent user attribute and a transient situation, exist only in the original utterance and are irreversibly lost the moment extraction produces a subject-relation-object triple. No downstream process can recover them. We propose Gated Memory, a lightweight, modular formation framework that interposes two decision checkpoints between conversation and storage: an admission gate that evaluates every candidate fact against the full utterance context before extraction runs, and a conditional enrichment stage that grounds admitted facts through an entity scope taxonomy with privacy constraints. The gate evaluates only the current exchange while using prior turns as read-only reference context, and produces a…
PDF arXiv
arXiv cs.LGOct 8
Read What Matters: Query-Adaptive Quantization for KV Caches
Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more…
PDF arXiv
arXiv cs.CVOct 8
Attributing HOW, Not Just WHICH: Counterfactual Response Trajectories for Diffusion Models
Haoqian Zhang, Ziyuan Yang, Zerui Shao, Yi Zhang
Diffusion models have achieved remarkable success in image generation, yet tracing their outputs to individual training examples remains challenging. Existing attribution methods often compress factor-specific effects into scalar responses, making distinct internal changes indistinguishable. This is particularly limiting for diffusion models, where semantic factors emerge through evolving representation dynamics during denoising. We therefore reformulate diffusion data attribution as attributing factor-induced internal response trajectories. In this paper, we propose a novel Concept Attribution method through Dynamic Trajectories(CADT). We argue that attribution should therefore ask not only \emph{which} examples matter, but also \emph{how} their influence unfolds during generation. Specifically, we construct matched counterfactual pairs at identical noisy states to isolate factor-specific representation displacements, and model their directional and magnitude evolution across denoising as dynamic attribution signatures. For each training example and generated query, CADT extracts stage-wise feature vectors and integrates them along the denoising process to form a trajectory…
PDF arXiv
arXiv cs.LGCodeOct 8▲ 5
The Lattice of Transition Laws
T. Y. Tsui, Jiatao Gu, Lingjie Liu
Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens. Recent work seeks to combine the advantages of the two models, and each hybrid fixes its decoding schedule by design. In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps. We describe diffusion, AR, and models in between as paths on one corruption lattice, and define the cost of a schedule as the dependence its parallel steps discard. The cost shows that the fewest steps of a zero-cost schedule are set by the geometry of the data, in the same way for tokens and for continuous fields. In particular, for data that are Markov on a graph and dependent along its paths, the fewest steps equal the graph's treedepth, which is logarithmic in the length of a sequence and linear in the side length of a grid. With fewer steps than the treedepth, every schedule pays a positive cost, whose ranking we predict before decoding with a kernel of pairwise dependence estimated from pretrained weights. Across text generation,…
PDF arXiv Code
arXiv cs.LGOct 8
Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing
KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference. This contrast raises the question of whether per-KV compression and multi-KV aggregation can be bridged within a single mechanism for efficient attention. We identify RAM-Net as such a bridge through soft assignments over a discrete address space. These assignments determine recurrent updates to the continuous slot state associated with each address. Under a restricted RAM-Net construction, we prove that soft address assignments extend hard quantized matching to a separable read-write overlap that locally approximates full-attention similarity and supports recurrent aggregation. These connections further enable Transformer-to-RAM-Net weight migration through a new path based on a soft-quantized intermediate construction. Across nine pretrained Transformer models from 0.3B to 7B parameters,…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 1h ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

283papers this week−6.6% vs last week

All papers · 1.6k With code · 63

Newest Most relevant

1.5k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.