Papers

Field

Top tools here

All tools

NVIDIA Dynamo+81% downloads · 16.5k/wk+81.4%
BentoML+32% downloads · 42.8k/wk+31.7%
MLX+27% downloads · 357.1k/wk+27.3%
vLLM1 release this week
ort+5% downloads · 922.2k/wk+5.2%

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management4 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compact, reusable representations.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level routing system efficiency1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes step desynchronization and high implementation complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent models2 papers raise it · 2 address it
Recurrent or looped models fail to extrapolate to much longer sequences than seen in training.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention still requires scoring all keys, limiting speedups despite reduced memory access.
- DoSA: Dots Sparse Attentionraises · addresses
Cache management for long trajectories1 paper raises it · 1 addresses it
KV cache I/O and storage dominate costs, and cache operations can invalidate states or require re-prefilling.
- Cache Engineering for Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Recently funded

Topics in this field

agent memory · memory construction · term memory262 papers42 this week
diffusion language models · language models diffusion · models diffusion language157 papers29 this week
gpus · gpu · throughput227 papers26 this week
looped language models · recurrent depth · loops95 papers22 this week
continual learning · catastrophic forgetting · forgetting244 papers21 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers20 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 2
ROUTEAUDIT: Interaction-Aware Identification for Budgeted Multi-Verifier Routing
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Adaptive multi-verifier systems are commonly compared through endpoint quality-cost gaps, even when the verifier catalog, availability, accounting, information filtration, or scorer changes with the policy. We formulate verifier routing as a contract-conditioned identification problem. The contract records request support, verifier catalog, realized availability, resource accounting, online filtration, and post-trace scoring; a matched route contrast changes only the policy coordinate. ROUTEAUDIT adds three measurable objects to this contract. A contract lattice averages coordinate increments over every admissible bridge order and reports the resulting attribution together with its path sensitivity. A policy-independent response tape identifies paired sequential contrasts when adaptive policies reveal different observations. For incomplete matching, request-level bounds use whichever potential outcome remains observed and give a sharp finite-population interval. The protocol commits paid observations and ledger events before the oracle join and returns an attribution certificate for each comparison. On two held-out raw-tail caches, matched static SF+SA equals the cascade,…
PDF arXiv
arXiv cs.AIOct 2
BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu +7
Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over…
PDF arXiv
arXiv cs.CLOct 2
When History Fails to Become Experience: Action Calibration in Language Agents
Jingyu Liu, Zhiwen Wang, Yuxin Jing, Huanyu Zhou, Yong Liu
Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we examine how agents use history. We find that history improves task completion overall, yet much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past actions and selectively records experience to guide subsequent decisions, improving task success beyond outcome labeling alone.
PDF arXiv
arXiv cs.LGOct 2
Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
Shivam Gupta
Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also available. We introduce Prefix-Certificate Retention (PCR), an exact finite-trace formulation for static, grouped, reset-on-access timeouts. Usable-prefix rewards become nodes whose prerequisites are timeout thresholds and preceding hit certificates. The resulting maximum-weight closure reduces to one minimum cut, with graph size linear in the number of block lookups and timeout choices. A breakpoint theorem extends the construction to all nonnegative timeouts without discretization error. We also derive a linear-time-in-grid-size dynamic program for ordered timeouts and bounds that certify the cost of this restriction. Exhaustive small-instance checks and chronological replay of 39,632 public Mooncake requests validate the formulation. On the fixed grid, ordered timeouts attain the unrestricted training optimum in 118 of 120 trace-grouping-price cases. Heterogeneous retention improves several held-out memory-time tradeoffs, but finer training optimization does…
PDF arXiv
arXiv cs.LGOct 2
A Controlled Audit of Personal AI Memory for Rating Prediction
Shivam Gupta
In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the exact rating distribution, item support, and metadata. We combine this control with full history, native memory extraction, and matched numerical readers in a publicly frozen evaluation of 400 held-out user profiles and 6,160 target ratings across Coat and MovieLens. On Coat, the tested Qwen-written Mem0 pipeline increases user-macro mean absolute error relative to full history by 0.084 for Qwen and 0.149 for Phi; both family-adjusted bootstrap intervals exclude zero. Correct historical assignments help both readers on Coat, but the corresponding MovieLens effects are smaller and inconclusive after adjustment. A history-only ridge reader outperforms Qwen in both domains and Phi on MovieLens, while the Coat Phi comparison is unresolved. All 2,800 reader calls, including 150 invalid outputs, are retained under a fixed fallback rule. A separate implementation verifies inputs, metrics, and all ten primary contrasts. The contribution is a reproducible…
PDF arXiv
arXiv cs.AIOct 2
Dynamic LLM Routers are Often Misguided
Sam Wang, Julia White, Sahibzada Allahyar, Dhruv Atreja, Urchade Zaratiana +1
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+81% downloads · 16.5k/wk+81.4%
BentoML+32% downloads · 42.8k/wk+31.7%
MLX+27% downloads · 357.1k/wk+27.3%
vLLM1 release this week
ort+5% downloads · 922.2k/wk+5.2%

Topics in this field

agent memory · memory construction · term memory262 papers42 this week
diffusion language models · language models diffusion · models diffusion language157 papers29 this week
gpus · gpu · throughput227 papers26 this week
looped language models · recurrent depth · loops95 papers22 this week
continual learning · catastrophic forgetting · forgetting244 papers21 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers20 this week

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 2
ROUTEAUDIT: Interaction-Aware Identification for Budgeted Multi-Verifier Routing
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Adaptive multi-verifier systems are commonly compared through endpoint quality-cost gaps, even when the verifier catalog, availability, accounting, information filtration, or scorer changes with the policy. We formulate verifier routing as a contract-conditioned identification problem. The contract records request support, verifier catalog, realized availability, resource accounting, online filtration, and post-trace scoring; a matched route contrast changes only the policy coordinate. ROUTEAUDIT adds three measurable objects to this contract. A contract lattice averages coordinate increments over every admissible bridge order and reports the resulting attribution together with its path sensitivity. A policy-independent response tape identifies paired sequential contrasts when adaptive policies reveal different observations. For incomplete matching, request-level bounds use whichever potential outcome remains observed and give a sharp finite-population interval. The protocol commits paid observations and ledger events before the oracle join and returns an attribution certificate for each comparison. On two held-out raw-tail caches, matched static SF+SA equals the cascade,…
PDF arXiv
arXiv cs.AIOct 2
BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu +7
Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over…
PDF arXiv
arXiv cs.CLOct 2
When History Fails to Become Experience: Action Calibration in Language Agents
Jingyu Liu, Zhiwen Wang, Yuxin Jing, Huanyu Zhou, Yong Liu
Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we examine how agents use history. We find that history improves task completion overall, yet much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past actions and selectively records experience to guide subsequent decisions, improving task success beyond outcome labeling alone.
PDF arXiv
arXiv cs.LGOct 2
Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
Shivam Gupta
Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also available. We introduce Prefix-Certificate Retention (PCR), an exact finite-trace formulation for static, grouped, reset-on-access timeouts. Usable-prefix rewards become nodes whose prerequisites are timeout thresholds and preceding hit certificates. The resulting maximum-weight closure reduces to one minimum cut, with graph size linear in the number of block lookups and timeout choices. A breakpoint theorem extends the construction to all nonnegative timeouts without discretization error. We also derive a linear-time-in-grid-size dynamic program for ordered timeouts and bounds that certify the cost of this restriction. Exhaustive small-instance checks and chronological replay of 39,632 public Mooncake requests validate the formulation. On the fixed grid, ordered timeouts attain the unrestricted training optimum in 118 of 120 trace-grouping-price cases. Heterogeneous retention improves several held-out memory-time tradeoffs, but finer training optimization does…
PDF arXiv
arXiv cs.LGOct 2
A Controlled Audit of Personal AI Memory for Rating Prediction
Shivam Gupta
In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the exact rating distribution, item support, and metadata. We combine this control with full history, native memory extraction, and matched numerical readers in a publicly frozen evaluation of 400 held-out user profiles and 6,160 target ratings across Coat and MovieLens. On Coat, the tested Qwen-written Mem0 pipeline increases user-macro mean absolute error relative to full history by 0.084 for Qwen and 0.149 for Phi; both family-adjusted bootstrap intervals exclude zero. Correct historical assignments help both readers on Coat, but the corresponding MovieLens effects are smaller and inconclusive after adjustment. A history-only ridge reader outperforms Qwen in both domains and Phi on MovieLens, while the Coat Phi comparison is unresolved. All 2,800 reader calls, including 150 invalid outputs, are retained under a fixed fallback rule. A separate implementation verifies inputs, metrics, and all ten primary contrasts. The contribution is a reproducible…
PDF arXiv
arXiv cs.AIOct 2
Dynamic LLM Routers are Often Misguided
Sam Wang, Julia White, Sahibzada Allahyar, Dhruv Atreja, Urchade Zaratiana +1
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 42m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

226papers this week−26.1% vs last week

All papers · 1.6k With code · 66

Newest Most relevant

1.5k papers this month

Open problems in Inference & serving

Efficient long-horizon context management4 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compact, reusable representations.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level routing system efficiency1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes step desynchronization and high implementation complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent models2 papers raise it · 2 address it
Recurrent or looped models fail to extrapolate to much longer sequences than seen in training.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention still requires scoring all keys, limiting speedups despite reduced memory access.
- DoSA: Dots Sparse Attentionraises · addresses
Cache management for long trajectories1 paper raises it · 1 addresses it
KV cache I/O and storage dominate costs, and cache operations can invalidate states or require re-prefilling.
- Cache Engineering for Agentsraises · addresses

Named by an AI from these papers' abstracts.