Papers

Field

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Recently funded

Topics in this field

agent memory · memory construction · term memory270 papers35 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers24 this week
continual learning · catastrophic forgetting · forgetting249 papers23 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers18 this week
looped language models · recurrent depth · loops98 papers17 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CLSep 28
Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier
Linye Wei, Shutian Zheng, Haoyu Zeng, Meng Li
Diffusion drafting with autoregressive (AR) verification has emerged as a promising paradigm for efficient speculative decoding. Recent self-speculation models, represented by Nemotron-Labs-Diffusion, further simplify the speculative pipeline by unifying drafting and verification within a shared backbone, while enabling longer acceptance lengths. However, the Pareto frontier between aggregate and per-request throughput remains underexplored. At low concurrency, sequential draft-verify execution requires two model forward passes per round, limiting the effective tokens per forward (TPF). By contrast, at high concurrency, longer drafts incur increasingly expensive computation, forcing individual requests to operate under constrained speculation budgets and preventing full exploitation of the full-backbone drafter. Our key observation indicates that drafting and verification exhibit reciprocal predictability. Draft logits can anticipate likely verification mismatches, while recent verification outcomes predict future drafting utility and suitable block sizes. Building on this observation, we introduce Reciprocal Guidance (RecGuide), a runtime draft-verify orchestration framework…
PDF arXiv
arXiv cs.CLSep 28
Just-In-Time Agent Memory with Runtime Agentic Research
Bingyu Yan, Chaofan Li, Hongjin Qian, Shuqi Lu, Chaozhuo Li +1
Memory is critical for AI agents. Many existing agent-memory systems follow an Ahead-of-Time (AOT) design, constructing memory before a specific request arrives. While this reduces online serving cost, such request-agnostic memory construction can discard fine-grained information that later becomes important. To address this limitation, we propose Just-In-Time Agent Memory (JAM), a trainable framework for query-conditioned context construction at runtime. A Memorizer preserves complete raw histories in a hierarchical page-store with compact navigational summaries, while a Researcher iteratively retrieves, inspects, and integrates evidence for each request. To train these memory-use behaviors, we introduce Memory-Gym, an evidence-grounded data synthesis pipeline covering nine task types across six domains, and optimize the Researcher through verified-trajectory supervised fine-tuning followed by Hint-guided Group Relative Policy Optimization. We demonstrate the effectiveness of JAM across a variety of benchmarks on agent memory and long-context processing, where it achieves stronger task performance than AOT-style memory systems while remaining substantially more efficient than…
PDF arXiv
arXiv cs.DCSep 28
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu, Wookeun Jung, Thanh Tuan Dao
Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that…
PDF arXiv
arXiv cs.AISep 28
PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agents
Hanzhong Zhang, Ziwei Xiang, Weicheng Xie, Shizhe Liu, Siyang Song
The profile of a role-playing agent usually depends on the pre-defined personality in a system prompt, whereas its memory processing pipeline, including prioritisation of stored memories and subsequent retrieval, remains independent of this personality. This separation causes the agent's memory processing to be inconsistent with the pre-defined personality, and makes it difficult to validate whether agent behaviours follow this personality. In this paper, we propose Personality-Integrated Memory (PersMem), which integrates personality into the agent's memory processing pipeline, making it consistently personality-dependent. PersMem processes memory using four steps, where the personality is mapped to operation-specific parameters controlling: (i) affective appraisal annotating emotion states of the user input; (ii) retention of previously stored memories along with the current input; (iii) passive affect-driven memory retrieval exploring memories similar to user input in semantics and personality-guided emotions; and (iv) active goal-driven memory retrieval that refines and selects passively retrieved memories for the reply. Consequently, consistency with the pre-defined…
PDF arXiv
arXiv cs.LGSep 28
Spexis: Speculative Lookahead Scheduling for LLM Inference
Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi, Sungkyun Kim, Jinho Lee +1
Spexis is a multi-GPU LLM inference framework that improves the efficiency of pipeline and tensor parallelism through speculative parallelism. Rather than using speculative decoding only to accelerate token generation, Spexis runs speculation in parallel with normal execution, introducing a new parallelism axis without increasing KV-cache memory usage. This improves memory efficiency and helps mitigate the bottlenecks of multi-GPU inference. Spexis further uses lookahead scheduling to predict speculation quality and future memory pressure, allowing it to reduce wasted speculation, KV-cache eviction, and recomputation. Built on top of vLLM, Spexis largely improves serving performance across a range of GPU configurations, achieving speedups of up to 34% over a baseline that uses the optimal combination of pipeline and tensor parallelism. Spexis's source code is publicly available at https://github.com/mlsys-seo/spexis.
PDF arXiv
arXiv cs.LGSep 28
FORGE: Form-Optimal Routing of Grounded Evidence for Frozen LLM Agents
Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Jialin Chen +5
In agentic AI systems, frozen foundation models are increasingly deployed as closed-weight API endpoints, making downstream adaptation possible only through the inputs and inference procedures surrounding the model. As a result, for each input query, two coupled decisions largely determine both answer quality and token cost: what evidence to provide and how much reasoning budget to allocate. Fixed defaults along these axes are often suboptimal, misallocating support form or reasoning depth on roughly 80% of queries in our analysis. To address this challenge, we propose FORGE, a unified framework for adapting frozen models through per-query routing over a joint action space that spans both support form and thinking depth. Under an entropy-regularized, cost-aware utility objective, we derive a closed-form Boltzmann routing target and instantiate the policy as a lightweight 269K-parameter factorized router. The routing policy is trained around the frozen host, without any weight access, through a three-stage pipeline: offline arm enumeration, supervised Kullback-Leibler (KL) distillation from the Boltzmann target, and Group Relative Policy Optimization (GRPO) refinement with host…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Topics in this field

agent memory · memory construction · term memory270 papers35 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers24 this week
continual learning · catastrophic forgetting · forgetting249 papers23 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers18 this week
looped language models · recurrent depth · loops98 papers17 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CLSep 28
Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier
Linye Wei, Shutian Zheng, Haoyu Zeng, Meng Li
Diffusion drafting with autoregressive (AR) verification has emerged as a promising paradigm for efficient speculative decoding. Recent self-speculation models, represented by Nemotron-Labs-Diffusion, further simplify the speculative pipeline by unifying drafting and verification within a shared backbone, while enabling longer acceptance lengths. However, the Pareto frontier between aggregate and per-request throughput remains underexplored. At low concurrency, sequential draft-verify execution requires two model forward passes per round, limiting the effective tokens per forward (TPF). By contrast, at high concurrency, longer drafts incur increasingly expensive computation, forcing individual requests to operate under constrained speculation budgets and preventing full exploitation of the full-backbone drafter. Our key observation indicates that drafting and verification exhibit reciprocal predictability. Draft logits can anticipate likely verification mismatches, while recent verification outcomes predict future drafting utility and suitable block sizes. Building on this observation, we introduce Reciprocal Guidance (RecGuide), a runtime draft-verify orchestration framework…
PDF arXiv
arXiv cs.CLSep 28
Just-In-Time Agent Memory with Runtime Agentic Research
Bingyu Yan, Chaofan Li, Hongjin Qian, Shuqi Lu, Chaozhuo Li +1
Memory is critical for AI agents. Many existing agent-memory systems follow an Ahead-of-Time (AOT) design, constructing memory before a specific request arrives. While this reduces online serving cost, such request-agnostic memory construction can discard fine-grained information that later becomes important. To address this limitation, we propose Just-In-Time Agent Memory (JAM), a trainable framework for query-conditioned context construction at runtime. A Memorizer preserves complete raw histories in a hierarchical page-store with compact navigational summaries, while a Researcher iteratively retrieves, inspects, and integrates evidence for each request. To train these memory-use behaviors, we introduce Memory-Gym, an evidence-grounded data synthesis pipeline covering nine task types across six domains, and optimize the Researcher through verified-trajectory supervised fine-tuning followed by Hint-guided Group Relative Policy Optimization. We demonstrate the effectiveness of JAM across a variety of benchmarks on agent memory and long-context processing, where it achieves stronger task performance than AOT-style memory systems while remaining substantially more efficient than…
PDF arXiv
arXiv cs.DCSep 28
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu, Wookeun Jung, Thanh Tuan Dao
Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that…
PDF arXiv
arXiv cs.AISep 28
PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agents
Hanzhong Zhang, Ziwei Xiang, Weicheng Xie, Shizhe Liu, Siyang Song
The profile of a role-playing agent usually depends on the pre-defined personality in a system prompt, whereas its memory processing pipeline, including prioritisation of stored memories and subsequent retrieval, remains independent of this personality. This separation causes the agent's memory processing to be inconsistent with the pre-defined personality, and makes it difficult to validate whether agent behaviours follow this personality. In this paper, we propose Personality-Integrated Memory (PersMem), which integrates personality into the agent's memory processing pipeline, making it consistently personality-dependent. PersMem processes memory using four steps, where the personality is mapped to operation-specific parameters controlling: (i) affective appraisal annotating emotion states of the user input; (ii) retention of previously stored memories along with the current input; (iii) passive affect-driven memory retrieval exploring memories similar to user input in semantics and personality-guided emotions; and (iv) active goal-driven memory retrieval that refines and selects passively retrieved memories for the reply. Consequently, consistency with the pre-defined…
PDF arXiv
arXiv cs.LGSep 28
Spexis: Speculative Lookahead Scheduling for LLM Inference
Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi, Sungkyun Kim, Jinho Lee +1
Spexis is a multi-GPU LLM inference framework that improves the efficiency of pipeline and tensor parallelism through speculative parallelism. Rather than using speculative decoding only to accelerate token generation, Spexis runs speculation in parallel with normal execution, introducing a new parallelism axis without increasing KV-cache memory usage. This improves memory efficiency and helps mitigate the bottlenecks of multi-GPU inference. Spexis further uses lookahead scheduling to predict speculation quality and future memory pressure, allowing it to reduce wasted speculation, KV-cache eviction, and recomputation. Built on top of vLLM, Spexis largely improves serving performance across a range of GPU configurations, achieving speedups of up to 34% over a baseline that uses the optimal combination of pipeline and tensor parallelism. Spexis's source code is publicly available at https://github.com/mlsys-seo/spexis.
PDF arXiv
arXiv cs.LGSep 28
FORGE: Form-Optimal Routing of Grounded Evidence for Frozen LLM Agents
Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Jialin Chen +5
In agentic AI systems, frozen foundation models are increasingly deployed as closed-weight API endpoints, making downstream adaptation possible only through the inputs and inference procedures surrounding the model. As a result, for each input query, two coupled decisions largely determine both answer quality and token cost: what evidence to provide and how much reasoning budget to allocate. Fixed defaults along these axes are often suboptimal, misallocating support form or reasoning depth on roughly 80% of queries in our analysis. To address this challenge, we propose FORGE, a unified framework for adapting frozen models through per-query routing over a joint action space that spans both support form and thinking depth. Under an entropy-regularized, cost-aware utility objective, we derive a closed-form Boltzmann routing target and instantiate the policy as a lightweight 269K-parameter factorized router. The routing policy is trained around the frozen host, without any weight access, through a three-stage pipeline: offline arm enumeration, supervised Kullback-Leibler (KL) distillation from the Boltzmann target, and Group Relative Policy Optimization (GRPO) refinement with host…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 5m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

203papers this week−31.0% vs last week

All papers · 1.6k With code · 70

Newest Most relevant

1.5k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.