Papers

Field

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
mlx-lm+8% downloads · 140.9k/wk+7.5%
vLLM1 release this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and reuse information without exceeding finite context windows or degrading performance.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems with desynchronization and complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality due to uniform computation allocation.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Recently funded

Topics in this field

agent memory · memory construction · term memory257 papers52 this week
gpus · gpu · throughput220 papers32 this week
diffusion language models · language models diffusion · models diffusion language150 papers29 this week
continual learning · catastrophic forgetting · forgetting237 papers25 this week
speculative decoding accelerates · self speculative decoding · speculative decoding89 papers24 this week
sparse attention · linear attention · attention layers134 papers23 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLOct 15
From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
Chaogui Gou
Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model…
PDF arXiv
arXiv cs.DCOct 15
Coordination-Free Workload Regularization for Variable-Cost Distributed Training
Tao Huang, Chen Zhu, Antonio Alonso, Daming Li, Adarsh Goura +2
Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude. For variable-length models, this converts input variation into idle time because each step waits for its most expensive rank-local batch. We present Batch Cost Lookahead (BCL), a coordination-free data-loading policy that buffers a bounded rank-local window and repartitions it into fixed-cardinality packs with balanced estimated cost. BCL preserves the trainer's record-count contract and requires neither cross-rank metadata exchange nor runtime modification. In simulation over public workload distributions, enabling BCL reduces per-rank estimated batch-cost variability by up to 82.8%. Across 13 NeMo-RL SFT corpora on eight H200 GPUs, rank-local BCL runs 1.050 to 1.167 times faster than pooled MFFD. Compiled H200 measurements on an internal sequence workload show speedups of 1.197 to 1.343 times over packed execution from one to four nodes. We also identify a memory-feasibility boundary: a p99 cap for high-variance or OOM-prone data bounds outliers before BCL balances the rest. These results show that substantial variable-cost imbalance can…
PDF arXiv
arXiv cs.AIOct 15
All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents
Bingbing Wang, Jingjie Lin, Weiming Qiao, Jing Li, Ruifeng Xu
Large language models increasingly rely on external memory to answer queries grounded in long-horizon histories such as multi-session dialogues and lengthy documents. However, existing memory methods often optimize retrieval and memory construction separately, without a shared reward space for evaluating search, writing, and stopping decisions. To address this, we propose All-in-One Memory (AiO-Mem), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action in a unified reward space. AiO-Mem compiles heterogeneous histories into source-traceable multi-view events, providing a shared substrate for retrieval and memory writing. A unified memory policy actor then rolls out complete interaction trajectories over this shared evidence state. A separate trajectory-conditioned process reward model is trained through evidence-grounded supervised fine-tuning and trajectory-level DPO to assign process scores to the collected trajectories. These scores are aggregated into an action-balanced trajectory reward and used by Process-Aligned DAPO (PA-DAPO) to optimize the memory policy. Across long-context memory…
PDF arXiv
arXiv cs.CLOct 15
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Wanyang Hong, Zhaoning Zhang
Sparse KV compression reduces long-context inference costs by retaining selected historical states, yet these states typically keep their coordinates from the full history. Under Rotary Position Embedding (RoPE), gaps left by evicted tokens continue to affect query-key matching, potentially limiting the use of retained evidence. We study post-selection positional geometry on a fixed selector-produced cache, examining how positional organization affects the use of retained information and which coordinate and continuation rules improve answer quality. To reorganize this geometry, we propose RoPERelay, which assigns compact, order-preserving RoPE coordinates, relocates cached key phases through inverse rotation and re-rotation, and synchronizes the current query and subsequent generation. It reuses existing K/V without renewed selection, historical re-prefill, or additional cache budget. Across 36 comparisons spanning two models, six sparse selectors, and three long-term memory tasks, RoPERelay improves answer quality in 32 cases. For Llama-3.1-8B-Instruct with BM25 on LongMemEval-S, accuracy increases from 23.6% to 35.4%. Measured complete-request time overhead is approximately…
PDF arXiv
arXiv cs.AIOct 15
MemComposer: Learning to Assemble Memories for LLM Agents via Reinforcement Learning
Yixuan Xie, Xueqiang Xu, Xinyi Fan, Xinxi Lyu, Tao Feng +6
Agent memory enables large language model (LLM) agents to reuse past experiences without updating their parameters. However, semantic similarity alone does not guarantee memory usefulness: downstream performance depends on how memories are selected and presented to the agent. We propose MemComposer, a memory composer that jointly examines the current query and retrieved candidates to decide which experiences to include, how many to select, and how to order them. We train the composer using Experience Policy Optimization (ExPO), a GRPO-based framework that evaluates assembled memory contexts through complete agent interactions and uses task-success feedback to optimize the assembly policy. Across three $τ^2$-bench domains with two frozen executors, MemComposer outperforms the strongest baseline for each executor while reducing average memory-token usage by 16.9% and 4.9%, respectively. The learned policy also transfers to unseen domains, executors, and memory banks, and benefits from memories generated by stronger models without further training. These results demonstrate the value of learning query-specific memory assembly using downstream task feedback.
PDF arXiv
arXiv cs.OSOct 15
Where the SRAM Goes: Per-Symbol Attribution of Static Memory in FreeRTOS-C and Embassy-Rust IoT Firmware on a 64 KB RISC-V Microcontroller
Romi Fadli, Jan Everhard Riwurohi, Wahyu Ramadani Putri Utami, Zulfa Malika Amana
Selecting a firmware stack for a memory-constrained microcontroller is usually guided by aggregate binary sizes, which report how much SRAM a build consumes but not where it goes or why. We address that gap with a per-symbol attribution method that decomposes static RAM into architectural categories - scheduler heap, network buffers and pools, task stacks, and linker-reserved regions - by bounding sized ELF symbols to the .data, .bss and .stack section ranges. The method is validated by reconciling every category sum exactly against the size-based totals for all sixteen builds. We apply it to eight cumulative firmware milestones, from a bare task scaffold to a full MQTT-over-Ethernet client, implemented twice, in FreeRTOS-C and in Embassy-Rust, on the CH32V203RBT6, a RISC-V microcontroller with 64 KB SRAM and an on-chip Ethernet MAC and PHY, with functional parity verified on hardware at every milestone. The decomposition shows that both stacks spend a nearly identical share of their RAM on networking (63.8% versus 63.9%), yet the absolute cost differs by 4.4 times: 40,719 bytes for lwIP against 9,316 bytes for smoltcp. FreeRTOS task stacks grow linearly at roughly 1 KB per task…
PDF arXiv

Papers

Topics in this field

agent memory · memory construction · term memory257 papers52 this week
gpus · gpu · throughput220 papers32 this week
diffusion language models · language models diffusion · models diffusion language150 papers29 this week
continual learning · catastrophic forgetting · forgetting237 papers25 this week
speculative decoding accelerates · self speculative decoding · speculative decoding89 papers24 this week
sparse attention · linear attention · attention layers134 papers23 this week

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CLOct 15
From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
Chaogui Gou
Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model…
PDF arXiv
arXiv cs.DCOct 15
Coordination-Free Workload Regularization for Variable-Cost Distributed Training
Tao Huang, Chen Zhu, Antonio Alonso, Daming Li, Adarsh Goura +2
Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude. For variable-length models, this converts input variation into idle time because each step waits for its most expensive rank-local batch. We present Batch Cost Lookahead (BCL), a coordination-free data-loading policy that buffers a bounded rank-local window and repartitions it into fixed-cardinality packs with balanced estimated cost. BCL preserves the trainer's record-count contract and requires neither cross-rank metadata exchange nor runtime modification. In simulation over public workload distributions, enabling BCL reduces per-rank estimated batch-cost variability by up to 82.8%. Across 13 NeMo-RL SFT corpora on eight H200 GPUs, rank-local BCL runs 1.050 to 1.167 times faster than pooled MFFD. Compiled H200 measurements on an internal sequence workload show speedups of 1.197 to 1.343 times over packed execution from one to four nodes. We also identify a memory-feasibility boundary: a p99 cap for high-variance or OOM-prone data bounds outliers before BCL balances the rest. These results show that substantial variable-cost imbalance can…
PDF arXiv
arXiv cs.AIOct 15
All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents
Bingbing Wang, Jingjie Lin, Weiming Qiao, Jing Li, Ruifeng Xu
Large language models increasingly rely on external memory to answer queries grounded in long-horizon histories such as multi-session dialogues and lengthy documents. However, existing memory methods often optimize retrieval and memory construction separately, without a shared reward space for evaluating search, writing, and stopping decisions. To address this, we propose All-in-One Memory (AiO-Mem), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action in a unified reward space. AiO-Mem compiles heterogeneous histories into source-traceable multi-view events, providing a shared substrate for retrieval and memory writing. A unified memory policy actor then rolls out complete interaction trajectories over this shared evidence state. A separate trajectory-conditioned process reward model is trained through evidence-grounded supervised fine-tuning and trajectory-level DPO to assign process scores to the collected trajectories. These scores are aggregated into an action-balanced trajectory reward and used by Process-Aligned DAPO (PA-DAPO) to optimize the memory policy. Across long-context memory…
PDF arXiv
arXiv cs.CLOct 15
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Wanyang Hong, Zhaoning Zhang
Sparse KV compression reduces long-context inference costs by retaining selected historical states, yet these states typically keep their coordinates from the full history. Under Rotary Position Embedding (RoPE), gaps left by evicted tokens continue to affect query-key matching, potentially limiting the use of retained evidence. We study post-selection positional geometry on a fixed selector-produced cache, examining how positional organization affects the use of retained information and which coordinate and continuation rules improve answer quality. To reorganize this geometry, we propose RoPERelay, which assigns compact, order-preserving RoPE coordinates, relocates cached key phases through inverse rotation and re-rotation, and synchronizes the current query and subsequent generation. It reuses existing K/V without renewed selection, historical re-prefill, or additional cache budget. Across 36 comparisons spanning two models, six sparse selectors, and three long-term memory tasks, RoPERelay improves answer quality in 32 cases. For Llama-3.1-8B-Instruct with BM25 on LongMemEval-S, accuracy increases from 23.6% to 35.4%. Measured complete-request time overhead is approximately…
PDF arXiv
arXiv cs.AIOct 15
MemComposer: Learning to Assemble Memories for LLM Agents via Reinforcement Learning
Yixuan Xie, Xueqiang Xu, Xinyi Fan, Xinxi Lyu, Tao Feng +6
Agent memory enables large language model (LLM) agents to reuse past experiences without updating their parameters. However, semantic similarity alone does not guarantee memory usefulness: downstream performance depends on how memories are selected and presented to the agent. We propose MemComposer, a memory composer that jointly examines the current query and retrieved candidates to decide which experiences to include, how many to select, and how to order them. We train the composer using Experience Policy Optimization (ExPO), a GRPO-based framework that evaluates assembled memory contexts through complete agent interactions and uses task-success feedback to optimize the assembly policy. Across three $τ^2$-bench domains with two frozen executors, MemComposer outperforms the strongest baseline for each executor while reducing average memory-token usage by 16.9% and 4.9%, respectively. The learned policy also transfers to unseen domains, executors, and memory banks, and benefits from memories generated by stronger models without further training. These results demonstrate the value of learning query-specific memory assembly using downstream task feedback.
PDF arXiv
arXiv cs.OSOct 15
Where the SRAM Goes: Per-Symbol Attribution of Static Memory in FreeRTOS-C and Embassy-Rust IoT Firmware on a 64 KB RISC-V Microcontroller
Romi Fadli, Jan Everhard Riwurohi, Wahyu Ramadani Putri Utami, Zulfa Malika Amana
Selecting a firmware stack for a memory-constrained microcontroller is usually guided by aggregate binary sizes, which report how much SRAM a build consumes but not where it goes or why. We address that gap with a per-symbol attribution method that decomposes static RAM into architectural categories - scheduler heap, network buffers and pools, task stacks, and linker-reserved regions - by bounding sized ELF symbols to the .data, .bss and .stack section ranges. The method is validated by reconciling every category sum exactly against the size-based totals for all sixteen builds. We apply it to eight cumulative firmware milestones, from a bare task scaffold to a full MQTT-over-Ethernet client, implemented twice, in FreeRTOS-C and in Embassy-Rust, on the CH32V203RBT6, a RISC-V microcontroller with 64 KB SRAM and an on-chip Ethernet MAC and PHY, with functional parity verified on hardware at every milestone. The decomposition shows that both stacks spend a nearly identical share of their RAM on networking (63.8% versus 63.9%), yet the absolute cost differs by 4.4 times: 40,719 bytes for lwIP against 9,316 bytes for smoltcp. FreeRTOS task stacks grow linearly at roughly 1 KB per task…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 51m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

268papers this week−12.7% vs last week

All papers · 1.5k With code · 62

Newest Most relevant

1.5k papers this month

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
mlx-lm+8% downloads · 140.9k/wk+7.5%
vLLM1 release this week

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and reuse information without exceeding finite context windows or degrading performance.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems with desynchronization and complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality due to uniform computation allocation.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.