Papers

Field

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
Open WebUI3 releases this week
mlx-lm+8% downloads · 140.9k/wk+7.5%

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use relevant information without exceeding finite context windows.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when they are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Recently funded

Topics in this field

agent memory · memory construction · term memory248 papers53 this week
gpus · gpu · throughput212 papers32 this week
diffusion language models · language models diffusion · models diffusion language145 papers29 this week
sparse attention · linear attention · attention layers131 papers23 this week
continual learning · catastrophic forgetting · forgetting230 papers21 this week
aware routing · model routing · model llm routing82 papers20 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLCodeOct 14▲ 8
Latent Compaction with Test-Time Training
Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han
Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32$\times$ compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 3
ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong +1
Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 9
The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
Ian Magnusson, Noah A. Smith, Pang Wei Koh
The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let…
PDF arXiv Code
arXiv cs.LGCodeOct 14▲ 67
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu +3
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose \textbf{LatentLoop}, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding $10^{21}$ FLOPs, LatentLoop consistently lowers validation NLL. At a…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 5
DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
Xiaocong Yang, Yuren Hao, ChengXiang Zhai
We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.
PDF arXiv Code
arXiv cs.AICodeOct 14▲ 50
Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory
Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang +75
Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience. We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II, Memory Management,…
PDF arXiv Code

Papers

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
Open WebUI3 releases this week
mlx-lm+8% downloads · 140.9k/wk+7.5%

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Topics in this field

agent memory · memory construction · term memory248 papers53 this week
gpus · gpu · throughput212 papers32 this week
diffusion language models · language models diffusion · models diffusion language145 papers29 this week
sparse attention · linear attention · attention layers131 papers23 this week
continual learning · catastrophic forgetting · forgetting230 papers21 this week
aware routing · model routing · model llm routing82 papers20 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CLCodeOct 14▲ 8
Latent Compaction with Test-Time Training
Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han
Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32$\times$ compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 3
ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong +1
Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 9
The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
Ian Magnusson, Noah A. Smith, Pang Wei Koh
The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let…
PDF arXiv Code
arXiv cs.LGCodeOct 14▲ 67
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu +3
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose \textbf{LatentLoop}, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding $10^{21}$ FLOPs, LatentLoop consistently lowers validation NLL. At a…
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 5
DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
Xiaocong Yang, Yuren Hao, ChengXiang Zhai
We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.
PDF arXiv Code
arXiv cs.AICodeOct 14▲ 50
Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory
Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang +75
Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience. We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II, Memory Management,…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 54m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

252papers this week−17.1% vs last week

All papers · 1.5k With code · 59

Newest Most relevant

1.4k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use relevant information without exceeding finite context windows.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when they are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.