Papers

Field

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Recently funded

Topics in this field

agent memory · memory construction · term memory262 papers49 this week
gpus · gpu · throughput227 papers33 this week
diffusion language models · language models diffusion · models diffusion language157 papers33 this week
continual learning · catastrophic forgetting · forgetting244 papers28 this week
looped language models · recurrent depth · loops95 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers22 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 5
What Must Replay Preserve? Separating Correctable Bias from Class Correspondence
BoRen Deng, Xiangyue Ma, Chenglong Li, Xiaoting Du
Class-incremental learning must recognize all classes seen so far without task labels. Logit replay methods such as DER and DER++ mitigate forgetting by matching the model's past predictions on stored examples. Deleting this matching reveals its benefit, but the resulting accuracy cost cannot show whether the stored scores themselves are needed, or whether the cost survives correction of the classifier's bias toward recent classes. We propose a diagnostic framework that treats a cached prediction as temporally heterogeneous supervision: it separates classes known when an example was stored from classes learned afterward, edits each group, and evaluates every model before and after a task-level offset that leaves within-task predictions unchanged. On CIFAR-100 with DER++, suitable fixed constants replace the unrefreshed stored scores of later-learned classes within an equivalence margin of 1 percentage point, and the offset reduces the cost of deleting their matching from 14.9 to 1.8 points. Reassigning the non-gold scores of classes known at storage, which preserves their values and each task's target probability, costs 4.3 points before and 4.0 after the offset, and a parallel…
PDF arXiv
arXiv cs.CLOct 5
D-Loop: Looped Diffusion Drafting for Speculative Decoding
Kecheng Chen, Yuyang He, Cheng Gong, Hui Liu, Guoping Long +6
Block diffusion accelerates speculative decoding by drafting multiple tokens in one forward pass. However, each position predicts a marginal distribution without observing earlier proposed tokens, limiting draft quality and acceptance length. We identify a concrete failure, the \emph{repetition trap}, in which neighboring positions produce redundant copies of the same token. We explain this tendency theoretically and empirically examine its association with shorter accepted drafts. Recent methods refine marginal predictions with an additional causal head or a separately trained drafter, increasing parameter storage and introducing separate training objectives. We instead propose D-Loop, which introduces \emph{intra-block causal conditioning} within the original diffusion drafter without additional model components. Inspired by semi-autoregressive generation and parameter sharing, D-Loop reuses the same backbone across looped passes. The first pass proposes a block, and the second conditions on a selected prefix to regenerate the suffix in parallel. A complementary prefix--suffix objective trains the shared drafter for both anchor-only prefix prediction and prefix-conditioned…
PDF arXiv
arXiv cs.CLOct 5
LRCC: Generalizing Low-Rank Compression with Conditional Computation
Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, Simone Scardapane
Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers' path choices.
PDF arXiv
arXiv cs.CLOct 5
Breaking the Tie: A Cluster-Aware Routing Framework for Large Language Models
Yao Lu, Zhaiyuan Ji, Yaxin Gao, Zeyu Wang, Zhe Tang +4
With the rapid development of artificial intelligence, the emergence of various Large Language Models (LLMs) has created a rich model ecosystem. However, this also brings a key challenge: how to select the optimal model for a specific user query. LLM routing addresses this need by dynamically assigning queries to the most suitable expert in the pool of candidate models. However, existing routing frameworks often simplify this process to a standard classification task; thus, a critical vulnerability is exposed when multiple candidate models correctly answer the same query. We formalize this capability overlap as routing noise, which misleads the router with arbitrarily correct candidate models, ultimately leading to routing collapse (a severe decline in generalization ability on unseen tasks). To address this problem, we propose a novel Cluster-Aware Soft-Labeling Routing (CASLR) framework. CASLR shifts the evaluation paradigm from the success of a single query to macro-domain consensus by replacing traditional one-hot vectors with a masked softmax mechanism. Specifically, for experts who answer incorrectly, we penalize their target probability to zero; for the remaining…
PDF arXiv
arXiv cs.CLCodeOct 5▲ 17
Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering
Sarim Hashmi, Mukul Ranjan, Abdelrahman Elsayed, Muhammad Umer Sheikh, Fahad Shamshad +1
Masked diffusion language models (dLLMs) generate text by iteratively denoising masked positions, re-predicting each token multiple times before it is committed. An autoregressive decoder exposes an answer's distribution once, at the step that commits it; a dLLM exposes it at every denoising step before commitment, and we show that an adversary can exploit this. Since an answer remains open to revision over many denoising steps, an adversary with access to internal activations can watch how likely the model is to produce a chosen answer and adjust the intervention accordingly. Building on this observation, we study targeted bias injection, an attack that steers a frozen dLLM toward a demographic answer selected by the adversary. The attack uses a simple proportional-integral (PI) controller that tracks the target-answer probability during denoising and adapts the strength of a steering vector on the fly. On ambiguous BBQ questions where the correct answer is abstention, our attack raises LLaDA-8B-Instruct's preference for the targeted group from 1.8 to 16.7 percentage points, more than three times the strongest fixed-strength steering baseline, and on SocialStigmaQA it raises the…
PDF arXiv Code
arXiv cs.LGOct 5
The Optimization Landscape of Learning Compacted Context Models
Thomas Villeneuve, Alex Sandomirsky, Charles O'Neill, Max Kirkby, Michael Psenka
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said context is akin to direct memory manipulation, without affecting the base model's weights. Many works pose KV compaction as an optimization problem: learn a smaller set of KV vectors that matches the behavior of the full KV cache. While this preserves base model behavior, optimizing through a frozen base model results in a highly nontrivial optimization problem with a brittle and flat loss landscape. In this paper, we characterize what makes these optimization problems difficult and demonstrate that a heavily simplified Perceiver-based architecture not only matches performance of a full Perceiver transformer in continuous context compaction, but outperforms baselines on compaction utility. Results are presented on MCQ tasks across Finance, Legal, Gutenberg, and Code.
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Topics in this field

agent memory · memory construction · term memory262 papers49 this week
gpus · gpu · throughput227 papers33 this week
diffusion language models · language models diffusion · models diffusion language157 papers33 this week
continual learning · catastrophic forgetting · forgetting244 papers28 this week
looped language models · recurrent depth · loops95 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers22 this week

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGOct 5
What Must Replay Preserve? Separating Correctable Bias from Class Correspondence
BoRen Deng, Xiangyue Ma, Chenglong Li, Xiaoting Du
Class-incremental learning must recognize all classes seen so far without task labels. Logit replay methods such as DER and DER++ mitigate forgetting by matching the model's past predictions on stored examples. Deleting this matching reveals its benefit, but the resulting accuracy cost cannot show whether the stored scores themselves are needed, or whether the cost survives correction of the classifier's bias toward recent classes. We propose a diagnostic framework that treats a cached prediction as temporally heterogeneous supervision: it separates classes known when an example was stored from classes learned afterward, edits each group, and evaluates every model before and after a task-level offset that leaves within-task predictions unchanged. On CIFAR-100 with DER++, suitable fixed constants replace the unrefreshed stored scores of later-learned classes within an equivalence margin of 1 percentage point, and the offset reduces the cost of deleting their matching from 14.9 to 1.8 points. Reassigning the non-gold scores of classes known at storage, which preserves their values and each task's target probability, costs 4.3 points before and 4.0 after the offset, and a parallel…
PDF arXiv
arXiv cs.CLOct 5
D-Loop: Looped Diffusion Drafting for Speculative Decoding
Kecheng Chen, Yuyang He, Cheng Gong, Hui Liu, Guoping Long +6
Block diffusion accelerates speculative decoding by drafting multiple tokens in one forward pass. However, each position predicts a marginal distribution without observing earlier proposed tokens, limiting draft quality and acceptance length. We identify a concrete failure, the \emph{repetition trap}, in which neighboring positions produce redundant copies of the same token. We explain this tendency theoretically and empirically examine its association with shorter accepted drafts. Recent methods refine marginal predictions with an additional causal head or a separately trained drafter, increasing parameter storage and introducing separate training objectives. We instead propose D-Loop, which introduces \emph{intra-block causal conditioning} within the original diffusion drafter without additional model components. Inspired by semi-autoregressive generation and parameter sharing, D-Loop reuses the same backbone across looped passes. The first pass proposes a block, and the second conditions on a selected prefix to regenerate the suffix in parallel. A complementary prefix--suffix objective trains the shared drafter for both anchor-only prefix prediction and prefix-conditioned…
PDF arXiv
arXiv cs.CLOct 5
LRCC: Generalizing Low-Rank Compression with Conditional Computation
Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, Simone Scardapane
Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers' path choices.
PDF arXiv
arXiv cs.CLOct 5
Breaking the Tie: A Cluster-Aware Routing Framework for Large Language Models
Yao Lu, Zhaiyuan Ji, Yaxin Gao, Zeyu Wang, Zhe Tang +4
With the rapid development of artificial intelligence, the emergence of various Large Language Models (LLMs) has created a rich model ecosystem. However, this also brings a key challenge: how to select the optimal model for a specific user query. LLM routing addresses this need by dynamically assigning queries to the most suitable expert in the pool of candidate models. However, existing routing frameworks often simplify this process to a standard classification task; thus, a critical vulnerability is exposed when multiple candidate models correctly answer the same query. We formalize this capability overlap as routing noise, which misleads the router with arbitrarily correct candidate models, ultimately leading to routing collapse (a severe decline in generalization ability on unseen tasks). To address this problem, we propose a novel Cluster-Aware Soft-Labeling Routing (CASLR) framework. CASLR shifts the evaluation paradigm from the success of a single query to macro-domain consensus by replacing traditional one-hot vectors with a masked softmax mechanism. Specifically, for experts who answer incorrectly, we penalize their target probability to zero; for the remaining…
PDF arXiv
arXiv cs.CLCodeOct 5▲ 17
Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering
Sarim Hashmi, Mukul Ranjan, Abdelrahman Elsayed, Muhammad Umer Sheikh, Fahad Shamshad +1
Masked diffusion language models (dLLMs) generate text by iteratively denoising masked positions, re-predicting each token multiple times before it is committed. An autoregressive decoder exposes an answer's distribution once, at the step that commits it; a dLLM exposes it at every denoising step before commitment, and we show that an adversary can exploit this. Since an answer remains open to revision over many denoising steps, an adversary with access to internal activations can watch how likely the model is to produce a chosen answer and adjust the intervention accordingly. Building on this observation, we study targeted bias injection, an attack that steers a frozen dLLM toward a demographic answer selected by the adversary. The attack uses a simple proportional-integral (PI) controller that tracks the target-answer probability during denoising and adapts the strength of a steering vector on the fly. On ambiguous BBQ questions where the correct answer is abstention, our attack raises LLaDA-8B-Instruct's preference for the targeted group from 1.8 to 16.7 percentage points, more than three times the strongest fixed-strength steering baseline, and on SocialStigmaQA it raises the…
PDF arXiv Code
arXiv cs.LGOct 5
The Optimization Landscape of Learning Compacted Context Models
Thomas Villeneuve, Alex Sandomirsky, Charles O'Neill, Max Kirkby, Michael Psenka
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said context is akin to direct memory manipulation, without affecting the base model's weights. Many works pose KV compaction as an optimization problem: learn a smaller set of KV vectors that matches the behavior of the full KV cache. While this preserves base model behavior, optimizing through a frozen base model results in a highly nontrivial optimization problem with a brittle and flat loss landscape. In this paper, we characterize what makes these optimization problems difficult and demonstrate that a heavily simplified Perceiver-based architecture not only matches performance of a full Perceiver transformer in continuous context compaction, but outperforms baselines on compaction utility. Results are presented on MCQ tasks across Finance, Legal, Gutenberg, and Code.
PDF arXiv

Activity · 12 weeks

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 1h ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

264papers this week−9.3% vs last week

All papers · 1.6k With code · 63

Newest Most relevant

1.5k papers this month