Papers

Field

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
mlx-lm+8% downloads · 140.9k/wk+7.5%
vLLM1 release this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and reuse information without exceeding finite context windows or degrading performance.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems with desynchronization and complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality due to uniform computation allocation.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Recently funded

Topics in this field

agent memory · memory construction · term memory257 papers52 this week
gpus · gpu · throughput220 papers32 this week
diffusion language models · language models diffusion · models diffusion language150 papers29 this week
continual learning · catastrophic forgetting · forgetting237 papers25 this week
speculative decoding accelerates · self speculative decoding · speculative decoding89 papers24 this week
sparse attention · linear attention · attention layers134 papers23 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.DSOct 15
Hard Instances on Demand
Jamie Fravel, Martin Koutecký, Pavlo Tsitsei
The conclusions of an experimental study of algorithms are only as good as the instances they are run on. Benchmark libraries for well-studied problems took decades to curate, while evaluations of new problems often fall back on random instances, which are known to be misleading: they may say more about the random generator than about the algorithmic question at hand. We treat instance generation as an optimization problem: parameterize the instance space, take the solver's running time as the objective, and let a black-box optimizer search for hard instances. We develop a widely applicable workflow built on the Nevergrad optimization platform and demonstrate it on four NP-hard problems: satisfiability, traveling salesperson, Hamiltonian cycle, and non-permutation flow-shop scheduling. The search needs only the solver, an instance encoder, and a compute budget. Our evolved instances beat the hardest instance of a budget-matched random draw in all four case studies, by an order of magnitude or more in three. They are often competitive with long-curated benchmark sets: our SAT champion is harder (for a three-solver portfolio) than 78% of the 498 SAT Competition formulas in its size…
PDF arXiv
arXiv cs.LGOct 15
On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Eliezer Shlomi, Eliron Rahimi, Avi Mendelson, Chaim Baskin
Diffusion language models (DLMs) promise parallel text generation, but in practice face an unresolved speed-quality trade-off. Exact decoding fills in only one masked position per model pass, while faster multi-token decoding schemes are lossy. Speculative decoding is the standard lossless acceleration for autoregressive models. However, it does not transfer directly to DLMs, which must choose both what to generate and where. We show that this order is the missing piece: the model's own predictions from the previous pass already give the right token at 92-100% of positions, but not which position comes next. Based on this observation, we introduce Speculating Positions Order Trajectory (SPOT), which learns to predict this order. A 0.62M-parameter model reads only the target's output distributions and proposes a tree of candidate continuations, and the target verifies all of them in a single pass by checking whether each equals its own next state. On the verification side, our analysis shows that prior block-diffusion verification approaches either assume $T=0$ or a sequential proposal interface. To support tree-based speculation at any temperature, we propose using coupled-noise…
PDF arXiv
arXiv cs.LGOct 15
Rethinking Routing Exploration in Modern Mixture-of-Experts
Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
PDF arXiv
arXiv cs.AIOct 15
Designing Transformers for Continuous Diffusion Language Models
Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading…
PDF arXiv
arXiv cs.LGOct 15
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron +4
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99\% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior…
PDF arXiv
arXiv cs.LGOct 15
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse $\relu$ and $\relu^2$ activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31\% and 30\% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1\% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.
PDF arXiv

Papers

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
mlx-lm+8% downloads · 140.9k/wk+7.5%
vLLM1 release this week

Topics in this field

agent memory · memory construction · term memory257 papers52 this week
gpus · gpu · throughput220 papers32 this week
diffusion language models · language models diffusion · models diffusion language150 papers29 this week
continual learning · catastrophic forgetting · forgetting237 papers25 this week
speculative decoding accelerates · self speculative decoding · speculative decoding89 papers24 this week
sparse attention · linear attention · attention layers134 papers23 this week

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 53m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

268papers this week−14.4% vs last week

All papers · 1.5k With code · 62

Newest Most relevant

1.5k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and reuse information without exceeding finite context windows or degrading performance.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems with desynchronization and complexity.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality due to uniform computation allocation.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Recent papers

All in Research →

arXiv cs.DSOct 15
Hard Instances on Demand
Jamie Fravel, Martin Koutecký, Pavlo Tsitsei
The conclusions of an experimental study of algorithms are only as good as the instances they are run on. Benchmark libraries for well-studied problems took decades to curate, while evaluations of new problems often fall back on random instances, which are known to be misleading: they may say more about the random generator than about the algorithmic question at hand. We treat instance generation as an optimization problem: parameterize the instance space, take the solver's running time as the objective, and let a black-box optimizer search for hard instances. We develop a widely applicable workflow built on the Nevergrad optimization platform and demonstrate it on four NP-hard problems: satisfiability, traveling salesperson, Hamiltonian cycle, and non-permutation flow-shop scheduling. The search needs only the solver, an instance encoder, and a compute budget. Our evolved instances beat the hardest instance of a budget-matched random draw in all four case studies, by an order of magnitude or more in three. They are often competitive with long-curated benchmark sets: our SAT champion is harder (for a three-solver portfolio) than 78% of the 498 SAT Competition formulas in its size…
PDF arXiv
arXiv cs.LGOct 15
On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Eliezer Shlomi, Eliron Rahimi, Avi Mendelson, Chaim Baskin
Diffusion language models (DLMs) promise parallel text generation, but in practice face an unresolved speed-quality trade-off. Exact decoding fills in only one masked position per model pass, while faster multi-token decoding schemes are lossy. Speculative decoding is the standard lossless acceleration for autoregressive models. However, it does not transfer directly to DLMs, which must choose both what to generate and where. We show that this order is the missing piece: the model's own predictions from the previous pass already give the right token at 92-100% of positions, but not which position comes next. Based on this observation, we introduce Speculating Positions Order Trajectory (SPOT), which learns to predict this order. A 0.62M-parameter model reads only the target's output distributions and proposes a tree of candidate continuations, and the target verifies all of them in a single pass by checking whether each equals its own next state. On the verification side, our analysis shows that prior block-diffusion verification approaches either assume $T=0$ or a sequential proposal interface. To support tree-based speculation at any temperature, we propose using coupled-noise…
PDF arXiv
arXiv cs.LGOct 15
Rethinking Routing Exploration in Modern Mixture-of-Experts
Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
PDF arXiv
arXiv cs.AIOct 15
Designing Transformers for Continuous Diffusion Language Models
Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading…
PDF arXiv
arXiv cs.LGOct 15
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron +4
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99\% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior…
PDF arXiv
arXiv cs.LGOct 15
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse $\relu$ and $\relu^2$ activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31\% and 30\% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1\% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.
PDF arXiv