Papers

Field

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Recently funded

Topics in this field

agent memory · memory construction · term memory262 papers48 this week
diffusion language models · language models diffusion · models diffusion language157 papers33 this week
gpus · gpu · throughput227 papers32 this week
continual learning · catastrophic forgetting · forgetting244 papers27 this week
looped language models · recurrent depth · loops95 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers22 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 5
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Jing Li, Jian Meng, Yingmeng Gao, Suming Qiu, Linyuan Qiu +6
Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit…
PDF arXiv
arXiv cs.CVCodeOct 5▲ 21
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
Zhuokun Chen, Feng Chen, Xi Lin, Xiyu Wu, Jiahao He +2
Long-horizon video world models require persistent memory to preserve scene consistency over extended rollouts. Softmax attention retains the full generation history through a growing KV cache, whereas recurrent linear attention compresses history into fixed-size states with substantially lower memory cost. However, we identify severe long-range forgetting in Gated DeltaNet (GDN), where information from distant but relevant scenes is progressively attenuated by subsequent state updates. To address this limitation, we propose HLA-WM, a training-free hybrid linear-attention framework that combines coarse-grained geometry-guided retrieval with fine-grained recurrent linear-state computation. HLA-WM exploits the affine structure of GDN to cache compact chunk-wise transition summaries, retrieve scene-relevant historical chunks using camera geometry, and recompose them into query-specific recurrent states. On the $60$-second SANA-WM-Bench, HLA-WM improves all six aggregate revisit-consistency and camera-control metrics of the base autoregressive generator without additional training, including a $0.74$ dB PSNR gain and a $28.5\%$ reduction in rotation error. The improvements persist…
PDF arXiv Code
arXiv cs.LGOct 5
PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents
Yiqi Wang, Jiaqi Liu, Jiaqi Zhang, Zhangkai Wu, Yiqun Duan +2
LLM agents rely on long-term memory to retain and reuse information when performing tasks over long horizons. Existing methods provide limited support for handling memories that become outdated as new observations or domain evidence arrive. Such outdated memories may remain semantically relevant, continue to affect dependent records, and retain value as historical evidence. This calls for two capabilities: dependency tracking to identify downstream effects and historical preservation to retain useful past records. We propose Provenance-Aware Cascading Memory Invalidation (PACMI), a framework that represents memories and new evidence in a provenance graph with typed dependency edges. PACMI assigns records to a four-state validity lattice, propagates validity changes to dependent memories, and uses the resulting states for retrieval and stale-premise detection. We also introduce a diagnostic benchmark with 100 cases and 300 queries across five domains. The evaluation separates node, context-, and answer-level performance. PACMI achieves the highest final-answer accuracy on this benchmark, and its paired difference from the strongest baseline is significant under an exact McNemar…
PDF arXiv
arXiv cs.LGOct 5
Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory
Parsa Hejabi, Morteza Dehghani, Payam Piray
Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how noisy each observation of them is. First, we show that the update of gated delta-rule memories is the form this filter takes under isotropic uncertainty. Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. A dense covariance would have to be propagated token by token, ruling out the parallel training these models depend on. We therefore give two assumed-density approximations, diagonal and quasi-diagonal, both of which leave the memory update in delta-rule form and reuse its chunked kernels. On controlled recall tasks in which associations change and observations are corrupted, Voltic leads all baselines. On the task combining volatility and stochasticity, its margin over the strongest baseline is larger…
PDF arXiv
arXiv cs.CLOct 5
Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning
Runguo Li
Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.
PDF arXiv
arXiv cs.PLOct 5
RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof
Ashkan Vedadi Gargary, Guido Martínez, Sebastian Burckhardt, Gabriel Ebner, Abhinav Jangda +2
AI systems can now write and optimize production GPU kernels, but validating them remains an important challenge. Evaluating the kernel on a few random inputs and checking that its outputs match a trusted reference kernel within numeric tolerances is not sufficient: races can cause nondeterministic behavior that fails to manifest in tests, and numeric tolerances can hide bugs and cause false positives even after extensive calibration. To address this challenge, we present RESOLVE, which combines testing and formal verification to build a comprehensive kernel validation pipeline. It operates in three steps: First, it tests for nondeterminism using binary instrumentation that perturbs execution timing to expose races. Second, an agent rewrites the candidate and reference kernels to obtain "reduced-concurrency" versions that are simpler to analyze but still produce bitwise-identical outputs in all tests. Third, the reduced kernels are formally analyzed in the F*/Pulse framework and prove that they perform the same computation on real numbers. This sidesteps the need for numeric tolerances. We show that RESOLVE can validate a broad selection of kernels using KernelBench, and prove…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+62% downloads · 16k/wk+62.3%
BentoML+40% downloads · 44.6k/wk+39.6%
mlx-lm1 release this week
MLX+26% downloads · 349.8k/wk+26.5%
vLLM1 release this week

Topics in this field

agent memory · memory construction · term memory262 papers48 this week
diffusion language models · language models diffusion · models diffusion language157 papers33 this week
gpus · gpu · throughput227 papers32 this week
continual learning · catastrophic forgetting · forgetting244 papers27 this week
looped language models · recurrent depth · loops95 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding92 papers22 this week

Recently funded

Market

Liufang TechnologySeries B+ · Oct 17$15M
EMode PhotonixPre-seed · Oct 16$1M
XHEARTSeries B+ · Oct 16undisclosed
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGOct 5
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Jing Li, Jian Meng, Yingmeng Gao, Suming Qiu, Linyuan Qiu +6
Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit…
PDF arXiv
arXiv cs.CVCodeOct 5▲ 21
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
Zhuokun Chen, Feng Chen, Xi Lin, Xiyu Wu, Jiahao He +2
Long-horizon video world models require persistent memory to preserve scene consistency over extended rollouts. Softmax attention retains the full generation history through a growing KV cache, whereas recurrent linear attention compresses history into fixed-size states with substantially lower memory cost. However, we identify severe long-range forgetting in Gated DeltaNet (GDN), where information from distant but relevant scenes is progressively attenuated by subsequent state updates. To address this limitation, we propose HLA-WM, a training-free hybrid linear-attention framework that combines coarse-grained geometry-guided retrieval with fine-grained recurrent linear-state computation. HLA-WM exploits the affine structure of GDN to cache compact chunk-wise transition summaries, retrieve scene-relevant historical chunks using camera geometry, and recompose them into query-specific recurrent states. On the $60$-second SANA-WM-Bench, HLA-WM improves all six aggregate revisit-consistency and camera-control metrics of the base autoregressive generator without additional training, including a $0.74$ dB PSNR gain and a $28.5\%$ reduction in rotation error. The improvements persist…
PDF arXiv Code
arXiv cs.LGOct 5
PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents
Yiqi Wang, Jiaqi Liu, Jiaqi Zhang, Zhangkai Wu, Yiqun Duan +2
LLM agents rely on long-term memory to retain and reuse information when performing tasks over long horizons. Existing methods provide limited support for handling memories that become outdated as new observations or domain evidence arrive. Such outdated memories may remain semantically relevant, continue to affect dependent records, and retain value as historical evidence. This calls for two capabilities: dependency tracking to identify downstream effects and historical preservation to retain useful past records. We propose Provenance-Aware Cascading Memory Invalidation (PACMI), a framework that represents memories and new evidence in a provenance graph with typed dependency edges. PACMI assigns records to a four-state validity lattice, propagates validity changes to dependent memories, and uses the resulting states for retrieval and stale-premise detection. We also introduce a diagnostic benchmark with 100 cases and 300 queries across five domains. The evaluation separates node, context-, and answer-level performance. PACMI achieves the highest final-answer accuracy on this benchmark, and its paired difference from the strongest baseline is significant under an exact McNemar…
PDF arXiv
arXiv cs.LGOct 5
Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory
Parsa Hejabi, Morteza Dehghani, Payam Piray
Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how noisy each observation of them is. First, we show that the update of gated delta-rule memories is the form this filter takes under isotropic uncertainty. Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. A dense covariance would have to be propagated token by token, ruling out the parallel training these models depend on. We therefore give two assumed-density approximations, diagonal and quasi-diagonal, both of which leave the memory update in delta-rule form and reuse its chunked kernels. On controlled recall tasks in which associations change and observations are corrupted, Voltic leads all baselines. On the task combining volatility and stochasticity, its margin over the strongest baseline is larger…
PDF arXiv
arXiv cs.CLOct 5
Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning
Runguo Li
Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.
PDF arXiv
arXiv cs.PLOct 5
RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof
Ashkan Vedadi Gargary, Guido Martínez, Sebastian Burckhardt, Gabriel Ebner, Abhinav Jangda +2
AI systems can now write and optimize production GPU kernels, but validating them remains an important challenge. Evaluating the kernel on a few random inputs and checking that its outputs match a trusted reference kernel within numeric tolerances is not sufficient: races can cause nondeterministic behavior that fails to manifest in tests, and numeric tolerances can hide bugs and cause false positives even after extensive calibration. To address this challenge, we present RESOLVE, which combines testing and formal verification to build a comprehensive kernel validation pipeline. It operates in three steps: First, it tests for nondeterminism using binary instrumentation that perturbs execution timing to expose races. Second, an agent rewrites the candidate and reference kernels to obtain "reduced-concurrency" versions that are simpler to analyze but still produce bitwise-identical outputs in all tests. Third, the reduced kernels are formally analyzed in the F*/Pulse framework and prove that they perform the same computation on real numbers. This sidesteps the need for numeric tolerances. We show that RESOLVE can validate a broad selection of kernels using KernelBench, and prove…
PDF arXiv

Activity · 12 weeks

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use information beyond finite context windows without losing critical details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Length generalization in state tracking1 paper raises it · 1 addresses it
Models trained on short sequences often fail to generalize to much longer sequences, limiting their practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but introduces synchronization and complexity challenges in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in long-term agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when those beliefs are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 53m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

260papers this week−11.0% vs last week

All papers · 1.6k With code · 63

Newest Most relevant

1.5k papers this month