Papers

Field

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
InevitableStage not given · Oct 26$16.8M
AilyticsSeries A · Oct 26$7.4M
Valor EnergyStage not given · Oct 26$5M
Assembler AISeed · Oct 26$4M

Recently funded

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers26 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
kv cache compression · kv cache · kv caches267 papers20 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers16 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers16 this week

Topics in this field

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AISep 24
Harness Tokenomics: A Router for the Enterprise Agentic Control Plane
Ted Kwartler, Alan Aqrawi, Arian Abbasi
Harnesses, the products that run AI coding agents, are multiplying, and enterprises are rolling them out to their employees. What started as pilots with a few hundred seats is now scaling to tens of thousands. Enterprises rarely build these harnesses and usually adopt the ones model vendors bundle with their seats and APIs, such as Anthropic's Claude Code or OpenAI's Codex. A harness decides which model answers, what the model reads, how the prompt cache is used and which subagents run, so it picks the rate on the price sheet and sets the volume bought at it. Enterprises that keep a proprietary or untuned harness at its defaults inherit these choices and their bill. We build a fast, customisable router in which Jev, a classifier with calibrated probabilities, labels every prompt against a bring-your-own taxonomy of agentic requests. One user turn is many requests over a prompt cache that belongs to one model, so the router moves work only where no running conversation has to rebuild its cache. It routes at session start, in side lanes and at subagent launch. From the price sheet we derive when a mid-task switch pays back. We also find a crossover, in which the highest-priced…
PDF arXiv
arXiv cs.DCSep 24
When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
Yiyu Liu, Minlan Yu, Juncheng Yang
Long-running LLM applications repeatedly send growing context, making prefix caching critical for reducing prefill cost. Yet prefix-cache behavior under agentic workloads remains poorly understood. We study production traces from two companies and evaluate 14 eviction algorithms across HBM-constrained and large memory-pool settings. Despite a large gap to Belady, sophisticated policies designed for traditional caches provide little benefit over LRU. The reason is structural: prefix reuse is dominated by the regular pacing of active sessions, making recency unusually predictive. Prefix caching nevertheless introduces new challenges, including heavy-tailed session footprints and highly variable miss costs as attention computation grows with sequence length. We introduce the compute-savings ratio and two offline oracles to quantify these effects. Our results show that effective prefix-cache management should retain recency as its foundation while selectively adding quick demotion for one-hit prefixes, compute-aware partial eviction for expensive misses, and capacity-dependent eviction granularity. We will release the traces and simulator to support future research.
PDF arXiv
arXiv cs.AISep 24
Neurosymbolic Routing for Reliable Reasoning on Resource-Constrained Edge Devices
Avyay Sadhu, Alvaro Velasquez, Lekai Chen
Running a language model on edge hardware provides private and low-latency reasoning without a network connection, and yet the small models that fit on such devices are unreliable on the tasks computers are expected to handle well, such as arithmetic, algebra, and formal logic problems. We argue that much of this unreliability is avoidable. Many queries appearing to demand reasoning are in fact structurally deterministic and permit fast and exact symbolic solutions. Therefore, forcing a probabilistic model to approximate them sacrifices accuracy and energy for little benefit. We present a neurosymbolic router that classifies each incoming query and dispatches it to the cheapest correct solver, sending structured tasks to deterministic engines and reserving the small language model (SLM) for open-ended word problems. Instead of hand-coding the routing logic, we learn a deterministic finite automaton (DFA) with the L* grammatical inference algorithm, using the SLM as a membership oracle and labeled data as an equivalence oracle. On a Raspberry Pi 4B (8 GB RAM, no GPU), evaluated on 100 untested prompts from DeepMind Mathematics, GSM8K, and RuleTaker, learned routing attains 100%…
PDF arXiv
arXiv cond-mat.dis-nnSep 23
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Qucheng Gao, Zuyi Yang, Xiao Chen
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $γ$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $β$. Attention condensation instead emerges in the scaling regime $β\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a…
PDF arXiv
arXiv cs.LGSep 23
Learning Holographic Reduced Representations with Clifford Variational Autoencoders
Mohamed Malek Abid, P. Michael Furlong
Vector Symbolic Algebras project data structures into a hyperdimensional vector space through the application of their vector algebras to randomly generated atomic vector symbols and fractional power encodings of real-valued data. Embedding unstructured data remains an open question. We present \textit{Clifford-VAE}, a variational autoencoder that learns to project data onto a Clifford torus in arbitrary dimensions. Experiments using the MNIST, FashionMNIST, and CIFAR-10 datasets demonstrate that Clifford-VAE produces representations that are competitive with those produced by Gaussian and Hyperspherical VAEs for semi-supervised classification tasks while outperforming Gaussian and Hyperspherical counterparts in the VSA benchmark tests of self-binding and unbinding, role-filler recovery, and bundle capacity. Clifford-VAE provides a principled technique for grounding perceptual data into a symbolic reasoning framework, providing a new approach to a long-standing problem in the VSA literature.
PDF arXiv
arXiv cs.LGCodeSep 23
Memory Attention
Jiale Kang
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.
PDF arXiv Code

Papers

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
InevitableStage not given · Oct 26$16.8M
AilyticsSeries A · Oct 26$7.4M
Valor EnergyStage not given · Oct 26$5M
Assembler AISeed · Oct 26$4M

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers26 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
kv cache compression · kv cache · kv caches267 papers20 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers16 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers16 this week

Recent papers

All papers in Research

arXiv cs.AISep 24
Harness Tokenomics: A Router for the Enterprise Agentic Control Plane
Ted Kwartler, Alan Aqrawi, Arian Abbasi
Harnesses, the products that run AI coding agents, are multiplying, and enterprises are rolling them out to their employees. What started as pilots with a few hundred seats is now scaling to tens of thousands. Enterprises rarely build these harnesses and usually adopt the ones model vendors bundle with their seats and APIs, such as Anthropic's Claude Code or OpenAI's Codex. A harness decides which model answers, what the model reads, how the prompt cache is used and which subagents run, so it picks the rate on the price sheet and sets the volume bought at it. Enterprises that keep a proprietary or untuned harness at its defaults inherit these choices and their bill. We build a fast, customisable router in which Jev, a classifier with calibrated probabilities, labels every prompt against a bring-your-own taxonomy of agentic requests. One user turn is many requests over a prompt cache that belongs to one model, so the router moves work only where no running conversation has to rebuild its cache. It routes at session start, in side lanes and at subagent launch. From the price sheet we derive when a mid-task switch pays back. We also find a crossover, in which the highest-priced…
PDF arXiv
arXiv cs.DCSep 24
When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
Yiyu Liu, Minlan Yu, Juncheng Yang
Long-running LLM applications repeatedly send growing context, making prefix caching critical for reducing prefill cost. Yet prefix-cache behavior under agentic workloads remains poorly understood. We study production traces from two companies and evaluate 14 eviction algorithms across HBM-constrained and large memory-pool settings. Despite a large gap to Belady, sophisticated policies designed for traditional caches provide little benefit over LRU. The reason is structural: prefix reuse is dominated by the regular pacing of active sessions, making recency unusually predictive. Prefix caching nevertheless introduces new challenges, including heavy-tailed session footprints and highly variable miss costs as attention computation grows with sequence length. We introduce the compute-savings ratio and two offline oracles to quantify these effects. Our results show that effective prefix-cache management should retain recency as its foundation while selectively adding quick demotion for one-hit prefixes, compute-aware partial eviction for expensive misses, and capacity-dependent eviction granularity. We will release the traces and simulator to support future research.
PDF arXiv
arXiv cs.AISep 24
Neurosymbolic Routing for Reliable Reasoning on Resource-Constrained Edge Devices
Avyay Sadhu, Alvaro Velasquez, Lekai Chen
Running a language model on edge hardware provides private and low-latency reasoning without a network connection, and yet the small models that fit on such devices are unreliable on the tasks computers are expected to handle well, such as arithmetic, algebra, and formal logic problems. We argue that much of this unreliability is avoidable. Many queries appearing to demand reasoning are in fact structurally deterministic and permit fast and exact symbolic solutions. Therefore, forcing a probabilistic model to approximate them sacrifices accuracy and energy for little benefit. We present a neurosymbolic router that classifies each incoming query and dispatches it to the cheapest correct solver, sending structured tasks to deterministic engines and reserving the small language model (SLM) for open-ended word problems. Instead of hand-coding the routing logic, we learn a deterministic finite automaton (DFA) with the L* grammatical inference algorithm, using the SLM as a membership oracle and labeled data as an equivalence oracle. On a Raspberry Pi 4B (8 GB RAM, no GPU), evaluated on 100 untested prompts from DeepMind Mathematics, GSM8K, and RuleTaker, learned routing attains 100%…
PDF arXiv
arXiv cond-mat.dis-nnSep 23
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Qucheng Gao, Zuyi Yang, Xiao Chen
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $γ$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $β$. Attention condensation instead emerges in the scaling regime $β\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a…
PDF arXiv
arXiv cs.LGSep 23
Learning Holographic Reduced Representations with Clifford Variational Autoencoders
Mohamed Malek Abid, P. Michael Furlong
Vector Symbolic Algebras project data structures into a hyperdimensional vector space through the application of their vector algebras to randomly generated atomic vector symbols and fractional power encodings of real-valued data. Embedding unstructured data remains an open question. We present \textit{Clifford-VAE}, a variational autoencoder that learns to project data onto a Clifford torus in arbitrary dimensions. Experiments using the MNIST, FashionMNIST, and CIFAR-10 datasets demonstrate that Clifford-VAE produces representations that are competitive with those produced by Gaussian and Hyperspherical VAEs for semi-supervised classification tasks while outperforming Gaussian and Hyperspherical counterparts in the VSA benchmark tests of self-binding and unbinding, role-filler recovery, and bundle capacity. Clifford-VAE provides a principled technique for grounding perceptual data into a symbolic reasoning framework, providing a new approach to a long-standing problem in the VSA literature.
PDF arXiv
arXiv cs.LGCodeSep 23
Memory Attention
Jiale Kang
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.
PDF arXiv Code

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Activity · 12 weeks

Fields · AI · updated 16m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

202papers this week−39.7% vs last week

All papers · 1.9k With code · 67

Newest Most relevant

1.6k papers this month

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.