Papers

Field

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
Open WebUI3 releases this week
mlx-lm+8% downloads · 140.9k/wk+7.5%

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use relevant information without exceeding finite context windows.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when they are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Recently funded

Topics in this field

agent memory · memory construction · term memory248 papers53 this week
gpus · gpu · throughput212 papers32 this week
diffusion language models · language models diffusion · models diffusion language145 papers27 this week
sparse attention · linear attention · attention layers131 papers22 this week
continual learning · catastrophic forgetting · forgetting230 papers21 this week
aware routing · model routing · model llm routing82 papers20 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLOct 19
Answering While You Type: Optimizing Time-to-Answer with Streaming Generation
Nikita Dragunov, Matvey Mikhalchuk, Anton Razzhigaev
Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete. The quantity a user experiences, however, is the interval between submitting a request and receiving a usable response, and that interval can be shortened without making decoding itself faster. While a request is being composed it is already partially specified, yet conventional deployments begin no computation until submission. We propose and study streaming generation, a decoding regime in which the model begins producing a response from an incomplete request and revises it as the remaining input arrives, so that only a small residue of the answer remains to be computed once the request is submitted. We evaluate the regime on autoregressive and diffusion language models, characterise the conditions under which it reduces user-visible latency, and analyse the mechanism responsible.
PDF arXiv
arXiv cs.AIOct 19
CoRS: Fast Neural Flexible Job-Shop Scheduling via Composite Interval State Reuse
Jiang Sutong, Zhao Peng, Wu Xuan, Wang Yan, Wang Jiaqi +4
Neural solvers for the Flexible Job-Shop Scheduling Problem (FJSP) typically refresh graph representations and update feasibility after every operation--machine decision. However, only feasibility must remain exact at every step, whereas graph encoding is substantially more expensive and can potentially be refreshed less frequently. We present CoRS, which decouples them: an interval-level graph representation is reused across multiple sequential decisions, while a stepwise feasibility state is updated after every action. This two-timescale design amortizes graph encoding without parallelizing action generation or relaxing scheduling constraints. CoRS further develops a feasibility-preserving composite-interval decoding scheme with dynamic constraint tracking and boundary state synchronization, which reduces representation refresh frequency while maintaining continuous feasible schedule construction. Furthermore, to alleviate representation staleness caused by interval-level state reuse, CoRS introduces enhanced static features as a compensation mechanism. By providing more comprehensive structural information of the scheduling problem, these features improve the model's…
PDF arXiv
arXiv cs.LGOct 19
Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks
Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-García
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves…
PDF arXiv
arXiv cs.IROct 19
FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
Yifan Wang, Yifei Du
Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a…
PDF arXiv
arXiv cs.CLOct 19
Balance-Aware Pivot Selection for Parallel Diffusion LLM Decoding
Yejin Lee, Hyundong Jin, Yo-Sub Han
Diffusion language models enable parallel token generation, offering a promising alternative to sequential autoregressive decoding. Existing acceleration methods use predictive confidence to enable parallel token commitment while preserving generation quality. However, confidence-based selection prioritizes reliable token predictions without explicitly considering how token positions shape the remaining decoding schedule. Moreover, such ordering does not always reflect inter-token dependencies, suggesting that confidence need not be the sole criterion for token selection. Motivated by this limitation, we propose Balance-Aware Pivot Decoding (BPD), which favors central pivots within contiguous masked intervals and decodes the selected positions in parallel. Balanced pivots yield logarithmic decoding depth in the interval length. Our practical method accounts for model uncertainty by combining confidence-based prefilling with confidence-thresholded center selection. We evaluate the resulting quality-efficiency tradeoff on LLaDA and Dream across five mathematical reasoning, code generation, and instruction-following benchmarks. Our implementation is available at:…
PDF arXiv
arXiv cs.LGOct 19
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Xuyuan Xiong, Siyu Chen, Zihan Shao, Yue Su, Weiyang Jin +3
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, although this may weaken the benefit of looping. To study this tradeoff systematically, we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower loss than the compute-optimal non-looped model with about 27% fewer…
PDF arXiv

Papers

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
BentoML+30% downloads · 41k/wk+29.7%
MLX+26% downloads · 342.6k/wk+25.5%
Open WebUI3 releases this week
mlx-lm+8% downloads · 140.9k/wk+7.5%

Topics in this field

agent memory · memory construction · term memory248 papers53 this week
gpus · gpu · throughput212 papers32 this week
diffusion language models · language models diffusion · models diffusion language145 papers27 this week
sparse attention · linear attention · attention layers131 papers22 this week
continual learning · catastrophic forgetting · forgetting230 papers21 this week
aware routing · model routing · model llm routing82 papers20 this week

Recently funded

Market

EMode PhotonixPre-seed · Oct 16$1M
MueonSeries A · Oct 15$65M
TrimagnetixSeed · Oct 15$1M
ModalSeries B+ · Oct 14$750M
Salience LabsSeries B+ · Oct 14$125M

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CLOct 19
Answering While You Type: Optimizing Time-to-Answer with Streaming Generation
Nikita Dragunov, Matvey Mikhalchuk, Anton Razzhigaev
Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete. The quantity a user experiences, however, is the interval between submitting a request and receiving a usable response, and that interval can be shortened without making decoding itself faster. While a request is being composed it is already partially specified, yet conventional deployments begin no computation until submission. We propose and study streaming generation, a decoding regime in which the model begins producing a response from an incomplete request and revises it as the remaining input arrives, so that only a small residue of the answer remains to be computed once the request is submitted. We evaluate the regime on autoregressive and diffusion language models, characterise the conditions under which it reduces user-visible latency, and analyse the mechanism responsible.
PDF arXiv
arXiv cs.AIOct 19
CoRS: Fast Neural Flexible Job-Shop Scheduling via Composite Interval State Reuse
Jiang Sutong, Zhao Peng, Wu Xuan, Wang Yan, Wang Jiaqi +4
Neural solvers for the Flexible Job-Shop Scheduling Problem (FJSP) typically refresh graph representations and update feasibility after every operation--machine decision. However, only feasibility must remain exact at every step, whereas graph encoding is substantially more expensive and can potentially be refreshed less frequently. We present CoRS, which decouples them: an interval-level graph representation is reused across multiple sequential decisions, while a stepwise feasibility state is updated after every action. This two-timescale design amortizes graph encoding without parallelizing action generation or relaxing scheduling constraints. CoRS further develops a feasibility-preserving composite-interval decoding scheme with dynamic constraint tracking and boundary state synchronization, which reduces representation refresh frequency while maintaining continuous feasible schedule construction. Furthermore, to alleviate representation staleness caused by interval-level state reuse, CoRS introduces enhanced static features as a compensation mechanism. By providing more comprehensive structural information of the scheduling problem, these features improve the model's…
PDF arXiv
arXiv cs.LGOct 19
Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks
Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-García
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves…
PDF arXiv
arXiv cs.IROct 19
FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
Yifan Wang, Yifei Du
Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a…
PDF arXiv
arXiv cs.CLOct 19
Balance-Aware Pivot Selection for Parallel Diffusion LLM Decoding
Yejin Lee, Hyundong Jin, Yo-Sub Han
Diffusion language models enable parallel token generation, offering a promising alternative to sequential autoregressive decoding. Existing acceleration methods use predictive confidence to enable parallel token commitment while preserving generation quality. However, confidence-based selection prioritizes reliable token predictions without explicitly considering how token positions shape the remaining decoding schedule. Moreover, such ordering does not always reflect inter-token dependencies, suggesting that confidence need not be the sole criterion for token selection. Motivated by this limitation, we propose Balance-Aware Pivot Decoding (BPD), which favors central pivots within contiguous masked intervals and decodes the selected positions in parallel. Balanced pivots yield logarithmic decoding depth in the interval length. Our practical method accounts for model uncertainty by combining confidence-based prefilling with confidence-thresholded center selection. We evaluate the resulting quality-efficiency tradeoff on LLaDA and Dream across five mathematical reasoning, code generation, and instruction-following benchmarks. Our implementation is available at:…
PDF arXiv
arXiv cs.LGOct 19
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Xuyuan Xiong, Siyu Chen, Zihan Shao, Yue Su, Weiyang Jin +3
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, although this may weaken the benefit of looping. To study this tradeoff systematically, we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower loss than the compute-optimal non-looped model with about 27% fewer…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 51m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

247papers this week−18.5% vs last week

All papers · 1.5k With code · 60

Newest Most relevant

1.4k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 3 address it
Long-horizon agents need to retain and use relevant information without exceeding finite context windows.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
Token-level LLM routing efficiency1 paper raises it · 1 addresses it
Fine-grained token-level routing improves cost-quality trade-offs but strains existing serving systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Memory-induced sycophancy in agents1 paper raises it · 1 addresses it
Persistent memories can cause agents to over-align with user beliefs, even when they are incorrect or outdated.
- Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancyraises · addresses
Quality gap in parallel token generation2 papers raise it · 2 address it
Diffusion language models generate tokens in parallel but lag behind autoregressive models in quality.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Modelsraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces computation but still requires scoring all keys, limiting efficiency gains.
- DoSA: Dots Sparse Attentionraises · addresses

Named by an AI from these papers' abstracts.