Papers

Field

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.

Recently funded

Topics in this field

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.

Recently funded

Topics in this field

gradient descent · learning rate · neural networks221 papers37 this week
graph learning · graph neural · graph neural networks263 papers32 this week
neural networks · neural network · arnold networks149 papers30 this week
memory efficient · optimizers · low rank adaptation157 papers24 this week
bit quantization · quantization aware training · quantization aware152 papers18 this week
linear attention · softmax attention · transformer attention106 papers12 this week

Topics in this field

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Recent papers

All in Research →

arXiv cs.LGOct 13
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Petar Basta
While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x…
PDF arXiv
arXiv cs.LGOct 12
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov, Dmitrii Vlashchuk, Vladimir Aletov +5
Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization but its dense polar update remains expensive. We propose Monarch-Muon, a structured matrix-aware optimizer for weights parameterized by Monarch factors. The resulting structure reduces weights, gradients, and optimizer state, and exposes small independent update subproblems in place of a single dense orthogonalization; we formalize this through a blockwise trust-region interpretation. In quality runs, Monarch-Muon improves over Monarch-AdamW at fixed structure: 3.06 vs. 3.09 in the dense 257M model and 2.68 vs. 2.77 in the 1B MoE model; the two-block MoE variant also matches dense AdamW while using 15% fewer active and 25.4% fewer total parameters. In 6.89B-parameter systems measurements, Monarch-Muon reduces peak memory from 68.9 to 52.9 GB, Muon optimizer state from 14.60 to 6.77 GB, and the Muon optimizer step from 1288 to 304 ms.
PDF arXiv
arXiv cs.LGOct 12
When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training
Mohammad Pasande, Mateusz Jaworski, Jeroen Rook, Elena Mocanu
Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity ($\ge 99\%$), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to $+14.4$ points at $99.9\%$ sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at $99.99\%$ sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available…
PDF arXiv
arXiv cs.LGOct 12
On Making $\mathrm{SE}(2)$-Invariant Networks Optimal
Tomas Karella, Emily Shinkle, Alice Elisabeth Anastasia Allen, Pieter J. Swart, Nicholas Lubbers +1
Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional neural networks are a canonical example, encoding translation equivariance. Extending this inductive bias to richer symmetry groups such as $\mathrm{SE}(2)$ in the most effective way possible is an ongoing architectural and computational challenge. Building on classical invariant theory, we derive invariant feature maps that are provably optimal: minimal and information-preserving with respect to the symmetry group. Our construction is designed to integrate seamlessly into standard architectures, requiring no modification to nonlinearities or normalization layers. As a concrete implementation, we introduce a plug-and-play module for convolutional neural networks that enforces $\mathrm{SE}(2)$ invariance, capturing both translations and rotations within standard architectures. The resulting models are both efficient and expressive. Experiments show competitive or improved accuracy compared to state-of-the-art equivariant methods at significantly lower computational cost. Finally, we provide a theoretical and empirical analysis of the relationship…
PDF arXiv
arXiv cs.LGOct 12
Measure Neural Mappings
Ricardo Baptista, Edoardo Calvello, Bohan Chen, Andrew Stuart, Urbain Vaes
We introduce and define measure neural mappings (MNMs), which combine measure-dependent neural transport with ensemble consistency for learning operators on probability measures. Ensemble consistency requires outputs from growing ensembles sampled from an input law to converge to the specified population output in probability. We establish quantitative continuum limits for attention on noncompact laws under strong sub-Gaussian and tail-integrability assumptions. These laws accommodate unbounded state spaces in scientific machine learning, including Bayesian inference and uncertainty quantification. For fixed self-attention transformers, the expected output Wasserstein error is bounded by the empirical input Wasserstein error plus an additional $N^{-1/2}$ attention term. These self-attention transformers are MNMs under the stated assumptions: pre-norm architectures follow from compact stability after normalization, while post-norm architectures are covered under the stated tail conditions. We prove corresponding cross-attention estimates and extend the results to multihead attention and grouped transformer evaluation, with convergence as the group size grows. For Set Transformers…
PDF arXiv
arXiv cs.LGOct 12
On the Statistical Mechanics of Attention
Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks221 papers37 this week
graph learning · graph neural · graph neural networks263 papers32 this week
neural networks · neural network · arnold networks149 papers30 this week
memory efficient · optimizers · low rank adaptation157 papers24 this week
bit quantization · quantization aware training · quantization aware152 papers18 this week
linear attention · softmax attention · transformer attention106 papers12 this week

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Recent papers

All in Research →

arXiv cs.LGOct 13
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Petar Basta
While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x…
PDF arXiv
arXiv cs.LGOct 12
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov, Dmitrii Vlashchuk, Vladimir Aletov +5
Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization but its dense polar update remains expensive. We propose Monarch-Muon, a structured matrix-aware optimizer for weights parameterized by Monarch factors. The resulting structure reduces weights, gradients, and optimizer state, and exposes small independent update subproblems in place of a single dense orthogonalization; we formalize this through a blockwise trust-region interpretation. In quality runs, Monarch-Muon improves over Monarch-AdamW at fixed structure: 3.06 vs. 3.09 in the dense 257M model and 2.68 vs. 2.77 in the 1B MoE model; the two-block MoE variant also matches dense AdamW while using 15% fewer active and 25.4% fewer total parameters. In 6.89B-parameter systems measurements, Monarch-Muon reduces peak memory from 68.9 to 52.9 GB, Muon optimizer state from 14.60 to 6.77 GB, and the Muon optimizer step from 1288 to 304 ms.
PDF arXiv
arXiv cs.LGOct 12
When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training
Mohammad Pasande, Mateusz Jaworski, Jeroen Rook, Elena Mocanu
Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity ($\ge 99\%$), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to $+14.4$ points at $99.9\%$ sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at $99.99\%$ sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available…
PDF arXiv
arXiv cs.LGOct 12
On Making $\mathrm{SE}(2)$-Invariant Networks Optimal
Tomas Karella, Emily Shinkle, Alice Elisabeth Anastasia Allen, Pieter J. Swart, Nicholas Lubbers +1
Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional neural networks are a canonical example, encoding translation equivariance. Extending this inductive bias to richer symmetry groups such as $\mathrm{SE}(2)$ in the most effective way possible is an ongoing architectural and computational challenge. Building on classical invariant theory, we derive invariant feature maps that are provably optimal: minimal and information-preserving with respect to the symmetry group. Our construction is designed to integrate seamlessly into standard architectures, requiring no modification to nonlinearities or normalization layers. As a concrete implementation, we introduce a plug-and-play module for convolutional neural networks that enforces $\mathrm{SE}(2)$ invariance, capturing both translations and rotations within standard architectures. The resulting models are both efficient and expressive. Experiments show competitive or improved accuracy compared to state-of-the-art equivariant methods at significantly lower computational cost. Finally, we provide a theoretical and empirical analysis of the relationship…
PDF arXiv
arXiv cs.LGOct 12
Measure Neural Mappings
Ricardo Baptista, Edoardo Calvello, Bohan Chen, Andrew Stuart, Urbain Vaes
We introduce and define measure neural mappings (MNMs), which combine measure-dependent neural transport with ensemble consistency for learning operators on probability measures. Ensemble consistency requires outputs from growing ensembles sampled from an input law to converge to the specified population output in probability. We establish quantitative continuum limits for attention on noncompact laws under strong sub-Gaussian and tail-integrability assumptions. These laws accommodate unbounded state spaces in scientific machine learning, including Bayesian inference and uncertainty quantification. For fixed self-attention transformers, the expected output Wasserstein error is bounded by the empirical input Wasserstein error plus an additional $N^{-1/2}$ attention term. These self-attention transformers are MNMs under the stated assumptions: pre-norm architectures follow from compact stability after normalization, while post-norm architectures are covered under the stated tail conditions. We prove corresponding cross-attention estimates and extend the results to multihead attention and grouped transformer evaluation, with convergence as the group size grows. For Set Transformers…
PDF arXiv
arXiv cs.LGOct 12
On the Statistical Mechanics of Attention
Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
PDF arXiv

Fields · AI · updated 22m ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

153papers this week−32.3% vs last week

All papers · 1k With code · 11

Newest Most relevant

1k papers this month