Papers

Field

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantization in RL training and rollout paths can diverge, reducing the effectiveness of low-precision methods.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not adequately evaluate System One models on graph tasks, limiting progress in this area.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Error propagation in low-rank weight factorization1 paper raises it · 1 addresses it
Local criteria for subspace selection ignore global error propagation, leading to performance collapse at high compression.
- Learning Functional Subspaces for Neural Network Compressionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

gradient descent · learning rate · neural networks233 papers41 this week
graph learning · graph neural · graph neural networks269 papers31 this week
neural networks · neural network · arnold networks155 papers30 this week
memory efficient · optimizers · low rank adaptation164 papers26 this week
bit quantization · quantization aware training · quantization aware158 papers18 this week
linear attention · softmax attention · transformer attention108 papers11 this week

Topics in this field

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 29
Traversing the solution space of neural networks with Hessian Null Space Continuation
Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman, Leo Kozachkov +1
On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network…
PDF arXiv
arXiv cs.LGSep 29
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset…
PDF arXiv
arXiv cs.LGSep 29
Kolmogorov-Arnold Classifier Systems as Universal Approximators
Hiroki Shiraishi, Hisao Ishibuchi, Masaya Nakata
As the input dimension $n$ grows, rule-based machine learning, such as Learning Classifier Systems (LCSs), faces a fundamental scalability bottleneck for function approximation: both rule count and parameter count grow exponentially with $n$. Traditional LCSs partition the $n$-dimensional input space directly, requiring $\mathcal{O}(m^n)$ rules for adequate coverage, where $m$ is the per-variable resolution. This article breaks from this paradigm by reorganizing rules dimension-wise, guided by the Kolmogorov-Arnold representation theorem: any continuous $n$-dimensional function can be expressed as a finite superposition of one-dimensional functions. The proposed Kolmogorov-Arnold Classifier System (KACS) decomposes the target function into one-dimensional subproblems and assigns a dedicated ruleset to each, reducing the worst-case rule count from $\mathcal{O}(m^n)$ to $\mathcal{O}(mn^2)$ and replacing $n$-dimensional local models with one-dimensional models requiring only two parameters per rule, independent of $n$. We also provide the first constructive proof that an LCS, namely KACS, is a universal approximator for continuous functions on compact domains. Evaluated against a…
PDF arXiv
arXiv cs.LGSep 29
Pattern Formation in Transformers
Erkan Turan, Gaspard Abel, Maks Ovsjanikov
What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems perspective, but relies on simplified architectural assumptions, and does not explain the rich structures observed in practice. This leaves a major open question: when a full Transformer escapes rank collapse, how does it structure token representations? Using pattern-formation theory, we show that the dynamical view of Transformers can account for Positional Encoding, Multi-Head Attention, and Output-Value geometry. We demonstrate that a full Transformer architecture imposes an inductive prior by selectively amplifying a rich set of previously unreported patterns, including traveling or rotating waves among others. We characterize the role of each architectural component in controlling which pattern is amplified, which ones stabilize, compete, or coexist. Finally, we show that these structures can act as a controllable dynamical prior that facilitates learning. By…
PDF arXiv
arXiv cs.LGSep 29
Behavioral Capacity Certificates for Quantized Language Models
Arian Eamaz, Mojtaba Soltanalian
Activation and key-value cache precision change what a quantized language model computes without altering its stored weights. Direct weight-code bounds, however, assign identical complexity to deployments that behave differently and charge separately for weight codes that behave identically. Behavioral Capacity Certificates (BCC) charge for behavior using the aggregate prior mass of complete implementations---weights, scales, activation and cache rules---that induce the same bounded loss. When quantization merges implementations, this shared mass lowers the complexity penalty, and a break-even law determines when the saving survives the cost of validating it. BCC supports a three-step deployment workflow, and our experiments verify each step. First, a forward-only screen shortlists per-layer bit-widths by how often candidate perturbations preserve the reference predictions, with quality comparable to Hessian-guided selection at lower preprocessing cost. Second, margin-certified cells identify weights that can be pruned or sign-flipped without changing the deployed behavior: every permitted combination preserves all declared predictions, and on OLMoE-1B-7B and SmolLM2-1.7B,…
PDF arXiv
arXiv cs.LGSep 29
TopoEmbedX: A General Framework for Representation Learning on Topological Domains
Florian Frantzen, Ibrahem AlJabea, Ines Henriques-Cadby, Theodore Papamarkou, Mustafa Hajij +1
Topological structures such as simplicial complexes, hypergraphs, and cell complexes extend standard graph models by modeling higher-order relationships. These structures appear in many modern datasets and require specialized methods for generating meaningful embeddings. In this paper, we introduce TopoEmbedX, a unified framework for embedding a wide range of topological domains into Euclidean spaces. The package brings together several existing topological embedding algorithms---DeepCell, Cell2Vec, CellDiff2Vec, HOLE, and HOGLEE---and introduces five new algorithms: ComplexNetMF, ComplexRep, ComplexRandNE, ComplexWalklets, and ComplexHeat. These algorithms extend well-known graph embedding techniques to higher-order settings using the augmented Hasse graph of a topological domain. TopoEmbedX provides a clear, consistent, and easy-to-use framework for topological representation learning. Experiments show that the embeddings generated by TopoEmbedX support tasks such as classification and regression across multidimensional data.
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks233 papers41 this week
graph learning · graph neural · graph neural networks269 papers31 this week
neural networks · neural network · arnold networks155 papers30 this week
memory efficient · optimizers · low rank adaptation164 papers26 this week
bit quantization · quantization aware training · quantization aware158 papers18 this week
linear attention · softmax attention · transformer attention108 papers11 this week

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Recent papers

All papers in Research

arXiv cs.LGSep 29
Traversing the solution space of neural networks with Hessian Null Space Continuation
Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman, Leo Kozachkov +1
On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network…
PDF arXiv
arXiv cs.LGSep 29
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset…
PDF arXiv
arXiv cs.LGSep 29
Kolmogorov-Arnold Classifier Systems as Universal Approximators
Hiroki Shiraishi, Hisao Ishibuchi, Masaya Nakata
As the input dimension $n$ grows, rule-based machine learning, such as Learning Classifier Systems (LCSs), faces a fundamental scalability bottleneck for function approximation: both rule count and parameter count grow exponentially with $n$. Traditional LCSs partition the $n$-dimensional input space directly, requiring $\mathcal{O}(m^n)$ rules for adequate coverage, where $m$ is the per-variable resolution. This article breaks from this paradigm by reorganizing rules dimension-wise, guided by the Kolmogorov-Arnold representation theorem: any continuous $n$-dimensional function can be expressed as a finite superposition of one-dimensional functions. The proposed Kolmogorov-Arnold Classifier System (KACS) decomposes the target function into one-dimensional subproblems and assigns a dedicated ruleset to each, reducing the worst-case rule count from $\mathcal{O}(m^n)$ to $\mathcal{O}(mn^2)$ and replacing $n$-dimensional local models with one-dimensional models requiring only two parameters per rule, independent of $n$. We also provide the first constructive proof that an LCS, namely KACS, is a universal approximator for continuous functions on compact domains. Evaluated against a…
PDF arXiv
arXiv cs.LGSep 29
Pattern Formation in Transformers
Erkan Turan, Gaspard Abel, Maks Ovsjanikov
What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems perspective, but relies on simplified architectural assumptions, and does not explain the rich structures observed in practice. This leaves a major open question: when a full Transformer escapes rank collapse, how does it structure token representations? Using pattern-formation theory, we show that the dynamical view of Transformers can account for Positional Encoding, Multi-Head Attention, and Output-Value geometry. We demonstrate that a full Transformer architecture imposes an inductive prior by selectively amplifying a rich set of previously unreported patterns, including traveling or rotating waves among others. We characterize the role of each architectural component in controlling which pattern is amplified, which ones stabilize, compete, or coexist. Finally, we show that these structures can act as a controllable dynamical prior that facilitates learning. By…
PDF arXiv
arXiv cs.LGSep 29
Behavioral Capacity Certificates for Quantized Language Models
Arian Eamaz, Mojtaba Soltanalian
Activation and key-value cache precision change what a quantized language model computes without altering its stored weights. Direct weight-code bounds, however, assign identical complexity to deployments that behave differently and charge separately for weight codes that behave identically. Behavioral Capacity Certificates (BCC) charge for behavior using the aggregate prior mass of complete implementations---weights, scales, activation and cache rules---that induce the same bounded loss. When quantization merges implementations, this shared mass lowers the complexity penalty, and a break-even law determines when the saving survives the cost of validating it. BCC supports a three-step deployment workflow, and our experiments verify each step. First, a forward-only screen shortlists per-layer bit-widths by how often candidate perturbations preserve the reference predictions, with quality comparable to Hessian-guided selection at lower preprocessing cost. Second, margin-certified cells identify weights that can be pruned or sign-flipped without changing the deployed behavior: every permitted combination preserves all declared predictions, and on OLMoE-1B-7B and SmolLM2-1.7B,…
PDF arXiv
arXiv cs.LGSep 29
TopoEmbedX: A General Framework for Representation Learning on Topological Domains
Florian Frantzen, Ibrahem AlJabea, Ines Henriques-Cadby, Theodore Papamarkou, Mustafa Hajij +1
Topological structures such as simplicial complexes, hypergraphs, and cell complexes extend standard graph models by modeling higher-order relationships. These structures appear in many modern datasets and require specialized methods for generating meaningful embeddings. In this paper, we introduce TopoEmbedX, a unified framework for embedding a wide range of topological domains into Euclidean spaces. The package brings together several existing topological embedding algorithms---DeepCell, Cell2Vec, CellDiff2Vec, HOLE, and HOGLEE---and introduces five new algorithms: ComplexNetMF, ComplexRep, ComplexRandNE, ComplexWalklets, and ComplexHeat. These algorithms extend well-known graph embedding techniques to higher-order settings using the augmented Hasse graph of a topological domain. TopoEmbedX provides a clear, consistent, and easy-to-use framework for topological representation learning. Experiments show that the embeddings generated by TopoEmbedX support tasks such as classification and regression across multidimensional data.
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 44m ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

157papers this week−29.9% vs last week

All papers · 1.1k With code · 11

Newest Most relevant

1k papers this month

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantization in RL training and rollout paths can diverge, reducing the effectiveness of low-precision methods.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not adequately evaluate System One models on graph tasks, limiting progress in this area.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Error propagation in low-rank weight factorization1 paper raises it · 1 addresses it
Local criteria for subspace selection ignore global error propagation, leading to performance collapse at high compression.
- Learning Functional Subspaces for Neural Network Compressionraises · addresses

Named by an AI from these papers' abstracts.