Papers

Field

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantization in RL training and rollout paths can diverge, reducing the effectiveness of low-precision methods.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not adequately evaluate System One models on graph tasks, limiting progress in this area.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Error propagation in low-rank weight factorization1 paper raises it · 1 addresses it
Local criteria for subspace selection ignore global error propagation, leading to performance collapse at high compression.
- Learning Functional Subspaces for Neural Network Compressionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

gradient descent · learning rate · neural networks233 papers46 this week
graph learning · graph neural · graph neural networks269 papers35 this week
neural networks · neural network · arnold networks155 papers33 this week
memory efficient · optimizers · low rank adaptation164 papers30 this week
bit quantization · quantization aware training · quantization aware158 papers24 this week
linear attention · softmax attention · transformer attention108 papers12 this week

Topics in this field

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 3
DASH: Fast, Valid Counterfactuals for Deep Networks via Batched Directional Search
Shraman Pal, Gabriel Medeiros, Clayton Escouper das Chagas, Can Li
Counterfactual explanations are most useful when they can be generated with low latency, remain close to the factual input, and satisfy input-domain, categorical, and actionability constraints. Achieving these objectives simultaneously is challenging for deep neural networks. Heuristic methods are often fast but may return invalid counterfactuals, whereas exact methods can certify global proximity but may not finish within practical time limits. We introduce DASH, a batched heuristic search method for finding close, valid, and actionable counterfactuals for deep neural networks under $\ell_1$, $\ell_2$, and $\ell_\infty$ objectives. DASH uses directional Lipschitz bounds and local affine models to generate anchors, then ranks and expands promising regions with batched network evaluations. We compare DASH against nine prior heuristic methods and time-limited exact mixed-integer baselines on four tabular datasets, with network depths from 2 to 32, and evaluate scalability on PBMC3k. Across 9,000 tabular query-norm cases, DASH returns a valid counterfactual within $5\%$ of the best heuristic-observed valid distance in $94.6\%$ of cases, with a median CPU search runtime of $0.061$ s.…
PDF arXiv
arXiv cs.CVOct 3
Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models
Bangwei Guo, Xujiang Zhao, Shengyu Chen, Yanchi Liu, Wei Cheng +4
Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller…
PDF arXiv
arXiv cs.CLOct 3
WNet: Discrete Wavelets Transform for Efficient Token Mixing
Rana Aref Salama, Abdou Youssef, Mona Diab
In a Transformer, token mixing is the step that lets each token draw information from other tokens, and it dominates the cost of encoding long sequences. Self-attention does this mixing very well: every token weighs every other token by content, which gives strong contextual modeling. That all-pairs comparison is also why its cost grows quadratically with sequence length. We introduce WNet, a Transformer encoder that replaces self-attention with token mixing based on the discrete wavelet transform (DWT). Three attention-free mixers recombine the scales: by linear fusion, by learned gating, or by letting each token choose its scales. A hybrid adds self-attention in the last layer only. A receptive-field analysis shows that wavelet mixers built from two-tap filters, such as Haar, never relate tokens outside fixed blocks, however deep the network, even when the filters are learned. Longer filters reach the whole sequence within two layers. We pre-train every model with masked language modeling on a fixed-token subset of C4 and fine-tune on GLUE, using one controlled setup with size-matched BERT and FNet baselines and a control that cannot mix tokens. The token-gated mixer trains as…
PDF arXiv
arXiv cs.CVOct 3
FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention
Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan, Shireen Y. Elhabian
High-resolution vision backbones have long been forced to trade away local token density to afford larger receptive fields. Hierarchical Swin transformers impose this compromise because standard windowed attention materializes an $M^2\times M^2$ score matrix per window, incurring $O(M^4)$ memory as windows or token grids grow. Furthermore, Swin adds a learned relative-position bias elementwise to attention scores, requiring full materialization of the score matrix and its gradient. This keeps Swin and SwinV2 trapped in a small-window($M=8,16$), coarse-token regime with patch size $4\times4$ ($p=4$), limiting performance for fine-grained tasks. We introduce FLASHSWIN, which replaces standard windowed attention with a FlashAttention implementation that computes exact softmax attention without materializing the score matrix, reducing per-window memory from $O(M^4)$ to $O(M^2)$. This enables higher token density and larger receptive fields without inflating memory overhead. Training memory is flat across window sizes: at a $32\times32$ window, FLASHSWIN-T requires only $12.4$\,GB, unchanged from $8\times8$, compared to $70/90$\,GB for SwinV2/V1-T. However, applying FlashAttention…
PDF arXiv
arXiv cs.LGOct 3
Path Laplacian Encodings for Directed Graphs
Lydia Mezrag, Semih Cantürk, Michael Perlmutter, Bastian Rieck, Guy Wolf
Directed graphs naturally model many real-world systems in which interactions are asymmetric, such as citation networks, web graphs, and information-flow networks. However, graph learning methods commonly rely on message passing with symmetrized graph representations or positional encodings that only partially exploit edge directionality. We introduce PathLapPE, a novel spectral positional encoding (PE) derived from the path Laplacian on directed graphs. PathLapPE provides node- and edge-level features that encode directional higher-order structure and can be incorporated into standard graph learning architectures. Empirical results on node- and graph-level benchmark tasks show that PathLapPE yields consistent improvements across several architectures, especially when combined with direction-aware message passing. Compared with magnetic Laplacian positional encodings, a widely studied spectral positional encoding for directed graphs, PathLapPE does not require additional fine-tuning of directionality hyperparameters while offering competitive runtime and performance.
PDF arXiv
arXiv cs.SDOct 3
GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
Ron Aluf, Alon Canfi, Eliya Nachmani
Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization…
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks233 papers46 this week
graph learning · graph neural · graph neural networks269 papers35 this week
neural networks · neural network · arnold networks155 papers33 this week
memory efficient · optimizers · low rank adaptation164 papers30 this week
bit quantization · quantization aware training · quantization aware158 papers24 this week
linear attention · softmax attention · transformer attention108 papers12 this week

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Activity · 12 weeks

Recent papers

All papers in Research

arXiv cs.LGOct 3
DASH: Fast, Valid Counterfactuals for Deep Networks via Batched Directional Search
Shraman Pal, Gabriel Medeiros, Clayton Escouper das Chagas, Can Li
Counterfactual explanations are most useful when they can be generated with low latency, remain close to the factual input, and satisfy input-domain, categorical, and actionability constraints. Achieving these objectives simultaneously is challenging for deep neural networks. Heuristic methods are often fast but may return invalid counterfactuals, whereas exact methods can certify global proximity but may not finish within practical time limits. We introduce DASH, a batched heuristic search method for finding close, valid, and actionable counterfactuals for deep neural networks under $\ell_1$, $\ell_2$, and $\ell_\infty$ objectives. DASH uses directional Lipschitz bounds and local affine models to generate anchors, then ranks and expands promising regions with batched network evaluations. We compare DASH against nine prior heuristic methods and time-limited exact mixed-integer baselines on four tabular datasets, with network depths from 2 to 32, and evaluate scalability on PBMC3k. Across 9,000 tabular query-norm cases, DASH returns a valid counterfactual within $5\%$ of the best heuristic-observed valid distance in $94.6\%$ of cases, with a median CPU search runtime of $0.061$ s.…
PDF arXiv
arXiv cs.CVOct 3
Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models
Bangwei Guo, Xujiang Zhao, Shengyu Chen, Yanchi Liu, Wei Cheng +4
Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller…
PDF arXiv
arXiv cs.CLOct 3
WNet: Discrete Wavelets Transform for Efficient Token Mixing
Rana Aref Salama, Abdou Youssef, Mona Diab
In a Transformer, token mixing is the step that lets each token draw information from other tokens, and it dominates the cost of encoding long sequences. Self-attention does this mixing very well: every token weighs every other token by content, which gives strong contextual modeling. That all-pairs comparison is also why its cost grows quadratically with sequence length. We introduce WNet, a Transformer encoder that replaces self-attention with token mixing based on the discrete wavelet transform (DWT). Three attention-free mixers recombine the scales: by linear fusion, by learned gating, or by letting each token choose its scales. A hybrid adds self-attention in the last layer only. A receptive-field analysis shows that wavelet mixers built from two-tap filters, such as Haar, never relate tokens outside fixed blocks, however deep the network, even when the filters are learned. Longer filters reach the whole sequence within two layers. We pre-train every model with masked language modeling on a fixed-token subset of C4 and fine-tune on GLUE, using one controlled setup with size-matched BERT and FNet baselines and a control that cannot mix tokens. The token-gated mixer trains as…
PDF arXiv
arXiv cs.CVOct 3
FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention
Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan, Shireen Y. Elhabian
High-resolution vision backbones have long been forced to trade away local token density to afford larger receptive fields. Hierarchical Swin transformers impose this compromise because standard windowed attention materializes an $M^2\times M^2$ score matrix per window, incurring $O(M^4)$ memory as windows or token grids grow. Furthermore, Swin adds a learned relative-position bias elementwise to attention scores, requiring full materialization of the score matrix and its gradient. This keeps Swin and SwinV2 trapped in a small-window($M=8,16$), coarse-token regime with patch size $4\times4$ ($p=4$), limiting performance for fine-grained tasks. We introduce FLASHSWIN, which replaces standard windowed attention with a FlashAttention implementation that computes exact softmax attention without materializing the score matrix, reducing per-window memory from $O(M^4)$ to $O(M^2)$. This enables higher token density and larger receptive fields without inflating memory overhead. Training memory is flat across window sizes: at a $32\times32$ window, FLASHSWIN-T requires only $12.4$\,GB, unchanged from $8\times8$, compared to $70/90$\,GB for SwinV2/V1-T. However, applying FlashAttention…
PDF arXiv
arXiv cs.LGOct 3
Path Laplacian Encodings for Directed Graphs
Lydia Mezrag, Semih Cantürk, Michael Perlmutter, Bastian Rieck, Guy Wolf
Directed graphs naturally model many real-world systems in which interactions are asymmetric, such as citation networks, web graphs, and information-flow networks. However, graph learning methods commonly rely on message passing with symmetrized graph representations or positional encodings that only partially exploit edge directionality. We introduce PathLapPE, a novel spectral positional encoding (PE) derived from the path Laplacian on directed graphs. PathLapPE provides node- and edge-level features that encode directional higher-order structure and can be incorporated into standard graph learning architectures. Empirical results on node- and graph-level benchmark tasks show that PathLapPE yields consistent improvements across several architectures, especially when combined with direction-aware message passing. Compared with magnetic Laplacian positional encodings, a widely studied spectral positional encoding for directed graphs, PathLapPE does not require additional fine-tuning of directionality hyperparameters while offering competitive runtime and performance.
PDF arXiv
arXiv cs.SDOct 3
GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
Ron Aluf, Alon Canfi, Eliya Nachmani
Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization…
PDF arXiv

Fields · AI · updated 1h ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

180papers this week−19.3% vs last week

All papers · 1.1k With code · 11

Newest Most relevant

1k papers this month

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantization in RL training and rollout paths can diverge, reducing the effectiveness of low-precision methods.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not adequately evaluate System One models on graph tasks, limiting progress in this area.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Error propagation in low-rank weight factorization1 paper raises it · 1 addresses it
Local criteria for subspace selection ignore global error propagation, leading to performance collapse at high compression.
- Learning Functional Subspaces for Neural Network Compressionraises · addresses

Named by an AI from these papers' abstracts.