Spotlight

Papers

Field

Top tools here

Open problems in Physics informed neural

Accurate long-term PDE dynamics prediction1 paper raises it · 1 addresses it
Reconstruction-based learning often fails to capture the true evolution of physical systems over time.
- PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEsraises · addresses
Numerical solver self-improvement without trial-and-error1 paper raises it · 1 addresses it
Current solvers lack structured ways to diagnose and fix poor performance in numerical methods.
- Training Numerical Intelligence via Auto-Diagnosis and Skill Discoveryraises · addresses
Graph decision-making benchmarks for System One models1 paper raises it · 1 addresses it
Existing benchmarks do not clearly evaluate System One models on graph tasks, leading to unclear capabilities.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Token collapse in graph soft prompting1 paper raises it · 0 address it
Graph-specific tokens in prompts often degenerate into generic signals, losing relational information.
- On the Token Collapse Phenomenon in Textual Graph Soft Promptingraises
Numerical stability in softmax and cross-entropy1 paper raises it · 1 addresses it
Standard implementations suffer from gradient bugs that harm training dynamics and convergence.
- A Fix for Softmax and Cross-Entropyraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

graph neural · graph learning · graph neural networks260 papers29 this week
physics informed neural · informed neural networks · neural operators296 papers22 this week
relu networks · deep neural · gradient descent158 papers21 this week
kernel · svm · gaussian125 papers13 this week
arnold networks · symmetries · parameter symmetries84 papers9 this week
muon optimizer · practical muon · gradient descent47 papers5 this week

Topics in this field

Active contributors · 90 days

Pietro LiòOn the Token Collapse Phenomenon in Textual Graph Soft Prompting9
Anima AnandkumarNeural Operator Splitting for Fluid Dynamics in Complex Geometry5
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Alessio BorgiLocaLeJEPA: Maximum Entropy Representations under Geometric Priors4
Jing XuSame Target, Different Origins: Aligning Latent Rollout for Consistent PDE Emulation4

Active contributors

Activity · 12 weeks

Activity

★ Spotlight this month

arXiv cs.LGOct 15

When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou

Muon's gains in data efficiency motivate evaluating its performance under explicit data, time, and tuning budgets. On a single A100 GPU, we compare a reference single-device Muon implementation with AdamW across 17.7M- and 33.6M-parameter causal language models and 2.70M-parameter vision transformers. A common 23-candidate development budget selects configurations evaluated with five matched seeds, full validation trajectories, optimizer-step timing, and a frozen one-shot test. Muon reduces validation-loss area under the curve by 3.25--22.44\% at equal data and lowers test loss in all 20 primary pairs. In this configuration, Muon improves time-normalized loss area} by 19.68--22.31\% on vision, while AdamW is better by 3.38--6.96\% on language. The full Muon optimizer call, including auxiliary AdamW updates, consumes 56.1--69.6\% of training time, compared with 6.1--9.1\% for the AdamW baseline. Muon also has higher expected regret under restricted access to these candidate pools. These results characterize small models with reference kernels; their wall-clock ordering does not establish a ranking for larger models or optimized implementations.

PDF arXiv page Details

Spotlight

Recent papers

All papers in Research

arXiv math.NAOct 22
Phase Transition from Random Features to Fully Trainable Networks
Xin He, Huan Lao, Xiliang Lu, Shijun Zhang
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $α>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((κ_P\log(m/δ)/m)^α)$ for two-layer networks of width $m$ and trainable proportion $P$, where $κ_P=(P/κ_1^*+(1-P)/κ_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-δ$. The constants $κ_0^*$ and $κ_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $α=(d+2)/(d-1)$ and $κ_0^*=4\sqrt{2}τ(d+1)/d$ for initialization scale $τ\ge1$, and $κ_1^*=1$ under ridge leverage…
PDF arXiv
arXiv cs.LGOct 22
A Fix for Softmax and Cross-Entropy
Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen
Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously-observed "Slingshot" phenomenon, speeds up grokking with weight decay, prevents accuracy collapse without weight decay, stops student networks from drifting away from the teacher during distillation, and shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's $ε$ hyperparameter.
PDF arXiv
arXiv cs.LGOct 22
On the Token Collapse Phenomenon in Textual Graph Soft Prompting
Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò +1
Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic…
PDF arXiv
arXiv math.OCOct 22
Direct Inference on Admissible Sets (Diona)
Lorenzo Sabug, Eric Kerrigan
We propose a new regression method named Direct Inference on Admissible Sets (Diona) that directly uses samples to estimate the state and its derivatives at a query point, along with their admissible bounds, without long global training. By casting the values and derivatives at the query and neighbouring samples as latent variables in a constrained optimisation problem, the burden of long training loops becomes one of on-demand local solves. We establish Diona's theoretical properties including finite-sample coverage and, in the noiseless case, convergence to the true state and derivatives under dense sampling. In a nonlinear regression benchmark on six functions of three to eight dimensions, Diona was more accurate than a Gaussian process (GP) on noiseless data especially with denser data sets. Furthermore, as with GP, it produced uncertainty bounds that contained the true value at more than 95% of query points in four functions. Diona's time to a first prediction after a data set update was three to four orders of magnitude shorter than that of the GP, and two to three orders shorter than that of a neural network, which was more accurate but offers no uncertainty…
PDF arXiv
arXiv stat.MLOct 22
Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting
Rémy Chapelle
Nonparametric estimation of joint probability distributions is a core problem in statistics, which is notoriously challenging with high-dimensional data. Such challenges also arise with conditional distribution estimation and have led to flexible algorithms that aim to tackle the ''curse of dimensionality'' by exploiting predictive structures of lower dimension than the ambient space. In this paper, we study the use of such conditional distribution estimators to estimate joint distributions by factorizing the latter in an ''autoregressive'' way. We show that for loss functions satisfying a subadditivity property, bounds on the convergence rate of an autoregressive estimator can be derived from bounds on the convergence rate of the conditional distribution estimators. These bounds relate the difficulty of estimating the joint distribution to that of estimating the most challenging factor in the autoregressive factorization. When the conditional estimators adapt to sparsity in their conditioning sets, we characterize this difficulty through variable elimination in an undirected dependence graph. This allows bounding the rates of convergence by quantities related to the sparsity of…
PDF arXiv
arXiv cs.CEOct 22
Structure-preserving reduced-order models of thin panels in supersonic and hypersonic flow
Miguel A. Aguilo
We build structure-preserving reduced-order models for both challenge problems of the AePW-4 High-Speed Working Group. They are a clamped thin panel at Mach 1.92 (RC-19) and a cantilever plate at Mach 5.85 (HyMAX). The published data for each case hold no time-resolved response and no repeated runs. The physics must therefore be carried by the form of the model. One construction gives two models per problem. A Ritz-Galerkin reference model describes the plate. A learned model keeps the reference model's exact potential, fits the remaining generalized forces, and trains only a dissipation that cannot add energy. Its energy balance holds for any weights by construction. On the trained model the worst relative residual is 2.2e-15. On RC-19, shock relations predict a pressure ratio of 1.521 across the panel. Pressure-sensitive paint measured the same value. Thirty runs of the learned model place the mean dominant frequency within 1.0% of the measured 236 Hz. At one periodic and one chaotic condition, the learned model reproduces the response regime of a published reduced-order model in all sixteen runs. It does so without that model's 113 Hz spectral peak, which the measurements do…
PDF arXiv

Papers

Topics in this field

graph neural · graph learning · graph neural networks260 papers29 this week
physics informed neural · informed neural networks · neural operators296 papers22 this week
relu networks · deep neural · gradient descent158 papers21 this week
kernel · svm · gaussian125 papers13 this week
arnold networks · symmetries · parameter symmetries84 papers9 this week
muon optimizer · practical muon · gradient descent47 papers5 this week

Active contributors · 90 days

Pietro LiòOn the Token Collapse Phenomenon in Textual Graph Soft Prompting9
Anima AnandkumarNeural Operator Splitting for Fluid Dynamics in Complex Geometry5
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Alessio BorgiLocaLeJEPA: Maximum Entropy Representations under Geometric Priors4
Jing XuSame Target, Different Origins: Aligning Latent Rollout for Consistent PDE Emulation4

Activity · 12 weeks

★ Spotlight this month

arXiv cs.LGOct 15

When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou

Muon's gains in data efficiency motivate evaluating its performance under explicit data, time, and tuning budgets. On a single A100 GPU, we compare a reference single-device Muon implementation with AdamW across 17.7M- and 33.6M-parameter causal language models and 2.70M-parameter vision transformers. A common 23-candidate development budget selects configurations evaluated with five matched seeds, full validation trajectories, optimizer-step timing, and a frozen one-shot test. Muon reduces validation-loss area under the curve by 3.25--22.44\% at equal data and lowers test loss in all 20 primary pairs. In this configuration, Muon improves time-normalized loss area} by 19.68--22.31\% on vision, while AdamW is better by 3.38--6.96\% on language. The full Muon optimizer call, including auxiliary AdamW updates, consumes 56.1--69.6\% of training time, compared with 6.1--9.1\% for the AdamW baseline. Muon also has higher expected regret under restricted access to these candidate pools. These results characterize small models with reference kernels; their wall-clock ordering does not establish a ranking for larger models or optimized implementations.

PDF arXiv page Details

Open problems in Physics informed neural

Accurate long-term PDE dynamics prediction1 paper raises it · 1 addresses it
Reconstruction-based learning often fails to capture the true evolution of physical systems over time.
- PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEsraises · addresses
Numerical solver self-improvement without trial-and-error1 paper raises it · 1 addresses it
Current solvers lack structured ways to diagnose and fix poor performance in numerical methods.
- Training Numerical Intelligence via Auto-Diagnosis and Skill Discoveryraises · addresses
Graph decision-making benchmarks for System One models1 paper raises it · 1 addresses it
Existing benchmarks do not clearly evaluate System One models on graph tasks, leading to unclear capabilities.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Token collapse in graph soft prompting1 paper raises it · 0 address it
Graph-specific tokens in prompts often degenerate into generic signals, losing relational information.
- On the Token Collapse Phenomenon in Textual Graph Soft Promptingraises
Numerical stability in softmax and cross-entropy1 paper raises it · 1 addresses it
Standard implementations suffer from gradient bugs that harm training dynamics and convergence.
- A Fix for Softmax and Cross-Entropyraises · addresses

Named by an AI from these papers' abstracts.

Recent papers

All papers in Research

arXiv math.NAOct 22
Phase Transition from Random Features to Fully Trainable Networks
Xin He, Huan Lao, Xiliang Lu, Shijun Zhang
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $α>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((κ_P\log(m/δ)/m)^α)$ for two-layer networks of width $m$ and trainable proportion $P$, where $κ_P=(P/κ_1^*+(1-P)/κ_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-δ$. The constants $κ_0^*$ and $κ_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $α=(d+2)/(d-1)$ and $κ_0^*=4\sqrt{2}τ(d+1)/d$ for initialization scale $τ\ge1$, and $κ_1^*=1$ under ridge leverage…
PDF arXiv
arXiv cs.LGOct 22
A Fix for Softmax and Cross-Entropy
Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen
Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously-observed "Slingshot" phenomenon, speeds up grokking with weight decay, prevents accuracy collapse without weight decay, stops student networks from drifting away from the teacher during distillation, and shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's $ε$ hyperparameter.
PDF arXiv
arXiv cs.LGOct 22
On the Token Collapse Phenomenon in Textual Graph Soft Prompting
Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò +1
Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic…
PDF arXiv
arXiv math.OCOct 22
Direct Inference on Admissible Sets (Diona)
Lorenzo Sabug, Eric Kerrigan
We propose a new regression method named Direct Inference on Admissible Sets (Diona) that directly uses samples to estimate the state and its derivatives at a query point, along with their admissible bounds, without long global training. By casting the values and derivatives at the query and neighbouring samples as latent variables in a constrained optimisation problem, the burden of long training loops becomes one of on-demand local solves. We establish Diona's theoretical properties including finite-sample coverage and, in the noiseless case, convergence to the true state and derivatives under dense sampling. In a nonlinear regression benchmark on six functions of three to eight dimensions, Diona was more accurate than a Gaussian process (GP) on noiseless data especially with denser data sets. Furthermore, as with GP, it produced uncertainty bounds that contained the true value at more than 95% of query points in four functions. Diona's time to a first prediction after a data set update was three to four orders of magnitude shorter than that of the GP, and two to three orders shorter than that of a neural network, which was more accurate but offers no uncertainty…
PDF arXiv
arXiv stat.MLOct 22
Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting
Rémy Chapelle
Nonparametric estimation of joint probability distributions is a core problem in statistics, which is notoriously challenging with high-dimensional data. Such challenges also arise with conditional distribution estimation and have led to flexible algorithms that aim to tackle the ''curse of dimensionality'' by exploiting predictive structures of lower dimension than the ambient space. In this paper, we study the use of such conditional distribution estimators to estimate joint distributions by factorizing the latter in an ''autoregressive'' way. We show that for loss functions satisfying a subadditivity property, bounds on the convergence rate of an autoregressive estimator can be derived from bounds on the convergence rate of the conditional distribution estimators. These bounds relate the difficulty of estimating the joint distribution to that of estimating the most challenging factor in the autoregressive factorization. When the conditional estimators adapt to sparsity in their conditioning sets, we characterize this difficulty through variable elimination in an undirected dependence graph. This allows bounding the rates of convergence by quantities related to the sparsity of…
PDF arXiv
arXiv cs.CEOct 22
Structure-preserving reduced-order models of thin panels in supersonic and hypersonic flow
Miguel A. Aguilo
We build structure-preserving reduced-order models for both challenge problems of the AePW-4 High-Speed Working Group. They are a clamped thin panel at Mach 1.92 (RC-19) and a cantilever plate at Mach 5.85 (HyMAX). The published data for each case hold no time-resolved response and no repeated runs. The physics must therefore be carried by the form of the model. One construction gives two models per problem. A Ritz-Galerkin reference model describes the plate. A learned model keeps the reference model's exact potential, fits the remaining generalized forces, and trains only a dissipation that cannot add energy. Its energy balance holds for any weights by construction. On the trained model the worst relative residual is 2.2e-15. On RC-19, shock relations predict a pressure ratio of 1.521 across the panel. Pressure-sensitive paint measured the same value. Thirty runs of the learned model place the mean dominant frequency within 1.0% of the measured 236 Hz. At one periodic and one chaotic condition, the learned model reproduces the response regime of a published reduced-order model in all sixteen runs. It does so without that model's 113 Hz spectral peak, which the measurements do…
PDF arXiv

Fields · AI · updated 41m ago

Physics informed neural

Found by the topic model: physics informed neural, graph neural, relu networks.

99papers this week−40.0% vs last week

All papers · 970 With code · 3

Newest Most relevant

821 papers this month