Research

• arXiv cs.AIOct 20

  Automatic Differentiation with Constraints in Forward and Inverse Problems

  Nikolay Yavich, Alexander Ryabov, Viacheslav Naumov, Sayan Ranu, N. M. Anoop Krishnan +2

  Approximate solutions of partial differential equations (PDEs) on unstructured grids can be represented on a graph. Numerical optimization involving stiff evolutionary and static PDEs often requires solving a large system of equations, which can be treated as a constraint. In this work, we present a fully differentiable graph-based implicit solver that incorporates this constraint. We evaluate its performance across four key tasks: a) linear and nonlinear forward modeling of evolutionary and static problems, b) solving inverse problems to estimate equation coefficients, c) solving inverse problems to estimate the right-hand side, and d) learnable graph coarsening to accelerate forward modeling. The stability and differentiability of our solver enable significant reductions in the complexity of forward modeling while efficiently addressing related inverse problems. These capabilities make it a powerful tool for different physics and engineering applications, in particular for geophysics, where inverse problems are commonly solved in order to reduce the model uncertainty.

  PDF
• arXiv cs.CVOct 20

  Manifold Prior Guided Deep Unfolding for Hyperspectral Image Reconstruction

  Shuqi Liu, Yingkai Zhang, Lin Gu, Tao Zhang, Ying Fu

  Hyperspectral Image (HSI) reconstruction has made substantial progress with the deep unfolding framework by decomposing the problem into a projection module and a denoiser module. Nevertheless, existing methods still exhibit limitations of insufficient matching with HSI data. The issues lie in two aspects: 1) the data-consistency projection module applies a fixed gradient descent step while ignoring the spectral content adaptivity of HSI; 2) the denoiser module either suffers from high computational complexity or is limited to the local receptive field, failing to efficiently capture the global spatial-spectral dependencies. In this paper, we propose a deep unfolding framework centered on a two-stage manifold learning strategy to obtain a degradation-free structural prior, which is then strategically injected into both the projection and denoiser modules. In the data-consistency projection, a prior-guided spectral weight mechanism is introduced to adaptively modulate the weight of spectral band. In the denoiser module, a manifold-guided Mamba is proposed to achieve the structure-aware extraction of spatial-spectral features through prior modulation. Quantitative and visual…

  PDF
• arXiv cs.LGOct 20

  Detection of Memorization in Diffusion Models Through Thermodynamic Singularity

  Iljung Kim, Kangwoo Park, Yung-Kyun Noh, Daniel D. Lee

  In diffusion models, proper distributional learning entails reversing a diffusion process over underlying density functions, whereas a key failure is memorization, in which the reverse process instead learns diffusion from individual training samples. We provide an analytical characterization of such memorized reverse processes, along with a criterion that detects the associated thermodynamic singularity. The proposed criterion reliably distinguishes between properly learned and memorizing networks across a range of neural network models designed to promote or mitigate memorization.

  PDF
• arXiv cs.LGOct 20

  Neural Operator Splitting for Fluid Dynamics in Complex Geometry

  Jianxiang Li, Xi Deng, Hong Chul Nam, Anima Anandkumar

  Solving dynamical systems in complex and diverse geometries remains challenging for both numerical solvers and learning-based surrogate models. We propose Neural Operator Splitting, a framework for learning fluid dynamics while preserving the physical structure of fractional-step numerical schemes. Instead of learning the full-step evolution monolithically, we decompose each time step into physically interpretable steps, and learn each operator separately using intermediate states from the underlying numerical solver. Guided by the linearity properties of the fractional steps, we introduce a conditional-linear Fourier neural operator (CL-FNO) whose linear state mappings are modulated by nonlinear conditioning on inputs such as boundary geometry. We further introduce two complementary strategies, symmetry sampling and corrective cycling, to mitigate error accumulation during autoregressive rollout. Experiments on 2D and 3D fluid flows with varying geometries and initial conditions show that our full model reduces mean relative $L^2$ rollout error by 68\% in 2D and 40\% in 3D compared with a single full-step neural operator. At comparable error levels, neural-operator inference is…

  PDF
• arXiv physics.geo-phOct 20

  HypoNet Nankai PS: rapid hypocenter determination for the Nankai Trough using physics-informed neural networks with three-dimensional P- and S-wave velocity models

  Ryoichiro Agata, Ayako Nakanishi, Dan Bassett, Satoru Baba, Yasuyuki Nakamura +1

  Rapid and accurate hypocenter determination based on three-dimensional (3D) seismic velocity models is essential for seismic monitoring in subduction zones, yet it remains computationally demanding. HypoNet Nankai, a hypocenter determination package that uses physics-informed neural network (PINN) travel-time inference for a realistic 3D P-wave velocity model of the Nankai Trough subduction zone, was designed to address this problem, but supported only P-wave arrival times. Here, we present HypoNet Nankai PS, which extends the framework to both P- and S-wave travel times. We first constructed an S-wave velocity model from the existing P-wave model using empirical conversion relations for marine sediments and the crust. Because the resulting S-wave velocity structure has lower velocities and therefore considerably steeper travel-time gradients, the original PINN training procedure designed for the P-wave velocity structure was insufficient. We therefore revised the training procedure by reconsidering the distribution of collocation points, the NN architecture, and the optimizer. Benchmark comparisons with the fast marching method showed that the revised approach substantially…

  PDF
• arXiv math.NAOct 20

  Soliton Profiles in 1D: Enhanced Neural-Network Solvers and Constrained Deep Ritz Methods

  Chandler Haight, Alex D Rodriguez, Svetlana Roudenko

  We develop and compare enhanced neural-network solvers for one-dimensional ground-state profiles, which are solutions of $$ Q^{\prime\prime}=bQ-γQ^{p},\quad Q>0,\quad p>1, \quad b, γ>0, $$ the stationary profile equation for the nonlinear Schrödinger, generalized Korteweg-de Vries, and nonlinear Klein-Gordon equations. Its explicit sech-type solutions serve as exact reference for error computation and benchmark hyperparameter selection. We improve on the standard Adam-trained physics-informed neural network (PINN) baseline by introducing higher-order optimization via Gauss-Newton/Levenberg-Marquardt and L-BFGS refinements, as well as Fourier-feature PINNs, and constrained Deep Ritz (DR) formulations. In DR approaches we introduce two new neural-network formulations, which implement two classical variational characterizations: a Nehari-constrained Deep Ritz method, in which the height is learned by projecting onto the Nehari/Pokhozhaev manifold, and a Weinstein Deep Ritz method, in which a normalized profile minimizes the Weinstein quotient. The Deep Ritz formulations perform best overall, in particular, the Weinstein variant gives the best error-runtime balance. The…

  PDF
• arXiv cs.CVOct 20

  Test-time Adaptation for Point Cloud Denoising under Unseen Noise

  Kun Jiang, Shida Fu, Fanhao Yi, Weiqing Li, Zhiyong Su

  Although deep learning-based point cloud denoising methods have demonstrated impressive results in recent years, their effectiveness usually degrades substantially in real-world scenarios where the noise distribution of test point clouds deviates from that of the training set. To address this challenge, we propose a test-time adaptive denoising (TTAD) framework that enables existing denoising approaches to adapt to unknown noise patterns at test time without relying on additional supervision or retraining from source data. Taking a pre-trained denoising model and a test noisy point cloud as input, TTAD first employs a residual-driven noise modeling (RDNM) module to estimate the sample-specific noise distribution through analyzing the deviation between the noisy input and its pseudo-clean version denoised by the initial model. Then, a noise-aware contrastive adaptation (NACA) module is designed to build sample-specific noisy-noisier test-time training pair from the denoised point cloud, and update the model parameters over a few adaptation steps per sample at test time through contrastive learning. Furthermore, TTAD naturally extends to a continual TTAD (CTTAD) setting,…

  PDF
• arXiv cs.LGOct 19

  VampPrior Unleashed

  Andrew A. Stirn, Abhejit Rajagopal, David A. Knowles, Grace Huynh

  Deep latent variable models (DLVMs) underpin many modern analysis pipelines, for example in single-cell genomics, yet their prior distributions have received markedly little attention. The VampPrior approximates the optimal prior (the aggregate posterior) and often attains high model evidence. Here, we identify two key limitations of the VampPrior: it ties prior and posterior uncertainty to a single amortized covariance and clusters poorly due to its fixed uniform mixture weights. We unleash the VampPrior by decoupling its prior and posterior covariances, resolving the conflict of generating high prior and low posterior entropy with the same shared parameters and substantially improving model evidence. We then replace the uniform mixture with a Dirichlet process. When combined, our proposals provide higher model evidence than the VampPrior and robust non-parametric clustering. Augmenting single cell Variational Inference (scVI) with our prior improves single-cell integration and clustering across six atlases while significantly raising model evidence. Our approach changes only the prior and thus can be applied to any DLVM with a continuous latent space.

  PDF
• arXiv math.NAOct 19

  Low Residuals, Wrong Value: Learning the Viscosity Solution of Hamilton-Jacobi-Isaacs Equations with PINNs

  Raul Jimenez, Pablo Tejerina-Pérez, Leonid Sarieddine, Pedro Tarancón-Álvarez, Marisol Traforetti +4

  Physics-informed neural networks (PINNs) and other residual-trained solvers judge a candidate solution by the size of its sampled PDE residual. For first-order Hamilton-Jacobi equations this is not sufficient: the same equation and boundary data admit Lipschitz almost-everywhere solutions that differ from the viscosity solution by $O(1)$ while violating the equation only on sets of measure zero. We study whether residual minimization selects such solutions in a fixed-horizon pursuit-evasion game whose Hamilton-Jacobi-Isaacs value function is known in closed form on the circle $S^1$ and on the flat tori $\mathbb{T}^{2}$ and $\mathbb{T}^{4}$. We train four neural parameterizations with ten seeds on each geometry, 120 runs in total. A standard network with periodic Fourier features converges to folded almost-everywhere solutions in all 30 runs: small residuals and the expected capture-layer gradient, yet values that are wrong over most of the domain. A network with periodicity imposed through the loss recovers the viscosity solution in one and two dimensions but becomes unreliable in four. Two parameterizations that receive the geodesic distance do substantially better: a single…

  PDF
• arXiv cs.LGOct 19

  Trajectory Replay: Continuous-Time Mixture Transport for Lifelong Generative Models

  Alessandro Palma, Charles Jones, Tristan Cinquin, Fabian J. Theis, Frederik Wenkel

  Adapting pre-trained continuous-time generative models to sequentially incoming data without forgetting earlier tasks is a central challenge in lifelong learning. A leading strategy, generative replay, uses a generative model trained on historical tasks to augment its own continual update with synthetic samples from experience. However, it requires full trajectory simulations at every training step, which accumulate discretization errors and can degrade the replayed historical distribution. Here, we propose trajectory replay, which bypasses sample generation by training an updated velocity field to match both a frozen historical velocity along its trajectories and the incoming task's velocity field, weighted by a stability-plasticity coefficient. Through the lens of continuous-time transport, we show the resulting optimum averages the two task velocities by their respective densities and generates the mixture of their probability paths. Combined with mean velocity distillation, trajectory replay accesses historical trajectories using only a few network evaluations per continual update. Empirically, it outperforms generative replay and distillation baselines on class-incremental…

  PDF
• arXiv cs.LGOct 19

  Cross-Fitting for Neural Posterior Estimation

  Jeffrey Regier

  Neural Posterior Estimation (NPE) fits an amortized conditional density estimator to a simulated training set of complete-data realizations. NPE's training objective, log-likelihood, favors posterior approximations that assign arbitrarily high density to the training points, and typical families of amortized conditional density estimators (e.g., normalizing flows, Gaussian encoders) are flexible enough to oblige. The resulting policy of extreme overconfidence translates poorly to held-out data. Early stopping and other regularizers can prevent this degeneracy, but they are limited to selecting posterior approximations along an unfavorable optimization trajectory set by in-sample residuals, where the posterior mean is underfit while the posterior variance is overfit. To avoid this trade-off, we propose Cross-Fitting NPE (CF-NPE), which uses sample splitting to separate conditional-mean estimation from the estimation of the posterior's higher moments. Using $K$-fold cross-fitting, CF-NPE fits $K$ posterior-mean regressors and then fits a conditional density estimator to the resulting out-of-fold residuals. In experiments with synthetic data and in a cosmological application, CF-NPE…

  PDF
• arXiv cs.CVOct 19

  MRI Reconstruction Risk Estimation from Inference Dynamics

  Tamir Shor, Alex Bronstein

  Iterative MRI reconstructors produce intermediate estimates before returning a final image. We study these inference trajectories as a model-internal representation of realized reconstruction risk, measured by log-NMSE. Model-native states are projected to magnitude images, summarized by a fixed 16-dimensional descriptor, and mapped to risk using a source-trained ridge readout. Across three reconstruction backbones, trajectory features are strongly predictive of reconstruction risk on held-out source data. Under zero-shot target shift, their behavior is backbone-dependent, exhibiting both preserved positive association and readout reversal. Conditional analyses show that trajectory information can remain associated with error beyond an initialization-only predictor. A prefix analysis further shows that useful trajectory information can emerge before reconstruction completes. Repeated-mask and low-shot calibration studies indicate that trajectories primarily encode persistent case difficulty, while their orientation and calibration can change under domain shift.

  PDF
• arXiv math.NAOct 19

  Residual-Based Adaptive Refinement in Hybrid Variational-Collocation PINNs for Singular Elliptic PDEs with Mixed Dirichlet-Neumann Boundary Conditions

  Somnath Giri, Muslim Malik, Rishikesh Yadav

  Solutions of second-order elliptic PDE problems on nonconvex or cracked domains with mixed Dirichlet-Neumann conditions typically lie outside $H^2(Ω)$. Near re-entrant corners, crack tips and points where the boundary condition changes type, the gradient grows like $r^{λ-1}$ with $λ<1$. We study residual-based adaptive refinement in hybrid variational-collocation PINNs. A multiscale Fourier-feature network is trained on a composite loss. The loss combines a variational residual tested against Legendre bubble functions, a strong-form collocation term and penalties for the Dirichlet and Neumann data. The variational term needs only first derivatives of the network, whereas the collocation term requires second derivatives and provides a residual that can be evaluated at any point. After each training stage, candidate points selected by this residual are added to the collocation set. The method is tested on five two dimensional benchmarks, namely a Mode~III crack, a cross-shaped domain with four re-entrant corners with unequal amplitude, a reaction-diffusion problem with four localized peaks, a monotone semilinear problem with a boundary layer and the Motz problem. Compared with…

  PDF
• arXiv stat.MLOct 19

  R2-GP: Variance-Calibrated Priors for Additive Gaussian Processes

  Javier Enrique Aguilar, Soham Mukherjee, David Kohns

  We develop R2-GP, an R2D2-type prior for Gaussian process (GP) generalised additive models (GAMs) that calibrates explained variance ($R^2$) for Matérn covariance kernels. The function variance for standard Matérn kernels depends jointly on the effect magnitude and length-scale. As such, smooth and rough components with identical amplitudes contribute very different amounts of variation. With independent priors on component amplitudes, this variation results in variance overestimation, especially in higher dimensions. We overcome this by taking direct control over each component's expected contribution to prior explained variance. We do this by rescaling each component based on its length scale and observed inputs, and modelling each rescaled variance as a share of the function variance induced by a global $R^2$ prior. This allows R2-GP to scale well to many components. To also scale to large sample sizes, we extend R2-GP to Hilbert space approximation methods, a framework for kernels exploiting their spectral decompositions for fast and efficient inference. Our simulations and real-world case studies demonstrate superior generalisation and parameter recovery over independent…

  PDF
• arXiv cs.LGOct 19

  CIS-Transolver: A Cross-Layer Incremental-State Neural Operator for Fine-Scale Automotive Aerodynamic Flow Prediction

  Jianglong Liu, Zhicheng Zhu, Jia Hao, Liangyue Jia

  Neural operators offer an efficient data-driven approach to rapid prediction of complex automotive aerodynamic flows. However, spectral bias and other factors limit accurate reconstruction of fine-scale flow variations near complex geometries. Existing studies improve the representation and fitting of fine-scale information mainly through information extraction and training objectives. Further study is needed to determine how extracted local details can be used continually in subsequent operator computations and contribute effectively to the overall flow-field representation. We propose CIS-Transolver, a cross-layer incremental-state neural operator that extends fine-scale modeling from local information extraction to its use across layers. The method extracts fine-scale information through local multiscale analysis and encodes it as geometric moment states that are propagated and updated across layers. By combining information from previous layers with feedback to the value representations in Physics-Attention, the method allows local details to contribute continually to subsequent modeling of global physical correlations. We systematically evaluate CIS-Transolver on three…

  PDF
• arXiv cs.LGOct 19

  Learning SOCP Solver Dynamics with Graph Neural Networks

  Adam Ghalem, Lou Salaün, Davide Rinaldi, Fragkiskos D. Malliaros

  Second-order cone programs (SOCPs) are a fundamental class of convex optimization problems widely applied across engineering domains. Classical interior-point methods are robust and accurate, but their iterative nature can be costly in real-time or repeated-solve settings. This has motivated learning-based solvers that approximate optimizer solutions with neural networks. Recent work has increasingly studied graph neural networks (GNNs) for linear, quadratic, and second-order cone optimization, leveraging their ability to model sparse relational structure among variables, constraints, and objective functions. Most existing GNN-based solvers compress the optimization process into a single forward pass, predicting the optimizer directly without using the intermediate trajectory produced by classical solvers. We introduce SOCPNet, a recursive GNN framework that uses solver trajectories to learn an iterative SOCP solution procedure. A shared GNN model is applied repeatedly, refining the current iterate at each step and enabling inference for a variable number of iterations. Rather than predicting the solution in a single pass, SOCPNet learns from intermediate iterates to…

  PDF
• arXiv cs.CVOct 19

  HP-PnP-Flow: Hybrid Time Scheduling and Projection for Plug-and-Play Image Restoration via Flow Matching

  Yong Li, Xiaojie Wang, Xiaoyan Zhang

  Image restoration is a fundamental inverse problem that seeks to recover high-quality images from degraded observations. Plug-and-Play (PnP) methods provide a flexible framework for integrating pretrained denoisers into iterative reconstruction, whereas flow matching offers an efficient generative modeling approach based on continuous-time ODE dynamics. Very recently, Martin et al.\ (ICLR, 2025) developed a PnP-Flow method, which integrates the PnP framework with flow matching through a time-dependent denoiser constructed from a pretrained flow matching model.By incorporating a different time schedule and an additional pixel-space projection operator, this paper proposes a novel efficient PnP-Flow algorithm (HP-PnP-Flow) that enjoys rigorous convergence guarantees under mild assumptions, without imposing restrictive conditions on iterates required by the aforementioned work. Numerical experiments on CelebA and AFHQ-Cat demonstrate that the newly proposed HP-PnP-Flow algorithm significantly outperforms PnP-Flow on several restoration tasks such as denoising, deblurring, super-resolution, random inpainting, and box inpainting. Notably, our algorithm achieves comparable or better…

  PDF
• arXiv cs.LGOct 19

  PnP-CF: A Consensus Flow-Matching Solver for General Inverse Problems

  Dibyanshu Kumar, Magda Gregorova

  Plug-and-Play (PnP) methods using continuous-time deep generative models as learned data distribution priors enable synthesizing high-quality solutions of model-based inverse problems. The quality of the solutions depends critically on the interplay and consensus between the inverse problem solver and the generative model sampler. We revisit a recently introduced flow-based PnP framework, which we refer to as PnP-CF (Consensus Flow), and extend its application to nonlinear inverse problems, including those with learned neural-network forward operators. PnP-CF combines a flow-matching denoiser with (i) ADMM-like dual variable updates and (ii) a time-adaptive pullback onto the intermediate flow probability path that combines a stochastic and a deterministic component. We discuss how these two features stabilize the algorithm and promote consensus, and demonstrate on a set of experiments that PnP-CF not only solves standard linear inverse problems with quality at least comparable to state-of-the-art baselines, but also extends gracefully to nonlinear inverse problems, including forward operators realized by pretrained neural networks, for which existing methods produce…

  PDF
• arXiv cs.CVOct 19

  LEAP: Accelerating ODE-based Diffusion Sampling with Optimal Linear Leaps

  Haowei Zheng, Zhenyu Zhou, Jiawei Chen, Can Wang, Defang Chen

  Diffusion models primarily rely on general-purpose ODE-based numerical methods for training-free accelerated sampling, but performance degrades with coarse time discretization. In this paper, we propose optimal linear leaps (LEAP), a specialized method grounded in theoretical and empirical investigations of sampling trajectory geometry. Built upon low-dimensionality and cross-trajectory consistency, LEAP formulates a supervised least-squares optimization problem within the local trajectory subspace, achieving a precise approximation of future states without model evaluations. Extensive experiments demonstrate that LEAP outperforms previous methods across diverse modalities, including text-to-image (Stable Diffusion 3.5, FLUX.1-dev) and image-to-3D (TRELLIS.2) generation. Furthermore, LEAP can be seamlessly integrated into existing solver-based methods, offering a plug-and-play enhancement for modern diffusion models. Code is available at https://github.com/zju-pi/LEAP.

  PDF
• arXiv q-fin.CPOct 19

  Arbitrage-Free Implied Volatility Surface Reconstruction via Additive Martingales

  Haochen Luo, Sifan Lin, Xiaolong Luo, An Binh Minh, Yuan Zhang +1

  This work focuses on reconstructing a continuous implied volatility (IV) surface from a small number of option quotes, a fundamental task for option pricing and risk management. We propose Additive Martingale (AddMart), a few-shot IV surface reconstruction framework that satisfies static no-arbitrage conditions by construction. Rather than predicting IVs directly, AddMart maps sparse quotes to non-negative increment densities of an exponential additive martingale, from which valid risk-neutral marginals and IVs are recovered. By incorporating the no-arbitrage structure directly into its architecture, rather than enforcing it through regularization in the learning objective, AddMart produces IV surfaces that satisfy static no-arbitrage conditions throughout the domain. Extensive experiments on ten real-option datasets demonstrate that AddMart achieves competitive reconstruction accuracy with no material static arbitrage violations, while directly enabling the estimation of VaR, CVaR, and crash probabilities.

  PDF

• Scalable and stable training for flow matching1 paper raises it · 1 addresses it

  Large-scale flow models struggle with training stability and integration error due to stochastic source-target pairing.

  • Scaling Online Optimal Transport Minibatches for Flow Matchingraises · addresses
• Efficient few-step generation without fixed discretization1 paper raises it · 1 addresses it

  Few-step generation often requires heuristic timestep discretization, limiting flexibility and performance.

  • Few-Step Generation via Data-Space Iterationraises · addresses
• Data-free and simulation-free distillation for diffusion/flow models1 paper raises it · 1 addresses it

  Distillation often relies on curated datasets or costly teacher rollouts, complicating training and scaling.

  • Consistent Distribution Matching for Data-Free Diffusion Distillationraises · addresses
• Dynamics-aligned latent representations for PDEs1 paper raises it · 1 addresses it

  Reconstruction-based learning for PDEs often fails to capture the evolution of physical fields accurately.

  • PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEsraises · addresses
• Detecting memorization in diffusion models1 paper raises it · 1 addresses it

  Memorization in diffusion models can lead to failure in proper distributional learning.

  • Detection of Memorization in Diffusion Models Through Thermodynamic Singularityraises · addresses

Named by an AI from these papers' abstracts.