TodayNewsResearchFieldsStacksMarketAnalytics
⌘K

Research

New papers ranked by attention, with their code, models and discussion linked. Pick a field on the left.

WeekMonthQuarterAll
NewestMost relevant

Papers

Open problems

Rising research topics

Ask the research

Sample
ByteTrail · AI and developer-tools intelligence
NewsResearchStacksAPI
Fields · this week
All fields2719Multimodal555AI agents504Frontier LLMs332Inference & serving237Teacher policy distillation201Medical image segmentation165Physics informed neural138Supervised & diffusion models114Forecasting time56Voice AI55Machine learned interatomic46Remote sensing45Electrocardiogram ecg42Federated learning37Deepfake detection37All 25 fields

Rising research topics

  • AI tutors20+81.8%
  • Channel state information17+70.0%
  • Character recognition OCR12+9.1%
  • Electrocardiogram ecg420.0%
  • Supervised & diffusion models114−0.9%
  • Physics informed neural138−4.2%
  • Machine learned interatomic46−4.2%
  • Anomaly detectors20−9.1%

Open problems across fields

  • Interface lock-in in multimodal spatial reasoning1 paper raises it · 1 addresses it

    Specializing a model on one spatial task (e.g., side detection) degrades performance on others (e.g., distance estimation), limiting general spatial intelligence.

    • ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligenceraises · addresses
  • Lack of benchmarks for agent harness reliability1 paper raises it · 1 addresses it

    Existing evaluations focus on model capabilities, not the reliability of the systems that deploy and manage agents.

    • Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
  • Negative net value of agent memory systems1 paper raises it · 1 addresses it

    Memory systems often underperform naive retrieval, as their design can obscure or remove critical evidence from raw history.

    • Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
  • Credit assignment in diffusion language models1 paper raises it · 2 address it

    Iterative denoising in DLMs makes it hard to attribute rewards to individual tokens, limiting reinforcement learning effectiveness.

    • DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Modelsraises · addresses
    • Multiscale Policy Optimization for Long-Horizon LLM Agentsaddresses
  • Uncertainty quantification failure under class imbalance1 paper raises it · 1 addresses it

    Standard uncertainty measures fail to reflect clinical ambiguity in imbalanced datasets, systematically penalizing minority classes.

    • When uncertainty quantification fails: structural vulnerabilities and corrections for selective prediction under clinical class imbalanceraises · addresses

Named by an AI from these papers' abstracts.

Papers

  • arXiv cs.AIOct 11

    Hidden in Plain Sight: Advancing Harmful Content Understanding via Perception-Aware Adversarial Self-Play

    Yisong Xiao, Benlei Cui, Haiwen Hong, Shiji Zhao, Zonghao Ying +6

    Despite significant advances in language understanding, Large Language Models (LLMs) can still miss meaning hidden in plain sight, failing to identify harmful intent in expressions readily understood by humans. Malicious users exploit this gap by strategically disguising harmful content to evade automated moderation, posing a serious threat to online platform safety. The vulnerability reflects a fundamental cognitive asymmetry between human readers, who integrate contextual, phonetic, and visual cues to decipher the intent behind deliberate obfuscation, and text-based LLMs, which rely primarily on token sequences and have limited exposure to evolving evasions. To bridge this asymmetry, we propose PASP (Perception-Aware Adversarial Self-Play), which combines multi-perceptual views with adversarial self-play to improve understanding of harmful intent as evasions evolve. Specifically, PASP constructs textual, phonetic, and visual views of each input, preserving its original context while exposing pronunciation and glyph cues obscured in token space. During adversarial self-play, an attacker uses defender feedback to generate intent-preserving evasions, while the defender is trained…

    PDF
  • arXiv cs.CVOct 11

    Rethinking Supervision in Document Unwarping: A Self-consistent Flow-free Approach

    Shaokai Liu, Hao Feng, Wengang Zhou

    In recent years, the proliferation of smartphones has led to an upsurge in the digitization of document files via these portable devices. However, images captured by smartphones often suffer from distortions, thereby negatively affecting digital preservation and downstream applications. To address this issue, we introduce DRNet, a novel deep network for document image rectification. Our approach is based on three key designs. Firstly, we exploit the intrinsic geometric consistency inherent in document images to guide the learning process of distortion rectification. Secondly, we design a coarse-to-fine rectification network to leverage the representations derived from the distorted document image, thereby enhancing the rectification result. Thirdly, we propose a unique perspective for supervising the learning of rectification networks, where undistorted document images are employed for supervision, which is free of warping mesh as ground truth in existing methods. Technically, both low-level pixel alignment and high-level semantic alignment jointly contribute to the learning of the mapping relationship between deformed document images and distortion-free ones. We evaluate our…

    PDF
  • arXiv cs.CVOct 11

    Harnessing SPAD Noise: Test-Time Gated Attention for Indoor Depth Estimation

    Yiquan Gao

    Monocular depth estimation (MDE) has emerged as a cost-effective alternative to hardware-dependent 3D sensing modalities, yet predicting precise metric depth remains a persistent challenge due to intrinsic scale ambiguity, edge bleeding, and limited zero-shot generalization in unseen environments. Recent multi-modal approaches incorporate 1D/3D Single-Photon Avalanche Diode (SPAD) data to resolve these ambiguities, but they mainly treat SPAD noise as a harmful artifact required for denoising or leave the noise ignored. In this paper, we present an essentially different insight: native SPAD noise is an informative prior that implicitly encodes deterministic environmental characteristics. Driven by this paradigm shift, we introduce a novel, lightweight Test-Time Gated Attention framework by Harnessing SPAD Noise, therefore dubbed HSN. Tailored for indoor scenarios, HSN leverages the synergistic combination of 1D SPAD histograms and Gate Adapters (GAs). Through three complementary optimization phases, it generates channel-wise gated attention vectors to dynamically adjust encoder feature maps at inference time, effectively resolving ambiguous scale and boundary challenges without…

    PDF
  • arXiv cs.LGOct 11

    World Modeling through Spectral Alignment

    Holger Molin, William Peng, Marco Bagatella, Randall Balestriero

    How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to…

    PDF
  • arXiv cs.LGOct 11

    Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning

    Chang-Hoon Jeong, Kisung Shin, Hanwool Sul, Byoung-Tak Zhang

    Context-based offline meta-reinforcement learning (COMRL) has advanced almost entirely through better task representations, which separate task identity from artifacts of the behavior policy so that the latent reflects the task alone. The implicit expectation has been that a clean representation will transfer to out-of-support (parameter-extrapolation) tasks. On such a task, however, the encoder produces a latent outside the training support, where the policy has never been trained. We address this gap with Pessimistic Latent Task-aware Optimization (PLATO), which exposes the policy to off-support latents during training. PLATO assumes a mild manifold hypothesis: task latents lie on a low-dimensional manifold, and out-of-support tasks lie farther from the training centroid. It exploits this geometry by perturbing inferred latents outward from the centroid to synthesize counterfactual tasks. A learned decoder ensemble then rolls out short trajectories at each perturbed latent, and the policy is updated on the imagined transitions under a pessimism weight that down-weights steps on which the ensemble disagrees. We prove that the perturbation leaves the training support by a…

    PDF
  • arXiv cs.LGOct 11

    Riemannian Graph Tokenizer for Structural Knowledge Transfer

    Qimin Zhou, Haibo Liu, Yujie Wang, Li Sun, Chuan Shi

    Foundation models are at the forefront of artificial intelligence. A tokenizer, converting the raw input into discrete representations that the model can understand, plays an important role to the success of foundation models. Unlike the text tokenizer that is well studied in large language models, graph tokenizer is still at its early stage, facing the challenges of tackling the non-Euclidean structures and capturing the structural semantics. How to design a graph tokenizer for structural knowledge transfer? To this end, we propose a Riemannian Graph Tokenizer (RGT) that bridges the structural knowledge and quantized representations to support cross-domain structural knowledge transfer. The connection is established by Riemannian geometry. Specifically, we first define the geometric vocabulary (trees, cycles and sequences), which captures fundamental structural patterns and reflects the intrinsic geometry of graph. Second, we construct a Riemannian quantizer with Riemannian Straight-Through Estimator to tokenise graph structures across multiple domains into discrete tokens. To ensure consistency and transferability across diverse geometric spaces, RGT further incorporates a…

    PDF
  • arXiv cs.AIOct 11

    SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

    Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu +1

    Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified…

    PDF
  • arXiv cs.CLOct 11

    Sensitivity or Criterion? Measuring and Moving When Language Models Act

    Sixun Dong, Yebowen Hu, Ming Yin, Yiran Chen, Fei Liu +1

    Before an agent acts, it must decide whether action is warranted. A single accuracy cannot distinguish failure to recognize a warranted action from reluctance to act. Signal-detection theory separates sensitivity from criterion, but criteria estimated from authored binary labels depend on each benchmark's example mix. We introduce FlipAct, a diagnostic of tool use in physical states. A rule over annotated tool properties computes each answer and a graded evidence score, fixing the Yes/No boundary at zero across models. Questions form 2-by-2 quads that hold required knowledge fixed while the state flips. The slope of the fitted Yes-rate curve measures sensitivity; its crossing at one half measures criterion. Across 44 open models, sensitivity rises with size within a family, and its ranking agrees with three of four external when-to-act benchmarks. Models respond to evidence sign rather than magnitude, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, most models are conservative across rephrasings; in some, this partly reflects a preference for No. Criterion is not determined by size. Activation steering shifts criterion with…

    PDF
  • arXiv cs.CVOct 11

    OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

    Hebaixu Wang, Jing Zhang, Haonan Guo, Di Wang, Jiayi Ma +1

    Multimodal remote sensing image generation emerges as a promising paradigm for recovering complete and complementary observations from heterogeneous and partially observed remote sensing data. However, existing methods predominantly depend on sensor-specific fine-tuning of natural image generation models, limiting unified multimodal generation capacity due to the absence of shared representations and robust conditioning mechanisms. To address these challenges, we propose OmniRSI, a unified multimodal remote sensing image generation model. Specifically, we develop a remote sensing-specific variational autoencoder that captures multimodal priors and promotes a locally smooth latent space for unified representation and robust reconstruction across diverse observed data. Building upon that, we fully adapt Stable Diffusion 3.5 with a ControlNet branch to support unified multimodal generation and translation conditioned on textual and visual cues. Besides, we devise a training-free cyclic latent alignment mechanism that iteratively enriches incomplete visual conditions with generated multimodal information, progressively improving guidance reliability while preserving observed…

    PDF
  • arXiv cs.CLOct 11

    From Candidate Tokens to Final Outcomes: Decoding Relative Long-Horizon Candidate Value from Reasoning Model Representations

    Yixin Yang, Yue Wang, Fandong Meng, Zhifang Sui

    Large reasoning models generate reasoning trajectories token by token. At a given prefix, multiple next-token candidates may be plausible, yet shape subsequent reasoning and differ in their chances of leading to the correct answer. Whether these differences are reflected in internal representations remains unclear. To study this question, we define relative long-horizon candidate value to quantify within-prefix differences and construct the Reasoning Candidate-Token Value (RCTV) dataset, pairing mathematical reasoning prefixes with plausible next-token candidates and their empirical long-horizon candidate values. Using RCTV, we examine hidden states, multilayer perceptron outputs, and attention outputs at candidate positions with sparse linear probes. Experimental results provide evidence that relative long-horizon candidate value is linearly decodable from these representations at multiple layers. Candidate-value probes can also outperform probability- and embedding-based baselines. Furthermore, a controlled online intervention experiment shows that using the decoded signals to guide token selection during generation can improve the chances of eventual success. Our findings…

    PDF
  • arXiv cs.AIOct 11

    CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

    Anahita Sharma, Apaar Shanker, Rishav Chakravarti, Sneha Srinivasan, Vijay Kalmath +4

    Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth -- a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA…

    PDF
  • arXiv cs.AIOct 11

    State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Study

    Dong Nie, Rui Lan, Yixin Nie, Lingzi Hong

    An end-to-end success rate alone cannot distinguish failure to learn an action from failure to apply it as task state changes. We study this distinction in deterministic tool workflows that provide the current artifact state and candidate action recipes. Using 1.7B, 8B, and 27B Qwen models, we compare supervised learning across workflow horizons and cross the state representation used for training with the one used at evaluation. Horizon-matched supervision yields high completion, but an H4-trained 1.7B model transfers poorly to longer workflows under full-state observations. In an H16 crossover, compact-trained 1.7B adapters complete 50/50 held-out tasks in compact view and 0/50 in full view for each of three paired training seeds; full-trained outcomes vary across seeds. A separate 8B adapter pair shows a similar asymmetry. The view dependence also appears in paired-choice tasks where both candidate calls are executable and the correct choice depends on table contents; here, the compact mask does not consult the target action. An explicit relevance cue improves full-view execution but does not close the gap for the tested checkpoint. In a separate 27B ScienceWorld study,…

    PDF
  • arXiv cs.AIOct 11

    A Decade of PhotoMappers: A Longitudinal Study of Crowdsourced Disaster Photos Geolocalization with Vision-Language Models and Geospatial Reasoning

    Hao Li, Wenping Yin, Fabian Deuser, Jia Jia, Ziqi Liu +3

    Crowdsourced disaster imagery provides timely, ground-level observations when conventional disaster mapping is delayed or incomplete, yet its geographic value depends on where observations are produced, how they are interpreted, and whether they can be reliably located as actionable Geospatial Intelligence (GEOINT). Using more than a decade of PhotoMappers, a crowdsourced disaster-mapping initiative, this study examines the changing geography of disaster information and asks how emerging GeoAI models compare with human geographic interpretation. We combine longitudinal spatial analysis with cross-view geolocalization using street view imagery (SVI) and remote sensing imagery (RSI), and evaluate vision-language models (VLMs) reasoning through 30 indicators spanning geographic cues and reasoning evidence. The results reveal a geographically and temporally uneven archive shaped by changing disaster types, Community Lifelines, and operational priorities. Integrating VGI, SVI, and RSI with Sinkhorn re-ranking achieves 40.90% R@1, 61.10% R@5, and 69.45% R@10, while yearly R@1 varies markedly from 25.00% to 65.07%. Geographic reasoning is similarly uneven: Qwen3.6-27B achieves an…

    PDF
  • arXiv cs.LGOct 11

    Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating

    Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang +2

    VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the…

    PDF
  • arXiv cs.LGOct 11

    GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

    Shengkun Tang, Cong Zeng, Zongfang Liu, Dan Alistarh, Zhiqiang Shen

    Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms…

    PDF
  • arXiv cs.LGOct 11

    WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

    Shengkun Tang, Dan Alistarh, Zhiqiang Shen

    Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation. However, the effect of coordinate deletion fundamentally depends on the basis in which pruning is performed. This raises a more fundamental question: how should the basis be chosen specifically for coordinate deletion? We introduce WUSPruner, a transform-based framework for structured pruning of large language models. We formulate basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning. Inspired by the second-moment transformation in WUSH, we consider a WUS transform constructed jointly from activation and weight statistics. We show that under positive-definite second moments, WUS transform minimizes the local reconstruction loss over all invertible changes of basis. In the resulting basis, the activation and weight second moments are simultaneously diagonalized and balanced, causing the deletion error to decompose into independent spectral contributions and yielding a principled criterion for coordinate selection. Guided by this result, we develop a scalable pruning method…

    PDF
  • arXiv cs.LGOct 11

    DC-Flow: Target-Conditioned Cyclic Peptide Design via Delayed-Coupling Flow Matching

    Yitian Wang, Minjie Cheng, Yaning Cui, Maohua Yang, Dongdong Wang +1

    Target-conditioned cyclic peptide design requires coordinating interfacial side chains with macrocyclic closure. Existing end-to-end generators concurrently update geometric and chemical variables from noise, assigning residues to disordered coordinates. Meanwhile, predictor-assisted workflows often get stuck in local traps because they lack feedback in the propose-and-filter stage and ignore 3D structural proposals during sequence search. To overcome the drawbacks of these two paradigms, we propose DC-FLOW, a staged generative framework built around Delayed-Coupling Flow Matching. In principle, DC-FLOW postpones sequence and side-chain activation until a target-conditioned backbone takes shape, resolving structural ambiguity before chemical assignment. Then, it feeds delayed activations back into the backbone, achieving co-evolution of backbone geometry and emerging residue identities by coupling their flows. This mechanism enables more reliable macrocyclic closure and favorable interface packing. For predictor-assisted workflows, DC-FLOW supplies 3D proposals with in-trajectory guidance, enabling cascaded test-time scaling to prune redundant backbones and terminate unpromising…

    PDF
  • arXiv cs.CROct 11

    From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents

    Binhong Wang, Haidong Kang, Yanfeng Zhang, Hong-Ning Dai, Guo Yu +1

    Audio Large Language Models (Audio LLMs) are embedded in agents for smart homes, vehicles, wearables, and embodied systems, where acoustic inputs can propagate from perception into persistent state and consequential execution. This setting extends security analysis beyond model behavior to the system conditions that determine how acoustic inputs can influence agent behavior and external actions. Existing surveys examine acoustic security, LLM-based agents, and edge deployment in isolation, leaving their security interactions in deployed Audio LLM-based agents insufficiently characterized. This survey introduces an Acoustic Sensing-to-Action (S2A) system model that traces these security implications along the path from acoustic sensing to external execution, while accounting for edge-cloud runtime and lifecycle transformation. We use the model to synthesize attacks, defenses, and evaluation methods across the acoustic, agent, and edge-system literature. The synthesis shows that security gaps can emerge within components and at the transitions between stages, where protections developed for one stage do not necessarily provide end-to-end assurance in the deployed system. These…

    PDF
  • arXiv cs.CVOct 11

    VideoEvoHarness: Self Evolving Agent Harness from Unlabeled Videos for Video Understanding

    Weixin Xu, Yifan Liu, Yi Wang, Mingfei Rong, Zhenyu Yang +3

    Video understanding agents rely on a harness to acquire evidence, use tools, and reason, but improving it requires substantial effort. Recent self evolution methods seek to optimize harnesses through execution feedback to reduce human intervention. However, existing methods often rely on annotated video questions and answers or focus on updating model parameters, leaving harness evolution from unlabeled videos underexplored. VideoEvoHarness jointly evolves a Questioner and a Solver harness from unlabeled videos with frozen model parameters. The Questioner constructs questions and computes reference answers directly from video memory records. Question quality assessments and Solver execution trajectories guide question generation policy revisions. The Solver supports evolving skills, strategies, hyperparameters, and tools using execution feedback. Executable strategies define workflows in code, reducing repeated interpretation of natural language instructions. Each round, candidate harnesses are compared on identical validation questions, with validated improvements retained for later use. Experiments on benchmarks and a dedicated benchmark question set, using videos disjoint from…

    PDF
  • arXiv cs.LGOct 11

    Approximation in Contrastive Representation Learning

    Yuanfan Li, Zihan Zhang, Yiming Ying, Ding-Xuan Zhou

    Contrastive representation learning (CRL) is a fundamental paradigm for learning transferable representations. Recent work identifies its population target as the pointwise mutual information (PMI) $\log\frac{p(x,y)}{p_X(x)p_Y(y)}$ between two modalities $X,Y$, where $p$ is the joint density function and $p_X,p_Y$ are the marginals. However, it remains unclear why standard inner-product scores $f(x)^\top g(y)$ can effectively approximate a generally nonseparable population target PMI. In this paper, we address this question from the aspect of approximation for CRL. By linking PMI to a compact operator, we show that $D$-dimensional inner-product models perform rank-$D$ spectral approximation, with error controlled by the spectral tail. We then study neural network realizations: in general settings, we show that deep neural network scoring functions are universally consistent, and the approximation error rates depend on the smoothness of the PMI function; under structured low-rank data models we obtain improved rates attained by moderate-width encoders. Finally, we establish a fast calibration rate from contrastive excess risk to downstream retrieval error under a density-ratio…

    PDF
Load more