Research

• Sensing communication16+23.1%
• Federated learning40+11.1%
• Agents491−2.2%
• Frontier LLMs424−9.4%
• Forecasting time75−16.7%
• Learning verifiable rewards164−17.6%
• Inference & serving246−18.3%
• Character recognition OCR15−21.1%

• arXiv cs.CLOct 19

  KEEP-Bench: Evaluating Evidence Preservation in Context Compression

  Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang, Kai-Wei Chang +1

  Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output…

  PDF
• arXiv stat.MLOct 19

  Teacher Supervision, Fixed Representations and Calibration: Exact Risk Calculations for Candidate-Scoring Decision Models

  Agus Sudjianto, Denis Burakov

  Candidate-scoring decision models return a probability distribution over a presented set of answers. Public documentation of one such system, Jev, and of an open counterpart, JevAny, describes three training ingredients that determine what these probabilities estimate: soft targets from larger models, a frozen pretrained backbone and temperature calibration. We analyze each ingredient in Gaussian surrogates where the finite-sample risk can be computed exactly. A student trained on teacher targets inherits the teacher's bias, which more student data does not remove and agreement with the teacher does not reveal. A fixed representation lowers the risk only below an explicit sample-size threshold. In a probit model the exact recalibration map is a temperature scaling, which removes all excess risk only when the teacher's coefficient has the correct direction and otherwise leaves a residual that calibration diagnostics do not detect. Under Luce's choice axiom, candidate scores are coherent across refinements of an event if and only if they aggregate by log-sum-exp. On 231,521 credit card complaints from the Consumer Financial Protection Bureau, where the decision is whether a…

  PDF
• arXiv cs.AIOct 19

  SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models

  Ruizhi Qian, Lei Shi, Tianyu Hu, Bo Yan, Song Wang +1

  Reasoning models often run into problems without noticing them, even when symptoms are visible in their own text. In humans, metacognition separates monitoring from control: fast, intuitive signals keep watch while slow, deliberate reasoning unfolds and help decide whether to continue, change course, or stop. System One (S1) decision models, such as Jev, are a natural candidate for this role: they return a calibrated probability for a natural-language question about a text in one fast call, yet they are used only as gates before reasoning. We introduce SOMA (System One Metacognitive Awareness), in which an S1 model continuously monitors a System Two (S2) reasoning model and intervenes when needed. SOMA has three layers: a scenario layer states each objective in natural language with predefined actions, an awareness layer lets S1 judge whether the described behavior appears, and a control layer accumulates these judgments, first reminds S2, and takes over only if the behavior persists. Sharing one S1 model and one controller across tasks, without training, SOMA addresses two different failures on two reasoning models: on overthinking, it cuts reasoning tokens by up to 36\% while…

  PDF
• arXiv cs.CLOct 19

  RESIN: Multi-Chain Reasoning with Entity-Structured Indexing

  Xuanyu Meng, Jash Rajesh Parekh, Jiawei Han

  Multi-chain reasoning is a challenging task not only because the multi-chain evidence is dispersed, but also because different parts of reasoning may require distinct, interacting chains of support. Flat retrieval can flood a model with loosely related text, whereas early selection of a single reasoning path can discard evidence that becomes useful only after other facts are considered. We introduce RESIN (Reasoning with Entity Structured Indexing), which uses structure to locate evidence while keeping the original source as the basis for reasoning. RESIN decomposes a question into focused one-hop needs, retrieves a small set of source-grounded units for each hop, and places only this relevant evidence in a compact reasoning buffer. The buffer preserves alternative supports until final composition, making intermediate decisions explicit without forcing a brittle path. On the full HolisQA benchmark, RESIN achieves 82.92 context recall and 76.70 answer relevancy over all questions, with higher reported CR and AR across five domains and higher F1 in four domains than the published baselines. Ablations show that both typed evidence diversity and hop-wise decomposition are essential,…

  PDF
• arXiv cs.CLOct 19

  EpiLens: Lightweight RAG for Epidemiological Knowledge Acquisition

  Vincenzo Perri, Ciro Cattuto, Daniela Paolotti

  To prepare for future outbreaks, researchers need to know what studies have been done and what data they use. Mapping a large literature by hand is slow, while large language models alone can misstate or conflate sources. We present EpiLens, a lightweight, open-source retrieval-augmented generation package that applies a fixed retrieval template and output schema to every paper in a corpus and links outputs to source passages. We build a corpus of 2,297 papers from the sources of a multi-pathogen parameter study and evaluate EpiLens's paper-classification workflow against 214 manually labelled papers. Without task-specific training data, the classifier matches or exceeds the tested supervised and zero-shot baselines across four tasks. The study's early extraction sheets cite published sources for 327 estimates. We link each estimate to a cited document or to corpus studies included by a review. Then, we use the classifier's labels to compare the papers associated with different parameters and pathogens. EpiLens labels more papers linked to reproduction numbers than fatality risks as model-based (54% versus 22%, compared with 62% versus 24% in the manual labels). The gap is…

  PDF
• arXiv cs.CLOct 19

  HoloCosm: Beyond Single-Metric Benchmark Compression Across Multiple Metrics

  Yupei Lin, Zhongzhan Huang, Junxin Li, Guoming Ling, Shanshan Zhong +3

  Benchmark compression reduces evaluation cost by selecting a small set of samples. Existing methods typically guide selection using a single metric, although benchmarks often assess models along multiple dimensions. We find that model rankings are often weakly correlated across metrics, indicating that different metrics capture distinct aspects of model performance. Samples selected for one metric do not reliably preserve scores and rankings on others when used for compressed evaluation. At 80\% compression, each of the five single-metric baselines yields higher reconstruction error than equal-size random sampling in over 70\% of cross-metric comparisons. To address this challenge, we present HoloCosm, which jointly minimizes normalized score-reconstruction error across source models and metrics while constraining ranking distortion. We evaluate HoloCosm across multiple compression rates on 15 language, multimodal, VLA, and agent benchmark configurations with 85 component metrics. With 80\% compression, HoloCosm outperforms Random, the strongest baseline overall: average NMAE drops by 35.3\% to 0.236, worst-component NMAE drops by 39.5\% to 0.330, and average Spearman correlation…

  PDF
• arXiv cs.LGOct 19

  Nested Shared-Operator Recursion on ARC-AGI-1: An Empirical Study of HRM, TRM and URM

  Sara Neri, Eugenio Gaeta, Andrea Ceschini, Andrea Facchinetti, Giuseppe Fico

  Recursive reasoning models refine persistent internal states by repeatedly applying learned operators. We investigate whether adding a persistent intermediate state that integrates repeated updates of a faster state through the same operator improves predictive performance relative to each model family's baseline at matched parameter counts. We define this organisation operationally and implement it within three architectures: the Hierarchical Reasoning Model (HRM), Tiny Recursive Model (TRM) and Universal Reasoning Model (URM). Using the official codebases on ARC-AGI-1, we compare each baseline with its parameter-matched nested variant. We evaluate held-out queries from tasks whose demonstrations were available during training. The experiments use a single-GPU budget far below the published training protocols. Under the tested conditions, nested variants show no consistent advantage in token accuracy, predictive loss or exact solving, which remains near zero for all configurations. The results therefore leave the hypothesis open. We specify a confirmatory comparison, with an informative baseline, multiple seeds, controls that isolate nesting and predefined criteria, to determine…

  PDF
• arXiv cs.CLOct 19

  The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

  Haonan Huang, Chengkun Cai

  Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal…

  PDF
• arXiv cs.CYOct 19

  Reasoning-Aware Multicalibrated Reliability Predictors for Language Models

  Ekaterina Fedorova, Alessandro Morosini, Chara Podimata

  As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable. In the context of hallucinations, existing work uses multicalibration to make such reliability predictions meaningful across application-relevant groups, such as prompt topics. Topic groups alone, however, can conceal systematic errors: within the same topic and confidence level, responses with different chain-of-thought (CoT) properties may exhibit opposing errors that cancel on average. In this work, we introduce reasoning-aware multicalibration, a framework that supplements application-relevant topic groups with groups derived from observable properties of CoT traces. We highlight that, theoretically, groupings based on CoT are "internally important", i.e., groupings for which the multicalibration guarantee is not societally crucial as it is for high-risk topics, but whose inclusion in the multicalibration procedure could easily and significantly improve predictive accuracy of the final reliability predictor. Empirically, we apply our framework to hallucination prediction…

  PDF
• arXiv cs.DCOct 19

  SafeWall: A Risk-Aware Job Walltime Recommendation for HPC Scheduling

  Kanglin Xu, Michael Papka, Zhiling Lan

  In high-performance computing (HPC), users must submit resource requirements for their jobs, with user-supplied walltime being a critical parameter. However, these estimates are notoriously inaccurate, as users tend to overestimate walltime to prevent the scheduler from terminating their jobs prematurely. While existing work on learning-based walltime prediction primarily focuses on maximizing prediction accuracy, such predictions are inherently risky for walltime recommendations. Specifically, underestimation leads to job termination before completion, resulting in lost progress and wasted resources. In this study, we present SafeWall, a novel recommendation method that adjusts walltime predictions by incorporating an explicit, nonnegative safety buffer to mitigate underestimation risk. The core of our approach is a risk-aware buffer that dynamically calibrates safety margins based on individual user's historical behavior. When user data is limited, the system employs statistical shrinkage to align these margins with a global baseline, ensuring robustness. This yields a lightweight mechanism that optimizes the trade-off between prediction accuracy and termination risk. SafeWall…

  PDF
• arXiv cs.LGOct 19

  Strength Controls Change How Interventions Compare Across Sparse Autoencoder Codes

  Jinchen Zhang, Zheyuan Liu, Yihan Zhu, Meng Jiang

  Changing sparse autoencoder (SAE) codes can alter both intervention direction and magnitude, so comparing intervention effects requires a strength control specifying what is held equal. Norm matching and output calibration provide different controls, but whether interventions yield different effects across codes after strength matching remains unclear. We compare intervention effects under these controls for fixed feature groups in Gemma and Llama and observe: (1) Whole-patch scaling. With one coefficient per input, full-output Kullback-Leibler (KL) calibration leaves smaller mean effect gaps from the encoder-based reference intervention than total-norm matching for nonnegative least-squares (NNLS) refits. On new Llama inputs, total-norm matching loses 11 reference transfers, whereas full-output calibration preserves every transfer decision. (2) Positionwise scaling. On a small matched gender-agreement subset, positionwise norm matching leaves smaller observed mean gaps than full-output calibration. (3) Answer exclusion. Compared with unscaled NNLS, answer-excluded calibration reduces gaps in the measured subject-verb agreement comparisons but increases them for gender refits.…

  PDF
• arXiv cs.CLOct 19

  Training Negatives Still Matter After Calibrating Answerability Readouts

  Jinchen Zhang, Zheyuan Liu, Yihan Zhu, Meng Jiang

  Answerability readouts predict whether a passage supplies an answer, but whether calibration can compensate for removing negative passages during training remains unclear. We present a controlled study of how training-negative construction affects calibrated prediction, ranking, and downstream answer retention. We compare native training, which retains negative passages, with empty training, which removes them, holding questions, labels, positive examples, and model weights fixed. Both are calibrated on the same labeled native examples. On SQuAD 2.0 and RepLiQA, native-trained logistic readouts at selected depths achieve lower Brier loss for Qwen2.5-7B-Instruct and Gemma-2-9B-it, with or without output scores. Native fusion also outperforms calibrated output alone. Post-hoc AUROC analysis shows better ranking under native training, beyond what threshold adjustment alone can explain. On the official SQuAD 2.0 development set, native readout policies improve F1 over empty policies but retain more answers to unanswerable questions. The advantage varies by estimator and depth, while RepLiQA's text shortcuts limit claims about contextual understanding. Unfinished generations prevent…

  PDF
• arXiv cs.SDOct 19

  Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente

  Marco Giordano, Stefano Giacomelli, Claudia Rinaldi, Fabio Graziosi

  Ground truth in Speech Emotion Recognition (SER) is often presented as a single categorical fact, yet an acted emotion and the emotion heard by listeners describe different aspects of the same recording. We examine this distinction in Emozionalmente, an acted Italian corpus that provides both an elicited production label and five listener judgements for every clip. Using frozen XLS-R, HuBERT, and WavLM representations with matched probes, we compare three label authorities: the expressed condition, the listeners' hard plurality, and their complete vote distribution. The results tell a consistent story. Recovering the expressed condition is easier than predicting a tie-broken listener plurality, but that gap compares two different questions rather than revealing a general loss of SER ability. As listener agreement weakens, the plurality becomes an increasingly brittle hard target. Training on the full vote distribution does not substantially change which class wins the model's hard decision, but it produces probability assignments that more closely follow the alternatives selected by listeners, especially for ambiguous clips. No authority therefore replaces the others. Expressed…

  PDF
• arXiv cs.CLOct 19

  WenSight-Decision-1 for Direct Decisions under E-commerce Rules

  Mingwen Liu, Jun Sun, Sichen Wang

  E-commerce decisions require applying natural-language rules to the available case evidence. We introduce WenSight-Decision-1, a shared model that scores supplied options directly from rules and case information for six operational tasks, with examples spanning six languages. We fine-tune Qwen3.5-9B with reference options as supervision, using the same option scores for training and inference. On 1,440 synthetic operational cases reserved for final evaluation, accuracy rises from 61.60% for Qwen3.5-9B to 92.92% for WenSight-Decision-1 under identical inputs and scoring. Joint fine-tuning improves all six tasks, bringing heterogeneous operational judgments under a common decision interface.

  PDF
• arXiv cs.CLOct 19

  SNO-PI: Label-Scarce SNOMED CT Entity Linking with Pre-trained Encoders and Curated Terminology Knowledge

  Piyawoot Songsiritat

  We present SNO-PI, the top-performing submission on SNOMED International's SNOMED CT Entity Linking Benchmark as of 18 October 2026. SNO-PI achieves a macro character IoU of 0.5387, outperforming the organisers' reference system (0.4582; based on the 2024 challenge winner's approach) by 18% relative. Linking clinical mentions to SNOMED CT is a zero- and few-shot problem over a very large label space: the 272 annotated MIMIC-IV discharge summaries cover only 6,595 of the roughly 219,000 eligible concepts, and a third of those occur only once. This motivates a design built on transfer from pre-trained models and human-curated knowledge. SNO-PI combines high-recall span proposal, SapBERT candidate retrieval and listwise ensemble reranking with three BERT-family encoders, each further pre-trained with masked language modelling on unlabelled MIMIC-IV notes and then on SNOMED CT synonyms and hierarchy before task fine-tuning. We find that curated knowledge helps most on concepts never seen in training, yet these concepts remain the main source of error.

  PDF
• arXiv cs.CLOct 19

  Self-Consistency of Entity Sets and Citations in a Deep Research System under Repeated Prompting

  Lauri Pykälä, Kimmo K. Kaski

  Deep research agents (DRA) are commonly used for open-ended synthesis tasks where closed-world QA benchmarks are poorly suited to characterize relevant aspects of performance. We study the self-consistency of one commercial deep research system under repeated identical prompting, representing each report with the set of named entities it mentions, categorised by a custom schema, and the set of citations it retrieves. Across repeated runs of a single open-ended cybersecurity related prompt, we quantify pairwise overlap to characterize how stable the retrieved and synthesized output structure is under repeated identical prompts. In ten runs, the pairwise entity Jaccard results in a median of 0.12 and the pairwise citation Jaccard is below 0.02, with discovery curves that do not visibly saturate. The magnitude and structure of (dis)agreement between entity sets varies by entity class, which suggests that self-consistency in a DRA is better understood as a function of the underlying entity space than as a stable property of the system.

  PDF
• arXiv cs.SIOct 19

  Where Does Rhetorical Distortion Live? A Multi-Platform, Multi-Language Measurement of Influencer Content

  Xucheng Yu, Jiangping Chen, Haohan Wang

  Much of the manipulation in online influencer content is not false but disproportionate: an accurate fact is framed as historic, urgent, or catastrophic. Fact-checking and clickbait detection are not designed to capture such framing, and little is known about where it originates: with the creator, the topic, the platform, or the audience that replies. We operationalize rhetorical distortion through five literature-grounded dimensions (Significance Inflation, Anxiety Manufacturing, Novelty Claims, Loaded Language, and Temporal Distortion) and measure it with a three-tier pipeline in which bilingual lexical rules propose candidates, negative filters cancel known false-positive contexts, and an LLM verifies or relabels the candidates. Applying the pipeline to 13,060 posts from 330 accounts on six platforms (RSS/newsletters, YouTube, Bluesky, Reddit, Weibo, and X/Twitter), we find that platform is far more strongly associated with account-level distortion than content category (Kruskal-Wallis $\varepsilon^2=0.33$; two-way ANOVA $η_p^2=0.49$ for platform vs. $0.09$ for category). X/Twitter scores highest (DI 7.74 vs. 0.78 for RSS). Classifying tweets and their replies separately shows…

  PDF
• arXiv cs.CVOct 19

  One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution

  Xiaokang Yuan, Binhui Yao, Luwei Yang, Ruimin Chen, Zeng Wang

  Document understanding is increasingly judged on where an answer comes from, scored against annotated gold locations as if each fact lived in one place. In long visual documents, a fact often lives in several places: a chart value is restated in the text, a key figure reappears in the summary. We ask how often such copies occur in standard benchmarks and what they do to evaluation. We annotate 420 questions from MMLongBench-Doc and MMDocRAG with the locations that independently establish each required fact, checked by two independent reviewers. Copies are common and often cross-modal: 19.5% of MMLongBench-Doc test questions have a strict copy outside the gold, and in 28% of MMDocRAG questions a non-gold candidate alone supplies a required fact. Single-gold scoring then fails in three predictable ways. A retriever that finds a copy is scored as a miss: page recall@1 of text and visual retrievers is underestimated by 13-24 points on questions with a cross-page copy. A model that cites a copy is scored as citing wrongly: crediting equivalent evidence reverses 11% of pairwise orderings among 65 competitive MMDocRAG systems. Masking the gold region leaves the fact available elsewhere:…

  PDF
• arXiv cs.CLOct 19

  How Benchmark Graph Construction Shapes Multi-Hop KG Navigation: A Controlled Audit of PathQuestion and PathQuestion-Large

  Eduin E. Hernandez, Sergio A. Diaz, Luis F. Garcia, Nurassyl Askar, Stefano Rini

  Multi-hop knowledge-graph question answering (KGQA) benchmarks are often treated as fixed language-and-reasoning tasks, but for navigation-based methods, difficulty also depends on the graph in which those tasks are instantiated. This makes graph construction a consequential but often entangled part of the navigation task. To isolate this effect, we audit PathQuestion (PQ) and PathQuestion-Large (PQL) as graph-navigation benchmarks under a controlled intervention: we hold questions, topic entities, released answers, released path annotations, and family-level splits fixed while replacing each released task-filtered graph with the larger source KG distributed with the benchmark. Source restoration expands the effective navigation search space and graph-consistent answer and path multiplicity while sharply reducing unguided reachability to released answers. We evaluate whether these structural changes persist across navigation policies using structural references, frozen small-language-model navigation policies, and trained MINERVA policies. Across both frozen and trained policies, restoration substantially reduces released-answer accuracy and trajectory fidelity, while…

  PDF
• arXiv cs.AIOct 19

  Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization

  Yupeng Chang, Linyuan Gao, Zhiyuan Li, Yuan Wu, Yi Chang

  An inference budget is useful only when the system delivers an answer within it. We study a fixed-budget policy combining first-valid-box stopping with a charged finalization cue. On 100 fresh, independently audited GSM8K test questions per model, it improves delivered endpoint accuracy over first-box stopping by 8.0 percentage points for Qwen3-4B-Thinking-2507 and 7.0 for DeepSeek-R1-Distill-Llama-8B. A prelocked repeat on the same questions gives 7.0 and 10.0 points; neither seed shows a gross harm relative to first-box stopping. The point estimates of incremental physical-slot savings are below one percent. A finalization-only ablation separates cue recovery from stopping, and a fixed scalar reader tests extraction as an alternative explanation. The cue is prior art; the contribution is a component and readout analysis with charged injections and verified paths. Results cover two checkpoints, one task, one prompt family, and two fixed seed conditions.

  PDF

• Structural decision-making in LLMs1 paper raises it · 3 address it

  Current LLMs lack native support for decision problems with structural requirements like null-awareness, permutation invariance, and scalability.

  • Jeva: System One Models from First Principlesraises · addresses
  • LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
  • SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
• Adaptive computation for decision models1 paper raises it · 1 addresses it

  Fixed-depth decision models waste computation on easy questions and cannot scale depth dynamically.

  • LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
• Reliable no-thinking behavior in LLMs1 paper raises it · 1 addresses it

  LLMs often ignore or misinterpret no-thinking instructions, leading to unwanted reasoning traces.

  • Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
• Memory consolidation in context-augmented LLMs1 paper raises it · 1 addresses it

  Gains from context are lost after its removal, and models struggle to retain learned knowledge without forgetting.

  • Cortex: Consolidating Context into Parametric Memoryraises · addresses
• Sustaining self-evolution in reasoning models1 paper raises it · 1 addresses it

  Self-generated training data degrades over rounds due to invalid or repetitive questions, causing performance collapse.

  • Questioning the Questions: Sustaining Self-Evolution in Reasoning Modelsraises · addresses

Named by an AI from these papers' abstracts.