Papers

Field

Top tools here

Coming with more sources: stars history

All tools

Transformers3 releases this week
faster-whisper+39% downloads · 3.7M/wk+38.6%
Whisper+38% downloads · 1.7M/wk+38.5%
Tokenizers+14% downloads · 46.6M/wk+14.2%
DeepSpeed+23% downloads · 319.1k/wk+23.1%

Top tools here

Open problems in Evals

Realistic end-to-end agent evaluation in ambiguous, live systems1 paper raises it · 1 addresses it
Current benchmarks often use well-specified tasks, failing to capture the ambiguity and dynamic constraints of real-world delivery work.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Scalable, realistic GUI agent training and evaluation environments1 paper raises it · 1 addresses it
GUI agent progress is limited by the cost and scarcity of interactive, realistic training and evaluation environments.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
Evaluating and engineering long-horizon agent harnesses1 paper raises it · 1 addresses it
Agent performance depends heavily on the harness, but existing benchmarks fix the harness and do not evaluate harness engineering itself.
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Comprehensive 3D consistency evaluation for generative video worlds1 paper raises it · 1 addresses it
Existing video generation benchmarks lack systematic, fine-grained evaluation of 3D consistency in interactive or revisited scenes.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
Fine-grained, interpretable evaluation of multimodal verifiers1 paper raises it · 1 addresses it
Binary verifiers provide limited feedback, and complex multimodal scenarios require decomposed, continuous scoring for better generalization.
- OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modelingraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLOct 14
From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence
Jiahong Qian
Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures actually consist of, and whether confidence signals can turn unreliable output into usable output. We run a preregistered, fully frozen evaluation of two new-generation low-cost LLMs (DeepSeek-V4-Flash, GLM-5.3-Flash) with a flagship-tier control (GPT-6.1-Sol) on a public corpus of 354 incident campaigns (315 unique reports, 630 model-report evaluations), scoring extraction quality against gold, confidence calibration, and robustness to single-sentence perturbations. Three findings emerge. (1) Unreliability persists across tiers: micro precision is 0.03-0.34 and verbalized calibration error (ECE) 0.53-0.94 across both splits; the gap does not close at the flagship tier. (2) The dominant failure is not hallucination: a full mention census plus a three-tier annotation of 360 sampled candidates (two independent LLM annotators from vendors outside the evaluated set, Cohen's kappa = 0.79 on the preregistered acceptance line, fully disclosed) back-projects to 65.6% background mentions and 19.9% gold-scope gaps, with only 1.2% unsupported…
PDF arXiv
arXiv cs.CLOct 14
From Triage to Impact: A Multi-Hop Reasoning Benchmark for Explainable Persian Hate Speech
Ghazal Kalhor, Sima Kamandloo, Mehrsa Shamsian, Behnam Bahrak
Hate speech has become increasingly prevalent on social media, raising concerns about its detection, interpretation, and moderation. Persian, a relatively low-resource language, lacks sufficient annotated data and computational resources for fine-grained hate speech analysis. In this study, we introduce a fine-grained, multi-hop benchmark for explainable Persian hate speech analysis, covering content triage, target identification, hate tactic and moral foundation analysis, textual evidence extraction, harmful narrative inference, and severity assessment. By evaluating state-of-the-art large language models (LLMs), we find that performance generally declines as tasks require more fine-grained semantic and inferential reasoning. Proprietary models generally outperform open-weight models, while reasoning-oriented models show no consistent advantage. Incorporating intermediate reasoning steps substantially improves severity prediction compared with one-shot assessment. Our analysis further reveals that Incitement to Violence or Harm is most strongly associated with higher severity, while political-affiliation targets are particularly associated with subversion-based moral framing.…
PDF arXiv
arXiv cs.CVOct 14
ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions
Tianyi Gao, Han Fang, Xin Wei, Yaoyao Zhong, Zhaoyan Pan +9
Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text-latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for…
PDF arXiv
arXiv cs.AIOct 14
Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry
Ren Zhuang, Ben Wang, Shuifa Sun
Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed reasoning state into a better one. We introduce Lifted State Policy Optimization (LSPO), which establishes this finite, cost-adjusted transition as the primary unit of learning. Each textual answer is augmented with a learned continuous coordinate, forming a lifted state that an energy function evaluates and orders. A shaped reward drives energy descent net of edit and step costs, and an internalization loss absorbs selected low-energy answers into the direct generator. On Qwen3.5-27B, the resulting single-pass policy reaches 71.8% aggregate accuracy across mathematics, science, code, and logic benchmarks, matching test-time reflection at 71.9% with 28% lower latency and 25.8% fewer tokens. The pattern holds across dense and mixture-of-experts backbones, and matched controls confirm that transition credit and the lifted coordinate each contribute, as does answer internalization. Our code is available at https://github.com/Venomrko/lspo.
PDF arXiv
arXiv cs.SECodeOct 14▲ 105
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu +6
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328…
PDF arXiv Code
arXiv cs.CVOct 14
Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Yuheng Chen, Yufan Chen, Zhuojun Cai, Ruiping Liu, Junwei Zheng +6
Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to…
PDF arXiv

Papers

Top tools here

Coming with more sources: stars history

All tools

Transformers3 releases this week
faster-whisper+39% downloads · 3.7M/wk+38.6%
Whisper+38% downloads · 1.7M/wk+38.5%
Tokenizers+14% downloads · 46.6M/wk+14.2%
DeepSpeed+23% downloads · 319.1k/wk+23.1%

Recent papers

All in Research →

arXiv cs.CLOct 14
From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence
Jiahong Qian
Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures actually consist of, and whether confidence signals can turn unreliable output into usable output. We run a preregistered, fully frozen evaluation of two new-generation low-cost LLMs (DeepSeek-V4-Flash, GLM-5.3-Flash) with a flagship-tier control (GPT-6.1-Sol) on a public corpus of 354 incident campaigns (315 unique reports, 630 model-report evaluations), scoring extraction quality against gold, confidence calibration, and robustness to single-sentence perturbations. Three findings emerge. (1) Unreliability persists across tiers: micro precision is 0.03-0.34 and verbalized calibration error (ECE) 0.53-0.94 across both splits; the gap does not close at the flagship tier. (2) The dominant failure is not hallucination: a full mention census plus a three-tier annotation of 360 sampled candidates (two independent LLM annotators from vendors outside the evaluated set, Cohen's kappa = 0.79 on the preregistered acceptance line, fully disclosed) back-projects to 65.6% background mentions and 19.9% gold-scope gaps, with only 1.2% unsupported…
PDF arXiv
arXiv cs.CLOct 14
From Triage to Impact: A Multi-Hop Reasoning Benchmark for Explainable Persian Hate Speech
Ghazal Kalhor, Sima Kamandloo, Mehrsa Shamsian, Behnam Bahrak
Hate speech has become increasingly prevalent on social media, raising concerns about its detection, interpretation, and moderation. Persian, a relatively low-resource language, lacks sufficient annotated data and computational resources for fine-grained hate speech analysis. In this study, we introduce a fine-grained, multi-hop benchmark for explainable Persian hate speech analysis, covering content triage, target identification, hate tactic and moral foundation analysis, textual evidence extraction, harmful narrative inference, and severity assessment. By evaluating state-of-the-art large language models (LLMs), we find that performance generally declines as tasks require more fine-grained semantic and inferential reasoning. Proprietary models generally outperform open-weight models, while reasoning-oriented models show no consistent advantage. Incorporating intermediate reasoning steps substantially improves severity prediction compared with one-shot assessment. Our analysis further reveals that Incitement to Violence or Harm is most strongly associated with higher severity, while political-affiliation targets are particularly associated with subversion-based moral framing.…
PDF arXiv
arXiv cs.CVOct 14
ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions
Tianyi Gao, Han Fang, Xin Wei, Yaoyao Zhong, Zhaoyan Pan +9
Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text-latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for…
PDF arXiv
arXiv cs.AIOct 14
Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry
Ren Zhuang, Ben Wang, Shuifa Sun
Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed reasoning state into a better one. We introduce Lifted State Policy Optimization (LSPO), which establishes this finite, cost-adjusted transition as the primary unit of learning. Each textual answer is augmented with a learned continuous coordinate, forming a lifted state that an energy function evaluates and orders. A shaped reward drives energy descent net of edit and step costs, and an internalization loss absorbs selected low-energy answers into the direct generator. On Qwen3.5-27B, the resulting single-pass policy reaches 71.8% aggregate accuracy across mathematics, science, code, and logic benchmarks, matching test-time reflection at 71.9% with 28% lower latency and 25.8% fewer tokens. The pattern holds across dense and mixture-of-experts backbones, and matched controls confirm that transition credit and the lifted coordinate each contribute, as does answer internalization. Our code is available at https://github.com/Venomrko/lspo.
PDF arXiv
arXiv cs.SECodeOct 14▲ 105
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu +6
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328…
PDF arXiv Code
arXiv cs.CVOct 14
Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Yuheng Chen, Yufan Chen, Zhuojun Cai, Ruiping Liu, Junwei Zheng +6
Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 34m ago

Evals

Benchmarks, evaluation methods, leaderboards and measuring model capabilities or failures.

108papers this week

All papers · 108 With code · 11

Newest Most relevant

108 papers this month

Active contributors · 90 days

Open problems in Evals

Realistic end-to-end agent evaluation in ambiguous, live systems1 paper raises it · 1 addresses it
Current benchmarks often use well-specified tasks, failing to capture the ambiguity and dynamic constraints of real-world delivery work.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Scalable, realistic GUI agent training and evaluation environments1 paper raises it · 1 addresses it
GUI agent progress is limited by the cost and scarcity of interactive, realistic training and evaluation environments.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
Evaluating and engineering long-horizon agent harnesses1 paper raises it · 1 addresses it
Agent performance depends heavily on the harness, but existing benchmarks fix the harness and do not evaluate harness engineering itself.
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Comprehensive 3D consistency evaluation for generative video worlds1 paper raises it · 1 addresses it
Existing video generation benchmarks lack systematic, fine-grained evaluation of 3D consistency in interactive or revisited scenes.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
Fine-grained, interpretable evaluation of multimodal verifiers1 paper raises it · 1 addresses it
Binary verifiers provide limited feedback, and complex multimodal scenarios require decomposed, continuous scoring for better generalization.
- OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modelingraises · addresses

Named by an AI from these papers' abstracts.