Spotlight

Papers

Field

Top tools here

Coming with more sources: stars history

All tools

Transformers3 releases this week
faster-whisper+39% downloads · 3.7M/wk+38.6%
Whisper+38% downloads · 1.7M/wk+38.5%
Tokenizers+14% downloads · 46.6M/wk+14.2%
DeepSpeed+23% downloads · 319.1k/wk+23.1%

Top tools here

Open problems in Evals

Realistic end-to-end agent evaluation in ambiguous, live systems1 paper raises it · 1 addresses it
Current benchmarks often use well-specified tasks, but real-world delivery requires handling ambiguity, live systems, and unstated constraints.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Scalable and realistic GUI agent training and evaluation environments1 paper raises it · 1 addresses it
GUI agent progress is limited by the high cost of interactive environments and scarcity of realistic training tasks.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
Evaluating long-horizon harness engineering for coding agents1 paper raises it · 1 addresses it
Existing benchmarks focus on task success under fixed harnesses, leaving harness engineering itself unexplored.
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Systematic 3D consistency evaluation for generative video worlds1 paper raises it · 1 addresses it
Current benchmarks do not comprehensively assess 3D consistency in interactive world models, leaving geometric and spatial stability underexplored.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
Fine-grained, interpretable evaluation for complex multimodal generation2 papers raise it · 2 address it
Binary or coarse-grained verifiers fail to provide detailed, actionable feedback for complex multimodal tasks like audio-video consistency or rule-governed world modeling.
- OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modelingraises · addresses
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

★ Spotlight this month

arXiv cs.AIOct 15

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng +5

Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. These findings suggest…

PDF arXiv page Details

Spotlight

Recent papers

All in Research →

arXiv cs.ROOct 15
LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation
Zhixu Li, Keqian Tang, Litian Gong, Jingyu Yao, Wenqian Zhang +5
Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a controlled benchmark for separating task completion from preference satisfaction. We introduce LIBERO-PeRM, a benchmark for preference learning in embodied agents. LIBERO-PeRM studies five questions: 1) how policies follow preferences in zero-shot settings; 2) how they learn single preferences from demonstrations; 3) how efficiently they acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and tasks; and 5) how policies compose preferences and behave under conflicts. We propose a controlled formulation and taxonomy of observable preferences in simulation, covering motion, skill, and user-profile choices. This formulation is restricted to language-expressible preferences that can be instantiated in LIBERO without changing the task goal, evaluated automatically, and reused across tasks. Building on LIBERO, our procedural pipeline generates paired demonstrations with and without preferences. We…
PDF arXiv
arXiv cs.CVOct 15
Grounding Generative Verification in 3D Content
Shuanglin Li, Jiaying Lin, Shuquan Ye
Recent advances in multimodal large language models (MLLMs) and 3D generation have expanded 3D understanding and synthesis. However, beyond visual fidelity, task-specific correctness grounded in 3D evidence remains a major challenge. We argue that building reliable 3D systems requires not only stronger generation and reasoning, but also verification feedback that identifies unmet requirements and explains their violations, allowing systems to reflect on and refine their outputs. Such feedback must integrate multi-view observations and distinguish genuine inconsistencies from projection-dependent appearances. We make three contributions. First, we introduce 3DVerifierBench, a benchmark spanning 10 tasks that evaluates decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study reveals a 21.6 percentage-point gap between humans and the strongest tested model, GPT-6 Astra, while explanation-aware analysis exposes weaknesses in rejection reliability. Second, we analyze verification across fine-grained factual categories, exposing errors in values and target binding. We develop a scalable pipeline combining multi-view fact…
PDF arXiv
arXiv cs.CLOct 15▲ 23
OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling
Bowei Liu, Xinchen Zhang, Xuhuan Li, Tianlin Zhang, Kaian Jiang +8
Generative visual verifiers play a crucial role in advancing multimodal intelligence. To enhance the robustness of visual verifiers in complex multimodal scenarios involving compositional requirements, we move beyond conventional binary verifiers that produce only True/False discrete judgments and introduce a verifier that automatically decomposes complex prompts or environmental rules into fine-grained components and assigns pointwise scores to provide a continuous and more informative critique signal. Our experiments further demonstrate that these structured and interpretable outputs improve the verifier's generalization to out-of-domain multimodal critic tasks. To train this verifier, we introduce a Decoupled RL training strategy that jointly optimizes sub-question-level binary judgment and global alignment scoring, using precise local feedback to develop fine-grained verification capabilities that provide a foundation for reliable continuous scoring at the global level. Under a general reinforcement learning framework, we first empirically demonstrate that the proposed pointwise graded verifier significantly outperforms binary verifiers in traditional text-to-image scenarios.…
PDF arXiv
arXiv cs.AICodeOct 15▲ 284
FDE-Gym: The Last Mile of AI Delivery
Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo +21
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56…
PDF arXiv Code
arXiv cs.LGOct 15
Lean Graph Search for Formal Theorem Proving
Yihan Xu, Jia-Chen Gu, Junyi Zhang, Di Wu, Haikang Deng +5
Retrieving relevant lemmas is crucial for formal theorem proving, enabling provers to build machine-checkable proofs from established mathematical results. Recent retrieval systems match library declarations to a target theorem or its intermediate proof steps, but make limited use of multi-hop dependencies among declarations. These dependencies can reveal supporting lemmas that are not necessarily lexically similar to the target theorem but recur in proofs of related problems. To this end, we propose LeanGraphSearch, a graph-based retrieval method for Lean 4 that constructs a graph from Mathlib's forward dependencies and traverses it to uncover supporting lemmas through multi-hop dependency chains. Specifically, LeanGraphSearch links each declaration to the premises it depends on, uses a graph-based ranking algorithm to propagate relevance from semantically matched declarations to their dependencies, and reranks the expanded candidates against the original query. Across two mathematical retrieval benchmarks, LeanGraphSearch improves recall by up to 12% over the strongest baseline, LeanSearch v2, on MathlibQR and increases coverage of complete annotated proof routes on MathlibMPR,…
PDF arXiv
arXiv cs.AIOct 15
Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
Trizal Garg, Sibayan Mitra, Arya Karanjkar, Advait Singh, Jagat Sesh Challa +4
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on average. The test cue also brings Indic refusal of harmful requests level with English, but raises refusal of benign Indic requests from 6.9% to 11.3%. We release all 47,880 judged responses to support further work. These results suggest that safety evaluations in Indic languages may not reflect how…
PDF arXiv

Papers

Top tools here

Coming with more sources: stars history

All tools

Transformers3 releases this week
faster-whisper+39% downloads · 3.7M/wk+38.6%
Whisper+38% downloads · 1.7M/wk+38.5%
Tokenizers+14% downloads · 46.6M/wk+14.2%
DeepSpeed+23% downloads · 319.1k/wk+23.1%

Activity · 12 weeks

Coming with more sources: repos

Active contributors · 90 days

★ Spotlight this month

arXiv cs.AIOct 15

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng +5

Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. These findings suggest…

PDF arXiv page Details

Fields · AI · updated 1h ago

Evals

Benchmarks, evaluation methods, leaderboards and measuring model capabilities or failures.

108papers this week

All papers · 108 With code · 11

Newest Most relevant

108 papers this month

Recent papers

All in Research →

arXiv cs.ROOct 15
LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation
Zhixu Li, Keqian Tang, Litian Gong, Jingyu Yao, Wenqian Zhang +5
Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a controlled benchmark for separating task completion from preference satisfaction. We introduce LIBERO-PeRM, a benchmark for preference learning in embodied agents. LIBERO-PeRM studies five questions: 1) how policies follow preferences in zero-shot settings; 2) how they learn single preferences from demonstrations; 3) how efficiently they acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and tasks; and 5) how policies compose preferences and behave under conflicts. We propose a controlled formulation and taxonomy of observable preferences in simulation, covering motion, skill, and user-profile choices. This formulation is restricted to language-expressible preferences that can be instantiated in LIBERO without changing the task goal, evaluated automatically, and reused across tasks. Building on LIBERO, our procedural pipeline generates paired demonstrations with and without preferences. We…
PDF arXiv
arXiv cs.CVOct 15
Grounding Generative Verification in 3D Content
Shuanglin Li, Jiaying Lin, Shuquan Ye
Recent advances in multimodal large language models (MLLMs) and 3D generation have expanded 3D understanding and synthesis. However, beyond visual fidelity, task-specific correctness grounded in 3D evidence remains a major challenge. We argue that building reliable 3D systems requires not only stronger generation and reasoning, but also verification feedback that identifies unmet requirements and explains their violations, allowing systems to reflect on and refine their outputs. Such feedback must integrate multi-view observations and distinguish genuine inconsistencies from projection-dependent appearances. We make three contributions. First, we introduce 3DVerifierBench, a benchmark spanning 10 tasks that evaluates decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study reveals a 21.6 percentage-point gap between humans and the strongest tested model, GPT-6 Astra, while explanation-aware analysis exposes weaknesses in rejection reliability. Second, we analyze verification across fine-grained factual categories, exposing errors in values and target binding. We develop a scalable pipeline combining multi-view fact…
PDF arXiv
arXiv cs.CLOct 15▲ 23
OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling
Bowei Liu, Xinchen Zhang, Xuhuan Li, Tianlin Zhang, Kaian Jiang +8
Generative visual verifiers play a crucial role in advancing multimodal intelligence. To enhance the robustness of visual verifiers in complex multimodal scenarios involving compositional requirements, we move beyond conventional binary verifiers that produce only True/False discrete judgments and introduce a verifier that automatically decomposes complex prompts or environmental rules into fine-grained components and assigns pointwise scores to provide a continuous and more informative critique signal. Our experiments further demonstrate that these structured and interpretable outputs improve the verifier's generalization to out-of-domain multimodal critic tasks. To train this verifier, we introduce a Decoupled RL training strategy that jointly optimizes sub-question-level binary judgment and global alignment scoring, using precise local feedback to develop fine-grained verification capabilities that provide a foundation for reliable continuous scoring at the global level. Under a general reinforcement learning framework, we first empirically demonstrate that the proposed pointwise graded verifier significantly outperforms binary verifiers in traditional text-to-image scenarios.…
PDF arXiv
arXiv cs.AICodeOct 15▲ 284
FDE-Gym: The Last Mile of AI Delivery
Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo +21
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56…
PDF arXiv Code
arXiv cs.LGOct 15
Lean Graph Search for Formal Theorem Proving
Yihan Xu, Jia-Chen Gu, Junyi Zhang, Di Wu, Haikang Deng +5
Retrieving relevant lemmas is crucial for formal theorem proving, enabling provers to build machine-checkable proofs from established mathematical results. Recent retrieval systems match library declarations to a target theorem or its intermediate proof steps, but make limited use of multi-hop dependencies among declarations. These dependencies can reveal supporting lemmas that are not necessarily lexically similar to the target theorem but recur in proofs of related problems. To this end, we propose LeanGraphSearch, a graph-based retrieval method for Lean 4 that constructs a graph from Mathlib's forward dependencies and traverses it to uncover supporting lemmas through multi-hop dependency chains. Specifically, LeanGraphSearch links each declaration to the premises it depends on, uses a graph-based ranking algorithm to propagate relevance from semantically matched declarations to their dependencies, and reranks the expanded candidates against the original query. Across two mathematical retrieval benchmarks, LeanGraphSearch improves recall by up to 12% over the strongest baseline, LeanSearch v2, on MathlibQR and increases coverage of complete annotated proof routes on MathlibMPR,…
PDF arXiv
arXiv cs.AIOct 15
Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
Trizal Garg, Sibayan Mitra, Arya Karanjkar, Advait Singh, Jagat Sesh Challa +4
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on average. The test cue also brings Indic refusal of harmful requests level with English, but raises refusal of benign Indic requests from 6.9% to 11.3%. We release all 47,880 judged responses to support further work. These results suggest that safety evaluations in Indic languages may not reflect how…
PDF arXiv

Open problems in Evals

Realistic end-to-end agent evaluation in ambiguous, live systems1 paper raises it · 1 addresses it
Current benchmarks often use well-specified tasks, but real-world delivery requires handling ambiguity, live systems, and unstated constraints.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Scalable and realistic GUI agent training and evaluation environments1 paper raises it · 1 addresses it
GUI agent progress is limited by the high cost of interactive environments and scarcity of realistic training tasks.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
Evaluating long-horizon harness engineering for coding agents1 paper raises it · 1 addresses it
Existing benchmarks focus on task success under fixed harnesses, leaving harness engineering itself unexplored.
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Systematic 3D consistency evaluation for generative video worlds1 paper raises it · 1 addresses it
Current benchmarks do not comprehensively assess 3D consistency in interactive world models, leaving geometric and spatial stability underexplored.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
Fine-grained, interpretable evaluation for complex multimodal generation2 papers raise it · 2 address it
Binary or coarse-grained verifiers fail to provide detailed, actionable feedback for complex multimodal tasks like audio-video consistency or rule-governed world modeling.
- OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modelingraises · addresses
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses

Named by an AI from these papers' abstracts.