Papers

Field

Top tools here

All tools

QwenPaw+467% downloads · 29.9k/wk+466.8%
jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%

Top tools here

Open problems in Agents

Evaluating long-horizon agentic behavior in real-world tasks4 papers raise it · 3 address it
Current benchmarks often simplify or omit the complex, multi-stage reasoning and exploration required for real-world professional tasks, limiting the assessment of true agentic capabilities.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver systems that meet the actual requirements, instead producing outputs that only appear correct.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents struggle to acquire and retain new knowledge through interaction in novel or dynamic settings, rather than relying on pretrained knowledge.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical context is relevant, leading to missed or misapplied information in long-term interactions.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Open-source, reproducible stacks for generalist computer-use agents1 paper raises it · 2 address it
Frontier implementations of generalist computer-use agents are often closed, limiting systematic study and community collaboration.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M
SunniesSeed · Oct 23$318.9k

Recently funded

Topics in this field

coding agents · code generation · coding agent318 papers63 this week
recursive self improvement · harness evolution · adaptive278 papers39 this week
agent · agentic · llm agents179 papers36 this week
skills · tasks · ai321 papers35 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 8
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu +2
We changed the agent: did it actually get better? Every self-improving agent loop answers this hundreds of times, and every answer comes from a verifier. On open-ended tasks none exists, so the loop is handed a hand-written rubric or a bare LLM judge grading output from a model like itself, inviting reward hacking and shared blind spots. We make the verifier the evolving object: an inspectable expression over small, mostly deterministic drawback detectors, synthesized from clustered failures, gated at birth, and selected for agreement with a ten-item anchored reference set plus consensus over unlabeled outputs, never for the agent's score. On MBPP+ it gains +0.21 held-out agreement over the hand-authored seed composition, on every seed, and ends ahead of the bare LLM judge it contains. One finding should change how co-evolved verifiers are validated: removing the anchor guards collapses the verifier into a vacuous always-pass grader, yet that collapsed verifier trains skills just as well. Downstream task score cannot certify a self-evolved verifier. Score does answer sufficiency, and there an evolved verifier can substitute: Double Ratchet, pairing the verifier with a…
PDF arXiv
arXiv cs.CLOct 8
SAIL: Scientific Agentic Intelligence via a Science-Aware Loop
SAIL Model Team, Boyuan Sun, Bryan Dai, Che Liu, Chi Liu +7
We introduce SAIL, an open model with 35B total and 3B active parameters for literature research, scientific coding, and multi-step research workflows. SAIL is developed through a science-aware improvement loop: agents built on frontier AI models analyze its task failures and construct training tasks that address the underlying capability gaps. The diagnosis examines search and evidence selection in literature tasks, scientific assumptions and reasoning in coding, and planning and revision in longer investigations. The agents draw on paper collections and scientific code repositories to build problems, interaction trajectories, and executable tasks with the required environments and tools. We repeat this loop over multiple development cycles and train SAIL through supervised fine-tuning, specialist training, multi-teacher on-policy distillation, and agentic reinforcement learning. SAIL achieves competitive performance across scientific research tasks with substantially fewer parameters than leading open-weight models.
PDF arXiv
arXiv cs.AIOct 8
Tracing the Thoughts of a Coding Agent Playing ARC-AGI-3: Lessons for Continual Learning
Chen Wu, Josh Passenger, Yin Song
We study how a coding agent learns across a sequence of abstract reasoning tasks. The agent runs on a frozen foundation model inside a fixed harness and acts by writing and running Python and shell scripts. It retains no state across turns other than its written artifacts, so every thought it forms, carries, corrects or abandons leaves a trace, where a thought is any belief, rule or plan committed to a file. We let the agent play ARC-AGI-3, a set of interactive reasoning games that provide no instructions. Each game is a sequence of levels, and a strategy that clears one level can fail on the next, so every new level is in effect a new task. The agent records what it learns as Python scripts and text notes, while the harness keeps a complete log of every action and observation. Our contribution is a measurement protocol that traces each thought through these files, from the task where it forms to the task where it is corrected or abandoned, applied to seven evaluation runs with three backbones from two model families. Scripts written for one task are almost never called again in a later task (33 of 630 references cross a task boundary), because most scripts embed the state of the…
PDF arXiv
arXiv cs.SEOct 8
Closed-loop evaluation of LLM agents for embedded software development
Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo
Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition…
PDF arXiv
arXiv cs.AIOct 8
Why machines will still not rule the world
Jobst Landgrebe, Barry Smith
In our book Why machines will never rule the world [13, 14] we argue that arti- ficial general intelligence is mathematically impossible. This is because the human beings and the processes which exhibit intelligence are complex systems whose be- haviour cannot be captured by the kinds of models that we can generate with or without computers. Proponents of contemporary machine intelligence respond with two lines of argument: a theoretical one, grounded in the universal approximation theorems for neural networks and the Church-Turing-Deutsch principle; and an em- pirical one, grounded in rapidly rising scores on standardized benchmarks. In this communication we examine and reject both responses. First, we show serious issues in the physicalist counter-argument based on the Church-Turing-Deutsch principle. Second, we review recent evidence to the effect that prominent benchmarks are compromised by training-data contamination, flawed test construction, and strate- gic optimization. Our central argument remains: That models required to perform cognitive behaviour in open-ended, thermodynamically complex and non-ergodic environments are not and will not become achievable.
PDF arXiv
arXiv cs.CLOct 8
Deception by Omission: Language Models Knowingly Hide Their Mistakes
Lucas Florin, Amelie Knecht, Ulysse Schaller, Thilo Hagendorff
Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed. Users then depend on the model to report what went wrong. An honest model discloses its mistakes, while a deceptive one conceals them. However, it is unclear how current LLMs behave in such situations. In this study, we prefill LLM trajectories with synthetic mistakes. The trajectories resemble real deployments in chat and agentic settings. Models fail to disclose their mistake in 36.4% of chat and 67.1% of agentic rollouts. In 2.4% and 5.3% of rollouts, respectively, they are aware of the mistake in their chain of thought but still deceptively conceal it. Rates vary by model: for instance, Gemini 3.5 Flash knowingly conceals mistakes in up to 19.9% of agentic rollouts. In 11.9% of chat and 51.8% of agentic rollouts, models show no awareness of mistakes, even though they reliably spot them when reviewing the same transcript as an outside observer. Our results show that, as agents take on more tasks with less oversight, users cannot rely on them to self-report possible mistakes. Developers should instead use independent monitors that review agent…
PDF arXiv

Papers

Top tools here

All tools

QwenPaw+467% downloads · 29.9k/wk+466.8%
jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%

Recently funded

Market

Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M
SunniesSeed · Oct 23$318.9k

Topics in this field

coding agents · code generation · coding agent318 papers63 this week
recursive self improvement · harness evolution · adaptive278 papers39 this week
agent · agentic · llm agents179 papers36 this week
skills · tasks · ai321 papers35 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week

Active contributors · 90 days

Activity · 12 weeks

Open problems in Agents

Evaluating long-horizon agentic behavior in real-world tasks4 papers raise it · 3 address it
Current benchmarks often simplify or omit the complex, multi-stage reasoning and exploration required for real-world professional tasks, limiting the assessment of true agentic capabilities.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver systems that meet the actual requirements, instead producing outputs that only appear correct.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents struggle to acquire and retain new knowledge through interaction in novel or dynamic settings, rather than relying on pretrained knowledge.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical context is relevant, leading to missed or misapplied information in long-term interactions.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Open-source, reproducible stacks for generalist computer-use agents1 paper raises it · 2 address it
Frontier implementations of generalist computer-use agents are often closed, limiting systematic study and community collaboration.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsaddresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 56m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

410papers this week−31.3% vs last week

All papers · 2.7k With code · 143

Newest Most relevant

2.4k papers this month

Recent papers

All papers in Research

arXiv cs.AIOct 8
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu +2
We changed the agent: did it actually get better? Every self-improving agent loop answers this hundreds of times, and every answer comes from a verifier. On open-ended tasks none exists, so the loop is handed a hand-written rubric or a bare LLM judge grading output from a model like itself, inviting reward hacking and shared blind spots. We make the verifier the evolving object: an inspectable expression over small, mostly deterministic drawback detectors, synthesized from clustered failures, gated at birth, and selected for agreement with a ten-item anchored reference set plus consensus over unlabeled outputs, never for the agent's score. On MBPP+ it gains +0.21 held-out agreement over the hand-authored seed composition, on every seed, and ends ahead of the bare LLM judge it contains. One finding should change how co-evolved verifiers are validated: removing the anchor guards collapses the verifier into a vacuous always-pass grader, yet that collapsed verifier trains skills just as well. Downstream task score cannot certify a self-evolved verifier. Score does answer sufficiency, and there an evolved verifier can substitute: Double Ratchet, pairing the verifier with a…
PDF arXiv
arXiv cs.CLOct 8
SAIL: Scientific Agentic Intelligence via a Science-Aware Loop
SAIL Model Team, Boyuan Sun, Bryan Dai, Che Liu, Chi Liu +7
We introduce SAIL, an open model with 35B total and 3B active parameters for literature research, scientific coding, and multi-step research workflows. SAIL is developed through a science-aware improvement loop: agents built on frontier AI models analyze its task failures and construct training tasks that address the underlying capability gaps. The diagnosis examines search and evidence selection in literature tasks, scientific assumptions and reasoning in coding, and planning and revision in longer investigations. The agents draw on paper collections and scientific code repositories to build problems, interaction trajectories, and executable tasks with the required environments and tools. We repeat this loop over multiple development cycles and train SAIL through supervised fine-tuning, specialist training, multi-teacher on-policy distillation, and agentic reinforcement learning. SAIL achieves competitive performance across scientific research tasks with substantially fewer parameters than leading open-weight models.
PDF arXiv
arXiv cs.AIOct 8
Tracing the Thoughts of a Coding Agent Playing ARC-AGI-3: Lessons for Continual Learning
Chen Wu, Josh Passenger, Yin Song
We study how a coding agent learns across a sequence of abstract reasoning tasks. The agent runs on a frozen foundation model inside a fixed harness and acts by writing and running Python and shell scripts. It retains no state across turns other than its written artifacts, so every thought it forms, carries, corrects or abandons leaves a trace, where a thought is any belief, rule or plan committed to a file. We let the agent play ARC-AGI-3, a set of interactive reasoning games that provide no instructions. Each game is a sequence of levels, and a strategy that clears one level can fail on the next, so every new level is in effect a new task. The agent records what it learns as Python scripts and text notes, while the harness keeps a complete log of every action and observation. Our contribution is a measurement protocol that traces each thought through these files, from the task where it forms to the task where it is corrected or abandoned, applied to seven evaluation runs with three backbones from two model families. Scripts written for one task are almost never called again in a later task (33 of 630 references cross a task boundary), because most scripts embed the state of the…
PDF arXiv
arXiv cs.SEOct 8
Closed-loop evaluation of LLM agents for embedded software development
Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo
Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition…
PDF arXiv
arXiv cs.AIOct 8
Why machines will still not rule the world
Jobst Landgrebe, Barry Smith
In our book Why machines will never rule the world [13, 14] we argue that arti- ficial general intelligence is mathematically impossible. This is because the human beings and the processes which exhibit intelligence are complex systems whose be- haviour cannot be captured by the kinds of models that we can generate with or without computers. Proponents of contemporary machine intelligence respond with two lines of argument: a theoretical one, grounded in the universal approximation theorems for neural networks and the Church-Turing-Deutsch principle; and an em- pirical one, grounded in rapidly rising scores on standardized benchmarks. In this communication we examine and reject both responses. First, we show serious issues in the physicalist counter-argument based on the Church-Turing-Deutsch principle. Second, we review recent evidence to the effect that prominent benchmarks are compromised by training-data contamination, flawed test construction, and strate- gic optimization. Our central argument remains: That models required to perform cognitive behaviour in open-ended, thermodynamically complex and non-ergodic environments are not and will not become achievable.
PDF arXiv
arXiv cs.CLOct 8
Deception by Omission: Language Models Knowingly Hide Their Mistakes
Lucas Florin, Amelie Knecht, Ulysse Schaller, Thilo Hagendorff
Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed. Users then depend on the model to report what went wrong. An honest model discloses its mistakes, while a deceptive one conceals them. However, it is unclear how current LLMs behave in such situations. In this study, we prefill LLM trajectories with synthetic mistakes. The trajectories resemble real deployments in chat and agentic settings. Models fail to disclose their mistake in 36.4% of chat and 67.1% of agentic rollouts. In 2.4% and 5.3% of rollouts, respectively, they are aware of the mistake in their chain of thought but still deceptively conceal it. Rates vary by model: for instance, Gemini 3.5 Flash knowingly conceals mistakes in up to 19.9% of agentic rollouts. In 11.9% of chat and 51.8% of agentic rollouts, models show no awareness of mistakes, even though they reliably spot them when reviewing the same transcript as an outside observer. Our results show that, as agents take on more tasks with less oversight, users cannot rely on them to self-report possible mistakes. Developers should instead use independent monitors that review agent…
PDF arXiv