Papers

Field

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+181% downloads · 13.5k/wk+180.6%
Vercel AI SDK400 releases this week
ONNX Runtime+35% downloads · 37.1M/wk+35.2%

Top tools here

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not capture the complexity of open, dynamic, and interactive web tasks, limiting the assessment of agent capabilities.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Memory systems often underperform naive retrieval, and their trade-offs between gains and losses (memory tax) are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Benchmark difficulty distribution mismatch and coverage gaps1 paper raises it · 1 addresses it
Benchmarks may misrepresent intended difficulty or miss key regions, leading to misleading evaluations of model progress.
- Your Benchmark Is an Empirical Measure Over Difficultyraises · addresses
Structural requirements for decision-native models1 paper raises it · 2 address it
Current decision models often lack null-awareness, permutation invariance, option scalability, and cacheability, limiting their robustness.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 2 address it
Agents struggle to extract and apply user habits from sparse or noisy historical interactions, limiting personalization.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
- Vela: User Behavior as Supervision Sparks LLM Intelligenceaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Kuaishou Keling AIStage not given · Oct 31undisclosed
Kling AIStage not given · Oct 30$3B
CohereSeries B+ · Oct 29$7B
VinciSeries B+ · Oct 29$250M
KineDBPre-seed · Oct 29$949.6k

Recently funded

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented291 papers38 this week
language models llms · large language models · language models314 papers34 this week
generative recommendation · sequential recommendation · personalized180 papers28 this week
memory construction · persistent memory · long term memory293 papers27 this week
deep research · scientific literature · citations163 papers21 this week
typed decision models · typed decision model · typed decisions160 papers20 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CLSep 27
Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$\rightarrow$FOL
Pu Suo, Ali Emami
A standard pipeline for symbolic reasoning over natural-language problems translates them into first-order logic and invokes a theorem prover. The translation step is the bottleneck: swap "every" for "some" and every inference that follows is corrupted. Yet today's metrics often score more broken translations higher than less broken ones, because BLEU, BERTScore, and Smatch++ reward surface overlap that the worst errors happen to preserve. We introduce SIV, which derives two kinds of probes from the target formula and uses a theorem prover to verify the candidate translation against each. Positive probes are statements the candidate must entail, which detect translations that drop content; contrastive probes are statements the candidate must not entail, which detect translations that assert more than the original. On a controlled pool of perturbed FOLIO translations, the severity of the error accounts for 80% of SIV's score variance, compared with at most 17% for any prior metric. Across six error classes on a disjoint pool, SIV scores the reference above the perturbed candidate in over 99% of pairs. Because each probe is labeled with what it tests, the failure pattern also…
PDF arXiv
arXiv cs.CRSep 27
The Price of Peeking: Anytime-Valid Leakage Detection on ML-KEM EM Traces
Georgios Feretzakis, Alexandros Papaspyridis
Side-channel evaluators routinely inspect leakage tests while acquisition is still running, and extend or stop the campaign based on what they see. Fixed-horizon screening such as the Welch $t$-test with threshold $|t|>4.5$ gives no error guarantee for this monitored decision rule. We study anytime-valid leakage detection based on testing by betting: SKIT-type swap-pair e-processes whose false-alarm probability is controlled uniformly over time under an explicit conditional symmetry null. In matched comparisons that share the frozen witness, rows and payoff, first-crossing detection needed 1.68-2.00$\times$ the traces of a fixed-horizon randomization test at 80% detection on synthetic streams, and 1.68-2.38$\times$ on degraded recordings from an open ML-KEM electromagnetic dataset with the primary Ridge witness at $α=0.05$. With the same primary witness and level, on undegraded reference and pqm4 recordings the monitored procedure stopped early: its median stopping point was 62-72 and 146-316 evaluation traces, i.e. 2-8% of a conservative 4096-trace budget. Under exact designed nulls on the recorded backgrounds, repeated-look $|t|>4.5$ screening over all 13000-20000 samples…
PDF arXiv
arXiv cs.CLSep 27
Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring
Gabriele Cinà
A speech neuroprosthesis decodes attempted speech from brain activity and ends by rescoring the decoder's candidate sentences with a language model of several billion parameters, the only component that needs a GPU. Replacing that model with a cheaper one is hard: general language models asked to pick one sentence from a list answer from where a label sits in the list rather than from the sentence itself. We pose rescoring as a single typed decision, one call that returns a probability for every candidate, served by Jev, a hosted model trained for calibrated decisions, and combine it with the decoder's own score. On 978 held-out sentences from a participant with ALS, where the published decoder alone reaches 8.1% word error, Jev reaches 7.5% against 7.8% for both OPT-6.7b and Qwen2.5-7B; with the decoder's weight re-tuned, 6.9% against 7.2% and 7.4%. Jev is ahead in all four comparisons and at most 0.2 points behind at the 95% bound. It costs 0.07 USD per thousand sentences and needs no GPU; a dedicated GPU running a 7B model is cheaper per sentence only above 43% utilisation, far beyond what one user generates. End-to-end latency over the internet is 262 ms, of which 62 ms is…
PDF arXiv
arXiv cs.CLSep 27
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi
Natural Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human languages, ranging from lexical ambiguity to high-level reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models' limitations and capabilities, so optimizing models' generalization. Notably, several benchmarks contain diagnostics datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics dataset, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN…
PDF arXiv
arXiv cs.AISep 27
What Happens During Autonomous Deep Research After the User Steps Away?
Yimin Liu, Yijia Zhang, Yanmin Li, Tangwen Luo, Yuze Li +2
In autonomous deep research, a user provides a task and relevant background, then leaves the agent to conduct an extended investigation without further human intervention. We study how this initial user information is reflected in intermediate actions and how these actions relate to final recommendations. We introduce DRaligned, a counterfactual behavioral evaluation framework built on PDR-Bench. By varying one task-relevant user factor while keeping the remaining context fixed, we compare acquisition requests, working drafts, and final reports. Source-grounded extraction, blinded local judgments, and deterministic aggregation yield coarse directional measurements while leaving ambiguous cases unresolved. Our experiments show that strong user-specific delivery can emerge from a largely shared research process: agents investigate similar broad questions but allocate requests differently, and final recommendations distinguish user conditions more clearly than explicit requests do. Reports can also integrate user factors that were not jointly visible during acquisition. In readable draft-to-report comparisons, recommendations often retain their coarse user-specific direction despite…
PDF arXiv
arXiv cs.AISep 27
When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routing
Jian Chen, Zixuan Yuan
Modern learned systems increasingly combine learned components with search, repair, or external solvers. Benchmarks often measure the resulting end-to-end system, while scientific claims may concern only one component, creating an attribution problem: evidence can fail to support the requested component-level claim while still supporting a positive conclusion about the larger system. Existing evidence-to-claim methods primarily calibrate claim strength. We argue that composite systems require a second dimension: scientific subject. We address this problem with subject-typed claim licensing, which separates weaker conclusions about the requested subject from positive but non-substitutive credit about another subject. We instantiate this idea in SCOPE-Routing for preference-conditioned multigraph routing. Non-authors reproducibly apply the declared semantics; held-out review yields fewer reference-relative upward deviations than unstructured review, while the difference from a strong evidence checklist remains unresolved; and a controlled routing study shows that score-optimal and claim-eligible methods can differ while valid hybrid-system credit is preserved. These results…
PDF arXiv

Papers

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+181% downloads · 13.5k/wk+180.6%
Vercel AI SDK400 releases this week
ONNX Runtime+35% downloads · 37.1M/wk+35.2%

Recently funded

Market

Kuaishou Keling AIStage not given · Oct 31undisclosed
Kling AIStage not given · Oct 30$3B
CohereSeries B+ · Oct 29$7B
VinciSeries B+ · Oct 29$250M
KineDBPre-seed · Oct 29$949.6k

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented291 papers38 this week
language models llms · large language models · language models314 papers34 this week
generative recommendation · sequential recommendation · personalized180 papers28 this week
memory construction · persistent memory · long term memory293 papers27 this week
deep research · scientific literature · citations163 papers21 this week
typed decision models · typed decision model · typed decisions160 papers20 this week

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 5m ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

286papers this week−36.4% vs last week

All papers · 2.7k With code · 80

Newest Most relevant

1.8k papers this month

Recent papers

All papers in Research

arXiv cs.CLSep 27
Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$\rightarrow$FOL
Pu Suo, Ali Emami
A standard pipeline for symbolic reasoning over natural-language problems translates them into first-order logic and invokes a theorem prover. The translation step is the bottleneck: swap "every" for "some" and every inference that follows is corrupted. Yet today's metrics often score more broken translations higher than less broken ones, because BLEU, BERTScore, and Smatch++ reward surface overlap that the worst errors happen to preserve. We introduce SIV, which derives two kinds of probes from the target formula and uses a theorem prover to verify the candidate translation against each. Positive probes are statements the candidate must entail, which detect translations that drop content; contrastive probes are statements the candidate must not entail, which detect translations that assert more than the original. On a controlled pool of perturbed FOLIO translations, the severity of the error accounts for 80% of SIV's score variance, compared with at most 17% for any prior metric. Across six error classes on a disjoint pool, SIV scores the reference above the perturbed candidate in over 99% of pairs. Because each probe is labeled with what it tests, the failure pattern also…
PDF arXiv
arXiv cs.CRSep 27
The Price of Peeking: Anytime-Valid Leakage Detection on ML-KEM EM Traces
Georgios Feretzakis, Alexandros Papaspyridis
Side-channel evaluators routinely inspect leakage tests while acquisition is still running, and extend or stop the campaign based on what they see. Fixed-horizon screening such as the Welch $t$-test with threshold $|t|>4.5$ gives no error guarantee for this monitored decision rule. We study anytime-valid leakage detection based on testing by betting: SKIT-type swap-pair e-processes whose false-alarm probability is controlled uniformly over time under an explicit conditional symmetry null. In matched comparisons that share the frozen witness, rows and payoff, first-crossing detection needed 1.68-2.00$\times$ the traces of a fixed-horizon randomization test at 80% detection on synthetic streams, and 1.68-2.38$\times$ on degraded recordings from an open ML-KEM electromagnetic dataset with the primary Ridge witness at $α=0.05$. With the same primary witness and level, on undegraded reference and pqm4 recordings the monitored procedure stopped early: its median stopping point was 62-72 and 146-316 evaluation traces, i.e. 2-8% of a conservative 4096-trace budget. Under exact designed nulls on the recorded backgrounds, repeated-look $|t|>4.5$ screening over all 13000-20000 samples…
PDF arXiv
arXiv cs.CLSep 27
Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring
Gabriele Cinà
A speech neuroprosthesis decodes attempted speech from brain activity and ends by rescoring the decoder's candidate sentences with a language model of several billion parameters, the only component that needs a GPU. Replacing that model with a cheaper one is hard: general language models asked to pick one sentence from a list answer from where a label sits in the list rather than from the sentence itself. We pose rescoring as a single typed decision, one call that returns a probability for every candidate, served by Jev, a hosted model trained for calibrated decisions, and combine it with the decoder's own score. On 978 held-out sentences from a participant with ALS, where the published decoder alone reaches 8.1% word error, Jev reaches 7.5% against 7.8% for both OPT-6.7b and Qwen2.5-7B; with the decoder's weight re-tuned, 6.9% against 7.2% and 7.4%. Jev is ahead in all four comparisons and at most 0.2 points behind at the 95% bound. It costs 0.07 USD per thousand sentences and needs no GPU; a dedicated GPU running a 7B model is cheaper per sentence only above 43% utilisation, far beyond what one user generates. End-to-end latency over the internet is 262 ms, of which 62 ms is…
PDF arXiv
arXiv cs.CLSep 27
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi
Natural Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human languages, ranging from lexical ambiguity to high-level reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models' limitations and capabilities, so optimizing models' generalization. Notably, several benchmarks contain diagnostics datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics dataset, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN…
PDF arXiv
arXiv cs.AISep 27
What Happens During Autonomous Deep Research After the User Steps Away?
Yimin Liu, Yijia Zhang, Yanmin Li, Tangwen Luo, Yuze Li +2
In autonomous deep research, a user provides a task and relevant background, then leaves the agent to conduct an extended investigation without further human intervention. We study how this initial user information is reflected in intermediate actions and how these actions relate to final recommendations. We introduce DRaligned, a counterfactual behavioral evaluation framework built on PDR-Bench. By varying one task-relevant user factor while keeping the remaining context fixed, we compare acquisition requests, working drafts, and final reports. Source-grounded extraction, blinded local judgments, and deterministic aggregation yield coarse directional measurements while leaving ambiguous cases unresolved. Our experiments show that strong user-specific delivery can emerge from a largely shared research process: agents investigate similar broad questions but allocate requests differently, and final recommendations distinguish user conditions more clearly than explicit requests do. Reports can also integrate user factors that were not jointly visible during acquisition. In readable draft-to-report comparisons, recommendations often retain their coarse user-specific direction despite…
PDF arXiv
arXiv cs.AISep 27
When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routing
Jian Chen, Zixuan Yuan
Modern learned systems increasingly combine learned components with search, repair, or external solvers. Benchmarks often measure the resulting end-to-end system, while scientific claims may concern only one component, creating an attribution problem: evidence can fail to support the requested component-level claim while still supporting a positive conclusion about the larger system. Existing evidence-to-claim methods primarily calibrate claim strength. We argue that composite systems require a second dimension: scientific subject. We address this problem with subject-typed claim licensing, which separates weaker conclusions about the requested subject from positive but non-substitutive credit about another subject. We instantiate this idea in SCOPE-Routing for preference-conditioned multigraph routing. Non-authors reproducibly apply the declared semantics; held-out review yields fewer reference-relative upward deviations than unstructured review, while the difference from a strong evidence checklist remains unresolved; and a controlled routing study shows that score-optimal and claim-eligible methods can differ while valid hybrid-system credit is preserved. These results…
PDF arXiv

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not capture the complexity of open, dynamic, and interactive web tasks, limiting the assessment of agent capabilities.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Memory systems often underperform naive retrieval, and their trade-offs between gains and losses (memory tax) are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Benchmark difficulty distribution mismatch and coverage gaps1 paper raises it · 1 addresses it
Benchmarks may misrepresent intended difficulty or miss key regions, leading to misleading evaluations of model progress.
- Your Benchmark Is an Empirical Measure Over Difficultyraises · addresses
Structural requirements for decision-native models1 paper raises it · 2 address it
Current decision models often lack null-awareness, permutation invariance, option scalability, and cacheability, limiting their robustness.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 2 address it
Agents struggle to extract and apply user habits from sparse or noisy historical interactions, limiting personalization.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
- Vela: User Behavior as Supervision Sparks LLM Intelligenceaddresses

Named by an AI from these papers' abstracts.