TodayNewsResearchFieldsStacksMarketAnalytics
⌘K

Research

New papers ranked by attention, with their code, models and discussion linked. Pick a field on the left.

WeekMonthQuarterAll
NewestMost relevant

Papers

Open problems

Rising research topics

Ask the research

Sample
ByteTrail · AI and developer-tools intelligence
NewsResearchStacksAPI
Fields · this week
All fields3549Multimodal578Agents548Frontier LLMs482Inference & serving282Image segmentation202Graph learning180Learning verifiable rewards1683D scene168Voice AI101Classifier & causal discovery89Forecasting time88Flow matching76Evals60Teacher policy distillation55Adversarial training53All 32 fields

Open problems across fields

  • General in-context learning for embodied agents1 paper raises it · 0 address it

    Improving adaptability of models to new tasks and environments without task-specific fine-tuning is critical for scalable embodied intelligence.

    • Has GPT-6-Astra Achieved Embodied Intelligence?raises
  • Cost and latency in agentic systems2 papers raise it · 1 addresses it

    High computational and time costs limit practical deployment of agentic systems for real-world tasks.

    • Has GPT-6-Astra Achieved Embodied Intelligence?raises
    • PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
  • Physical-state verification and failure recovery1 paper raises it · 0 address it

    Robustness to irreversible failures and accurate state tracking are essential for reliable embodied control.

    • Has GPT-6-Astra Achieved Embodied Intelligence?raises
  • Evidence-grounded multimodal reasoning benchmarks1 paper raises it · 1 addresses it

    Current benchmarks often fail to ensure models use necessary audio-visual evidence, weakening evaluation reliability.

    • EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoningraises · addresses
  • Length generalization in sequential state tracking1 paper raises it · 1 addresses it

    Models struggle to maintain performance on problems much longer than their training data, limiting scalability.

    • PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses

Named by an AI from these papers' abstracts.

Rising research topics

  • Training & fine-tuning42+90.9%
  • AI safety29+61.1%
  • Federated learning53+39.5%
  • Quantum machine learning29+38.1%
  • Computer vision29+31.8%
  • Evals60+25.0%
  • Code models13+18.2%
  • Forecasting time88+12.8%

Papers

  • arXiv cs.CVComputer visionOct 15

    Motion- and Gravity-Aware Feedforward 3D Reconstruction

    Shiyao Qian, Yuan Ren, Dongfeng Bai

    Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues. In this work, we introduce a motion- and gravity-aware framework under the Learning Using Privileged Information (LUPI) paradigm. During training, residual optical flow and gravity direction are provided as privileged signals to guide geometric representation learning; during inference, the model uses standard visual inputs only. We integrate these priors into transformer backbones with lightweight architectural changes, preserving feed-forward efficiency. Experiments on standard benchmarks show consistent gains in reconstruction accuracy and robustness, especially in scenes with dynamic objects, ambiguous structure, and challenging camera motion. Our analysis further shows that the optimal injection stage is backbone-dependent: middle-layer integration is generally preferable for VGGT, while late-stage integration is more effective for π3. These results suggest that LUPI offers a practical way to inject motion and physical priors into feed-forward 3D…

    AI60PDF
  • arXiv cs.CLAgentsOct 15▲ 54

    Mosaic: Multi-Session Agentic Reinforcement Learning

    Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan +1

    Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and…

    AI70Code
  • arXiv cs.SIOct 15

    High prevalence of antagonizing rhetoric in trending topics on Twitter/X

    Armin Pournaki, Eckehard Olbrich

    Disagreement between political opponents is a central element of democracy. This disagreement can be voiced in different ways. An opponent can be antagonized, i.e. portrayed as an enemy that needs to be overcome, or they can be criticized, i.e. treated as a legitimate adversary with whom one can engage. While the latter is a fundamental part of democratic exchange, the former can be detrimental to public discourse. Political disagreement increasingly takes place on social media, where negativity towards political out-groups has been found to drive engagement. However, existing research has largely treated such negativity as a single dimension, leaving open whether it is antagonization or criticism that circulates on these platforms. This work addresses this gap by introducing a computational approach to detect these two types of rhetoric in text and identify its targets. On a corpus of over 17 million tweets from trending topics in the German Twittersphere, the method reveals that about a third of all tweets contain antagonizing rhetoric. Comparing tweets from trending topics to a random sample shows that the discourse in trending topics is systematically more antagonizing than…

    PDF
  • arXiv cs.CLAI safetyOct 15▲ 55

    Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

    Jian Gu, Hongyu Zhang, Chunyang Chen, Aldeida Aleti

    The vocabulary of a language model is its native semantics, and the embedding and unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods reuse a fixed lens or treat states as a point cloud, and neither reads a state as the model does. We argue that every hidden state has a native reference frame, induced by the layers that remain and adaptive to the context. Lexical reference frames approximate it in a model-native and context-aware way, reading a state through the remaining layers with a given number of rounds of attention between positions and a typical attention output for the rest, fitting nothing to the model's output. No lens with fixed directions per layer can reach a mean cosine with the native frame's lexical readouts above the square root of their coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B. LexFrame[1], of order 1, reaches 0.97 and 0.97, and its readings are on average within 0.16 and 0.15 nats of the model's output. Its coefficient differences attribute predictions more faithfully than attribution patching, and its lexical basis steers predictions while sparing other tokens. On average, its…

    AI60Code
  • arXiv cs.CLAgentsOct 15

    CLUE: Uncovering Social Context for Pragmatic Understanding

    Chengfei Wu, Zhaoqing Wu, Dan Goldwasser

    Understanding a conversation requires connecting what is said to the speakers' relationships, expectations, and earlier exchanges. Prior work shows that supplying relevant context can improve models' social judgments, but much of this information may already be present in the conversation. This motivates training models to discover the cues that matter and use them to support an interpretation. We propose CLUE, a two-stage training pipeline that combines supervised evidence reasoning with GRPO guided by Grounded Information Utilization ($\mathrm{IU}_{\mathrm{G}}$). This reward credits new, evidence-supported social inferences while accounting for reasoning effort. We train on a small set of conversational examples to assess effectiveness and generalization across tasks, languages, and genres. At inference time, the trained model reasons over the supplied interaction in a self-contained manner. Compared with base Qwen, CLUE achieves relative gains of 17.3% in-domain and 9.3% out-of-domain in average weighted F1, with higher F1 on four of five out-of-domain evaluations. These gains are accompanied by over 80% fewer reasoning tokens on average in both domains. Human evaluation…

    AIEvals50PDF
  • arXiv cs.CLOct 15

    Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

    Zhiyuan Wang

    Running prompt tokens through fewer routed experts during prefill is a simple way to make mixture-of-experts (MoE) serving cheaper, yet it is unclear what this damages. We locate the damage with token-, layer- and attention-path interventions on short-answer question answering in four MoE families. At half the experts the F1 loss of three families is mostly answer form: the model still finds the answer but stops answering briefly. In Qwen3-30B-A3B this loss is made at the two chat-template tokens before the answer, and in DeepSeek-V2-Lite at the first 8-32 prompt tokens, where its instruction sits; by answer likelihood these positions account for about half and a fifth of the damage. Reduction changes the first token's attention mass, norm and largest activation by less than 4%, and restoring super experts or massive activations recovers at most about a quarter of the loss, well below keeping the sensitive positions full. On real prompts with about 4% of tokens at full experts, placing them on the task interface keeps F1 within one point of the unmodified model, while placing them at random in the documents is about 3 points worse (two of five prespecified primary comparisons;…

    PDF
  • arXiv cs.AIRAGOct 15

    Multi-sourced AI Tutor for Non-Native English Speaker Students

    Yiyin Shen, Xinyi Ai, Weibing Wang, Rogers Jeffrey Leo John, Meenakshi Syamkumar

    Large language models (LLMs) are increasingly being adopted as teaching tools in computing education, particularly for developing AI tutors capable of answering student questions. With the rise in enrollment of international students in computing programs, researchers have begun to explore the difficulties that non-native English speaker (NNES) students face in computing courses due to language barriers. To aid such students, we developed a general-purpose AI tutor system that can process student questions in foreign languages (currently focused on Mandarin) and provide bilingual responses. Our system also incorporates the functionality of generating a glossary of technical terms and their explanations to enhance comprehension. Our approach utilizes a specialized Retrieval-Augmented Generation (RAG) pipeline with a novel question extraction mechanism, enabling us to effectively utilize a diverse array of course materials for answering student queries. We evaluated our AI tutor using 91 Piazza posts, demonstrating that it could answer 83.52% of the posts correctly and in a helpful manner. The tutor also provided high-quality translations for 76.92% of the questions and generated…

    AIDeveloper tooling60PDF
  • arXiv cs.CLMultimodalOct 15

    EMO: A Closed-Loop Test-Time Computation Framework via Alternating Context Optimization and Logits Steering

    Wei Xia, Yehui Tang, Zhi-Hong Deng

    Test-time context optimization adapts frozen language models through textual feedback, but ordinary conditional generation provides no separate control over how strongly to reinforce the distributional changes induced by a context update. We introduce EMO, a training-free framework inspired by expectation--maximization that alternates question-specific context estimation (E-step) and distribution refinement (M-step). The E-step revises a system context using textual gradients from an external judge without access to gold answers. The M-step contrasts next-token distributions under the revised and preceding contexts at a shared prefix to construct a confidence-constrained logit correction. Feedback on the steered response drives the next context update, coupling adaptation across rounds with intervention within each generation. Theoretically, we connect parameter and context optimization through a shared target-KL objective and additive logit representations, and derive the steering vector from context-induced logit displacements. Under context-improvement and faithful-steering assumptions, we prove a one-round decrease in target KL divergence for sufficiently small steering…

    AIInference & serving70PDF
  • arXiv cs.CLCode modelsOct 15

    PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

    Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu +1

    Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.

    AIEvals80PDF
  • arXiv cs.LGOct 15

    The importance of tensor handling and implementation of physics-informed DeepONets

    Nahum Banks, Gustavo A. M. de Almeida, Sergio Maldonado

    Physics-informed neural networks (PINNs) reduce or eliminate the need for large labelled datasets by enforcing governing equations during training. Physics-informed Deep Operator Networks (PI-DeepONets) extend this approach to learning solution operators, enabling rapid prediction across families of problems. Despite these advantages, we show that their application to higher-dimensional problems remains limited by the computational cost of training. To address this issue, we propose a modified PI-DeepONet architecture that reduces this cost by removing redundancy in how the Branch network processes input functions across collocation points. The proposed formulation is mathematically equivalent to the original architecture but reduces input tensor sizes and the number of Branch network evaluations. We analyse the computational complexity of both architectures and derive theoretical scaling laws for training time with respect to the number of collocation points and input functions. The proposed architecture is evaluated on 1D, 2D, and 3D linear advection problems. In all cases, it achieves comparable predictive accuracy to the original PI-DeepONet while consistently reducing…

    AI20PDF
  • arXiv cs.AIAgentsOct 15

    Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

    Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng +5

    Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. These findings suggest…

    AIEvals90PDF
  • arXiv cs.LGOct 15

    Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld

    Hilda Adwubi Osei

    Reinforcement learning agents optimize exactly the reward signal they are given, which can diverge sharply from an intended true objective when that signal is only a proxy for it; this failure mode is known as specification gaming. Existing evidence for this failure mode is largely observational, with comparatively few demonstrations pairing a controlled, deliberately constructed proxy/true gap with a mitigation that carries a verifiable guarantee rather than only an empirical improvement. We construct a minimal gridworld in which a cleaning-robot agent can either genuinely remove dirt or conceal it from a cheap sensor, and show that a standard deep Q-network trained only on the sensor's proxy reward converges to a policy that games it completely, reaching near-maximal proxy reward while never achieving the true objective. We then introduce a constrained reinforcement learning fix, penalizing the specific behaviors through which the agent exploits the gap, and show that it eliminates the gaming behavior from the final policy while substantially recovering true-objective performance. We further derive and empirically verify an exact accounting identity relating the proxy reward,…

    AI30PDF
  • arXiv cs.AIRAGOct 15

    RAG-RST: Retrieved Exemplars as In-Context Discourse Schema for RST Parsing

    Mohammadreza Sediqin, Shlomo Engelson Argamon

    In Rhetorical Structure Theory (RST), each edge of a discourse tree carries a nuclearity label, marking which span holds the central content, and a relation drawn from a closed inventory of functional categories. Neither label is a surface property: the same two clauses can be Elaboration or Background depending on rhetorical intent, and the boundary between them is fixed by annotation convention rather than by lexical cues. Predicting these labels therefore requires a formal discourse schema, and exposure to discourse-rich pretraining text does not supply it: given only a target EDU pair, a strong LLM recovers nuclearity at 0.794 F1 and relation at 0.738 F1. We propose RAG-RST, a retrieval-augmented framework that supplies this schema as in-context demonstration rather than through parameter updates, raising nuclearity to 0.846 F1 and relation to 0.788 F1 with no training. Two configurations are compared: a precomputed strategy that labels all EDU pairs before parsing, and a dynamic strategy that retrieves at each reduce step, so that the exemplars reaching the generator depend on the reductions the parser has already committed to. Both outperform existing RST parsers across…

    AICode models70PDF
  • arXiv cs.CVRoboticsOct 15

    FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving

    Haohua Que, Mingkai Liu, Handong Yao

    A perception system that must brake for a pedestrian does not need a low average latency. It must deliver an answer before the deadline on every frame, the worst case included. Commodity edge accelerators optimize the mean through dynamic scheduling, OS jitter, thermal throttling, and contention, so their per-frame latency can be measured but not bounded. We build FALCON, a camera-based 3D occupancy front-end on the AMD Kria KR260, an embedded FPGA drawing 4.3 W idle and up to 6.6 W under inference, and make the latency guarantee layered and explicit: programmable-logic (PL) compute is cycle-bounded analytically; inference is a measured envelope over 100k-frame traces; and a watchdog enforces output delivery, publishing the previous output flagged stale when a frame is late. The deadline runs use a fixed-work proxy of six ResNet-50 DPU inferences per frame, not the full occupancy network: across three 100k-frame enforced runs, all 300k proxy frames delivered an output within the 100 ms deadline plus at most 10.2 ms, the worst of the three runs. The full six-camera FlashOcc pipeline composes to about 1.3 s per frame on the board and is not yet a 10 Hz system. The two operations…

    AIInference & serving80PDF
  • arXiv cs.AIEvalsOct 15

    Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

    Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari

    Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer…

    AIAI safety70PDF
  • arXiv cs.AIAgentsOct 15

    Believed but Not Said: Measuring Hidden Belief Collapse in Multi-Agent LLM Discussions

    Liangjie Huang, Henry Peng Zou, Wei-Chieh Huang, Wooseong Yang, Yuwei Han +1

    LLM systems increasingly rely on groups of agents that discuss, critique, and check one another, assuming that peers will catch individual errors. We test this assumption against a minimal adversary. In a ten-agent factual discussion, one or three adversarial agents repeatedly assert a plausible false answer. After each round we probe each benign agent's private answer through a separate call whose response is hidden from its peers, and we separately label its public stance toward the claim. Across six models and three factual-QA datasets, groups remain stable without an attacker: only 0.6% of benign agents finish with the false answer. A single adversarial agent increases this rate to 44.8% and three to 60.5%. Studies of conformity typically report public agreement without private change; we find the reverse. With three adversarial agents present, benign agents hold the false answer in 51.6% of exposed turns but publicly endorse it in only 19.7%. Much of the influence therefore remains invisible in the discussion, and the false answers persist when agents are asked for their own views. Agents that retain the true answer often fail to defend it; once adopted, the false answer…

    AIEvals90PDF
  • arXiv cs.AIMultimodalOct 15

    CART: Compatible Routing for Multimodal Continual Instruction Tuning

    Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou

    Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert…

    AITraining & fine-tuning80PDF
  • arXiv cs.CLMultimodalOct 15▲ 25

    Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

    Tong Zhu, Zhe Xu, Zefeng He, Yunkai Chai, Xuyang Hu +5

    Interactive agents repeatedly choose among a finite set of actions, labels, or ordinal outcomes. Generating a free-form response for each choice adds latency and complicates the interface between a model and its environment. We present Intern-Decision, a family of 0.8B, 2B, and 4B multimodal decision models that return structured choices and candidate probabilities from text and optional images. Our autoregressive masked language modeling objective places one decision placeholder per field in a complete assistant JSON skeleton. Ground-truth symbols appear only in supervision, while all fields are evaluated in a single causal forward pass using the backbone's native language-model head. The 4B model achieves a 90.02% unweighted average accuracy across seven evaluation suites, compared with 88.74% for Jev v1.13.0. For a fixed 289-token request with three fields on one RTX 4090, mean latency is 44.16 ms for the 4B model and approximately 33--34 ms for the smaller models. On a separate 96-case known-distribution pilot, temperature scaling reduces expected multiclass Brier from 0.628 to 0.550 and expected calibration error from 0.213 to 0.089. A confidence-based handoff study on 111…

    AIAgents80Code
  • arXiv cs.AIInference & servingOct 15

    DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction

    Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu +6

    Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth. While recent eviction methods have moved beyond simple attention-score heuristics to incorporate value representations, they typically face a fundamental trade-off: the underlying optimization for minimizing output perturbation is an NP-hard combinatorial problem, leading to mathematically suboptimal heuristics or implementations that are difficult to kernelize. In this work, we propose DropKV, a simple principled eviction framework based on Decoupled Residual-Output Perturbation. By decoupling the joint eviction decision into independent per-token scoring, DropKV circumvents combinatorial intractability and admits a provable constant-factor approximation guarantee under a cone condition that we empirically verify on three long-context LLMs. To ensure practical viability, we provide a high-performance implementation using fused Triton kernels that avoid materializing the full attention matrix, delivering a 19.8x scoring-kernel speedup that translates to up to a 9.9% matched-batch end-to-end prefill speedup. Extensive…

    AI80PDF
  • arXiv cs.CLAI safetyOct 15▲ 3

    Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization

    Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein

    Personalization of Large Language Models (LLMs) is increasingly important for user-centric applications, yet prior methods mainly adapt dense LLMs, whose polysemantic neurons, each encoding multiple unrelated concepts, hinder localizing the components behind personalization, and thus understanding and controlling it. We propose to personalize Mixture-of-Experts (MoE) LLMs, which sparsely route each token to a few specialized experts, reducing polysemanticity and enabling such localization. We adapt three open-weight MoE LLMs on a large-scale personalization dataset and train crosscoders on paired activations of the personalized and non-personalized models to learn interpretable personalized latents, from which we identify the experts encoding distinct personalization features. The personalized MoE LLMs perform competitively with dense LLMs while activating only a fraction of their parameters, exhibit lower polysemanticity than dense LLMs with similar active parameters, and, within the same model family, yield personalized latents that are less shared with the non-personalized model. Causal interventions show that these latents suffice to recover the personalized model's behavior…

    AITraining & fine-tuning70PDF
Load more
PDF
PDF
PDF