Lucy: edgerunning agentic web search on mobile with machine generated task vectors

Alan Dao (Gia Tuan Dao); Alex Nguyen; Dinh Bach Vu; Norapat Buppodom

REVIEW 4 major objections 5 minor 10 references

Lucy: edgerunning agentic web search on mobile with machine generated task vectors

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Lucy, a 1.7B-parameter model, reaches 78.3% on SimpleQA—matching DeepSeek-671B—by treating its <think> tags as a trainable dynamic task vector machine.

desk verdict A useful small-model RL recipe and released weights, but the headline parity claim rests on an uncontrolled, self-inconsistent benchmark number. read the letter →

arxiv 2508.00360 v1 pith:NXS3IKE5 submitted 2025-08-01 cs.CL

Alan Dao (Gia Tuan Dao) , Dinh Bach Vu , Alex Nguyen , Norapat Buppodom This is my paper

classification cs.CL

keywords smalllanguagemodelsagenticwebsearchreinforcementlearningtaskvectorschain-of-thoughtSimpleQAmodelcontextprotocoledgedeployment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a small language model can rival much larger ones on knowledge-intensive agentic search if the model's internal reasoning is treated not as a byproduct but as a trainable mechanism—a 'dynamic task vector machine' in which the text between and tags is where the model builds and adjusts its task representation at test time. This paper trains Lucy, a 1.7B-parameter model, with a two-stage reinforcement-learning reward scheme that penalizes inefficient searching and overlong thinking, and it reports 78.3% accuracy on the SimpleQA benchmark through a Model Context Protocol (MCP) web-search server, roughly matching DeepSeek-671B's 78.2%. If the result holds, it matters because it suggests that scale is not the only route to capable agentic behavior: structured, self-constructed reasoning plus targeted rewards may let small models run on phones and edge devices while doing useful tool-based research. The paper frames the contribution as a recipe for stabilizing reasoning rather than as a new architecture or a larger dataset.

What carries the argument

The dynamic task vector machine is the paper's central object: the generation delimited by <think> and </think> tags is interpreted as a computational process in which the model constructs, updates, and refines its task representation on the fly, rather than as a trace to be read. The argument is carried by a composite reward function—correctness gated by a log-scaled behavioral score in Stage 1, then a binary product of correctness, format, and XML gates in Stage 2—with the efficient-thinking reward $r_{\text{think}}(x) = SN(x; \mu=35, \sigma=150, \alpha=-5)$ and the visit/search ratio reward shaping behavior while avoiding supervised fine-tuning. The Model Context Protocol (MCP) server supplies the standardized tool interface for web search and scraping, and the reward functions are what make the 1.7B model's reasoning stable enough for multi-turn tool use.

What would settle it

A decisive check: score Lucy's released checkpoint on a fresh set of short-form factuality questions that were not used to tune rewards or evaluation thresholds, via an independently run MCP search server, and score DeepSeek-671B on the same server; if the accuracy gap is much larger than the reported 0.1 percentage point, the parameter-efficiency claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the <think>...</think> span can be converted from a passive trace into a dynamic task vector machine: as the model generates reasoning, it constructs and refines its own task vectors, and this self-modifying process can be optimized by reinforcement learning with verifiable rewards (RLVR) instead of by supervised fine-tuning. Lucy is trained on 10,325 question-answer pairs derived from the MuSiQue-Ans dataset, using a two-stage reward design: Stage 1 gates a log-scaled sum of behavioral rewards (tool execution, format adherence, XML validity, visit/search ratio, and an efficient-thinking reward shaped toward about 35 tokens) behind a correctness reward, and Stage 2 enforces a binary product of correctness, format, and XML gates. Evaluated on SimpleQA through an MCP server, Lucy reaches 78.3% accuracy at 1.7B parameters, compared with 78.2% for DeepSeek-671B and 80.7% for the 4B Jan-Nano baseline. The authors also report an emergent behavior—the model skips <think> tags before low-uncertainty reading steps after the efficiency reward is applied—and they interpret this as evidence that the optimized task-vector machine can allocate reasoning capacity dynamically.

Load-bearing premise

The whole result depends on the 78.3% SimpleQA score being a genuine measure of web-search ability, not an artifact of reward tuning or an evaluation pipeline that the authors themselves control.

Editorial extensions

If this is right

A 1.7B-parameter model with structured think-tag reasoning and RLVR can score 78.3% on SimpleQA, within a tenth of a point of DeepSeek-671B, so parameter count is not a hard ceiling for agentic search.
The same two-stage reward recipe, applied to a small model trained on about 10,000 QA pairs without SFT, can produce coherent multi-turn search behavior, suggesting that data scale is not the sole bottleneck for agentic capability.
Penalizing overlong thinking yields emergent skipping of think tags before deterministic reading steps, cutting latency by 17.8x in the observed scenario while preserving accuracy.
Agentic web search on mobile and edge devices becomes plausible, because the gain comes from reasoning stabilization plus small-model efficiency rather than from a large context window or many parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the mechanism would be to measure the hidden-state task vectors before and after each tool call and check whether the RLVR training makes those vectors more stable; the paper does not report such internal-state measurements, so the 'machine' is currently inferred from behavior rather than observed in activations.
The near-tie with DeepSeek-671B might also indicate that SimpleQA-with-search is close to saturation for tool-enabled models; harder multi-hop or contradictory-source benchmarks would reveal whether the 1.7B result is a genuine parameter-efficiency win or a ceiling effect.
The paper itself notes in Section 5.2 that Lucy often circulates incorrect assumptions when it lacks entity knowledge and that its training barely kept pace with Jan-Nano, so the edge over larger models is likely confined to benchmarks where the search tool can supply the missing facts.
Systematically sweeping the skew-normal reward's $\mu$, $\sigma$, and $\alpha$ values across different tool-use tasks would show whether the reported efficient-thinking behavior is a robust property of the recipe or a heuristic tuned to this particular benchmark.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this paper has a real artifact—a trained 1.7B agentic search model with released weights—and a concrete new training ingredient, the skew-normal think-length reward. The rest is a framing ("dynamic task vector machine") that never becomes operational, and the headline claim that Lucy matches DeepSeek-V3 on SimpleQA is not supported by the evidence as written.

What's actually new: the two-stage RLVR recipe with a composite reward that includes a think-length penalty is a sensible attempt to stop small models from overthinking during tool use. They train on 10k MuSiQue-derived pairs without SFT, and they report honest limitations in Section 5.2: small models circulate incorrect assumptions and test-time compute can't fix missing knowledge. Appendix A is also a decent probe—moving tool calls/responses relative to tags changes accuracy by 18 points on 500 samples, which at least motivates the tag structure. Releasing the weights is useful.

Soft spots. The evaluation is the load-bearing problem. The abstract, body, and Table 1 name three different DeepSeek variants (V3, 67B, 671B-OpenRouter), so the parity claim is unstable before you even ask about noise. A 0.1-point gap is far below sampling error. Section 4.1's claim of a 19.1-point improvement over a 4B baseline is directly contradicted by Table 1, where the two Jan-Nano 4B rows score 80.7 and 83.2—both above Lucy's 78.3. There is no eval code, no question count, no search budget, no answer-extraction rule, and no error bars. The "dynamic task vector machine" is asserted in the abstract but never formalized; the only steering evidence (Appendix B) is circular because the vector is built from the same problem's CoT, and it rests on two examples. These are not minor omissions; they are the difference between a useful technical report and a credible benchmark result.

Who benefits: people working on small-model RL for tool use and edge deployment will get something from the recipe, the weights, and the Appendix A probe. But nobody should quote the 78.3% number until the eval is redone with a public harness and an honest comparator.

My recommendation: it deserves a serious referee, not a desk reject, because the trained model and the think-length reward are real and reusable. I'd ask for a corrected SimpleQA eval with error bars, a fix to the contradictory baseline claims, and a demotion of the task-vector framing to speculation. That would turn a borderline paper into a genuinely useful one.

Referee Report

4 major / 5 minor

Summary. The paper introduces Lucy, a 1.7B-parameter small language model trained with a two-stage reinforcement learning framework for agentic web search over a Model Context Protocol (MCP) server. The central conceptual claim is that the content inside <think> tags constitutes a 'dynamic task vector machine' that the model constructs and refines during inference, and that optimizing this process with behavior-targeted rewards yields strong performance on knowledge-intensive tasks. The headline result is 78.3% accuracy on the SimpleQA benchmark, which the authors claim matches much larger models such as DeepSeek-V3 (also referred to as DeepSeek-67B and DeepSeek-671B). The paper also includes two appendix experiments: a prompt-template analysis on Qwen3-4B using Jan-Nano evaluation logs, and a two-problem task-vector steering study on AIME-2024.

Significance. If the headline result were credible, the paper would be significant: it would demonstrate that a 1.7B model can match a 671B model on an external factuality benchmark through a lightweight RL recipe, and it would give empirical weight to a 'task vector' interpretation of chain-of-thought. The authors also ship model weights on Hugging Face, which is a concrete asset for the community. However, the significance is almost entirely conditional on the benchmark comparison, and that comparison is not currently valid: the evaluation protocol is inherited from an unpublished self-citation, no harness or code is released, the comparator identity is inconsistent, and the claimed improvement over a 4B baseline is contradicted by the paper's own Table 1. The 'dynamic task vector machine' is never operationalized, so the theoretical contribution is terminological rather than demonstrated. Section 5.2 contains an honest acknowledgment that Lucy 'barely kept pace' with Jan-Nano, which further weakens the parameter-efficiency narrative. The released weights are useful, but they do not substitute for a controlled evaluation.

major comments (4)

[Section 4.1 / Table 1 / Abstract] The headline comparison is internally inconsistent. The abstract names DeepSeek-V3, Section 4.1 names DeepSeek-67B, and Table 1 names DeepSeek-671B (OpenRouter). These are different models with different parameter counts; a claim of parity with a named model must specify which model was actually evaluated. In addition, Section 4.1 states a '19.1 percentage point improvement over the 4B parameter baseline,' but Table 1 lists two 4B Jan-Nano models with scores of 80.7% and 83.2%, both above Lucy's 78.3%. No 4B baseline near 59.2% appears anywhere in the paper. The Section 5.2 statement that 'the training process barely kept pace with Jan-nano's baseline' reinforces the internal contradiction. This undermines the central parameter-efficiency claim.
[Section 4.1] The evaluation is not reproducible as described. The protocol is dismissed as 'following the evaluation protocol established in the Jan-Nano project,' a self-citation to a separate technical report at arXiv:2506.22760. The paper reports no SimpleQA question count, no search budget, no answer-extraction rule, no tool environment specification, and no variance estimates (seeds, repeated runs, or confidence intervals). The 0.1-point gap between Lucy (78.3%) and DeepSeek (78.2%) is far smaller than expected run-to-run noise on a benchmark like SimpleQA. Without a released evaluation harness, the benchmark score is a single uncontrolled number and cannot support the paper's central claim.
[Section 3 / Appendix A / Appendix B] The 'dynamic task vector machine' is not operationalized. Section 3 defines reward functions and training stages but never defines what a task vector is mathematically, how it is extracted from or injected into hidden states, or how the generation inside <think> tags updates it. The appendices do not fill this gap: Appendix A tests prompt templates on Qwen3-4B using Jan-Nano logs, not on the trained Lucy model, and the results (63.0-82.2%) are not connected to the proposed mechanism. Appendix B claims task-vector steering on two AIME problems, but the vector is derived from the model's own chain-of-thought for the very same problem being tested, which is circular; two examples are also too few to support any generalization. The paper therefore provides no evidence that 'task vector' is more than a new name for chain-of-thought.
[Section 3.1.2 / Section 3.2] The key reward hyperparameters are described as empirically set with no sensitivity analysis. The think-length reward uses a skew-normal with (μ=35, σ=150, α=-5), the behavioral weights include a large w_v/s=3.0, and Stage 2 introduces a 'specific threshold' for g_format without stating its value. Since these parameters directly shape the trained policy, and the SimpleQA result is not accompanied by ablations or variance estimates, it remains possible that the reported score is an artifact of tuning on the evaluation benchmark. At minimum, the paper should report the grid or procedure used to set these values and show robustness to reasonable perturbations.

minor comments (5)

[Abstract / Section 4.1 / Table 1] Please standardize the comparator model name (DeepSeek-V3 vs. DeepSeek-67B vs. DeepSeek-671B) throughout. The reader should not have to guess whether one or three models were evaluated.
[Section 3.1.1] Equation (1) introduces N_turn and other counts without a precise definition; please define each N in a sentence and clarify whether the reward is computed per turn or over the entire episode. The visit/search reward formula also appears with an odd superscript '0.25' and should be rewritten for clarity.
[Appendix A] The prompt templates in Listings 1-5 contain obvious tokenization artifacts (e.g., 'a ss is ta nt', 'R ea so ni ng :') that make the exact prompts ambiguous. Please provide clean, tokenizer-independent prompt strings.
[Section 5.1] The 17.8× latency improvement is claimed for a single 'representative scenario' with no measurement protocol, no number of trials, and no absolute latency numbers. Please either provide a proper measurement or label the value explicitly as an anecdotal observation.
[References / Appendix B] Hendel et al. 2023 is cited twice (2023a and 2023b) with overlapping content; check whether both are intended. Also, Table 3's column 'With Think (CoT)' is not clearly defined relative to the 'Baseline' row for problem I-1, where the baseline is reported as failing initially but succeeding after multiple attempts; please clarify the protocol.

Circularity Check

1 steps flagged · score 5.0 of 10

The task-vector steering case study in Appendix B is circular by construction, while the headline SimpleQA comparison is externally grounded but rests on a self-cited protocol with inconsistent comparators.

fitted input called prediction [Appendix B: Case Study on Task Vector Steering for AIME-2024]
"We compare a baseline zero-shot prompt against a modified prompt where a task vector, derived from the model's own chain-of-thought (CoT) processing for the *same problem*, is injected into its hidden states."

The steering vector is computed from a CoT-prompted run on the exact AIME problem that the 'Baseline + Task Vector' condition is then evaluated on. For problems 2024-I-1 and 2024-I-4, the CoT run already produced the correct answer (204 and 116), so the injected layer-23 vector carries the target problem's solution. The successful steered answer is therefore the model re-reading its own correct CoT, not an independent prediction. This is a fitted input (vector = f(CoT(target))) renamed as evidence that task-vector steering improves reasoning; it would provide the same support if the vector were simply a cached copy of the answer. No held-out problem is used, so the demonstration is equivalent, by construction, to grading a prompt augmented with the test answer.

full rationale

The central claim that Lucy reaches 78.3% on SimpleQA is not circular: SimpleQA is an external benchmark and the MCP-augmented evaluation is an empirical measurement, not a quantity derived from the training data or rewards. The 19.1-point improvement claim and the DeepSeek comparison are nonetheless weakened by internal inconsistencies: the text names DeepSeek-67B, Table 1 names DeepSeek-671B (OpenRouter), the abstract names DeepSeek-V3, and Table 1 lists Jan-Nano 4B rows at 80.7% and 83.2%, both above Lucy, despite Section 4.1 claiming a 19.1-point gain over a 4B baseline; Section 5.2 even concedes that 'The training process barely kept pace with Jan-nano's baseline.' These are correctness and reproducibility concerns, not circularity. The one directly quotable circular step is Appendix B: the steering vector is extracted from the model's own CoT-prompted run on the same problem that the steered baseline is then evaluated on, and that CoT run already contained the correct answer. Thus the steering result is a prediction made from the answer of the same problem, i.e., a fitted input renamed as an independent demonstration. The evaluation protocol is also delegated to the authors' Jan-Nano report, a self-citation, but because SimpleQA itself is external this does not by itself make the headline result circular. The net score reflects one clearly circular supporting demonstration while the main benchmark result retains independent external content.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The headline benchmark is externally grounded by SimpleQA, but the central conceptual claim rests on an unmeasured interpretation of think-tag content, hand-set reward hyperparameters, an inherited self-cited evaluation protocol, and an assumption that the training distribution transfers to SimpleQA.

free parameters (3)

think-length reward parameters = mu=35, sigma=150, alpha=-5
Set empirically in Section 3.1.2 for the skew-normal efficient-thinking reward; no ablation or validation on held-out data is reported.
behavioral reward weights = wtool=0.2, wformat=0.2, wthink=0.1, wxml=0.1, wv/s=3.0
Hand-chosen weights in Section 3.2, Equation 2, with no sensitivity analysis or alternative weight comparisons.
Stage 2 format threshold = unspecified
The binary gate gformat in Section 3.2 uses 'a specific threshold' that is never reported, making the reward function incomplete.

assumptions (5)

ad hoc to paper Content inside <think> tags corresponds to a task vector that can be optimized directly.
Section 1 and Appendix A assert this framing based on Hendel et al., but Lucy's training never extracts or probes these vectors, and no evidence ties the RL rewards to task-vector dynamics.
domain assumption The MuSiQue-Ans adaptation of 10,325 QA pairs is an adequate training distribution for SimpleQA agentic search.
Section 3.3 provides no domain-shift analysis, contamination check, or validation of whether this training set covers the knowledge types in SimpleQA.
domain assumption The Jan-Nano evaluation protocol fairly and accurately scores agentic search on SimpleQA.
Section 4.1 inherits the protocol from a self-cited technical report with no public code or independent verification, so the 78.3% figure depends on this unstated measurement assumption.
domain assumption The skew-normal reward shape induces efficient thinking without reducing correctness.
Section 3.1.2 states the shape was chosen heuristically from observations of overthinking; no controlled comparison against other reward shapes is given.
domain assumption The MCP server provides reliable search results that are correctly integrated into the evaluation.
Section 4.1 assumes tool responses are accurate and complete enough to support the reported benchmark; no analysis of search failures or server errors is presented.

invented entities (1)

Dynamic task vector machine
purpose: Conceptual framing for optimizing think-tag reasoning as a self-modifying task-vector mechanism during inference.
No measurement, equation, or ablation in the paper ties Lucy's training rewards to task-vector dynamics. The only steering evidence in Appendix B uses a vector taken from the same problem's chain of thought, which does not independently validate the concept.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lucy: edgerunning agentic web search on mobile with machine generated task vectors." pith.science (2026). https://pith.science/paper/NXS3IKE5

@misc{pith2026250800360,
  author       = {Pith},
  title        = {Pith review of: Lucy: edgerunning agentic web search on mobile with machine generated task vectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXS3IKE5}},
  note         = {Machine review of arXiv:2508.00360}
}

read the original abstract

Small language models (SLMs) are inherently limited in knowledge-intensive tasks due to their constrained capacity. While test-time computation offers a path to enhanced performance, most approaches treat reasoning as a fixed or heuristic process. In this work, we propose a new paradigm: viewing the model's internal reasoning, delimited by <think> and </think> tags, as a dynamic task vector machine. Rather than treating the content inside these tags as a mere trace of thought, we interpret the generation process itself as a mechanism through which the model \textbf{constructs and refines its own task vectors} on the fly. We developed a method to optimize this dynamic task vector machine through RLVR and successfully trained an agentic web-search model. We present Lucy, a 1.7B-parameter SLM that leverages this dynamic reasoning mechanism with MCP integration to achieve 78.3% accuracy on the SimpleQA benchmark, performing on par with much larger models such as DeepSeek-V3. This demonstrates that small models can rival large ones when equipped with structured, self-constructed task reasoning.

Figures

Figures reproduced from arXiv: 2508.00360 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages

[1]

This provided a consistent set of contextual information (tool calls and tool responses) for each question

We sampled 500 questions from the evaluation logs of the Jan-Nano-128k run on the SimpleQA benchmark. This provided a consistent set of contextual information (tool calls and tool responses) for each question

work page
[2]

These prompts is different on how tool calls and responses were placed relative to the <think> tags

We prompted Qwen3-4B to generate a final answer using this same context, but with five different prompt strategies. These prompts is different on how tool calls and responses were placed relative to the <think> tags

work page
[3]

The five templates evaluated are reproduced in Table 2

We evaluated the accuracy of the model’s generated answers against the ground-truth short answers, using the same evaluation pipeline from the Jan-Nano benchmark. The five templates evaluated are reproduced in Table 2. Scores in bold denote accuracy. T emplate Model Accuracy Observation

work page
[4]

Tools outside, responses inside<think> 4 Qwen3-4B 73.8% Model returns empty string

work page
[5]

Baseline Prompt 1 Jan-Nano-128K 81.2% -

work page
[6]

Baseline Prompt 1 Qwen3-4B 81.6% -

work page
[7]

Think Prompt 2 Qwen3-4B 82.2% -

work page
[8]

Tools and responses inside<think> 3 Qwen3-4B 77.4% Model tries to call tools

work page

Show all 10 references

[10]

Sister , Sister

Sequenced <think> blocks 5 Qwen3-4B 63.0% Model returns empty string Table 2: Dry-run results on the eval logs of Jan-Nano-128K on SimpleQA(500 samples). <| im_start | > system { s y s t e m _ p r o m p t } <| im_end | > <| im_start | > user What episode of the TV show " Siste...

2025
[2025]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al

URL https://arxiv.org/abs/2502.09977. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in ne...

2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

This provided a consistent set of contextual information (tool calls and tool responses) for each question

We sampled 500 questions from the evaluation logs of the Jan-Nano-128k run on the SimpleQA benchmark. This provided a consistent set of contextual information (tool calls and tool responses) for each question

work page

[2] [2]

These prompts is different on how tool calls and responses were placed relative to the <think> tags

We prompted Qwen3-4B to generate a final answer using this same context, but with five different prompt strategies. These prompts is different on how tool calls and responses were placed relative to the <think> tags

work page

[3] [3]

The five templates evaluated are reproduced in Table 2

We evaluated the accuracy of the model’s generated answers against the ground-truth short answers, using the same evaluation pipeline from the Jan-Nano benchmark. The five templates evaluated are reproduced in Table 2. Scores in bold denote accuracy. T emplate Model Accuracy Observation

work page

[4] [4]

Tools outside, responses inside<think> 4 Qwen3-4B 73.8% Model returns empty string

work page

[5] [5]

Baseline Prompt 1 Jan-Nano-128K 81.2% -

work page

[6] [6]

Baseline Prompt 1 Qwen3-4B 81.6% -

work page

[7] [7]

Think Prompt 2 Qwen3-4B 82.2% -

work page

[8] [8]

Tools and responses inside<think> 3 Qwen3-4B 77.4% Model tries to call tools

work page