CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories

Jialong Wu; Jianguo Lin; Keping Yang; Ting Liang; Yilong Lai; Yipin Yang; Zhenglin Wang

REVIEW 3 major objections 4 minor 1 cited by

CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CRMWeaver shows a 4-billion-parameter agent trained on synthetic CRM tasks, reinforcement learning, and shared task memories can match or beat models roughly sixty times larger on the CRMArena-Pro business benchmark.

desk verdict The 4B-vs-235B headline does not hold up as an apples-to-apples comparison; the full system uses a test-time o3-mini oracle that baselines do not get, but the w.o.-Shared-Memory variant is still a meaningful result. read the letter →

arxiv 2510.25333 v2 pith:N5FSY7Z6 submitted 2025-10-29 cs.CL

Yilong Lai , Yipin Yang , Ting Liang , Jialong Wu , Zhenglin Wang , Jianguo Lin , Keping Yang This is my paper

classification cs.CL

keywords businessagentscustomerrelationshipmanagementreinforcementlearningsyntheticdatagenerationlong-termmemorytoolusemulti-hopreasoningCRMArena-Pro

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a lightweight language model (Qwen3-4B) can become a competitive business agent if it is trained on synthetic queries generated from the relational structure of CRM data, optimized with reinforcement learning against a rule-based reward, and augmented at inference with a retrievable memory of task guidelines. If true, this means enterprise automation no longer requires the largest, most expensive frontier models: a small model can handle complex multi-table database queries, workflow execution, and policy checks. The authors show this on CRMArena-Pro, where their model averages 55.6 (B2B) and 57.1 (B2C), matching or exceeding Qwen3-235B-A22B-Instruct and Gemini 2.5-Pro, and decisively beating them on database tasks. The result matters because business agents must operate over intricate, interconnected enterprise data, and the current cost of high-performing agents is prohibitive for many deployments.

What carries the argument

The load-bearing object is the synthetic-data construction: a graph G whose nodes are records and whose edges are inter-table foreign-key relationships. A random walk extracts a path Ψ=(e1,…,ek), and an LLM iteratively generates sub-queries qi = LLM({e_{i-1}, e_i} | q_{i-1}, A), where A is the unique ID of the seed record e1; this produces long, self-contained questions that force multi-hop search and reasoning. On top of this, the training pipeline uses rejection-sampled SFT with observation-token loss masking, then DAPO-based RL with a rule-based reward, and finally an inference-time long-term memory module that stores distilled workflow guidelines E = LLM(H_adv, Q) in an index, retrieves

What would settle it

Take a random sample of 100 synthetic QA pairs from the pipeline of Eq. (2) and have independent annotators (or a held-out strong LLM with no access to the answer) derive the intended record ID from the query alone; measure the fraction of queries whose answer is uniquely derivable. If that fraction is materially below the model's accuracy on CRMArena-Pro (e.g., <60% instead of >70%), the synthetic grounding assumption fails and the training signal is suspect. Alternatively, retrain the full pipeline on a human-validated subset of synthetic queries and check whether CRMArena-Pro scores drop; i

Watch

Extended reading notes

Core claim

The core claim is that the recipe of (1) synthesizing complex multi-hop training questions by walking a graph of interconnected business records and having a strong LLM iteratively fold record pairs into increasingly complex queries, (2) bootstrapping the 4B policy with rejection-sampled, distilled trajectories, (3) refining it with DAPO reinforcement learning where the reward is 0.9×answer accuracy (exact match for data tasks, F1 for QA) plus 0.1×format, and (4) injecting retrieved task guidelines at inference into the system prompt, yields an agent that scores 55.6/57.1 on CRMArena-Pro B2B/B2C and 73.0/72.8 on database querying. The ablation without shared memory drops to 54.5/55.3, and th

Load-bearing premise

The synthetic training questions generated by Eq. (2) are taken to be well-posed, unambiguous, and correctly answered by the target record ID, with no human validation; if any substantial fraction are ambiguous or incorrectly grounded, the SFT/RL training signal is corrupted and the benchmark gains could reflect noise or shortcut learning.

Editorial extensions

If this is right

If the reported numbers hold, a 4B-parameter model can substitute for 100B+ parameter models on realistic enterprise CRM tasks, which would make business-agent deployment far cheaper and more feasible on-premise.
The large RL-driven gain on workflow execution (about +23 points in both B2B and B2C) suggests that procedural adherence, not world knowledge, is the main bottleneck these tasks pose—an insight that transfers to other tool-using agents.
Explicit transfer from CRMArena training to the CRMArena-Pro environment indicates the synthetic-data + RL recipe generalizes to new schemas, not just new queries on the same schema.
The database-querying scores above 72 for a 4B model imply that SQL/SOSL generation over interconnected tables can be largely learned from synthetic walks, reducing the need for large amounts of human-annotated business trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the synthetic-data method (random walk + iterative LLM query chaining) is a general procedure, not CRM-specific; it should transfer to other multi-table enterprise domains (ERP, healthcare records), and the paper gives no reason it would not.
My inference: the shared-memory module effectively distills procedural knowledge from a stronger reasoning model into a lightweight policy; the ~1-2 point ablation gap suggests memory is a supplement, and most of the performance comes from training—so the headline claim would survive even if the memory index were removed.
My inference: because the RL reward is answer-based (exact match / F1) and the authors themselves note a policy-task score drop caused by the metric rewarding shortcut 'None' answers, the method may be optimizing for benchmark metrics rather than robust business reasoning; a stress test with adversarially paraphrased queries would clarify this.
My inference: scaling this recipe to larger backbones (14B/32B) is a natural next step that the authors state hardware constraints prevented; if the memory and RL gains persist, such a model could plausibly exceed all frontier baselines on text and policy tasks, not just database tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Open with the punchline: the headline comparison is confounded. The full CRMWeaver system relies on a memory module that, for queries below the similarity threshold, calls o3-mini to solve the test query, converts that solution into a guideline, and prepends it before the 4B policy answers the same query. None of the baselines get that help, so the 55.6/57.1 averages are not an apples-to-apples measure of the trained model.

What is actually new and good: the random-walk synthetic data generation over the record graph is a genuine twist, and the SFT + DAPO + memory recipe is a coherent extension of the WebDancer line. The RL-versus-SFT comparison is well designed, and the Workflow gains are large. The memory module is precisely described in Algorithm 1, and the paper's limitation section is candid. The w.o.-Shared-Memory row reaching 54.5/55.3 with a 4B backbone is itself a competitive result against much larger models.

The stress-test concern lands. For exactly the "unseen scenario" cases where memory is claimed to help most, o3-mini has already produced a solution to the test question, and the 4B model executes under a strong hint. The Database Querying scores of 73.0/72.8 are the most suspicious, because a distilled guideline can hand over the SQL join strategy. Even retrieval hits draw from memories built from stronger models' successful trajectories. The reported ~3% update frequency limits the aggregate circularity, but it doesn't remove the confound for the queries most likely to drive the headline.

The fix is straightforward: run baselines with identical memory injection, or reposition the primary claim to the w.o.-Shared-Memory variant. I would also ask for code and data; without them, the synthetic-data quality in Eq. (2) is impossible to audit. Minor issues: no error bars, single benchmark, and several baseline scores are taken from the original paper rather than re-run under the same ReAct configuration.

Bottom line: the paper deserves a serious referee, but the main comparison must be reframed before publication. If the authors reposition to the ablation variant, this is a solid systems result for the CRM-agent community.

Referee Report

3 major / 4 minor

Summary. CRMWeaver proposes a training-and-inference recipe for a 4B-parameter business agent: synthetic data generated from random walks over a record graph, SFT on rejected/trajectory-distilled trajectories, DAPO-based agentic RL, and a retrieval-augmented long-term memory module that injects workflow guidelines into the system prompt. The model is trained on CRMArena and evaluated on CRMArena-Pro in a single-turn, multi-tool setting. The paper reports that the Qwen3-4B-based system achieves average scores of 55.6 (B2B) and 57.1 (B2C), comparable to Qwen3-235B-A22B-Instruct and Gemini-2.5-Pro, with a particularly large advantage in Database Querying (73.0/72.8). Ablations show that removing the shared-memory module lowers averages to 54.5/55.3, and that RL improves over SFT, especially on Workflow tasks.

Significance. If the empirical claims are taken at face value, the paper contributes a useful recipe: a small open-weight model can be made competitive on a realistic enterprise CRM benchmark by combining synthetic multi-table supervision, agentic RL, and inference-time reuse of distilled guidelines. The transparent ablations (w.o. Shared Memory, SFT vs. RL), the explicit limitations section, and the use of locally deployable frameworks are strengths. However, the headline comparison is not yet trustworthy because the memory module uses o3-mini to solve out-of-memory test queries before the 4B policy attempts them, and no baseline receives the same aid. The w.o.-Shared-Memory variant is a fairer statement of what the trained 4B model achieves, and it is still competitive; the paper's contribution would be better framed around that variant, with the memory module treated as a separate, explicitly measured system component.

major comments (3)

[§3.4, Algorithm 1, Appendix B; Table 1] The headline comparison in Table 1 is confounded by a test-time oracle. For any query whose Top-1 retrieval similarity is below Φ=0.7, Algorithm 1 invokes Update(Q,M), which Appendix B implements by having o3-mini solve that exact test query, a second o3-mini pass check the solution, and GPT-4.1 convert the successful trajectory into a guideline E that is prepended to the system prompt before the 4B policy solves the same query. Thus, for these out-of-memory cases, the test instance has already been solved by a stronger model, and the reported agent executes under a hint derived from that solution. The §4.4 statement that update frequency is only ~3% does not remove the confound: memory hits also inject guidelines distilled from successful trajectories of stronger models. None of the baselines (GPT-4o, Gemini-2.5-Pro, Qwen3-235B, Kimi-K2) receives the same augmentation. The w.o.-Shared-M
[Table 1; §4.1–4.2] The evaluation rests on a single benchmark, a single run, and no error bars or significance tests. OpenAI and Gemini scores are taken from the original CRMArena-Pro paper rather than re-run under the same harness, while Qwen3-235B and Kimi-K2 are run with ReAct; the prompt and tool-interface match with the original protocol is not quantified. Without variance estimates or matched controls, the small differences supporting 'comparable' (e.g., B2B average 55.6 vs. 54.3 for Kimi-K2) are not established. The database-querying advantage in particular needs a per-task breakdown and evidence that SQL/SOSL execution and result handling are identical across compared systems.
[§3.1, Eq. (2)] The synthetic data construction assumes that GPT-4.1-generated queries from random walks over the record graph are well-posed, unambiguous, and uniquely answered by the target record ID. No human validation, error analysis, or ambiguity rate is reported for the generated QA pairs. The prompt principles in Table 2 do not guarantee that a question cannot be satisfied by a different record or that the natural-language attributes map to a unique record. If a nontrivial fraction of synthetic questions is under-specified, the SFT and RL rewards in Eqs. (4)–(7) would reward behavior that fits a noisy generator rather than genuine business reasoning. A human-validated sample (e.g., 100–200 items), per-category pass rates, and an analysis of whether the final policy uses surface cues from the synthetic prompts would substantially strengthen the training-recipe claim.

minor comments (4)

[§4.4; Algorithm 1] The few-shot LLM classifier used to suppress memory updates for knowledge-QA tasks is not introduced in §3.4 or in Algorithm 1. Its accuracy, prompting, and any effect on the reported update statistics should be documented, since it is an additional off-menu component of the inference pipeline.
[§4.1; Appendix C] 'metioned' typo; minor formatting issues ('et.al, 2024', 'llama-factory' capitalization). The example trajectory in Table 6 contains ellipses inside SQL IN clauses; if this is an abridged illustration rather than a literal execution log, it should be labeled as such.
[§3.3.2, Eq. (7)] The reward function weights are clear (0.1/0.9), but the definition of score_answer is fuzzy for numbers and strings: specify exact-match normalization for dates, decimals, and multi-value answers. This matters for interpreting RL-driven gains.
[General] The term 'long-term memory' is somewhat misleading: the module is a static index of guidelines distilled by stronger models, not a memory that the agent itself updates during deployment. Clarifying this distinction would improve positioning relative to the cited AgentKB/Mem0/Agent Workflow Memory literature.

Circularity Check

1 steps flagged · score 6.0 of 10

Test-time o3-mini oracle solves the same query in the memory-miss path; the 'shared memory' gain on unseen scenarios is not an independent prediction.

fitted input called prediction [§3.4 (Algorithm 1), Eq. (8), Appendix B]
"For tasks that do not have a direct precedent in the indexed memories, we employ an offline refinement step. In this step, a more powerful reasoning model attempts to solve the query and generates a detailed corresponding workflow guideline. This guideline is then added to the memory index... To generate the long-term memory unit E, we use o3-mini as the advanced reasoning model. After obtain the trajectories Hadv, we use GPT4.1 with prompt described in Table 4 to generate the guideline for the given task."

In Algorithm 1, any test query Q whose Top-1 similarity to stored memories is below the threshold Φ=0.7 enters the else branch, Update(Q,M). Appendix B reveals that update uses o3-mini to produce a trajectory Hadv for exactly that test query Q, and Eq. (8), E = LLM(Hadv, Q), converts the successful o3-mini trajectory into a guideline E. That guideline is then appended to the system prompt before the 4B policy θ solves the same Q (S ← Solve(Q|P;θ)). Thus the memory module's benefit on 'unseen scenarios' is not a generalization prediction: the test query itself has already been routed through a stronger model that attempted to solve it, and the reported agent executes under an oracle-derived hint. The Table 1 baselines receive no such injection, so the comparison is confounded for the memory

full rationale

The training-time components of CRMWeaver are self-contained: synthetic data is generated from known record paths with gold answers, SFT and DAPO RL are evaluated on the distinct CRMArena-Pro benchmark after training on CRMArena, and the rule-based reward is checked against gold answers. No circularity arises there. The one significant circular element is the inference-time memory module. In the memory-miss branch of Algorithm 1, the test query itself is solved by o3-mini, converted into a guideline via Eq. (8), and then injected into the same query's context before the 4B policy answers it. Consequently, the reported score for those 'unseen' queries reflects the advanced model's reasoning more than an independent capability of the trained 4B model, and none of the Table 1 baselines receives the same oracle-derived help. The paper's own w.o.-Shared-Memory ablation shows the trained model alone is still competitive (54.5 B2B / 55.3 B2C), and the update frequency is small (~3% B2B), so this is a partial circularity rather than a total collapse of the central claim. The self-citations to prior agentic-RL works are not load-bearing for the benchmark comparison. The headline 'comparable to Qwen3-235B / Gemini 2.5-Pro' should be repositioned to the w.o.-Shared-Memory variant, or baselines should be augmented with the same memory mechanism.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the quality of automatic synthetic data, the separation of train/test environments, and the correctness of the oracle-generated memory guidelines.

free parameters (4)

Memory similarity threshold Φ = 0.7
Set by hand in Algorithm 1; controls when memory is injected, affects performance.
Reward weights (format/answer) = 0.1 / 0.9
Chosen in Eq. 7; emphasizes answer correctness, hand-tuned.
DAPO clip bounds ε_low, ε_high = 0.2, 0.28
Hyperparameters of the RL update in Eq. 5; standard choices.
Rollout size G = 16
Number of rollouts per question in RL, chosen for stability/cost.

assumptions (3)

domain assumption CRMArena and CRMArena-Pro are distinct environments with no test-set leakage.
The paper asserts this in footnote 1 to support the generalization claim.
domain assumption LLM-generated synthetic queries (GPT-4.1) are correct and unambiguous.
No human validation is reported; training relies on these questions.
domain assumption The evaluation benchmark CRMArena-Pro is reliable and its published baseline numbers are truthful.
The paper copies GPT/Gemini results from the original CRMArena-Pro paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories." pith.science (2026). https://pith.science/paper/N5FSY7Z6

@misc{pith2026251025333,
  author       = {Pith},
  title        = {Pith review of: CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N5FSY7Z6}},
  note         = {Machine review of arXiv:2510.25333}
}

read the original abstract

Recent years have witnessed the rapid development of LLM-based agents, which shed light on using language agents to solve complex real-world problems. A prominent application lies in business agents, which interact with databases and internal knowledge bases via tool calls to fulfill diverse user requirements. However, this domain is characterized by intricate data relationships and a wide range of heterogeneous tasks, from statistical data queries to knowledge-based question-answering. To address these challenges, we propose CRMWeaver, a novel approach that enhances business agents in such complex settings. To acclimate the agentic model to intricate business environments, we employ a synthesis data generation and RL-based paradigm during training, which significantly improves the model's ability to handle complex data and varied tasks. During inference, a shared memories mechanism is introduced, prompting the agent to learn from task guidelines in similar problems, thereby further boosting its effectiveness and generalization, especially in unseen scenarios. We validate the efficacy of our approach on the CRMArena-Pro dataset, where our lightweight model achieves competitive results in both B2B and B2C business scenarios, underscoring its practical value for real-world applications.

Figures

Figures reproduced from arXiv: 2510.25333 by the authors.

**Figure 2.** The overall pipeline of CRMWeaver. We first construct both simple and complex synthetic data by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Performance comparison with SFT and RL on CRMArena-Pro across different tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
cs.CL 2026-06 unverdicted novelty 5.0 of 10

This survey categorizes agentic environments for LLMs by eight attributes and domains, introduces symbolic and neural synthesis paradigms with evaluation, and outlines four agent evolution pathways plus three environm...

Reference graph

Works this paper leans on

29 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

If no such records exist, return ’None’

Which live chat transcript documents a case where a Portland-based customer, previously known for resolving product and sizing com- plaints in 2022, reached out on November 30, 2023, to request an immediate exchange af- ter receiving a blue hoodie instead of black running shoes, and successfully initiated the return process with photo documentation throug...

2022
[2]

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma

Sciarena: An open evaluation platform for 10 foundation models in scientific literature tasks.arXiv preprint arXiv:2507.01001. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma

arXiv
[3]

Q": "generated question

Identify the durable and waterproof item, ideal for harsh weather conditions, that was added as a new selection by the customer whose previous order item was related to outdoor hiking, after a warranty issue with defective equipment was resolved in Portland and re- sulted in a successful transaction activation on March 19, 2022. What is the unique catalog...

2022
[5]

If no such records exist, return ’None’

During the second week of January 2022, a customer with an account connected to a con- tact named Isabella Adams, who has an email address of isabella.morgan@example.com and a shipping address in Austin, Texas, placed an order that was activated and used a price book featuring the promotion ’Start 2022 with unbeatable deals on sports gear.’ Which price bo...

2022
[8]

Allow deriving answer A by using the source and target records and connect key Question Generation Principles:
[9]

Exact correspondence − Each question must fully base on the original conclusion, with the answer being its core content
[16]

Q": "generated question

Language consistency − The language of each question must be the same as the conclusion's language. Table 2: Prompt for Generating Seed Query 13 Prompt for Initialize Seed Query You are a synthesis data generator. I will give you an existing question and its question. Apart from this, I will also give you a schema and a record. The schema and record inclu...
[17]

Be a complete natural language question

Show all 29 references

[18]

Allow deriving answer A by using the source and target records and the connect key Question Generation Principles:
[19]

Exact correspondence − Each question must be fully based on the original conclusion, with the answer being its core content
[20]

Derivability − The original conclusion must be directly derivable from the question and be the only correct answer
[21]

Self−containment − Questions must be complete and independent, not relying on external references or unspecified context
[22]

Information hiding − Do not reveal specific sources or data paths, but can include search hints
[23]

Specificity and clarity − Questions should include details like specific times to ensure unique answers
[24]

Single question − Generate only one question per conclusion
[25]

If the conclusion can only be obtained from input content, include hints via data source identifiers in the question
[26]

Table 3: Prompt for Generating Deeper Challenge Queries 14 Prompt for Generating Guideline Please provide a concise guideline for the following execution log

Language consistency − The language of each question must be the same as the conclusion's language. Table 3: Prompt for Generating Deeper Challenge Queries 14 Prompt for Generating Guideline Please provide a concise guideline for the following execution log. Break down the gui...
[27]

Focus exclusively on behavioral improvements derived from similar task patterns and experience
[28]

Format output strictly as:
[29]

[Specific suggestion 1]
[30]

[Specific suggestion 2]
[31]

## Detailed Schema Information {schema_information} ## Execution logs: {successful_trajectories} Table 4: The prompt used to generating the guideline of memory unitE

No headings, explanations, or markdown. ## Detailed Schema Information {schema_information} ## Execution logs: {successful_trajectories} Table 4: The prompt used to generating the guideline of memory unitE. Key:Can we qualify this lead based on their recent discussions?... Value:
[32]

Always use the execute tool to pull every relevant transcript and pricing/knowledge record before judging any BANT factor, ensuring no critical data is missed
[33]

After data retrieval, perform an explicit cost−vs−budget calculation inside the analysis step to avoid incorrect 'Budget' conclusions
[34]

When Authority is not clearly confirmed in the transcript, trigger an additional execute search (e.g., manager titles, purchasing roles) before deciding it is fulfilled or unmet
[35]

Parse Timeline details quantitatively (days, dates) rather than qualitatively; if uncertain, run another execute query for scheduling constraints to improve accuracy
[36]

Accepted

Maintain a consistent workflow order −> execute −> analyze −> respond to increase reliability and reproducibility across similar tasks. Table 5: An example of Memory Unit E, where theKeyis the indexed query used to match similar queries, andValueis the corresponding guideline....
[2024]

InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand

Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Wanjun Zhong, L...

2024
[2025]

Xingyao Wang, Boxuan Li, Yufan Song, Frank F

Webshaper: Agentically data synthesizing via information-seeking formalization.arXiv preprint arXiv:2507.15061. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, M...

2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

If no such records exist, return ’None’

Which live chat transcript documents a case where a Portland-based customer, previously known for resolving product and sizing com- plaints in 2022, reached out on November 30, 2023, to request an immediate exchange af- ter receiving a blue hoodie instead of black running shoes, and successfully initiated the return process with photo documentation throug...

2022

[2] [2]

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma

Sciarena: An open evaluation platform for 10 foundation models in scientific literature tasks.arXiv preprint arXiv:2507.01001. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma

arXiv

[3] [3]

Q": "generated question

Identify the durable and waterproof item, ideal for harsh weather conditions, that was added as a new selection by the customer whose previous order item was related to outdoor hiking, after a warranty issue with defective equipment was resolved in Portland and re- sulted in a successful transaction activation on March 19, 2022. What is the unique catalog...

2022

[4] [5]

If no such records exist, return ’None’

During the second week of January 2022, a customer with an account connected to a con- tact named Isabella Adams, who has an email address of isabella.morgan@example.com and a shipping address in Austin, Texas, placed an order that was activated and used a price book featuring the promotion ’Start 2022 with unbeatable deals on sports gear.’ Which price bo...

2022

[5] [8]

Allow deriving answer A by using the source and target records and connect key Question Generation Principles:

[6] [9]

Exact correspondence − Each question must fully base on the original conclusion, with the answer being its core content

[7] [16]

Q": "generated question

Language consistency − The language of each question must be the same as the conclusion's language. Table 2: Prompt for Generating Seed Query 13 Prompt for Initialize Seed Query You are a synthesis data generator. I will give you an existing question and its question. Apart from this, I will also give you a schema and a record. The schema and record inclu...

[8] [17]

Be a complete natural language question