Pith. sign in

REVIEW 4 major objections 6 minor 19 references

AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that an LLM-driven pipeline can synthesize 5,404 high-quality data-driven discovery training tasks from GitHub research code, and that a 32B open-weight model fine-tuned on them matches GPT-4o on ScienceAgentBench.

desk verdict A genuinely new auto-collection pipeline for scientific coding tasks, with modest but real gains on external benchmarks—but the dataset-quality headline rests on a 256-task self-evaluation. read the letter →

arxiv 2506.08140 v1 pith:C7D2IZB2 submitted 2025-06-09 cs.LG cs.CL

Yifei Li , Hanane Nour Moussa , Ziru Chen , Shijie Chen , Botao Yu , Mingyi Xue , Benjamin Burns , Tzu-Yao Chiu
show 11 more authors
Vishal Dey Zitong Lu Chen Wei Qianheng Zhang Tianyu Zhang Song Gao Xuhui Huang Xia Ning Nesreen K. Ahmed Ali Payani Huan Sun
This is my paper
classification cs.LGcs.CL
keywords scientificdiscoverydata-drivencodegenerationautomateddatasetcollectionopen-weightLLMssupervisedfine-tuningScienceAgentBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that the bottleneck preventing open-weight AI co-scientists—the scarcity of high-quality coding tasks for data-driven scientific discovery—can be broken by an automatic pipeline rather than slow manual annotation. It presents AutoSDT, which searches GitHub and PapersWithCode for research code, filters for programs that ingest datasets and produce scientific outputs, adapts them into standalone runnable programs, and back-translates them into task instructions; the result is AutoSDT-5K, 5,404 tasks across four disciplines at an average cost of $0.55 per task. To support the claim, nine domain experts reviewed 256 sampled tasks and judged 93% ecologically valid and 92.2% of the synthesized programs correct. The paper further claims that supervised fine-tuning on this dataset delivers concrete gains: AutoSDT-Coder-32B doubles its base model's success rate on ScienceAgentBench, from 3.9% to 7.8%, matching GPT-4o from 2024-05-13, and lifts the DiscoveryBench hypothesis matching score from 6.9 to 8.1. If these claims hold, AI co-scientists no longer need proprietary models or expensive manual datasets.

What carries the argument

The load-bearing object is the AutoSDT pipeline, a three-stage automatic dataset factory. AutoSDT-Search expands seed discipline keywords with an LLM and crawls GitHub and PapersWithCode for research repositories; AutoSDT-Select uses LLM judgments to keep only code files that ingest datasets and emit scientific outputs, then extracts the dependencies needed to make a compact runnable workspace (40.42 MB average versus 264.98 MB per repository); AutoSDT-Adapt rewrites each selected program for standalone execution, iterating through self-debugging execution loops up to three times, then back-translates the working program into a domain-language task instruction. The final object is AutoSDT-5K, the 5,404 instruction-solution pairs produced by this factory, which the paper uses both as the evidence that automatic collection works and as the training signal for AutoSDT-Coder.

What would settle it

Sample, say, 300 AutoSDT-5K tasks with independent domain experts who are not connected to the authors, execute each provided solution from scratch in a clean environment, and compare outputs to the stated task goal; if the expert-validity rate or the executable-solution rate falls far below the reported 93% and 92.2%, or if a large share of tasks fail because the LLM-selected dependencies are wrong, then the dataset-quality premise collapses. A second decisive check: retrain AutoSDT-Coder-32B on a cleaned subset from which all tasks that fail independent execution are removed and see whether the ScienceAgentBench gains persist.

Watch

Extended reading notes

Core claim

The central claim is that data-driven scientific discovery tasks—which previously required 2.5–3 hours of expert annotation per instance—can be synthesized automatically from naturally occurring scientist-authored code, at a scale and cost that make them usable for training rather than only evaluation. AutoSDT-5K, built this way, is asserted to be the largest and the only automatically collected open dataset for data-driven scientific discovery, spanning bioinformatics, computational chemistry, geographic information science, and psychology and cognitive neuroscience, with 756 unique Python packages. The paper's evidence is two-part: expert review of 256 tasks reports 93% ecological validity and 92.2% functional correctness of the synthesized solutions, and fine-tuning Qwen2.5-Coder-Instruct on the full dataset yields the AutoSDT-Coder series, with the 32B model doubling its ScienceAgentBench success rate to 7.8%—the same level as GPT-4o (2024-05-13)—and raising its DiscoveryBench hypothesis matching score by 17.4% relative to the base model.

Load-bearing premise

The entire argument rests on the assumption that the LLM judgments used to filter repositories, select scientific programs, extract dependencies, adapt code, and generate instructions are accurate enough that the 5,404 tasks in AutoSDT-5K are predominantly real, executable, and correctly solved—and that the 256 tasks checked by the paper's own experts are representative of the whole dataset.

Editorial extensions

If this is right

  • AutoSDT-5K makes it feasible to train—not just evaluate—models on data-driven discovery, since each task costs about $0.55 in API fees rather than 2.5–3 hours of expert annotation.
  • Fine-tuning on AutoSDT-5K improves open-weight code models on both ScienceAgentBench and DiscoveryBench, and the gains grow with model size: the 32B model's success rate doubles from 3.9% to 7.8%, the 14B improves to 5.9%, and the 7B shows no success-rate gain.
  • The trained models transfer across disciplines to some degree, with discipline-specific training best for specialization and multi-discipline training best for broad coverage.
  • Larger models continue to benefit from more AutoSDT-5K data beyond 5,000 examples, while the 14B model saturates near 2,500, so the dataset's value should grow as open-weight models scale up.
  • AutoSDT-Coder-32B reaches the performance level of GPT-4o (2024-05-13) on ScienceAgentBench, but a gap to reasoning models such as o1-preview and Claude-3.7-Sonnet remains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to run AutoSDT on a fifth discipline with abundant research code, such as ecology or econometrics, and check whether expert-validity rates and downstream fine-tuning gains reproduce; success there would confirm the pipeline rather than the four chosen fields.
  • Adding outcome-based evaluation scripts that compare model outputs to the gold program results would unlock reinforcement learning and rejection sampling; the paper notes this verification is nontrivial because adapted programs lack unit tests.
  • Because instructions are back-translated from code, they inherit the source repositories' missing context—the paper reports 26.7% of sampled instructions lack clarity—so injecting README or paper context during instruction generation is a natural next experiment.
  • Read editorially, the absolute success rate of 7.8% is low, so matching GPT-4o signals that automatic data collection removes a data bottleneck but probably needs to be combined with reasoning traces or agent scaffolds to produce practically useful co-scientists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AutoSDT, a fully automatic pipeline that mines GitHub and PapersWithCode repositories to produce AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven scientific discovery across four disciplines (bioinformatics, computational chemistry, geographic information science, and psychology/cognitive neuroscience). The pipeline uses LLMs for keyword expansion, repository/filtering, dependency extraction, program adaptation, and instruction generation. The authors report that expert evaluation of 256 sampled tasks gives 93% ecological validity and 92.2% code correctness, and that supervised fine-tuning of Qwen2.5-Coder-Instruct models on AutoSDT-5K yields substantial improvements on ScienceAgentBench and DiscoveryBench, with the 32B model reaching a 7.8% success rate on ScienceAgentBench, comparable to GPT-4o (2024-05-13).

Significance. If the claims hold, this is a valuable contribution: it offers a low-cost (0.55 USD/task) automatic pipeline for scaling training data for open-weight AI co-scientists, and the dataset is released publicly. The training evaluation is strengthened by using external benchmarks whose repositories are explicitly excluded, and the paper reports standard deviations over multiple runs. The release of data and code, the cost transparency, and the cross-disciplinary scope are concrete assets. However, the central 'high-quality AutoSDT-5K' claim rests on a small, non-independent expert evaluation, and the benchmark comparisons include some overstatements relative to the reported error bars and to the original DiscoveryBench results. The usefulness of the dataset as a high-quality training resource therefore needs additional verification or more carefully hedged claims.

major comments (4)
  1. [§3.2 and Author Contributions] The expert evaluation of 256 tasks is not an independent validation: the Author Contributions section lists M. Xue, B. Burns, T. Chiu, V. Dey, Z. Lu, C. Wei, Q. Zhang, T. Zhang, and S. Gao as the subject-matter experts who validated the 256 tasks, and all nine are co-authors. The reported 93% ecological validity, 92.2% code correctness, and 84.4% functional equivalence are therefore self-assessments. Because the abstract and §1 assert that AutoSDT-5K is 'high-quality' and the training gains are attributed to that quality, these claims need independent support. Please provide an external audit by non-affiliated domain experts, release a larger verified subset, or substantially temper the quality claims.
  2. [Limitations and §2.3] The Limitations section concedes that AutoSDT does not generate evaluation scripts and that 'it is non-trivial to ensure the complete correctness of the programs adapted through AutoSDT without human or even subject matter expert intervention.' The self-debugging loop in §2.3 verifies only that the adapted program executes without errors; it does not verify that the program's scientific output is correct. Since only 256 of 5,404 tasks (4.7%) have any human review, and functional equivalence to the original code was judged in only 84.4% of those, the attribution of benchmark gains to the high quality of the full dataset is not established. Please analyze the sensitivity of the fine-tuning gains to dataset quality (e.g., compare training on verified vs. unverified subsets, or filter by expert-rated quality) or provide additional automated/independent verification.
  3. [Table 4 and §4.2] The text states that 'AutoSDT-Coder-32B outperforms GPT-4o (2024-05-13),' but the reported numbers do not support this: in Table 4, AutoSDT-Coder-32B has SR 7.8 (±1.4) vs. GPT-4o's 7.5 (±0.5) and VER 36.0 (±5.2) vs. GPT-4o's 42.2 (±1.6). The confidence intervals overlap substantially and GPT-4o is numerically higher on VER. The abstract's more cautious phrasing 'reaches the same level of performance' is appropriate, but §4.2 overstates the result. Please either report a significance test or revise the wording to reflect that the difference is not statistically established.
  4. [Appendix E and §4.1] The DiscoveryBench comparison is based on the authors' re-implementation of the benchmark pipeline, not on the official numbers from the DiscoveryBench paper. Table E.1 shows that the reproduction gives GPT-4o (2024-05-13) an HMS of 10.4, whereas the original DiscoveryBench paper reports 15.5 for the same model. The claim that AutoSDT-Coder-32B 'closes the gap' to GPT-4o is therefore against a modified evaluation setup, and the gap to the original benchmark results remains large. Please report the original DiscoveryBench numbers alongside the re-implemented ones and clarify which numbers are used for the claims in the abstract and §4.2.
minor comments (6)
  1. [Abstract and §3.1] The claim that AutoSDT-5K is 'the only automatically collected and the largest open dataset for data-driven scientific discovery' is a strong literature claim; please provide a systematic comparison in the main text or explicitly note the date of the literature search, since concurrent works may affect this statement.
  2. [§3.2] The expert evaluation reports that 26.7% of task instructions lack clarity, with the main issue being missing methodological guidance. This is a nontrivial fraction for a training dataset, and the paper would benefit from a more detailed breakdown of the types of missing information and how this affects the fine-tuning results.
  3. [§2.1 and Appendix A.1] The pipeline includes several arbitrary thresholds (e.g., minimum 10 stars, maximum 1,000 lines, self-debugging loop at most three iterations). The paper does not analyze the sensitivity of the final dataset to these choices; please add a short discussion or ablation acknowledging these as design decisions.
  4. [§3.1] The cost of 0.55 USD per task covers only API costs. The expert evaluation time and the compute for fine-tuning are not included, so the phrase 'as low as 0.55 USD per task' could be misread; please clarify the scope of the cost figure.
  5. [Listing B.1] The example task instruction says to save masks to the directory specified by 'output_mask_path', but the code solution in Listing B.3 overrides the argument to 'pred_results/pred_road_masks'. This mismatch between instruction and code in the displayed example should be fixed, and it would be reassuring if the dataset were checked for such path inconsistencies.
  6. [Appendix A.1 and Listing B.2] There are several typos: 'AutoDCT-Search' in Table A.1 should be 'AutoSDT-Search', and 'Unavilable' in Listing B.2 (line 21) should be 'Unavailable'. A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 2.0 of 10

Main training claim is externally benchmarked; only the code-solution validity metric is a back-translation closed loop.

  1. self definitional [Section 2.3 (Task Instruction Generation); Section 3.2 (Code Solution Correctness); Appendix Table A.5]
    "Given an adapted program, we prompt an LLM to back-translate it into a clear task instruction (Appendix Table A.5) that explicitly includes the task goal, required input data and/or model files, and expected output files ... the correctness of the adapted code is still high, with 92.2% of programs deemed correct solutions to their task instructions."

    The task instruction is generated from the code solution via back-translation, and the same code is then judged to be a valid solution to that instruction. A faithful back-translation will by construction produce high agreement, because the instruction is written to describe the code rather than to specify an independently grounded task. Thus the 92.2% 'code solution correctness' figure largely measures instruction-generation fidelity, not an independent verification that the program solves a pre-existing task. The separate functional-equivalence check against the original GitHub program (84.4%) is non-circular, but the headline 'functionally correct' number is a closed loop.

full rationale

The paper's central value claim is that training on AutoSDT-5K improves data-driven discovery performance. That claim is tested on ScienceAgentBench and DiscoveryBench, and the paper explicitly states that repositories used in those benchmarks are excluded from AutoSDT-5K ('we eliminate duplicate repositories and ensure that there is no overlap with the repositories utilized in existing benchmarks which we use for evaluation'). DiscoveryBench is an external benchmark from a different research group, so the observed gains (e.g., 3.9% to 7.8% success rate on ScienceAgentBench; hypothesis matching score from 6.9 to 8.1 on DiscoveryBench) are genuine held-out predictions rather than fits to the training data. The only circular element I found is the quality-validation loop: task instructions are back-translated from the code solutions, and the same code is then rated as a correct solution to those instructions (Section 2.3 and Section 3.2). That makes the 92.2% correctness statistic partially self-confirming. Separately, the expert validators are listed as co-authors in the Author Contributions section, which is an independence concern rather than a circularity per se; the paper's own Limitations section also concedes that ensuring complete correctness of adapted programs 'is non-trivial ... without human or even subject matter expert intervention.' These caveats weaken the dataset-quality evidence but do not collapse the main training claim, which rests on external benchmarks. Overall, the circularity is confined to a supporting validation metric, so a score of 2 is appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper does not derive equations; its central claims rest on assumptions about source repositories, LLM-based filtering and adaptation, and extrapolation from a co-author expert evaluation. No invented scientific entities are introduced. The heuristic pipeline thresholds (minimum stars, file-length cutoff, self-debugging iterations) are treated as free design choices.

free parameters (3)
  • minimum repository stars = 10
    Search inclusion criterion in Appendix A.1; an arbitrary threshold that affects the pool of repositories and therefore the task distribution.
  • maximum source file length = 1000 lines
    Rule-based filter in Section 2.2; excludes longer programs and affects the complexity and difficulty distribution of tasks.
  • self-debugging iterations = 3
    AutoSDT-Adapt in Section 2.3 caps the adaptation and execution loop at three attempts; tasks still failing are discarded, affecting yield and quality.
assumptions (4)
  • domain assumption GitHub and PapersWithCode contain repositories that are representative of real-world data-driven discovery workflows.
    The pipeline in Section 2.1 assumes code hosting platforms provide ecologically valid scientific code; only 256 sampled tasks are human-checked.
  • domain assumption LLM parametric knowledge can accurately identify data-driven scientific code and extract dependencies.
    Section 2.2 relies on GPT-4o for scientific-code filtering and dependency extraction; systematic LLM errors would propagate into the dataset.
  • domain assumption The original repository code is scientifically correct and the LLM-based adaptation preserves its functionality.
    Section 2.3 modifies code for standalone executability without unit tests; expert review found only 84.4% functional equivalence to the original code.
  • domain assumption Expert ratings on 256 sampled tasks generalize to all 5,404 tasks.
    Section 3.2 uses a random 256-task sample, but the evaluators are also listed as authors and no inter-rater reliability or independent audit is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists." pith.science (2026). https://pith.science/paper/C7D2IZB2

@misc{pith2026250608140,
  author       = {Pith},
  title        = {Pith review of: AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7D2IZB2}},
  note         = {Machine review of arXiv:2506.08140}
}
read the original abstract

Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o.

Figures

Figures reproduced from arXiv: 2506.08140 by the authors.

Figure 1
Figure 1. Performance of our AutoSDT-Coder in com [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our AutoSDT collects data-driven discovery tasks in three steps: (1) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Distribution of subtasks in our dataset. Tasks in AutoSDT-5K are multi-step research workflows [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of training set size on ScienceAgent [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    Functionality: the functionality of the given program should be related to tasks in a scientific workflow. These tasks include but are not limited to feature engineering, machine learning, deep learning, computational analysis, data visualization, model training, numerical calculation/analysis, statistical methods, domain-specific analysis/simulation, etc

    work page

  2. [2]

    a scientific task

    Input: the program should receive at least one or multiple datasets as input. In other words, the program is dealing with a dataset and conducting analysis or experiments on top of the data. The data can either be loaded through built-in functions or be loaded from local files. If the current program does not receive and process any data, it cannot be con...

    work page

  3. [3]

    A code file is considered a scientific task ONLY IF it completely satisfied the three dimensions above

    Output: the program should output numerical or visualization results that can be further evaluated. A code file is considered a scientific task ONLY IF it completely satisfied the three dimensions above. For example, code files that purely contain modeling, training/testing, data pre-processing, or only consist of utility functions or class definitions, a...

    work page

  4. [4]

    In the case of a “Yes”, make sure to put the relative paths to all dependencies as a list of paths in the MODULE_PATHS field, based on the GitHub file tree given to you

    work page

  5. [5]

    pred_results/pred_[code_file_name].[extension]

    If based on the code alone you can only identify the folder that contains the dependency but not the exact file only return the path to the folder. This is because you might sometimes not be able to know which file the dependency is exactly located in based on only looking at the file tree. Thus, to stay on the safe side, just give the path to the folder ...

    work page 2024

  6. [7]

    Yes” in DATASET_LABEL field. Otherwise, you should output “No

    Recognize whether the code makes use of a dataset. The dataset can either be loaded via built-in library functions (e.g., data = MNIST ()) or loaded from a local file in the repository (csv, jsonl, xls, txt, parquet, or any other file type). If the dataset(s) used in the code are either loaded through built-in library functions or contained within the rep...

    work page

  7. [8]

    You will list the paths to all datasets used in the code as a list of paths after the field DATASET_PATHS

    In the case where the dataset used in the code is contained within the repository, you also have to find the relative path to the dataset file, based on the GitHub file tree that will be given to you. You will list the paths to all datasets used in the code as a list of paths after the field DATASET_PATHS

    work page

  8. [9]

    Yes” in the MODULE_LABEL. Otherwise, output a “No

    Besides the dataset, now you have to identify all other in-repository dependencies that the code uses, and extract their relative paths based on the file tree given to you. These can be modules, classes, models, or any other dependency that the code imports from a folder within the repository. If you identify that there are in-repository dependencies used...

    work page

Show all 19 references
  1. [13]

    Is this a meaningful and realistic scientific data analysis task that a scientist in your field would perform in their research?[Yes/No]

  2. [14]

    Is the instruction correctly expressed in the domain scientific language?[Yes/No]

  3. [15]

    Is the instruction clear and contains all required information needed to complete the task - goal, methods, input, and output? In other words, if you were given this instruction as a task, would you have the information you need to start writing a solution?[Yes/No]

  4. [16]

    In this section you will answer two questions about the functionality equivalence with the original program on GitHub and the program correctness

    If your answer to the previous question was “No”, what is missing? Program Solution. In this section you will answer two questions about the functionality equivalence with the original program on GitHub and the program correctness

  5. [17]

    Does the program perform the same functionality as the original program on GitHub? There might be changes to the program in the adaptation process to make it executable in a standalone environment (e.g. changes to the import statements, to the input / output routines, etc.) Pl...

  6. [18]

    In this section you will answer rate the task difficulty

    Does the program represent a valid solution to the task? There could be multiple possible solutions to a task, here you should just determine whether the program is a valid solution and correctly addresses the goal of the task.[Yes/No] Task Difficulty. In this section you will...

  7. [19]

    do not assume familiarity with certain libraries/methods/packages.)

    How would you rate the difficulty level of the task? Your judgment about the task difficulty should be realistic ( i.e. do not assume familiarity with certain libraries/methods/packages.). In other words, if you had to write a solution for the task right now, how long would th...

    2024

  8. [1981]

    InProceedings of the 7th International Joint Confer- ence on Artificial Intelligence - Volume 1, IJCAI’81, page 121–126, San Francisco, CA, USA

    Bac0n.5: the discovery of conservation laws. InProceedings of the 7th International Joint Confer- ence on Artificial Intelligence - Volume 1, IJCAI’81, page 121–126, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. Patrick W. Langley, Herbert A. Simon, Gary Bradshaw, an...

    1987 arXiv

  9. [2018]

    Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose

    Scanpy: large-scale single-cell gene expres- sion data analysis.Genome Biology, 19:15. Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. 2024. Codebench- gen: Creating scalable execution-based code genera- tion benchmarks.Preprint, arXiv:2404....

    2024 arXiv

  10. [2025]

    citation,

    Swe-smith: Scaling data for software engineer- ing agents.Preprint, arXiv:2504.21798. Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas R. Campion, Zhiyong Lu, Chunhua Weng, and...

    2024 arXiv

  11. [8192]

    geojson_roads_speed/

    Warmup is turned off for 7B/14B and turned on for 32B. Training is done on 4 NVIDIA H100 96G GPUs (for 7B/14B) and 8 for 32B models. Inference.We use the vLLM library (Kwon et al., 2023) to deploy LLM servers and con- duct inference experiments. For all the infer- ence in Scie...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.