Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code Workflows

Emily Chen; Jingfei Chen; Maggie Baird; Orlando Marquez Ayala; Patrice Bechard

REVIEW 4 major objections 4 minor 37 references

Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code Workflows

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a 12-billion-parameter small language model yields higher-quality low-code workflows than prompting much larger LLMs, improving Flow Similarity by about 10% on average.

desk verdict A genuinely useful industry case study, but the headline 10% FlowSim gain is likely an upper bound: the metric is validated only on the fine-tuned SLM's outputs and the prompt comparison is not per-model tuned. read the letter →

arxiv 2505.24189 v2 pith:S5CYRGCW submitted 2025-05-30 cs.LG cs.AIcs.CL

Orlando Marquez Ayala , Patrice Bechard , Emily Chen , Maggie Baird , Jingfei Chen This is my paper

classification cs.LGcs.AIcs.CL

keywords low-codeworkflowsfine-tuningsmalllanguagemodelsretrieval-augmentedgenerationtreeeditdistancestructuredoutputserroranalysisworkflow

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether, for a domain-specific task that demands structured outputs, fine-tuning a small language model still beats prompting a large one. It studies enterprise low-code workflow generation, where a natural-language requirement must become a JSON workflow whose steps and inputs have to match a specific installation. The authors fine-tune Mistral-Nemo-12B-Base on two stages of the pipeline and compare it with six prompted LLMs, including GPT-4o and Gemini 2.0 Flash. Using a tree-edit-distance metric, Flow Similarity, the fine-tuned SLM is about 10% better on outline-plus-inputs across in-domain and out-of-domain test sets, and it also has fewer structure errors than most baselines. The result matters because falling token prices erode the cost rationale for small models; this is evidence that a quality rationale can remain.

What carries the argument

The pipeline decomposes generation into two sub-tasks: createFlow, which produces the workflow outline with an annotation per step, and populateInputs, which fills each step's inputs using retrieved environment data. RAG supplies the available step names and table/column/value names so the output matches each installation. Flow Similarity (FlowSim) measures quality by representing workflows as trees and computing tree edit distance, and the paper validates it against human scores (Pearson 0.78 for outline plus inputs). The fine-tuned model is Mistral-Nemo-12B-Base trained with standard supervised fine-tuning, with the training data as the key ingredient.

What would settle it

Run the same TEST and OOD evaluation with per-model prompt optimization, such as model-specific instructions or few-shot examples, on GPT-4o and Gemini; if either model reaches or passes the fine-tuned SLM's Flow Similarity on outline plus inputs, the paper's central claim would fail.

Watch

Extended reading notes

Core claim

The central claim is that for generating low-code workflows in JSON, supervised fine-tuning of a small language model outperforms prompting large language models. On the TEST set, the fine-tuned Mistral-Nemo-12B-Base reaches 86.0 Flow Similarity for outline plus inputs, versus 75.3 for GPT-4o and 77.5 for Gemini-2.0-Flash; on the out-of-domain set the numbers are 79.8 versus 67.4 and 67.8, a 12.4% gap. The paper also reports that most remaining errors are model errors rather than retrieval errors: substituting perfect retrieval for the real retriever improves the top models by at most 4%. Error analysis across 24 features shows the fine-tuned SLM leads on enterprise-specific and input features, while it lags on structural logic such as FOREACH and PARALLEL.

Load-bearing premise

The comparison assumes that the single hand-written prompt template works equally well for every prompted LLM; if each large model were given a prompt tuned to its own instruction-following style, the measured quality gap could shrink or disappear.

Editorial extensions

If this is right

On structured, domain-specific tasks with strict output schemas, fine-tuning a small model can beat prompting frontier LLMs, not just match them.
The gap is larger when full inputs are generated (outline plus inputs) than for outline alone, so the harder sub-task benefits more from fine-tuning.
Since perfect retrieval improves the best models by at most 4%, further gains should come from model improvements rather than better retrieval.
The fine-tuned SLM is weaker on structural logic steps such as FOREACH and PARALLEL; targeted training data could close that gap.
The FlowSim metric with structure validation offers an automated way to evaluate workflow generation without human review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Per-model prompt optimization could narrow the 7–12% gap, since the paper uses one template per task for all baselines without tuning.
The two-stage outline-plus-inputs decomposition likely transfers to other structured-output domains, such as API call generation or config-file synthesis.
As token prices fall, the economic argument for SLMs weakens, but the quality argument may persist specifically for schema-constrained domain outputs.
The binary feature matrix for error analysis is a reusable diagnostic; adding training data for the weak structural-logic features could close that gap.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a solid, honest industry case study, and the core question it asks is timely. But the headline number - roughly 10% FlowSim improvement for a fine-tuned 12B SLM over prompted frontier LLMs - should be read as an upper bound. The stress-test concern is real: FlowSim compares against a single labeled reference workflow with exact string matches, and Appendix A validates the metric only on 30 samples generated by the fine-tuned SLM, not on the LLMs' outputs. Since the SLM is fine-tuned to mimic the reference style, the gap may partly measure stylistic closeness to the reference, not higher quality. The paper acknowledges the single-reference limitation but does not quantify its effect across models, so the 7.2% (TEST) and 12.4% (OOD) gaps are likely overstated. What the paper does well: the two-stage decomposition with annotations as glue is a practical design worth borrowing; the large OOD evaluation and a separately collected TEST set simulating real user behavior are more than most workshop papers do; the binary-feature error analysis is a genuine, reusable methodology; and the structure-error rates plus the perfect-RAG ablation give useful supplementary evidence. The authors are transparent about their limitations, including the metric's flaws, the proprietary data, and the single prompt template used for all LLMs. The soft spots, in proportion: the prompt-fairness issue is real but secondary - one template per task, no per-model tuning, could easily shrink the gap. The bigger issue is the metric. No confidence intervals or significance tests anywhere, and the human-correlation validation on 30 samples, while statistically significant, does not establish that FlowSim is fair across models. None of these are fatal for an industry case study, but they mean the paper's central claim is conditional: fine-tuning beats prompting under this particular single-reference, exact-match metric with a hand-tuned prompt template. Who this is for: practitioners building structured-output or enterprise-workflow generators, and researchers studying fine-tuning versus prompting in domain-specific DSLs. It is a useful data point, not a rigorous benchmark. If I were an editor, I would send it to peer review rather than desk reject - the question is important, the method is detailed, and the limitations are honestly stated. But I would ask for per-model prompt tuning, a multi-reference or softer metric, and some uncertainty quantification before treating the 10% as established. As is, conditionally accept.

Referee Report

4 major / 4 minor

Summary. The paper reports a case study in which the authors fine-tune Mistral-Nemo-12B-Base on a collection of low-code workflow generation tasks (outline generation and input population, both in JSON) and compare it against prompting several much larger LLMs, including GPT-4o, Gemini-2.0-Flash, Llama-3.3-70B-Instruct, and o3-mini. The authors introduce a custom evaluation metric, Flow Similarity (FlowSim), based on tree edit distance over workflow trees, and report that the fine-tuned SLM outperforms all prompted LLMs by 7.2% on the TEST set and 12.4% on the OOD set for outline-and-inputs, roughly 10% on average. They also report fewer structure errors, a perfect-RAG experiment suggesting that most outline errors are model errors rather than retrieval errors, and a feature-based error analysis on the TEST set. The paper's central claim is that, for this domain-specific structured-output task, fine-tuning a small model gives a real quality advantage over prompting much larger general-purpose models.

Significance. If the comparison is fair and the metric is unbiased across models, the result is a practically useful data point for enterprise applications: it suggests that for structured, domain-specific generation, a fine-tuned 12B model can beat much larger prompted models in output quality while presumably being cheaper and faster to serve. The paper also contributes a reusable two-stage pipeline (outline then inputs), a new dataset of labeled workflows, and a systematic error-analysis procedure that goes beyond a single aggregate metric. The authors are transparent about some limitations of FlowSim and provide a human-correlation study in Appendix A. However, the significance is currently tempered by two load-bearing methodological concerns: the metric is validated only on outputs of the fine-tuned SLM, and the prompted baselines use a single hand-written prompt template with no per-model tuning. These concerns need to be addressed before the central quantitative claim can be taken at face value.

major comments (4)

[§4.3, Appendix A] The FlowSim metric is validated against human ratings only on 30 TEST-set samples generated by the fine-tuned SLM (Appendix A, Table 6). The central claim, however, is a cross-model comparison: the 7.2% and 12.4% gaps in Figure 4 are computed with the same metric on outputs of GPT-4o, Gemini-2.0-Flash, and the other prompted models. Because FlowSim compares against a single reference workflow and requires exact string matches for inputs, it can systematically penalize valid alternative workflows or synonymous input values, which may be more common in instruction-tuned LLM outputs. The authors acknowledge the single-reference limitation in §4.3, but they do not quantify how it affects the cross-model comparison. To support the headline claim, the authors should validate FlowSim (or a version of it) against human judgments on outputs from at least the two best-performing LLMs (GPT-4o and Gemini-2.0-Flash), or provide evidence that exact-match penalties are approximately equally distributed across models.
[§4.2] The comparison treats the two manually crafted prompt templates as equally effective for all six prompted models. Section 4.2 states that one template per task is used for every non-fine-tuned model, with no per-model prompt tuning or hyperparameter search. Since instruction-tuned LLMs differ in their sensitivity to prompt formatting, guidelines, and examples, the measured gap may conflate model capability with prompt fit. A stronger comparison would include a small prompt-tuning step per model (e.g., adapting the examples or instruction phrasing) or at least a robustness check with two or three template variants. Without such evidence, the conclusion that fine-tuning an SLM is superior to 'prompting an LLM' is too strong; it should be stated as superior to prompting with the authors' default template.
[§5, Figure 4] All reported FlowSim scores are point estimates without variance, confidence intervals, or significance tests. The main quantitative claim is a 7.2% gap on TEST (108 samples) and a 12.4% gap on OOD (1,072 samples), but the paper does not show whether these gaps are statistically distinguishable from noise. For the TEST set, bootstrapped confidence intervals or a paired nonparametric test (e.g., Wilcoxon signed-rank on per-sample FlowSim scores) would substantially strengthen the claim. This is especially important because the FlowSim distribution is likely skewed and the sample size is modest.
[§5, Table 3] The conclusion that 'most errors are model errors' is drawn from the perfect-RAG experiment, which is limited to outline generation on the TEST set. The paper itself states in §5 that the gap between the fine-tuned SLM and LLMs is larger when generating complete workflows (outline and inputs) than when generating outlines alone. Therefore, the perfect-RAG result does not directly support a claim about the sources of errors in the populateInputs stage. The authors should either qualify the 'most errors are model errors' statement to outline generation only, or run a similar perfect-RAG analysis for input population with oracle artifacts.

minor comments (4)

[§4.3] The paper clearly lists the limitations of FlowSim, which is good practice; however, the second limitation (exact string matching for inputs) is likely to affect different models differently, and a sentence quantifying this (e.g., how many of the 108 TEST samples contain valid but non-identical input values) would help the reader assess the risk.
[§6.2] In the sentence 'the SLM frequently misses the dependency steps associated with them (e.g. FOREACH is frequently paired with a prior look_up_records step, PARALLEL should always consist of more than one branch)', the grammar is slightly awkward; consider rewording to 'a preceding look_up_records step'.
[§2] The citation style is inconsistent: 'previous work (2, 18, 29, 31, inter alia)' uses parentheses with numbers, while other citations use bracketed numbers; please unify the style throughout.
[Appendix A] The human-evaluation correlation study uses 30 samples; it would be helpful to state the number of human raters and whether each sample was scored by multiple raters, since inter-rater reliability affects the strength of the validation claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fine-tuning advantage is an empirical comparison on held-out test sets, and the only self-citation is not load-bearing for the central claim.

full rationale

The central claim is an empirical result: a fine-tuned Mistral-Nemo-12B-Base is compared against prompted LLMs on TEST and OOD workflows using FlowSim, a tree-edit-distance metric applied to labeled expected workflows. No parameter is fitted from the evaluation data and then renamed as a prediction; the reported 7.2% and 12.4% gaps come directly from comparing generated outputs to independent ground-truth workflows. FlowSim is a standard tree-edit-distance similarity (Zhang-Shasha) applied to a workflow-to-tree encoding, and Appendix A validates it against human scores, although only on samples generated by the fine-tuned SLM; this is a measurement-validity limitation, not a circular reduction, and the paper explicitly acknowledges the single-reference and exact-string-match limitations in Section 4.3. The only potentially self-referential element is the statement in Section 3.2 that 'RAG is necessary to reduce hallucinations when generating structured outputs [2, 3]', where reference [2] is prior work by two of the authors; however, this claim motivates the shared retrieval pipeline and is not used to establish the fine-tuning advantage, since both the fine-tuned SLM and the prompted LLMs use the same RAG infrastructure. The central comparison therefore retains independent content and does not reduce by construction to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

No math derivation or fitted constants underpin the comparison; the ledger records the three domain assumptions the empirical result relies on: metric validity, dataset representativeness, and prompt fairness. There are no free parameters, and the only introduced construct is the FlowSim metric.

assumptions (3)

domain assumption FlowSim, a tree edit distance with exact input string matching, is a valid proxy for user-perceived workflow quality
Used for all quantitative comparisons; Appendix A reports Pearson/Spearman correlations of 0.76-0.83 with human scores on 30 samples, but the metric has acknowledged limitations (single reference workflow, exact match, ignores structural validity).
domain assumption The TEST and OOD evaluation sets are representative of real usage and are out-of-domain relative to the training data
TEST is 108 expert-submitted requirements; OOD is roughly 100 workflows each from 10 customer deployments. The paper states steps and table names differ from training, but no leakage analysis is provided.
ad hoc to paper The manually crafted prompt templates are equally well suited to all six prompted LLMs
Section 4.2 states one template per task is populated for all non-fine-tuned models; no per-model prompt optimization or hyperparameter search is reported, so the comparison assumes prompt fit is not a confounder.

invented entities (1)

FlowSim (Flow Similarity metric) independent evidence
purpose: Quantifies similarity between generated and expected workflow trees for automated evaluation
Validated against human scores (Appendix A, Pearson 0.78) and used as the paper's primary outcome; it is a custom construct rather than an established benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code Workflows." pith.science (2026). https://pith.science/paper/S5CYRGCW

@misc{pith2026250524189,
  author       = {Pith},
  title        = {Pith review of: Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5CYRGCW}},
  note         = {Machine review of arXiv:2505.24189}
}

read the original abstract

Large Language Models (LLMs) such as GPT-4o can handle a wide range of complex tasks with the right prompt. As per token costs are reduced, the advantages of fine-tuning Small Language Models (SLMs) for real-world applications -- faster inference, lower costs -- may no longer be clear. In this work, we present evidence that, for domain-specific tasks that require structured outputs, SLMs still have a quality advantage. We compare fine-tuning an SLM against prompting LLMs on the task of generating low-code workflows in JSON form. We observe that while a good prompt can yield reasonable results, fine-tuning improves quality by 10% on average. We also perform systematic error analysis to reveal model limitations.

Figures

Figures reproduced from arXiv: 2505.24189 by the authors.

**Figure 2.** Sample JSON representation of the trigger and first [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** System architecture with UI, AI, and data layers. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Flow similarity results obtained on the TEST and OOD sets for Outline and Outline with inputs. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Percentage of examples with structure errors on the [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Workflow in Figure 1 represented as a tree. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: FlowSim results obtained on the TEST and OOD sets for Outline and Outline with inputs, after structure validation is applied [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

[1]

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. https: //www.anthropic.com/news/claude-3-opus-sonnet-haiku. Model card

work page 2024
[2]

Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured out- puts via Retrieval-Augmented Generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar (Eds.). Asso...

doi:10.18653/v1/2024.naacl-industry.19 2024
[3]

Nastaran Bassamzadeh and Chhaya Methani. 2024. A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation. arXiv preprint arXiv:2407.02742 (2024)

arXiv 2024
[4]

Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. 2020. Tide: A general toolbox for identifying object detection errors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 558–573

work page 2020
[5]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020
[6]

Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024. Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. Advances in Neural Information Processing Systems 37 (2024), 129672–129695

work page 2024
[7]

Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. arXiv preprint arXiv:2407.06153 (2024)

arXiv 2024
[8]

Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models. arXiv preprint arXiv:2411.05451 (2024)

arXiv 2024

Show all 37 references

[9]

Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Tn. 2024. Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?. In Proceedings of the 2024 Conference of the North American Chapter of ...

2024
[10]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, et al . 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. CoRR (2023)

2023
[11]

Gabrielle Gauthier-melancon, Orlando Marquez Ayala, Lindsay Brin, Chris Tyler, Frederic Branchaud-charron, Joseph Marinier, Karine Grande, and Di Le. 2022. Azimuth: Systematic Error Analysis for Text Classification. In Proceedings of the 2022 Conference on Empirical Methods in...

2022 doi
[12]

Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al
[13]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

2024 arXiv
[14]

Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 (2020)

2020 arXiv
[15]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

2024 arXiv
[16]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

2020
[17]

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Ko- cetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al
[18]

Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. 2024. Autoflow: Automated workflow generation for large language model agents. arXiv preprint arXiv:2407.12821 (2024)

2024 arXiv
[19]

Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

2023 arXiv
[20]

Mistral. 2024. Mistral Nemo. https://mistral.ai/news/mistral-nemo Accessed: 2024

2024
[21]

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 (2018)

2018 arXiv
[22]

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071 (2023)

2023 arXiv
[23]

OpenAI. 2025. OpenAI o3-mini System Card. https://cdn.openai.com/o3-mini- system-card-feb10.pdf. Accessed: March 19, 2025

2025
[24]

Fouad Trad and Ali Chehab. 2024. Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models. Machine Learning and Knowledge Extraction 6, 1 (2024), 367–384. doi:10.3390/make6010018

2024 doi
[25]

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al
[26]

David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error Analysis of Statistical Machine Translation Output. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC‘06) , Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, ...

2006
[27]

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

2021 arXiv
[28]

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al
[29]

Michael Wornow, Avanika Narayan, Krista Opsahl-Ong, Quinn McIntyre, Nigam Shah, and Christopher Re. 2024. Automating the enterprise with foundation models. Proceedings of the VLDB Endowment 17, 11 (2024), 2805–2812

2024
[30]

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann
[31]

Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso. 2023. FlowMind: automatic workflow genera- tion with LLMs. In Proceedings of the Fourth ACM International Conference on AI in Finance. 73–81

2023
[32]

Noam Wies, Yoav Levine, and Amnon Shashua. 2023. Sub-Task Decomposi- tion Enables Learning in Sequence to Sequence Tasks. In The Eleventh Interna- tional Conference on Learning Representations. https://openreview.net/forum? id=BrJATVZDWEH

2023
[35]

arXiv preprint arXiv:2303.17564 (2023)

Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023)

2023 arXiv
[37]

Kaizhong Zhang and Dennis Shasha. 1989. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems. SIAM J. Comput. 18, 6 (1989), 1245–1262. doi:10.1137/0218082 arXiv:https://doi.org/10.1137/0218082 A Correlation of Flow Similarity with Human Evaluation...

1989 doi
[2021]

arXiv preprint arXiv:2110.08207 (2021)

Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 (2021)

2021 arXiv
[2023]

arXiv preprint arXiv:2312.11805 (2023)

Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

2023 arXiv
[2024]

Nejm Ai 1, 3 (2024), AIoa2300138

Towards generalist biomedical AI. Nejm Ai 1, 3 (2024), AIoa2300138

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. https: //www.anthropic.com/news/claude-3-opus-sonnet-haiku. Model card

work page 2024

[2] [2]

Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured out- puts via Retrieval-Augmented Generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar (Eds.). Asso...

doi:10.18653/v1/2024.naacl-industry.19 2024

[3] [3]

Nastaran Bassamzadeh and Chhaya Methani. 2024. A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation. arXiv preprint arXiv:2407.02742 (2024)

arXiv 2024

[4] [4]

Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. 2020. Tide: A general toolbox for identifying object detection errors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 558–573

work page 2020

[5] [5]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020

[6] [6]

Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024. Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. Advances in Neural Information Processing Systems 37 (2024), 129672–129695

work page 2024

[7] [7]

Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. arXiv preprint arXiv:2407.06153 (2024)

arXiv 2024

[8] [8]

Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models. arXiv preprint arXiv:2411.05451 (2024)

arXiv 2024