Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read QA-TOOLBOX adds 200,841 grounded QA pairs for factory task guidance and argues that an LLM-as-a-judge can grade answers experts trust.

desk verdict A useful first-cut benchmark for manufacturing task-guidance QA whose central validity rests on unvalidated LLM-generated specs and narrations; the expert validation of the judge is the strongest piece. read the letter →

arxiv 2412.02638 v1 pith:WFEKEPZQ submitted 2024-12-03 cs.CL cs.AI

Ramesh Manuvinakurike , Elizabeth Watkins , Celal Savur , Anthony Rhodes , Sovan Biswas , Gesem Gudino Mejia , Richard Beckwith , Saurav Sahay
show 2 more authors
Giuseppe Raffa Lama Nachman
This is my paper · ORCID
classification cs.CLcs.AI
keywords manufacturingtaskguidancequestionansweringLLM-as-a-judgedataaugmentationAssembly101proceduralinstructionsreference-freeevaluationLLMbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a benchmark built by having LLMs fill in missing specs, narrations, and questions for an existing assembly video dataset can stand in for real manufacturing task guidance, and that an LLM-as-a-judge can grade answers without reference responses. If correct, the field gains a reusable, IP-safe testbed for comparing models and an automated evaluator that experts prefer over crowd-worker ratings. The claim rests on QA-TOOLBOX, a dataset of 200,841 question-answer pairs grounded in Assembly101 and seeded by questions real technicians said they would ask. The paper also reports a blind expert study in which experts significantly preferred GPT-4o-as-judge ratings over crowd-worker ratings (Student's t-test, n=150, p=0.04; Cohen's kappa = 0.61).

What carries the argument

The central machinery is a three-stage augmentation-and-evaluation pipeline. First, internal manufacturing specs, narrations, and technician questions are anonymized by replacing semantic entities with placeholders, and Mixtral-8x7b is prompted to generate Assembly101-specific specs, narrations, and questions; this step is what makes the video dataset usable for QA. Second, baseline LLMs (Phi-3, Gemma, Mistral, Llama-3, Flan-T5) answer questions given the spec and past narration, with no vision input. Third, LLM-as-a-judge scores correctness, conciseness, completeness, and groundedness on a 0–10 scale without reference answers, and the paper selects GPT-4o as the preferred judge based on expert preference.

What would settle it

Have manufacturing experts independently rate a sample of the generated Assembly101 specs and narrations for fidelity to real factory documents and check whether the 200,841 questions are answerable from their paired specs; if experts find systematic mismatches or unanswerable questions, the benchmark's model rankings would reflect generation quality rather than task-guidance ability.

Watch

Extended reading notes

Core claim

QA-TOOLBOX is, to the paper's knowledge, the first manufacturing process task-guidance QA dataset that couples procedure specifications, technician-style narrations, video demonstrations, and grounded question-answer pairs. The questions fall into process-execution, referent, and task-guidance categories, with the specs and narrations generated by Mixtral-8x7b from anonymized internal manufacturing data and Assembly101 annotations. The second claim is that reference-free evaluation works: experts preferred GPT-4o-as-judge ratings over crowd-worker ratings when scoring Llama-3-8b-Instruct responses, so an LLM-as-a-judge can substitute for human crowds in this setting. Among the tested open-source models, Phi-3-medium-128k received the highest judge scores across categories.

Load-bearing premise

The benchmark assumes that the Mixtral-written specs, narrations, and questions faithfully represent real manufacturing task guidance; the paper itself notes that no quantitative study validated the quality of that generated content, so if the synthetic documents are unrepresentative the evaluation measures models on invented procedures rather than on what technicians actually face.

Editorial extensions

If this is right

  • QA-TOOLBOX gives the community a reusable testbed for comparing open-source and proprietary models on manufacturing task guidance.
  • Reference-free LLM judging, if trusted, removes the need for crowd-worker ratings in this domain, cutting evaluation cost and latency.
  • The augmentation pipeline can be applied to other video-procedure datasets that lack textual specs or narrations.
  • Grounding in the spec is scored separately, so the benchmark can expose cases where a model is correct but not grounded, or grounded but incomplete.
  • The best result at 14 billion parameters suggests open-source models under 15B parameters can carry task guidance, which matters for on-premise deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the judge-validity result transfers, the same blind expert-preference protocol could be reused to audit LLM judges in other safety-relevant procedural domains such as aviation maintenance or clinical checklists.
  • Because the generated specs and narrations were not quantitatively validated, benchmark scores likely measure performance against synthetic reference documents; how much that deviates from real technician-facing documents remains an open empirical question the paper leaves for future work.
  • The question taxonomy could be used as diagnostic axes to see which model families fail on groundedness versus completeness, guiding targeted fine-tuning or retrieval augmentation.
  • A direct next experiment would compare LLM-generated specs and narrations against human-written versions in a blind style-and-accuracy test with technicians; the benchmark's validity depends on that comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces QA-TOOLBOX, a dataset of 200,841 question-answer pairs for manufacturing task-guidance QA, built on the Assembly101 video dataset. The authors use Mixtral-8x7b to generate procedure specs, narrations, and questions from anonymized internal manufacturing data, then benchmark several open-source LLMs (Phi-3, Llama-3, Gemma, Mistral, Flan-T5) in a reference-free setup using LLM-as-a-judge with GPT-4o, Nemotron, and Mixtral. They report that GPT-4o ratings are preferred over crowd-worker ratings by domain experts in a blind study (Student's t-test, n=150, p=0.04), and that Phi-3-medium-128k performs best across correctness, conciseness, completeness, and groundedness.

Significance. If validated, the dataset and evaluation pipeline would provide a reusable testbed for a relatively underserved domain, with a participatory design (Section 3.2.2) and a concrete anonymization procedure. The expert validation of one judge (GPT-4o) against crowd ratings is a useful step toward reference-free evaluation. However, the benchmark's validity rests on the quality and representativeness of LLM-generated specs and narrations, which are never quantitatively verified; this makes the central claim currently weaker than the presentation suggests.

major comments (4)
  1. [Section 3.3.2, Section A.2] The generated specs and narrations are load-bearing: they are the only grounding for baseline answers (Section 4) and the explicit reference for judge prompts (Section A.2, 'Use spec as a reference'). The paper admits 'no quantitative studies were conducted' on their quality. Without human verification that these documents are faithful to real manufacturing procedures, the benchmark and the judge scores measure fidelity to synthetic content, not to true task-guidance correctness. The citations [50,17,30] for LLM augmentation quality do not validate this specific dataset. This is a central assumption that must be tested—e.g., by having manufacturing experts rate or correct a sample of the generated specs and narrations.
  2. [Section 6, R2; Table 4] The expert validation study covers only Llama-3-8b-Instruct responses and only compares GPT-4o versus crowd ratings. It does not validate the judge scores for the other five baseline models, nor the model rankings in Table 4. The conclusion that 'GPT-4o as a judge' is the best choice for the whole benchmark extrapolates from a single response distribution. The manuscript should either provide validation across multiple models or explicitly scope the expert result to a proof-of-concept.
  3. [Table 4, Section 6 R3] Table 4 reports means on a 0-10 scale without sample sizes, standard deviations, or pairwise significance tests. The claim that Phi-3-medium-128k is 'consistently better' appears to be based on eyeballing the table; no statistical support is given. Given the judge-score variability implied by Figure 4 (different judges give different absolute scores), the ranking claim is not substantiated without error bars and an appropriate multiple-comparison procedure.
  4. [Section 5, Section 6 R2] The reference-free evaluation relies on LLM judge prompts that all instruct 'Use spec as a reference,' but the spec itself is generated without ground truth. This creates a circular evaluation: a hallucinated step in the generated spec will be rewarded as correct in both the 'correctness' and 'groundedness' judges. The manuscript should discuss this circularity explicitly and ideally include a small set of human-verified 'golden' answers against which judge scores can be calibrated.
minor comments (6)
  1. [Section 2] Typo: 'scenarions' should be 'scenarios'.
  2. [Section 6 R2] Typo: 'generted' should be 'generated'.
  3. [Section 3.3.2] The qualitative analysis of the generated narrations and specs is not described with enough detail to be reproducible; specify how many samples were inspected and by whom.
  4. [Section 3.3] The prompt templates in Section A.1 are shown in equation-like form, but the 'n samples' placeholder and the number of in-context examples are not specified. Provide the actual prompts used for reproducibility.
  5. [Abstract and Introduction] The paper claims the dataset will be open-sourced, but no link or repository is provided. Please include a URL or state the intended release venue.
  6. [Section 4] The unimodal language-only setup is a reasonable scoping decision, but the phrase 'we release the evaluations from unimodal scenario' is unclear; please rephrase to 'we evaluate the unimodal setting'.

Circularity Check

1 steps flagged · score 4.0 of 10

Evaluation loop is self-referential: the LLM-generated spec is both the grounding for baseline answers and the judge's reference for correctness, so model scores measure fidelity to synthetic content rather than independently verified manufacturing guidance.

  1. self definitional [Section 3.3.2, Section 4, and Section A.2 (correctness_prompt / groundedness_prompt)]
    "'While no quantitative studies were conducted, an avenue for future work is to study the quality of narrations and spec documents generated.' ... 'Deduct points for not being accurate or including any irrelevant phrases or sentences. Use spec as a reference.' ... 'The spec for the process: Assembly101spec. Now answer the question: Question.'"

    The judge's definition of correctness and groundedness is explicitly anchored to the spec ('Use spec as a reference'), and that spec is Assembly101spec, generated by Mixtral-8x7b with no quantitative validation. Baseline models are given the same Assembly101spec as the only task context. Therefore a model score is, by construction, a measure of consistency with the LLM-generated spec, not with any externally verified manufacturing procedure. If Mixtral inserted a hallucinated tool, ordering, or rationale, the judge would reward an answer that repeats it and penalize an answer that is true to the actual Assembly101 video but contradicts the synthetic spec.

full rationale

The paper contains no load-bearing self-citation chain: the cited prior work on LLM augmentation quality and LLM-as-a-judge is external, and the expert study provides an independent human check on judge ratings. However, the central evaluation loop is self-referential in a specific, quotable way. The reference documents used to define correct and grounded answers (Assembly101spec and Assembly101narration) are produced by Mixtral-8x7b, and the paper explicitly states that no quantitative studies were conducted on their quality. Because the same generated spec is fed to the baseline models as context and is then used by the LLM judge as the reference for correctness, completeness, conciseness, and groundedness, the reported model rankings are by construction scores of adherence to LLM-generated specifications rather than to verified manufacturing procedures. This is a partial circularity rather than a full one: the expert preference for GPT-4o ratings is an external signal about judge quality, and the dataset construction pipeline is transparent. The weakness is concentrated in the unvalidated synthetic reference, which is both an input to the system and the standard against which outputs are measured. If the generated spec were externally verified or released for independent checking, the loop would be broken; the paper does not provide such validation or release.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

There are no fitted numerical parameters. The central reliance is on unvalidated LLM-generated reference content, the accuracy of Assembly101 annotations, and the representativeness of a small internal sample of technician questions.

assumptions (4)
  • domain assumption Assembly101 action labels and video annotations are accurate enough to serve as the structural basis for generating specs and narrations.
    The augmentation pipeline in Section 3.3 prompts Mixtral with Assembly101 action labels; if these labels are noisy, all generated specs and narrations inherit that noise.
  • ad hoc to paper LLM-generated specs and narrations are representative of real manufacturing spec documents and technician narrations.
    Section 3.3.2 states 'no quantitative studies were conducted' on the quality of the generated narrations and specs, yet the benchmark's correctness and groundedness judgments depend on this assumption.
  • domain assumption LLM-as-a-judge scores, specifically GPT-4o ratings, reflect the quality that domain experts would assign.
    The paper supports this with a small expert preference study (n=150, p=0.04, kappa=0.61), but the study covered only Llama-3-8b responses and did not validate the generated reference documents.
  • domain assumption The 20 internal technician recordings and participatory questions are representative of manufacturing task guidance needs.
    A small sample from one facility was used as few-shot exemplars to generate all 200,000+ questions; representativeness across manufacturing settings is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing." pith.science (2026). https://pith.science/paper/WFEKEPZQ

@misc{pith2026241202638,
  author       = {Pith},
  title        = {Pith review of: QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFEKEPZQ}},
  note         = {Machine review of arXiv:2412.02638}
}
read the original abstract

In this work we explore utilizing LLMs for data augmentation for manufacturing task guidance system. The dataset consists of representative samples of interactions with technicians working in an advanced manufacturing setting. The purpose of this work to explore the task, data augmentation for the supported tasks and evaluating the performance of the existing LLMs. We observe that that task is complex requiring understanding from procedure specification documents, actions and objects sequenced temporally. The dataset consists of 200,000+ question/answer pairs that refer to the spec document and are grounded in narrations and/or video demonstrations. We compared the performance of several popular open-sourced LLMs by developing a baseline using each LLM and then compared the responses in a reference-free setting using LLM-as-a-judge and compared the ratings with crowd-workers whilst validating the ratings with experts.

Figures

Figures reproduced from arXiv: 2412.02638 by the authors.

Figure 1
Figure 1. Example of task guidance QA ex￾changes. Manufacturing is a human-centric domain where tech￾nicians must follow complex instructions provided by a spec (task specification manual). A spec typi￾cally consists of natural language description of sev￾eral processes that needs to be accomplished towards reaching an end goal. Manufacturing industry faces several challenges. Within a spec, a typical process can contain anyw… view at source ↗
Figure 2
Figure 2. Workflow for creating QA-TOOLBOX dataset, baseline, and evaluations. Open-sourced [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure shows an example from the QA-TOOLBOX. Demonstrations and action labels is [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Radar charts of LLM-as-a-judge for the six models. i) Left chart shows different models’ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Shows the three different augmentation approaches and format of the data [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios

    cs.CV 2025-05 conditional novelty 7.0 of 10

    IndustryEQA offers 1,344 video-based question-answer pairs across six categories, with a focus on equipment and human safety, plus evaluations of several vision-language models.

Reference graph

Works this paper leans on

56 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    A dataset and benchmark for hospital course summarization with adapted large language models

    Asad Aali, Dave Van Veen, Yamin Ishraq Arefeen, Jason Hom, Christian Bluethgen, Ed- uardo Pontes Reis, Sergios Gatidis, Namuun Clifford, Joseph Daws, Arash S Tehrani, et al. A benchmark of domain-adapted large language models for generating brief hospital course summaries. arXiv preprint arXiv:2403.05720, 2024

    work page Pith review arXiv 2024

  2. [2]

    Phi-3 technical re- port: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Longhealth: A question answering benchmark with long clinical documents

    Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXiv:2401.14490, 2024

    arXiv 2024

  4. [4]

    Nemotron-4 340b technical report

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024

    arXiv 2024

  5. [5]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md

    2024

  6. [6]

    Learning from narrated instruction videos

    Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Learning from narrated instruction videos. IEEE transactions on pattern analysis and machine intelligence, 40(9):2194–2208, 2017

    work page 2017

  7. [7]

    More than reading comprehension: A survey on datasets and metrics of textual question answering

    Yang Bai and Daisy Zhe Wang. More than reading comprehension: A survey on datasets and metrics of textual question answering. arXiv preprint arXiv:2109.12264, 2021

    arXiv 2021

  8. [8]

    Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark

    Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark. arXiv preprint arXiv:2305.14938, 2023

    arXiv 2023

Show all 56 references
  1. [9]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent...

    2022 arXiv

  2. [10]

    Clue: A clinical language understanding evaluation for llms

    Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Kora¸ s, Constantin Marc Seibold, Kaleb E Smith, and Jens Kleesiek. Clue: A clinical language understanding evaluation for llms. arXiv preprint arXiv:2404.04067, 2024

    2024

  3. [11]

    Wizard of oz studies: why and how

    Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. Wizard of oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces, pages 193–200, 1993

    1993

  4. [12]

    Laiw: A chinese legal large language models benchmark (a technical report)

    Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. Laiw: A chinese legal large language models benchmark (a technical report). arXiv preprint arXiv:2310.05620, 2023

    arXiv

  5. [13]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on comput...

    2018

  6. [14]

    Developing Virtual Partners to Assist Military Personnel

    DARPA. Developing Virtual Partners to Assist Military Personnel. https://www.darpa. mil/news-events/2021-03-03 , 2021. [Online; accessed 24-June-2024]

    2021

  7. [15]

    On the limitations of reference-free evaluations of generated text

    Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference-free evaluations of generated text. arXiv preprint arXiv:2210.12563, 2022. 9

    2022 arXiv

  8. [16]

    Simsensei kiosk: A virtual human interviewer for healthcare decision support

    David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. Simsensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference...

    2014

  9. [17]

    Data augmentation using llms: Data perspectives, learning paradigms and challenges

    Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data augmentation using llms: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02990, 2024

    2024 arXiv

  10. [18]

    Lawbench: Benchmarking legal knowledge of large language models

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289, 2023

    2023 arXiv

  11. [19]

    doc2dial: A goal-oriented document-grounded dialogue dataset

    Song Feng, Hui Wan, Chulaka Gunasekara, Siva Sankalp Patel, Sachindra Joshi, and Luis A Lastras. doc2dial: A goal-oriented document-grounded dialogue dataset. arXiv preprint arXiv:2011.06623, 2020

    2011 arXiv

  12. [20]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural...

    2024

  13. [21]

    Cuad: An expert-annotated nlp dataset for legal contract review

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268, 2021

    2021 arXiv

  14. [22]

    Construction of a japanese financial benchmark for large language models

    Masanori Hirano. Construction of a japanese financial benchmark for large language models. arXiv preprint arXiv:2403.15062, 2024

    2024 arXiv

  15. [23]

    A taxonomy and review of generalization research in nlp

    Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pi- mentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, et al. A taxonomy and review of generalization research in nlp. Nature Machine Intelligence, 5(10): 1161–...

    2023

  16. [24]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  17. [25]

    Wikibench: Community-driven data curation for ai evaluation on wikipedia

    Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. Wikibench: Community-driven data curation for ai evaluation on wikipedia. InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1...

    2024

  18. [26]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  19. [27]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Pr...

    2024

  20. [28]

    Materiality, sociomateriality, and socio-technical systems: What do these terms mean? how are they different? do we need them

    Paul M Leonardi. Materiality, sociomateriality, and socio-technical systems: What do these terms mean? how are they different? do we need them. Materiality and organizing: Social interaction in a technological world, 25(10):1093, 2012

    2012

  21. [29]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems...

    2020

  22. [30]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024. 10

    2024 arXiv

  23. [31]

    Alphafin: Benchmarking financial analysis with retrieval-augmented stock-chain framework

    Xiang Li, Zhenyu Li, Chen Shi, Yong Xu, Qing Du, Mingkui Tan, Jun Huang, and Wei Lin. Alphafin: Benchmarking financial analysis with retrieval-augmented stock-chain framework. arXiv preprint arXiv:2403.12582, 2024

    2024 arXiv

  24. [32]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  25. [33]

    Large language models in healthcare: A comprehensive benchmark

    Fenglin Liu, Hongjian Zhou, Yining Hua, Omid Rohanian, Lei Clifton, and David Clifton. Large language models in healthcare: A comprehensive benchmark. medRxiv, pages 2024–04, 2024

    2024

  26. [34]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  27. [35]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  28. [36]

    How can large language models help humans in design and manufacturing? part 2: Synthesizing an end-to-end llm-enabled design and manufacturing workflow

    Liane Makatura, Michael Foshey, Bohan Wang, Felix Hähnlein, Pingchuan Ma, Bolei Deng, Megan Tjandrasuwita, Andrew Spielberg, Crystal Owens, Peter Yichen Chen, et al. How can large language models help humans in design and manufacturing? part 2: Synthesizing an end-to-end llm-e...

    2024

  29. [37]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/ . Ac- cessed: 2024-03-18

    2024

  30. [38]

    Deloitte and the manufacturing institute: Big gains in perceptions of us manufacturing as innovative, critical and high- tech, 2022

    PR Newswire. Deloitte and the manufacturing institute: Big gains in perceptions of us manufacturing as innovative, critical and high- tech, 2022. URL https://www.prnewswire.com/news-releases/ deloitte-and-the-manufacturing-institute-big-gains-in-perceptions-of-us-manufacturing...

    2022

  31. [39]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  32. [40]

    A simple recipe for multilingual grammatical error correction

    Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. A simple recipe for multilingual grammatical error correction. arXiv preprint arXiv:2106.03830, 2021

    2021 arXiv

  33. [41]

    Targeting the benchmark: On methodology in current natural language processing research

    David Schlangen. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Vo...

    2021

  34. [42]

    Sener, D

    F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. CVPR, 2022

    2022

  35. [43]

    Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors

    Hong Shen, Alicia DeV os, Motahhare Eslami, and Kenneth Holstein. Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–29, 2021

    2021

  36. [44]

    A roadmap to pluralistic alignment

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070, 2024

    2024 arXiv

  37. [45]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1207–1216, 2019

    2019

  38. [46]

    Gemma Team. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301. URL https://www.kaggle. com/m/3301. 11 Correctness Conciseness Completeness GroundednessGPT4o-as-JudgePE (Corr) Ref (Corr) TG (Corr)PE (Conc) Ref (Conc) TG (Conc)PE (Comp) Ref (Comp) TG (Comp)PE (Grn) Ref (Grn) TG (Grn)Phi-...

    2024 doi

  39. [47]

    Fingpt: Instruction tuning benchmark for open-source large language models in financial datasets

    Neng Wang, Hongyang Yang, and Christina Dan Wang. Fingpt: Instruction tuning benchmark for open-source large language models in financial datasets. arXiv preprint arXiv:2310.04793, 2023

    2023 arXiv

  40. [48]

    Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. InProceedings of the IEEE/C...

    2023

  41. [49]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  42. [50]

    Llms as workers in human- computational algorithms? replicating crowdsourcing pipelines with llms

    Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, et al. Llms as workers in human- computational algorithms? replicating crowdsourcing pipelines with llms. arXiv preprint arXiv:2307.10168, 2023

    2023 arXiv

  43. [51]

    Pixiu: A large language model, instruction data and evaluation benchmark for finance

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443, 2023

    2023 arXiv

  44. [52]

    The finben: An holistic financial benchmark for large language models

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. The finben: An holistic financial benchmark for large language models. arXiv preprint arXiv:2402.12659, 2024

    2024 arXiv

  45. [53]

    Superclue-fin: Graded fine-grained analysis of chinese llms on diverse financial tasks and applications

    Liang Xu, Lei Zhu, Yaotong Wu, and Hang Xue. Superclue-fin: Graded fine-grained analysis of chinese llms on diverse financial tasks and applications. arXiv preprint arXiv:2404.19063, 2024

    2024 arXiv

  46. [54]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  47. [55]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018

    2018

  48. [56]

    "" conciseness_prompt =

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. A Appendix A...

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.