Pith. sign in

REVIEW 4 major objections 6 minor 72 references

Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An eight-agent system claims that for complex tasks, asking the right questions before generation beats repairing answers through multi-turn chat; in a four-person pilot every participant's prompts were more complete, outputs rated higher,

desk verdict Honest, well-scoped pilot of an eight-agent prompt-formulation system; the direction-consistency results are encouraging, but the completeness metric is circular and the 'asking questions' mechanism lacks a proper control. read the letter →

arxiv 2608.01366 v2 pith:2VUKQKQI submitted 2026-08-02 cs.MA cs.AI

B. Sankar , Pawni Yadav , Srinidhi Ranjini Girish , Amogh A. S This is my paper
classification cs.MAcs.AI
keywords PromptformulationMulti-agentsystemsQuestion-askingCognitiveloadHuman-AIinteractionengineeringLLMevaluationEEG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PAWNI is an eight-agent conversational system whose design claim is that for complex tasks, the limiting factor in LLM output quality is the prompt, not the model. Its authors observed that users default to short prompts and repair them through multi-turn conversation, where models lose accuracy and errors compound. So PAWNI front-loads the repair: a questioner agent interviews the user until the request covers an 18-element checklist of prompt ingredients, then a generation-evaluation-polish loop verifies and refines the prompt before the user ever sends it. In an exploratory within-subjects study of four people on four different complex tasks, prompts produced this way scored 42% to 91% structural completeness, were rated higher on every output-quality dimension, and reached satisfactory answers in one LLM turn versus one to twelve unaided. The paper is careful that four participants cannot establish the effect, and offers the consistency of direction as its primary evidence.

What carries the argument

The load-bearing object is a three-tier, 18-element prompt framework (Essential, Enhancement, Elevation) that doubles as the system's target: the Architect agent asks questions until the Essential elements are covered, the Scribe enforces all 18, and the Judge scores against the same rubric. Around this sits the reverse-reasoning loop: the Forge drafts an ideal answer from the structured summary before the prompt is final, the Judge evaluates it with multiple models, the Mirror turns the criticism into edits, and the Scribe produces the polished prompt, iterating until a quality threshold (default 8/10) or three loops. A self-evolving pattern file stores weighted domain knowledge that the qu

What would settle it

Take the same four tasks with a larger, counterbalanced sample and compare three arms: PAWNI, a fixed 18-question form with no dynamic questioner or pattern knowledge, and a naive condition with a time-matched writing task. If the fixed form matches PAWNI on completeness and output ratings, the self-evolving knowledge base and question prioritisation are not doing the work; if the naive-but-similar-length condition matches PAWNI, then length or effort, not structural completeness, is the active variable.

Watch

Extended reading notes

Core claim

The paper's central claim is that output quality in LLMs is gated by prompt completeness, not by the model's generation ability: ask the right questions up front and a single well-formed prompt outperforms repeated corrective turns. To test that, the authors built PAWNI, a pipeline of eight specialised agents that turns an unstructured query into a structured prompt through a guided interview. In an exploratory within-subjects study (four participants, four different complex tasks, each on a different commercial LLM), PAWNI prompts scored 91.3% structural completeness versus 42.5% for unaided prompting, output-quality ratings rose on every dimension (group mean 4.04 to 6.50 on a 1–7 scale),

Load-bearing premise

The load-bearing premise is that the 18-element checklist really is what makes a prompt good; the system trains itself to that checklist, judges itself against it, and the paper's headline evidence is the gap between checklist scores, so if the checklist is the wrong or incomplete measure, the central explanation falls away even if the raw outcome numbers stand.

Editorial extensions

If this is right

  • Correction cycles move before generation: clarification happens in the Architect interview, not as post-hoc repairs to model output.
  • Single-shot execution becomes viable: all four participants reached a satisfactory deliverable in one LLM turn, versus one to twelve unaided.
  • Output ratings improve across all quality dimensions and across four different commercial models, implying the lever is the specification, not any one model's behaviour.
  • Users report less subjective workload despite a longer structured interview, because the extra effort is productive rather than corrective.
  • Prompt length and completeness are separable: one naive user wrote more than 1,300 words across twelve turns and still scored below 50 percent completeness, so the benefit is not simply verbosity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If completeness is the active ingredient, the cheapest falsifiable version of PAWNI is a static 18-question form; a head-to-head would tell whether dynamic, knowledge-weighted questioning earns its complexity.
  • The paper's line between eliciting what the user did not say and reformulating what they did say predicts that vendor-side 'improve my prompt' tools will hit a ceiling on novel complex tasks, and that a combined elicit-then-optimise system should dominate either alone.
  • The proposed tutor-versus-prosthesis test (measuring unaided prompt completeness before, after, and after a washout of PAWNI use) would tell whether repeated structured questioning changes how users specify tasks, not just what they paste in the moment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents PAWNI, an eight-agent conversational pipeline that transforms an unstructured user query into a structured prompt via Sentinel classification, Scout knowledge retrieval, Architect-led Q&A, Forge–Judge–Mirror–Scribe self-evaluation, and optional POML conversion. The authors introduce a three-tier, 18-element prompt framework and report an exploratory within-subjects pilot (N=4) with four diverse tasks, combining 32-channel EEG, NASA-TLX, behavioral counts, and blind prompt-completeness ratings. The headline results are that PAWNI-generated prompts score higher on structural completeness (42.5% to 91.3%), LLM outputs are rated higher on all quality dimensions, NASA-TLX workload drops (39.6 to 21.7), and every participant reaches a satisfactory output in a single LLM turn versus 1–12 turns unaided. The authors are unusually explicit that N=4 precludes inference, that effect sizes are unstable/estimator-dependent, that ordering was fixed, and that self-report measures carry demand-characteristic risk; they frame direction consistency as the primary evidence and describe the study as a formative evaluation. The central thesis is that 'asking the right questions'—front-loaded intent clarification—rather than response generation is the main lever for human-AI collaboration.

Significance. If the causal claim could be supported, the paper would make a useful contribution: it shifts attention from response-side optimization to prompt formulation, and it demonstrates a concrete, integrated architecture that combines question-asking, domain-knowledge retrieval, and self-evaluation. The system is described in enough detail to be reimplemented, and the measurement protocol (EEG phase segmentation, TLX, criteria checklists, interaction coding) is a reusable template. The paper is also honest in ways that are rare at this sample size: effect sizes are reported with Hedges correction, bootstrap intervals, and a clear statement that Wilcoxon p-values cannot reach significance; the SUS result is explicitly labeled uninterpretable; P2's task drift is disclosed; and §7.8 concedes the absence of ablation evidence. These strengths do not, however, overcome the core circularity and missing-control problems: the completeness metric is simultaneously PAWNI's optimization target, the Judge's rubric, and the mechanism measure, and no condition separates question-asking from mechanical prompt elaboration. The paper is best read as a system/feasibility report; as evidence for the 'asking

major comments (4)
  1. [§6.3.3, §7.1; cf. §4.3.5, §4.3.7, §5.5.4] The structural-completeness measure is circular as mechanism evidence. The Scribe is required to 'ensure completeness of all 18 elements' (§4.3.7), the Judge evaluates 'the presence of all 18 prompt elements' (§4.3.5), and the blind raters use a 10-item checklist derived from the same framework (§5.5.4). The 42.5%→91.3% gap in Figure 18 is therefore guaranteed by PAWNI's construction. It cannot independently support the claim in §7.1 that 'the mediating mechanism is prompt structural completeness.' Either the framework must be externally validated (e.g., completeness scored against a rubric not known to the system, or against an independent outcome), or the mechanistic claim must be removed from the abstract and §7.1. As written, the central causal chain from 'asking questions' to 'better answers' is not established.
  2. [§7.6, §7.8; §6.3.3] The paper argues that automatic rewriting cannot supply user-only constraints and therefore Q&A is necessary, but no condition isolates the Q&A component. PAWNI prompts are ~4× longer than cumulative naive prompts, contain Scout-retrieved domain knowledge, and are formatted with labelled sections; any of these factors, alone or combined, could explain the output-quality, TLX, and turn-count differences. §7.8 explicitly concedes there is no ablation or counterfactual for any agent. The 'expansion control' is missing: a condition that fixes the prompt-expansion/formatting function while removing the Architect's adaptive Q&A (e.g., a static comprehensive form, or auto-expansion without user elicitation) is needed before the observed gains can be attributed to 'asking the right questions.' This is load-bearing for the abstract's first sentence and for the paper's stated thesis.
  3. [§5.5.1, §5.3.1, §6.1; Table 4; §7.4] RQ1's primary outcome—output quality—is measured by unblinded participant ratings. Table 4 correctly classifies these as 'Moderate' risk and §5.3.1 lists safeguards, but the output-quality ratings are the central evidence for 'better answers,' and with N=4 and fixed ordering the demand-characteristic risk is substantial: every participant knew the PAWNI condition was the researchers' own system. The problem is compounded by P2's task drift between conditions (§7.4), yet P2's output-quality ratings remain pooled with the others in the 4/4 aggregate. The paper should either re-analyze RQ1 with P2 excluded or provide an independent blind quality rating (human expert or a held-out LLM judge) before claiming consistent output-quality improvement.
  4. [§6.7, §7.8; §8] The component-level process logs do not provide the causal evidence the narrative implies. That the Judge scored the Forge draft at 3.97/10 and the Scribe output at 9.27/10 shows the pipeline is not a rubber stamp on its own intermediate products, but it does not show that the Architect's front-loaded questions—rather than the Scout's retrieved knowledge or the Scribe's formatting—drove the output-quality gains. The paper's own §7.8 acknowledges this, and §8 defers ablation work. Given this, the abstract's claim that the results 'support the hypothesis that optimising prompt formulation front-end is a critical lever' should be softened to a feasibility/measurement-protocol claim until the missing control is run.
minor comments (6)
  1. [Abstract; §6.3.3] The abstract says '42% to 91%'; the body reports 42.5% to 91.3%. Use consistent rounding throughout.
  2. [§4.3.5, §5.5.4] The relationship between the 18-element framework and the 10-item checklist is unclear. Are items such as 'Audience' and 'Domain Specifics' aggregates of multiple 18-element entries? State the mapping explicitly.
  3. [§5.7] The footnote marker after 'Wilcoxon p=0.375' is not typeset as a real footnote; the footnote text appears inline in the main flow. Fix the formatting.
  4. [§6.3.1] The sentence 'P2 therefore contributes no within-participant improvement to the turn-count measures, which is why their direction consistency is 3/4' is ambiguous. The direction consistency belongs to the measures, not to P2; rephrase.
  5. [Table 5] The 'Prompt specificity' row has an empty bootstrap CI. Either report the interval or explicitly state why it is not computed (e.g., non-normalized ratio).
  6. [§4.3] The system name is spelled inconsistently: 'Prompt Architecture Wizard using Neural Intelligence' in the abstract vs. 'Prompt Architecture Wizard with Neural Intelligence' in §4.3. Pick one.

Circularity Check

2 steps flagged · score 6.0 of 10

Completeness is both PAWNI's optimization target and its mechanism evidence; the 42%→91% gap reduces by construction, while output-quality ratings remain independent.

  1. self definitional [Section 4.3.7 (Scribe), Section 5.5.4 (Prompt Quality Measures), Section 7.1 (Discussion)]
    "The mediating mechanism is prompt structural completeness. Naive prompts scored a mean of 42% on structural completeness—meaning that over half of the essential prompt elements were absent even after multiple rounds of iterative refinement. PAWNI-generated prompts scored 91%. This gap directly explains why the first LLM response in Condition B was rated 6.5/7 for quality, compared to 3.8/7 in Condition A..."

    The outcome measure used as the mediator is the same construct PAWNI is engineered to maximize. Section 5.5.4 scores structural completeness against a 10-item checklist covering Context, Objective, Constraints, Output Format, Evaluation Criteria, Audience, Scope, Examples, Tone/Style, and Domain Specifics. Section 4.3.7 says the Scribe 'enforces hard constraints: ... ensuring completeness of all 18 elements.' Section 6.3.3 reiterates that PAWNI 'systematically ensures the presence of nearly all 18 prompt elements.' Thus the 42%→91% gap is the system succeeding at its own optimization target; invoking that gap as the 'mediating mechanism' for output quality uses a measure defined by the system's objective as evidence for that objective's causal value. It cannot distinguish front-loaded Q&A

  2. other [Section 4.3.5 (Judge Agent) and Section 6.7 (Component-Level Evidence)]
    "A multi-model judge that assigned a mean of 3.97/10 to its own pipeline’s intermediate output is not behaving as a rubber stamp, which is a necessary—though not sufficient—condition for the quality-assurance loop to be meaningful."

    The Judge's rubric is the same 18-element framework that the Scribe is constrained to satisfy. Section 4.3.5 states the Judge evaluates 'the presence of all 18 prompt elements,' and Section 6.7 reports the Forge draft scoring 3.97/10 while the Scribe output scores 9.03–9.47/10. This rise simply reflects convergence on the system's own internal objective; it is internal consistency checking, not independent validation of prompt quality. The paper hedges it as 'necessary—though not sufficient,' and the step is not the central claim, but it is still self-referential evidence offered in support of the QA loop.

full rationale

The paper's central empirical evidence for the mechanism is prompt structural completeness: naive prompts score a mean of 42% and PAWNI prompts 91% (Section 6.3.3), and Section 7.1 presents this gap as the mediator 'directly explaining' the output-quality advantage. That specific link is circular in the sense defined here: the 10-item completeness checklist operationalizes the same 18-element framework that the Scribe agent is required to enforce (Section 4.3.7) and that the Judge uses as its rubric (Section 4.3.5). A system that is explicitly built to maximize completeness will, if it works as designed, score high on a completeness measure derived from its own target; observing that gap does not test whether 'asking the right questions' rather than mechanical prompt elaboration drives the outcome. The internal Judge scores are also self-referential, though the paper appropriately labels this evidence as merely necessary. However, the central claim does not reduce entirely to the circular measure: output-quality ratings (4/4), NASA-TLX (4/4), turn counts (3/4), and EEG theta/alpha (3/4) are external, directionally consistent observations that give the paper independent content. The paper's own Section 7.8 concedes there is no ablation or counterfactual evidence for any agent, and Section 7.6's argument that rewriting cannot supply user-only constraints is plausible but untested. This is a partial circularity in the mechanism evidence, not a fully tautological derivation, so a score of 6 rather than 8 or 10 is appropriate.

Assumptions & free parameters 7 free parameters · 5 assumptions · 3 invented entities

The central claim rests on several author-chosen constructs: the 18-element framework (both objective and metric), the theta/alpha EEG load index borrowed from the literature, hand-set thresholds (0.85, 0.82, 8/10, 3 iterations), and the four-model judge assumption. These are not fitted to data in the usual sense, but they are chosen by hand and some are load-bearing for the completeness mechanism. The invented entities are software components and a taxonomy, not physical postulates; none has an independent falsifiable handle.

free parameters (7)
  • taxonomy_match_high_threshold = 0.85 cosine similarity
    Cosine similarity above which the Sentinel uses an existing taxonomy entry automatically (Section 4.3.1). Hand-set; affects classification behavior.
  • taxonomy_match_confirm_threshold = 0.70-0.85 cosine similarity
    Range requiring user confirmation for taxonomy match (Section 4.3.1). Hand-set.
  • pattern_file_merge_threshold = 0.82 cosine similarity
    Similarity above which fetched knowledge points are merged and weights incremented (Section 4.4). Controls knowledge-base evolution and convergence.
  • judge_threshold = 8/10 (default)
    Score threshold for the Scribe-Judge-Mirror loop to terminate (Section 4.3.5). Hand-configured; determines whether the loop iterates.
  • eval_loop_max_iterations = 3 (default)
    Maximum number of Scribe-Judge-Mirror iterations (Section 4.3.7). Hand-set.
  • eeg_sliding_window_parameters = 2 s window, 0.5 s step, 4 s smoothing
    EEG timeline parameters (Section 5.5.2). Affect the continuous cognitive-load indices; chosen by authors.
  • iaf_band_definitions = theta 4 Hz to IAF-2 Hz; alpha IAF +/- 2 Hz
    Individual alpha frequency band calibration rule (Section 5.5.2). Follows Klimesch but threshold conventions are author-chosen.
assumptions (5)
  • domain assumption Frontal theta / parietal alpha (theta/alpha) ratio is a valid index of cognitive load
    Adopted from cited literature (Chikhi et al. 2022; Gevins and Smith 2003) and used as the primary EEG load measure in Section 5.5.2; not independently validated in this study.
  • domain assumption The 10-item task criteria checklists capture output quality for each of the four tasks
    Checklists were constructed by the authors for the four tasks (Section 5.2); no reliability or validity evidence is reported.
  • ad hoc to paper The 18-element prompt framework is the correct completeness target
    The framework (Section 3.2) is author-derived from a review and serves simultaneously as the system's design objective and the evaluation rubric; its necessity and sufficiency are assumed.
  • standard math ICA and artefact rejection sufficiently remove EEG movement artefacts
    The preprocessing pipeline (Section 5.5.2) uses ICA via MNE-Python; this is standard practice, but the paper itself notes movement artefacts are only partially mitigated (Section 7.4).
  • domain assumption The four-model Judge aggregation provides meaningful prompt-quality scores
    The Judge combines scores from GPT, Claude, GLM, and Grok (Section 4.3.5); inter-rater reliability of these models is not assessed.
invented entities (3)
  • PAWNI eight-agent pipeline (Sentinel, Scout, Architect, Forge, Judge, Mirror, Scribe, Cipher)
    purpose: Transforms unstructured user queries into structured prompts through sequential agentic processing
    The system under evaluation; no falsifiable handle exists outside the pilot results reported in this paper.
  • Three-tier 18-element prompt framework (Essential, Enhancement, Elevation)
    purpose: Defines the completeness target for prompt generation and the rubric for prompt evaluation
    Author-derived taxonomy with no independent validation that these 18 elements are necessary or sufficient for prompt quality.
  • Self-evolving pattern-file knowledge base
    purpose: Accumulates and weights domain best practices that guide the Architect's question selection
    Internal mechanism; only evidence is the process-log summaries in Section 6.7, which describe behavior but not causal contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution." pith.science (2026). https://pith.science/paper/2VUKQKQI

@misc{pith2026260801366,
  author       = {Pith},
  title        = {Pith review of: Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VUKQKQI}},
  note         = {Machine review of arXiv:2608.01366}
}
read the original abstract

Large language models (LLMs) are integral to complex intellectual tasks, yet output quality remains constrained by user-provided prompts. Iterative multi-turn prompting often leads to context degradation and diminishing cognitive returns. We present PAWNI (Prompt Architecture Wizard using Neural Intelligence), an agentic conversational interface of eight agents that transforms unstructured queries into structured prompts through guided question-and-answer dialogue informed by a self-evolving knowledge base. Rather than optimising the model's response, PAWNI optimises the question itself by front-loading intent clarification. We also propose a three-tier framework of 18 prompt elements across Essential, Enhancement, and Elevation categories. To evaluate system behaviour and validate a measurement protocol, we conducted an exploratory within-subjects study (N=4) across four complex tasks, integrating 32-channel EEG, NASA-TLX workload, and behavioural metrics. Participants produced more structurally complete prompts with PAWNI (42% to 91% of assessed elements), rated LLM outputs higher across all quality dimensions, and reported lower workload (39.6 vs. 21.7 NASA-TLX). Every participant reached satisfactory output in a single turn, compared to 1-12 turns unaided. While effect sizes are unstable due to sample size, direction consistency supports the hypothesis that optimising prompt formulation front-end is a critical lever for human-AI collaboration.

Figures

Figures reproduced from arXiv: 2608.01366 by the authors.

Figure 1
Figure 1. PAWNI pipeline architecture. Eight agents process the user’s raw input sequentially. The Architect agent conducts multi-turn [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. The PAWNI system landing page interface. The page header defines PAWNI as a ’Prompt Architecture Wizard using Neural [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Active PAWNI application interface demonstrating a fully executed optimization pipeline. The top navigation bar confirms [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (28 more)
Figure 4
Figure 4. Figure 4: Participant wearing a detailed multi-electrode EEG cap (EMOTIV FLEX2) is seated at a wooden desk and interacting with [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: 32-channel EEG electrode montage used in the study. Electrode positions follow the 10-20 international system. Colours [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Mean task criteria quality scores (1–5 scale) per participant and group mean. All four participants showed improvement with [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Output quality ratings across six dimensions (1–7 Likert). PAWNI-assisted outputs (orange) were rated higher on all dimensions [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: System Usability Scale comparison. PAWNI (orange) scored in the “Best Imaginable” range ( [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: NASA-TLX workload profile (group average, 0–100). For all demand sub-scales, lower values indicate less workload (better). For [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: NASA-TLX Overall Workload paired dot plot. Every participant showed reduced workload with PAWNI. Group mean decreased [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Continuous cognitive load timelines (𝜃/𝛼 ratio) for all participants. Left column: Condition A (Naive LLM). Right column: Condition B (PAWNI-assisted). Background colours indicate interaction phases (writing, waiting, reading, evaluating for Condition A; answering, wa…
Figure 12
Figure 12. Figure 12: Group-average overlaid cognitive load timeline. Both conditions are mapped to normalised session progress (0–100%). The [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Group-average EEG topographic maps of theta power (z-scored within participant). Left: Naive LLM. Right: PAWNI-assisted. [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Group-average phase-averaged cognitive load ( [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: EEG–NASA-TLX cross-validation scatter plot. Each point represents one participant. The [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Number of conversational turns per participant. PAWNI reduced all interactions to a single LLM turn. Percentage labels [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: Time-on-task comparison. Blue bars: Condition A total time. Orange (dark): PAWNI pipeline time (Architect Q&A + agent [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: Prompt structural completeness (percentage of 10 assessed elements present). Naive prompts (blue) averaged 42%; PAWNI [PITH_FULL_IMAGE:figures/full_fig_p032_18.png]
Figure 19
Figure 19. Figure 19: Per-participant difference topographic maps ( [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: PAWNI process experience assessment (1–7 Likert, higher = more agreement). Individual participant dots are overlaid. Wait [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]
Figure 21
Figure 21. Figure 21: Comparative assessment: which process was better? (1 = Naive LLM much better, 4 = no difference, 7 = PAWNI much better). [PITH_FULL_IMAGE:figures/full_fig_p034_21.png]
Figure 22
Figure 22. Figure 22: Conversation turn type composition. Left panel: Condition A (Naive LLM), showing stacked turn types per participant. Right [PITH_FULL_IMAGE:figures/full_fig_p037_22.png]
Figure 23
Figure 23. Figure 23: Per-participant EEG topographic maps of theta power (z-scored within participant). Left column: Naive LLM. Right column: [PITH_FULL_IMAGE:figures/full_fig_p047_23.png]
Figure 24
Figure 24. Figure 24: Task criteria coverage trajectory. Lines show cumulative coverage (%) after each conversational turn in Condition A. Stars [PITH_FULL_IMAGE:figures/full_fig_p048_24.png]
Figure 25
Figure 25. Figure 25: Summary heatmap of all key metrics across participants and conditions. Colour intensity reflects normalised performance (1 [PITH_FULL_IMAGE:figures/full_fig_p049_25.png]
Figure 26
Figure 26. Figure 26: Master comparison dot plots for six key metrics. Each line connects one participant’s Naive LLM (N) and PAWNI-assisted (P) [PITH_FULL_IMAGE:figures/full_fig_p049_26.png]
Figure 27
Figure 27. Figure 27: Per-participant band power spectra at frontal (Fz, FC1, FC2) and parietal (Pz, CP1, CP2) regions. Five frequency bands shown: [PITH_FULL_IMAGE:figures/full_fig_p050_27.png]
Figure 28
Figure 28. Figure 28: Group-average band power spectra (𝑁 = 4, ±1 SD) at frontal and parietal ROIs. Large error bars reflect inter-participant variability in absolute power [PITH_FULL_IMAGE:figures/full_fig_p051_28.png]
Figure 29
Figure 29. Figure 29: Group-average difference topographic map ( [PITH_FULL_IMAGE:figures/full_fig_p051_29.png]
Figure 30
Figure 30. Figure 30: Per-participant overlaid cognitive load timelines ( [PITH_FULL_IMAGE:figures/full_fig_p052_30.png]
Figure 31
Figure 31. Figure 31: Per-participant phase-averaged cognitive load ( [PITH_FULL_IMAGE:figures/full_fig_p053_31.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 17 canonical work pages

  1. [1]

    Sai Teja Reddy Adapala. 2025. Cognitive Load Limits in Large Language Models: Benchmarking Multi-Hop Reasoning.arXiv preprint arXiv:2509.19517 (2025). arXiv:2509.19517 [cs.AI] doi:10.48550/arXiv.2509.19517

    work page doi:10.48550/arxiv.2509.19517 2025

  2. [2]

    Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2024. STaR-GATE: Teaching Language Models to Ask Clarifying Questions.arXiv preprint arXiv:2403.19154(2024). arXiv:2403.19154 [cs.CL] doi:10.48550/arXiv.2403.19154 Published at COLM 2024

    arXiv doi:10.48550/arxiv.2403.19154 2024

  3. [3]

    Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara van Gog. 2010. Using Electroencephalography to Measure Cognitive Load.Educational Psychology Review22, 4 (2010), 425–438. doi:10.1007/s10648-010-9130-y

    doi:10.1007/s10648-010-9130-y 2010

  4. [4]

    Levendowski, Michelle N

    Chris Berka, Daniel J. Levendowski, Michelle N. Lumicao, Alan Yau, Gene Davis, Vladimir T. Zivkovic, Richard E. Olmstead, Patrice D. Tremoulet, and Patrick L. Craven. 2007. EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks.A viation, Space, and Environmental Medicine78, 5, Suppl. (2007), B231–B244

    work page 2007

  5. [5]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. AAAI Press, Vancou...

    doi:10.1609/aaai.v38i16.29720 2024

  6. [6]

    Koretsky, Maya Willey, Chelsea X

    Owen Bianchi, Mathew J. Koretsky, Maya Willey, Chelsea X. Alvarado, Tanay Nayak, Adi Asija, Nicole Kuznetsov, Mike A. Nalls, Faraz Faghri, and Daniel Khashabi. 2025. Hidden in the Haystack: Smaller Needles Are More Difficult for LLMs to Find.arXiv preprint arXiv:2505.18148(2025). arXiv:2505.18148 [cs.CL] doi:10.48550/arXiv.2505.18148

    work page doi:10.48550/arxiv.2505.18148 2025

  7. [7]

    John Brooke. 1996. SUS: A Quick and Dirty Usability Scale. InUsability Evaluation in Industry, P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I. L. McClelland (Eds.). Taylor & Francis, London, 189–194. doi:10.1201/9781498710411-35

    doi:10.1201/9781498710411-35 1996

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv doi:10.48550/arxiv.2005.14165 2020

Show all 72 references
  1. [9]

    Eason Chen, Danyang Wang, Luyi Xu, Chen Cao, Xiao Fang, and Jionghao Lin. 2024. A Systematic Review on Prompt Engineering in Large Language Models for K-12 STEM Education.arXiv preprint arXiv:2410.11123(2024). arXiv:2410.11123 [cs.CY] doi:10.48550/arXiv.2410.11123

    2024 doi arXiv

  2. [10]

    Shuaihang Chen, Yuanxing Liu, Wei Han, Weinan Zhang, and Ting Liu. 2024. A Survey on LLM-Based Multi-Agent System: Recent Advances and New Frontiers in Application.arXiv preprint arXiv:2412.17481(2024). arXiv:2412.17481 [cs.CL] doi:10.48550/arXiv.2412.17481

    2024 doi arXiv

  3. [11]

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou

  4. [12]

    Xin Chen, Feng Jiang, Yiqian Zhang, Hardy Chen, Shuo Yan, Wenya Xie, Min Yang, and Shujian Huang. 2026. Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers.arXiv preprint arXiv:2601.22139(2026). arXiv:2601.22139 [cs....

    2026 doi arXiv

  5. [13]

    Samy Chikhi, Nadine Matton, and Sophie Blanchet. 2022. EEG Power Spectral Measures of Cognitive Workload: A Meta-Analysis.Psychophysiology 59, 6 (2022), e14009. doi:10.1111/psyp.14009

    2022 doi

  6. [14]

    1988.Statistical Power Analysis for the Behavioral Sciences(2nd ed.)

    Jacob Cohen. 1988.Statistical Power Analysis for the Behavioral Sciences(2nd ed.). Lawrence Erlbaum Associates, Hillsdale, NJ

    1988

  7. [15]

    Ramtin Ehsani, Sakshi Pathak, and Preetha Chatterjee. 2025. Towards Detecting Prompt Knowledge Gaps for Improved LLM-Guided Issue Resolution. InProceedings of the 22nd IEEE/ACM International Conference on Mining Software Repositories (MSR ’25). IEEE, Ottawa, ON, Canada, 699–71...

    2025 arXiv

  8. [16]

    Lucile Favero, Daniel Frases, Juan Antonio Pérez-Ortiz, Tanja Käser, and Nuria Oliver. 2025. ELLIS Alicante at CQs-Gen 2025: Winning the Critical Thinking Questions Shared Task: LLM-Based Question Generation and Selection. InProceedings of the 12th Workshop on Argument Mining ...

    2025 doi

  9. [17]

    Georgiou

    Georgios P. Georgiou. 2025. ChatGPT Produces More “Lazy” Thinkers: Evidence of Cognitive Engagement Decline.arXiv preprint arXiv:2507.00181 (2025). arXiv:2507.00181 [cs.CY] doi:10.48550/arXiv.2507.00181

    2025 doi arXiv

  10. [18]

    Alan Gevins and Michael E. Smith. 2003. Neurophysiological Measures of Cognitive Workload During Human–Computer Interaction.Theoretical Issues in Ergonomics Science4, 1–2 (2003), 113–131. doi:10.1080/14639220210159717

    2003 doi

  11. [19]

    Smith, Harrison Leong, Linda McEvoy, Susan Whitfield, Robert Du, and Georgia Rush

    Alan Gevins, Michael E. Smith, Harrison Leong, Linda McEvoy, Susan Whitfield, Robert Du, and Georgia Rush. 1998. Monitoring Working Memory Load During Computer-Based Tasks with EEG Pattern Recognition Methods.Human Factors40, 1 (1998), 79–91. doi:10.1518/001872098779480578

    1998 doi

  12. [20]

    Engemann, Daniel Strohmeier, Christian Brodbeck, Lauri Parkkonen, and Matti S

    Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Lauri Parkkonen, and Matti S. Hämäläinen. 2014. MNE Software for Processing MEG and EEG Data.NeuroImage86 (2014), 446–460. doi:10.1016/j.neuroimage.2013.10.027

    2014 doi

  13. [21]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large Language Model Based Multi-Agents: A Survey of Progress and Challenges. InProceedings of the Thirty-Third International Joint Conference on Artifici...

    2024 doi

  14. [22]

    Sandra G. Hart. 2006. NASA-Task Load Index (NASA-TLX); 20 Years Later. InProceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50. SAGE Publications, Thousand Oaks, CA, USA, 904–908. doi:10.1177/154193120605000909

    2006 doi

  15. [23]

    Hart and Lowell E

    Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. InHuman Mental Workload, P. A. Hancock and N. Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, Amsterdam, 139–183. doi:10.101...

    1988 doi

  16. [24]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborat...

    2024 doi arXiv

  17. [25]

    Magdalena Kaiser, Rishiraj Saha Roy, and Gerhard Weikum. 2024. Robust Training for Conversational Question Answering Models with Reinforced Reformulation Generation. InProceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). ACM, New York,...

    2024

  18. [26]

    Sheer Karny, Anthony Baez, and Pat Pataranutaporn. 2026. Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI. InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). ACM, New Yor...

    2026

  19. [27]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Stat...

    2024 doi arXiv

  20. [28]

    Wolfgang Klimesch. 1999. EEG Alpha and Theta Oscillations Reflect Cognitive and Memory Performance: A Review and Analysis.Brain Research Reviews29, 2–3 (1999), 169–195. doi:10.1016/S0165-0173(98)00056-3

    1999 doi

  21. [29]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models Are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 22199–22213. arXiv:2205....

    2022 doi arXiv

  22. [30]

    Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Paweł W. Woźniak. 2023. A Survey on Measuring Cognitive Workload in Human–Computer Interaction.Comput. Surveys55, 13s (2023), 283:1–283:39. doi:10.1145/3582272

    2023 doi

  23. [31]

    Nataliya Kosmyna, Eugene Hauptmann, Ye Tong Yuan, Jessica Situ, Xian-Hao Liao, Ashly Vivian Beresnitzky, Iris Braunstein, and Pattie Maes. 2025. Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task.arXiv preprint arXiv:2506.08...

    2025 doi arXiv

  24. [32]

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. LLMs Get Lost in Multi-Turn Conversation.arXiv preprint arXiv:2505.06120 (2025). arXiv:2505.06120 [cs.CL] doi:10.48550/arXiv.2505.06120 44 B. Sankar et al

    2025 doi arXiv

  25. [33]

    Questions

    Shuyue Stella Li, Jimin Mun, Faeze Brahman, Pedram Hosseini, Bryceton G. Thomas, Jessica M. Sin, Bing Ren, Jonathan S. Ilgen, Yulia Tsvetkov, and Maarten Sap. 2025. ALFA: Aligning LLMs to Ask Good Questions. A Case Study in Clinical Reasoning. InProceedings of the Second Confe...

    2025 doi arXiv

  26. [34]

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A Survey on LLM-Based Multi-Agent Systems: Workflow, Infrastructure, and Challenges. Vicinagearth1, 1 (2024), 9. doi:10.1007/s44336-024-00009-2

    2024 doi

  27. [35]

    Xujian Liang and Zhaoquan Gu. 2025. Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. AAAI Press, 24558–24566. doi:10.1609/aaai.v39i23.34635

    2025 doi

  28. [36]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.Comput. Surveys55, 9 (2023), 1–35. doi:10.1145/3560815

    2023 doi

  29. [37]

    Qibang Liu, Wenzhe Wang, and Jeffrey Willard. 2025. Effects of Prompt Length on Domain-Specific Tasks for Large Language Models.arXiv preprint arXiv:2502.14255(2025). arXiv:2502.14255 [cs.CL] doi:10.48550/arXiv.2502.14255

    2025 doi arXiv

  30. [38]

    Yao-Yang Liu, Zhen Zheng, Feng Zhang, Jin-Cheng Feng, Yi-Yang Fu, Ji-Dong Zhai, Bing-Sheng He, Xiao Zhang, and Xiao-Yong Du. 2026. A Comprehensive Taxonomy of Prompt Engineering Techniques for Large Language Models.Frontiers of Computer Science20, 3 (2026), 2003601. doi:10.100...

    2026 doi

  31. [39]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: It...

    2023 doi arXiv

  32. [40]

    Davide Mazzaccara, Alberto Testoni, and Raffaella Bernardi. 2024. Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain. InFindings of the Association for Computational Linguistics: EMNLP 2024. Association for Computat...

    2024 arXiv

  33. [41]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, ...

    2022 doi arXiv

  34. [42]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23...

    2023

  35. [43]

    Soya Park, Hari Subramonyam, and Chinmay Kulkarni. 2023. Thinking Assistants: LLM-Based Conversational Assistants That Help Users Think by Asking Rather Than Answering.arXiv preprint arXiv:2312.06024(2023). arXiv:2312.06024 [cs.HC] doi:10.48550/arXiv.2312.06024

    2023 doi arXiv

  36. [44]

    Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. 2025. Agentic Large Language Models, a Survey.Journal of Artificial Intelligence Research84 (2025). doi:10.1613/jair.1.18675 Article 29

    2025 doi

  37. [45]

    Pope, Edward H

    Alan T. Pope, Edward H. Bogart, and Debbie S. Bartolome. 1995. Biocybernetic System Evaluates Indices of Operator Engagement in Automated Task.Biological Psychology40, 1–2 (1995), 187–195. doi:10.1016/0301-0511(95)05116-3

    1995 doi

  38. [46]

    Gradient Descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic Prompt Optimization with “Gradient Descent” and Beam Search. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Comput...

    2023 arXiv

  39. [47]

    Bujar Raufi and Luca Longo. 2022. An Evaluation of the EEG Alpha-to-Theta and Theta-to-Alpha Band Ratios as Indexes of Mental Workload. Frontiers in Neuroinformatics16 (2022), 861967. doi:10.3389/fninf.2022.861967

    2022

  40. [48]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications.arXiv preprint arXiv:2402.07927(2024). arXiv:2402.07927 [cs.AI] doi:10.48550/arXiv...

    2024 doi arXiv

  41. [49]

    2011.A Practical Guide to the System Usability Scale: Background, Benchmarks and Best Practices

    Jeff Sauro. 2011.A Practical Guide to the System Usability Scale: Background, Benchmarks and Best Practices. Measuring Usability LLC, Denver, CO

    2011

  42. [50]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay S...

    2024 doi arXiv

  43. [51]

    Hong Su. 2026. Autonomous Question Formation for Large Language Model-Driven AI Systems.arXiv preprint arXiv:2602.01556(2026). arXiv:2602.01556 [cs.AI] doi:10.48550/arXiv.2602.01556

    2026 doi

  44. [52]

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D. Nguyen. 2025. Multi-Agent Collaboration Mechanisms: A Survey of LLMs.arXiv preprint arXiv:2501.06322(2025). arXiv:2501.06322 [cs.AI] doi:10.48550/arXiv.2501.06322

    2025 doi arXiv

  45. [53]

    Shubham Vatsal and Harsh Dubey. 2024. A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks.arXiv preprint arXiv:2407.12994(2024). arXiv:2407.12994 [cs.CL] doi:10.48550/arXiv.2407.12994 Asking Questions the Right Way: A Multi-Agent Conversatio...

    2024 doi arXiv

  46. [54]

    Wenxuan Wang, Juluan Shi, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2025. Learning to Ask: When LLM Agents Meet Unclear Instruction. InProceedings of the 2025 Conference on Empirical Methods in Natur...

    2025 arXiv

  47. [55]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InProceedings of the International Conference on Learning Representations (ICLR). ...

    2023 doi arXiv

  48. [56]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Ass...

    2022 doi arXiv

  49. [57]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt

  50. [58]

    Williams, Genevieve M

    Nikolas S. Williams, Genevieve M. McArthur, Bianca de Wit, George Ibrahim, and Nicholas A. Badcock. 2020. A Validation of Emotiv EPOC Flex Saline for EEG and ERP Research.PeerJ8 (2020), e9713. doi:10.7717/peerj.9713

    2020 doi

  51. [59]

    arXiv:2302.11382 [cs.SE] doi:10.48550/arXiv.2302.11382

    A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.arXiv preprint arXiv:2302.11382(2023). arXiv:2302.11382 [cs.SE] doi:10.48550/arXiv.2302.11382

    2023 doi arXiv

  52. [60]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 36. Curran Associates, In...

    2023 doi arXiv

  53. [61]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. In Proceedings of the International Conference on Learning Representations (ICLR). OpenReview.net. arXiv:2309.03409 doi:10.48550/arXiv.2309.03409

    2024 doi arXiv

  54. [62]

    Murong Yue. 2025. A Survey of Large Language Model Agents for Question Answering.arXiv preprint arXiv:2503.19213(2025). arXiv:2503.19213 [cs.CL] doi:10.48550/arXiv.2503.19213

    2025 doi arXiv

  55. [63]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of the International Conference on Learning Representations (ICLR). OpenReview.net, Kigali, Rwanda. arXiv...

    2023 doi arXiv

  56. [64]

    Dumais, Nick Craswell, Paul N

    Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. 2020. Generating Clarifying Questions for Information Retrieval. InProceedings of The Web Conference 2020 (WWW ’20). ACM, New York, NY, USA, 418–428. doi:10.1145/3366423.3380126

    2020

  57. [65]

    Differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. TextGrad: Automatic “Differentiation” via Text.arXiv preprint arXiv:2406.07496(2024). arXiv:2406.07496 [cs.CL] doi:10.48550/arXiv.2406.07496

    2024 doi arXiv

  58. [66]

    Xuan Zhang, Yang Deng, Zifeng Ren, See-Kiong Ng, and Tat-Seng Chua. 2024. Ask-before-Plan: Proactive Language Agents for Real-World Planning. InFindings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA...

    2024 arXiv

  59. [67]

    Zamfirescu-Pereira, Richmond Y

    J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). ACM, New York, NY, USA, ...

    2023

  60. [68]

    Le, and Ed H

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. InProceedings of the International Conferen...

    2023 doi arXiv

  61. [69]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen

  62. [70]

    arXiv:2303.18223 [cs.CL] doi:10.48550/arXiv.2303.18223

    A Survey of Large Language Models.arXiv preprint arXiv:2303.18223(2023). arXiv:2303.18223 [cs.CL] doi:10.48550/arXiv.2303.18223

    2023 doi arXiv

  63. [72]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models Are Human-Level Prompt Engineers. InProceedings of the International Conference on Learning Representations (ICLR). OpenReview.net. arXiv:2211.01...

    2023 doi arXiv

  64. [2023]

    arXiv:2311.17311 [cs.CL] doi:10.48550/arXiv.2311.17311 Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution 43

    Universal Self-Consistency for Large Language Model Generation.arXiv preprint arXiv:2311.17311(2023). arXiv:2311.17311 [cs.CL] doi:10.48550/arXiv.2311.17311 Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution 43

    2023 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.