{"id":"246685c8-75a6-429c-980a-a844ec14ab2a","arxiv_id":"2412.12644","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"iPrOp is an interactive prompt optimization framework with a human in the loop; a simulation that selects prompts by training F1 shows rising validation F1 on three emotion classification datasets.","lead":"The paper introduces iPrOp, an interactive prompt optimization system for large language models that lets users compare prompt variants using predictions, explanations, and performance scores. It reports rising F1 scores across 15 simulated optimization steps on three emotion classification datasets, though no human evaluation is included.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central interactive claim rests on an untested proxy: the evaluation replaces human assessment with automatic F1 selection, so the paper's evidence does not validate the human-in-the-loop aspect it claims.","rationale":"The reader's weakest_assumption correctly identifies that the simulation substitutes automatic F1 selection for human assessment, and that no user study tests the interactive claim. My stress-test agrees with this diagnosis and sharpens it by noting that the failure is not merely a missing user study: the formal definition of iPrOp in Section 3 centers on a human assessment H that integrates multiple information sources, and the evaluation uses none of them. Consequently, the experiment cannot validate the method as presented. Additionally, even the automatic result is not internally robust because Figure 4 lacks error bars, statistical tests, seeds, or baselines, and the paper's own limitations admit prompt insensitivity and the possibility that naive prompts perform well. These points reinforce the reader's conditional verdict rather than overturning it: the paper's contribution is a system demo with a plausible but unverified workflow, and the evidence is insufficient to accept the central claim. The verdict should remain CONDITIONAL, requiring a proper user study and stronger experimental controls before acceptance.","tokens_in":10404,"tokens_out":3231,"duration_ms":32202,"concrete_test":"Run a real-user study (N ≥ 20) on the same three emotion classification datasets using the actual iPrOp interface, with participants choosing between prompt pairs based on the displayed instances, explanations, and metrics over 15 iterations. Compare the validation F1 of the prompts selected by users against those selected by the automatic weighted-F1 proxy, starting from identical initial prompts. If user-selected prompts differ systematically from proxy-selected prompts, or if user selections yield lower or no better validation F1, the automatic simulation is not representative of human-in-the-loop behavior and the central interactive claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that iPrOp, an interactive prompt optimization approach with a human in the loop, can generate improved prompts and enhance task performance. The formalization in Section 3 defines the user's assessment H as the selection mechanism, incorporating prompt performance, readability, explanations, and alignment with user expectations. However, the evaluation in Section 4 completely replaces H with automatic selection by weighted F1 on a fixed training subset, stating that 'the prompt is selected corresponding to the weighted F1 score over a fixed subset of the training data' and 'we only consider automated classification performance scores.' No user is involved, and the other components of I(pi)—explanations, readability, alignment—are not tested at all. Therefore, the experiment validates a fully automatic F1-driven search, not the interactive iPrOp system. The load-bearing assumption—that automatic F1 selection is a faithful stand-in for human assessment—is asserted but never tested; the paper itself defers a user study to future work. Furthermore, even as an evaluation of automatic optimization, the result is weak: Figure 4 shows training and validation F1 trends over 15 iterations with no error bars, no multiple seeds, no statistical test, and no comparison to baselines such as random prompt selection or best-of-N without iteration. The paper's own Limitations section notes that some datasets are insensitive to prompt variation and that 'a simple or even naive initial prompt' can achieve superior performance, which further undermines the interpretation of the observed trend. Thus, the evidence does not support the abstract's claim that the evaluation shows the interactive approach's capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces iPrOp, an interactive prompt optimization framework for LLM-based text classification, formalized as an iterative loop in which a user assesses candidate prompts on the basis of performance metrics, predictions with explanations, informative instances, and readability, and selects the preferred prompt for further paraphrasing. The authors describe a prototype web interface and report a simulation experiment on three emotion classification datasets (TEC, GROUNDED-EMOTIONS, TALES-E-MOTION) using llama3.1:8b-instruct-fp16, in which the user assessment is replaced by automatic selection according to the weighted F1 score on a fixed training subset. The paper claims that the approach can generate improved prompts and lead to enhanced task performance, based on an observed increasing trend in training and validation F1 over 15 iterations.","tokens_in":10704,"tokens_out":4179,"duration_ms":35729,"significance":"If the central claim were fully supported, iPrOp would offer a useful bridge between manual prompt engineering and automatic prompt optimization, with a structured interface for non-technical users and a modular workflow that could support systematic studies of prompt properties. Strengths include the clear formalization of the user assessment function, the publicly available prototype web interface and explanation video, and a candid Limitations section that acknowledges several unresolved challenges. However, the significance is currently limited by the fact that the reported experiment does not actually test the human-in-the-loop component: it substitutes automatic F1-based selection for human judgment, and it provides no baselines, error bars, or statistical tests. The result is a plausible pilot for an automatic optimization loop, but it does not yet substantiate the interactive contribution claimed in the title and abstract.","major_comments":[{"comment":"The central empirical claim—that iPrOp benefits from a human in the loop—is not evaluated. The formal definition of user assessment H in Eq. (1) includes prompt performance, readability, explanations, and alignment with user expectations, but the experiment replaces H with automatic selection by weighted F1 on a fixed training subset, stating that \"the prompt is selected corresponding to the weighted F1 score over a fixed subset of the training data\" and that \"we only consider automated classification performance scores.\" A user study is deferred to future work. Consequently, the reported results validate a fully automatic F1-driven prompt search, not the interactive human-in-the-loop process that the title and abstract claim. To support the central claim, the authors need either a user study or a carefully validated simulation of human preferences (e.g., a model trained on actual human judgments), and the abstract and title should be tempered to match the actual evaluation.","section":"Section 4 (Evaluation) and Section 3 (Methods)"},{"comment":"The evidence for the effectiveness of the optimization loop is statistically weak. The F1 curves over 15 iterations are presented without error bars, multiple random seeds, or significance tests, and no baselines are provided (e.g., random prompt paraphrasing, best-of-N selection without iteration, or a fixed initial prompt). Without such controls, the observed upward trend cannot be attributed to the iterative refinements or to the information presented to the simulated user; the trend could in principle arise from the paraphrase model's drift or from variance. The paper should include at least one baseline and repeat the experiment with different seeds and train/validation splits, reporting variance, in order to substantiate the claim that the approach \"has the capability to generate improved prompts.\"","section":"Section 4, Figure 4"},{"comment":"The paper's own limitations weaken the generality of the empirical claim. It states that \"certain datasets exhibit limited sensitivity to divergent prompts, allowing a simple or even naive initial prompt to achieve superior performance.\" This is directly relevant to the three emotion datasets evaluated: if one or more of them is insensitive to prompt variation, the rising F1 curve may reflect noise or incidental improvements rather than the method's capability. The manuscript should report per-dataset initial vs. final performance and explicitly discuss which datasets support the claimed improvement, rather than aggregating the trend across datasets.","section":"Section 5 (Limitations)"}],"minor_comments":[{"comment":"There is a notation inconsistency: the argmax ranges over p ∈ P ∪ M(P), but I(pi) uses an index i that is not introduced; it should be I(p) for consistency. Also, the definition of M(P) as \"a prompt paraphrasing model that varies the prompts P selected from the previous iteration\" could be made more precise by distinguishing the set of candidate prompts P from the set of paraphrased variants M(P).","section":"Section 3, Eq. (1)"},{"comment":"The dual y-axes and color-only legend make the figure difficult to read, especially in grayscale; consider plotting each dataset in a separate panel with a shared scale, or using distinct line styles in addition to color.","section":"Figure 4"},{"comment":"The sentence \"Wu et al.'s (2024) analysis of strategies to enhance the transparency of LLMs.\" is a sentence fragment; it should be revised into a complete sentence, for example by adding a main verb and the finding or contribution.","section":"Section 2.3"},{"comment":"The phrase \"but also enable to study the intrinsic parameters\" is ungrammatical; it should be \"but also enables the study of the intrinsic parameters.\"","section":"Abstract"},{"comment":"The dataset name \"TALES-E-MOTION\" appears inconsistently as \"TALES -E MOTION\" and \"TALES -E MOTION\"; please standardize the formatting throughout the text and figure.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is closer to a system/demonstration paper than to a fully validated research contribution. The prototype web interface and the simulation are promising, and the authors are transparent about the limitations, but the mismatch between the title/abstract's human-in-the-loop claim and the automatic F1-based evaluation is substantial. I would encourage the editor to require a user study or a much more faithful simulation of human preferences, along with baselines and statistical rigor, before considering publication. If the authors are unwilling or unable to add a user study, the manuscript should be reframed as a pilot evaluation of an automatic prompt search that is intended to support future interactive use."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper describes a human-in-the-loop prompt optimization framework and then does not actually test the human part. The evaluation replaces the user with automatic F1-based prompt selection on a fixed training subset, and the paper says a user study is future work. So the abstract's claim that \"the evaluation shows our approach has the capability to generate improved prompts\" overstates what the experiments show.\n\nWhat it does well is set out a clean formalization: the user's assessment H over a presented property set I(pi) = (pi, T_alpha^pi, E(T_alpha, pi), F1(T_beta^pi)). The workflow in Figure 2 is clear, and the modular design lets each component be either human or simulated. The web prototype is a real artifact. The paper is also transparent about key limitations: it notes that certain datasets are insensitive to prompt variation and that a simple initial prompt can already be near-optimal.\n\nThe central problem is proportionality. The experiment has no baselines (e.g., random rephrasing or best-of-N without iteration), no multiple seeds, no error bars, and no statistical test. Fifteen iterations on three emotion datasets show a rising training and validation F1 trend, but given the selection criterion is F1 on the training subset, the validation trend is a mild independent check. Still, without a comparison to non-iterative selection, the trend does not show that the iterative loop itself helps. More importantly, the unique components of iPrOp—explanations, instance selection, readability, alignment with user expectations—are not evaluated at all. As an automatic prompt optimizer, the result is already established by APO, Promptbreeder, and others. As an interactive system, there is no evidence.\n\nThis is a workshop-level system paper. It would be useful for researchers starting work on interactive prompt optimization or HCI-for-NLP tooling. It does not currently support the stronger claims in the abstract, but the framework and prototype are a legitimate starting point.\n\nI would send it to review, but expect that a serious referee would ask for a real user study, baselines, and more rigorous reporting. The honest limitations section is a point in its favor; it is not trying to hide the gaps. Whether it is acceptable at a workshop depends on whether the framework alone is enough.","headline":"A tidy framework paper whose headline result is a simulation of the interactive loop, not a test of it.","tokens_in":11241,"tokens_out":2967,"would_cite":false,"duration_ms":27584,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces iPrOp, an interactive prompt optimization loop with a human in the loop, and claims it improves prompt performance by generating and selecting among paraphrased prompts over repeated iterations.","keywords":["interactive prompt optimization","human-in-the-loop","prompt engineering","large language models","emotion classification","prompt paraphrasing","explainability","F1 evaluation"],"falsifier":"Run the same 15-iteration optimization on the three datasets using a train/validation/test split and report F1 on the held-out test fold; if the final prompt does not outperform the initial seed prompt on that test fold, the central claim of enhanced task performance fails.","tokens_in":10204,"feed_emoji":"📈","tokens_out":7377,"duration_ms":56811,"temperature":0.7,"pith_summary":"This paper introduces iPrOp, an interactive prompt optimization workflow that places a human in the loop between automatic prompt rephrasing and performance evaluation. The goal is to make prompt engineering usable by non-experts while letting the user weigh competing criteria: classification performance, prompt readability, explanation quality, and alignment with user expectations. The central claim is that this loop yields better prompts: in a simulation over 15 iterations on three emotion classification datasets, F1 scores on both training and validation data rise. If correct, iPrOp provides a practical middle ground between fully manual prompt crafting and fully automated prompt search, and a platform for studying how example selection, explanation strategy, and model choice influence prompt optimization.","feed_headline":"Human-in-the-loop prompt loop lifts F1 over 15 rounds","feed_subtitle":"Guided rephrasing and choice improve prompts on three emotion datasets without prompt-engineering skill.","key_machinery":"The central mechanism is the iPrOp loop, formalized as $p^* = \\arg\\max_{p \\in P \\cup M(P)} H(I(p_i))$, where $M(P)$ is a prompt paraphrasing model that generates variants, and $I(p_i)$ packages the properties presented to the user: the prompt itself, a sample of informative training instances with the model's labeled predictions and explanations, and an F1 score computed on a held-out subset. The loop is iterative: the user (or, in the simulation, the F1-based selector) chooses the preferred prompt, which becomes the seed for the next round of paraphrasing. The workflow is designed so that each step can be performed by the user or automatically, and it is this interchangeability that lets iPrOp serve both interactive and simulated modes.","core_discovery":"The paper proposes iPrOp and claims that its iterative workflow—starting from an initial seed prompt, repeatedly generating paraphrased prompt variants, selecting the preferred prompt on the basis of informative examples, model-produced explanations, and performance metrics—can generate improved prompts that lead to enhanced task performance. In the reported experiment, the selection step is simulated: a program picks the prompt with the highest weighted F1 on a fixed training subset at each iteration, using the Llama 3.1 8B instruct model on the TEC, GROUNDED-EMOTIONS, and TALES-EMOTION datasets. Over 15 iterations, F1 increases on both the training subset and a validation set for all three datasets, which the paper takes as evidence for the approach's effectiveness. The paper also formalizes the selection process, where the user preference function H chooses the prompt maximizing over the presented prompt properties.","pith_inferences":["The reported simulation suggests the performance gain may come primarily from the rephrasing-and-score loop itself, not from the human; a controlled comparison between full automatic selection and human-guided selection would reveal how much the human judgment actually adds.","If the F1-based selection is a faithful proxy, then the same loop should transfer to other classification tasks and even to non-classification tasks by replacing F1 with a suitable metric, a testable extension the paper does not run.","The limitation that some datasets are insensitive to rephrasing implies that an upfront diagnostic—e.g., measuring how much paraphrases change the model's predictions—could predict which tasks will benefit from interactive optimization.","A user study comparing iPrOp against both manual prompt engineering and fully automatic optimization would settle whether the interactive interface actually reduces effort while improving or matching end performance."],"forward_implications":["Users without prompt-engineering expertise can reach competitive prompts for text classification by following a structured, information-rich selection loop rather than writing prompts from scratch.","The loop makes trade-offs explicit: a user can reject a high-F1 prompt in favor of one that is more readable or whose explanations better match their expectations, which is not possible in fully automatic optimization.","Because each step can be automated or human-performed, the approach supports partial automation, letting the human intervene only on decisions that most need judgment.","The framework provides a testbed for studying how example selection, explanation generation, and LLM choice affect the trajectory of prompt optimization across iterations."],"supporting_citations":[{"why":"Supplies the Llama 3.1 8B instruct model used for all prompt evaluations and paraphrasing in the experiment.","marker":"(Dubey et al., 2024)"},{"why":"Provides the TEC emotion classification dataset used in the evaluation.","marker":"(Mohammad, 2012)"},{"why":"Provides the GROUNDED-EMOTIONS dataset used in the evaluation.","marker":"(Liu et al., 2017)"},{"why":"Provides the TALES-EMOTION dataset used in the evaluation.","marker":"(Alm and Sproat, 2005)"}],"fun_headline_variants":["Interactive prompt optimization lifts F1 over 15 rounds","Human-in-the-loop prompt refinement boosts task performance","Guided prompt selection improves F1 without expert prompt engineering","iPrOp: iterative human-guided prompt optimization lifts scores","No prompt expertise needed: interactive optimization lifts F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments replace the human assessment that the method is designed around with automatic selection by weighted F1 on a training subset, so the claim that the interactive loop improves prompts depends on this score-based proxy standing in for real user choices and on the small validation sets supporting generalization.","fun_headline_variants_meta":{"raw":{"variants":["Interactive prompt optimization lifts F1 over 15 rounds","Human-in-the-loop prompt refinement boosts task performance","Guided prompt selection improves F1 without expert prompt engineering","iPrOp: iterative human-guided prompt optimization lifts scores","No prompt expertise needed: interactive optimization lifts F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2526,"prompt_tokens":879,"completion_tokens":1647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1570}},"tokens_in":495,"tokens_out":1647,"duration_ms":11832,"temperature":1.0,"reasoning_tokens":1570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:52:10.015359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 15-iteration optimization on the three datasets using a train/validation/test split and report F1 on the held-out test fold; if the final prompt does not outperform the initial seed prompt on that test fold, the central claim of enhanced task performance fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TALES-EMOTION dataset used in the evaluation."}],"review_version":1}