{"id":"b21f73ac-f4e5-4fbd-bda0-051ccf6d1742","arxiv_id":"2502.13149","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Bi-Fact, a bidirectional fact-level LLM-based metric, reports higher agreement with human judgments than existing metrics when scoring intent extraction from GUI trajectories.","lead":"This paper introduces Bi-Fact, an evaluation method that splits an intended task description and a model's prediction into atomic facts, then checks which facts are supported in each direction to score precision and recall. On two small datasets, Bi-Fact agreed with human judgments more often than existing metrics such as BLEU, ROUGE, and NLI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-conditioned decomposition of predicted intents in Section 2.2 Stage 1 may omit spurious facts, inflating precision and the reported F1; the central claim is not tested against a neutral decomposition.","rationale":"The reader correctly identified the representation assumption (atomic facts and bidirectional entailment) as a conceptual vulnerability. My stress-test focuses on a more specific and testable implementation detail that sits inside that assumption: the predicted-intent decomposition is conditioned on the gold facts, which can distort the precision component. This is the single most load-bearing point because the paper's central claim is that Bi-Fact gives an accurate automated evaluation of intent extraction; if the predicted-fact list is incomplete by construction, then even a perfectly consistent LLM cannot recover a faithful precision/recall measurement. I do not claim the method is incoherent or fraudulent; the design is transparent, and the prompt is included in the appendix, which is a genuine strength. But the empirical support is not yet sufficient to rule out this bias. The paper lacks significance tests on the main comparison, an LLM specification, and code/data release, but those are secondary to the decomposition-conditioning issue because fixing statistics would not fix a systematic measurement bias. My proposed test is lightweight and would settle the question: compare the gold-conditioned prompt with a neutral decomposition prompt. If performance collapses or changes materially, the headline claim depends on the biased protocol; if it does not, the concern is resolved. I therefore keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT. The condition should be explicitly added: demonstrate that the published results are robust to a neutral, gold-blind decomposition step or remove the gold-conditioning from the method description.","tokens_in":7394,"tokens_out":5510,"duration_ms":56744,"concrete_test":"Run Bi-Fact on the same 400 gold-predicted pairs with two Stage 1 variants: (A) the published gold-conditioned prompt, and (B) a neutral prompt that asks for an exhaustive atomic decomposition of the predicted intent without showing the gold facts. Compare (i) the average number of predicted facts per pair and (ii) precision, recall, F1, and the binarized intent-match F1/Kappa. Additionally, have two independent annotators, blind to the gold facts, mark predicted-fact lists for a sample of 50 pairs. If variant B produces materially more predicted facts or changes the binary judgments, the published metric is biased; if the two variants agree, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is an internal design property, not a statistical footnote. In Section 2.2 Stage 1, the LLM decomposes the predicted intent while receiving the gold fact list as input and is instructed to decompose 'in accordance with' that list. This can systematically suppress predicted facts that have no counterpart in the gold decomposition, so precision is computed over a filtered set rather than over all facts actually present in the prediction. Since Bi-Fact's F1, and therefore its binarized intent-match threshold, depends directly on precision, the headline numbers (F1 0.722, Kappa 0.508) may reflect a lenient fact-recovery mechanism rather than true semantic agreement. The same prompt also contains a 'prerequisite counts as covered' rule for recall, which further loosens the comparison. The fact-level validation does not control for this: the 36 pairs are manually factorized and entailment-labeled by the authors, and it is unclear whether predicted intents are factorized independently or with the gold facts visible. No inter-annotator agreement or LLM ablation is reported, so the paper does not yet establish that Bi-Fact tracks human intent-equivalence judgments rather than its own gold-conditioned decomposition choices.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Bi-Fact, a fact-level metric for evaluating predicted natural-language intents extracted from GUI trajectories. The method decomposes gold and predicted intents into atomic facts and uses an LLM to assess, bidirectionally, whether each fact of one intent is entailed by the other intent as a whole, yielding precision, recall, and F1. The metric is validated on 400 human-annotated gold–predicted intent pairs with binary task-equivalence judgments, where a binarized version of Bi-Fact achieves the highest F1 (0.722) and Cohen's kappa (0.508) over BLEU, ROUGE, METEOR, sentence T5 similarity, bidirectional NLI, and AutoRater; and on 36 manually factorized intent pairs, where Bi-Fact's F1 has a Pearson correlation of 0.781 with manual fact-level F1, outperforming NLI (0.517).","tokens_in":7552,"tokens_out":3763,"duration_ms":27227,"significance":"If valid, Bi-Fact would be a genuinely useful evaluation method for intent extraction from UI trajectories: it is more informative than sentence-level overlap or NLI because it produces fact-level feedback on which parts of an intent are correct or missing, and it is validated against human judgment in a task-specific, outcome-based sense of intent equivalence. Evidence for the claimed advantage is presented in the form of correlation with human annotations rather than only illustrative examples, and the checklists and prompts are included in the appendix, which strengthens reproducibility. The central weakness is that the evidence base is small (400 binary pairs, 36 fact-level pairs), no significance tests or confidence intervals are reported for Table 1, and the automatic gold-conditioned decomposition of predictions creates a risk that precision is inflated; the paper would need a stronger, more controlled validation to support the strongest claims of superiority.","major_comments":[{"comment":"The automatic evaluation is vulnerable to a gold-conditioning bias in the decomposition of the predicted intent. In Stage 1, the predicted sentence is decomposed 'in accordance with' the fixed gold fact list, and the only worked example in Figures 3–5 shows the predicted facts being produced with the expert facts visible. This setup can suppress predicted facts that have no counterpart in the gold decomposition, so precision is computed over a filtered set rather than over all facts actually present in the prediction. Because the headline binary-intent-match F1 (0.722) and kappa (0.508) depend on a precision computed from this filtered set, the metric may be measuring fact recovery under a gold-aligned decomposition rather than faithful fact-level agreement. The 36-pair manual validation does not rule this out unless the manual factorization of predicted intents was performed independently of the gold facts; the paper does not state that it was, and no LLM-ablation or inter-annotator agreement is reported. The authors should either justify that the gold-conditioned decomposition does not systematically omit spurious predicted facts, or modify the protocol so that prediction decomposition is independent of the gold facts (e.g., decomposed once, before the gold facts are shown), and re-run the validation.","section":"Section 2.2, Stage 1 / Prompt in Figure 3"},{"comment":"Table 1 reports point estimates of precision, recall, F1, and kappa over 400 intent pairs, but no confidence intervals or significance tests are given for the comparison of Bi-Fact (F1 0.722, kappa 0.508) against the second-best metric (NLI: F1 0.614, kappa 0.467; T5-sim F1 0.669). With 400 binary decisions, the difference in F1 between 0.722 and 0.669 could easily be within sampling error, and the kappa differences are also modest. The claim that Bi-Fact 'notably achieves the highest F1 and Kappa ... indicating better agreement' needs either a paired significance test (e.g., McNemar or bootstrap with a defined resampling unit) or confidence intervals on the aggregate, and the thresholds should be acknowledged as selected on the dev set; otherwise the central claim of superiority over existing metrics is not statistically established.","section":"Section 3.2, Table 1"},{"comment":"The fact-level validation rests on 36 intent pairs. The Pearson correlation of 0.781 (p < 0.001) is reported, but with n=36 and no confidence interval the precision of the correlation estimate is quite low, and the comparison against NLI (r=0.517) is likewise reported without an interval or a test of the difference between the two dependent correlations. More importantly, the manual annotations were produced by the authors using the same atomic-fact decomposition and entailment notion that defines Bi-Fact, so the validation is partly a self-consistency check of the protocol. The authors should report the manual annotation procedure in full (who annotated, whether predicted intents were factorized with or without the gold facts visible, whether multiple annotators labeled the entailment decisions, and inter-annotator agreement), and provide a confidence interval or bootstrap for the correlation.","section":"Section 3.3 and Section 3.1 'Fact-Level Data'"},{"comment":"The core assumption that two intents are equivalent exactly when each atomic fact of one is entailed by the other as a whole, with 'atomic facts' as the representational units, is reasonable for the UI domain but is not independently established. The paper's example (book a business-class flight to Paris vs. book a flight to Paris) shows how fact-level scores can be more informative than a binary judge, but nothing in the 400-pair or 36-pair evaluations tests how robust the atomic-fact decomposition is to paraphrase variation or to implicit intent that is not lexically realized. The authors should at least discuss or ablate the decomposition granularity (e.g., compare against a baseline that uses gold facts only for recall and an independently produced fact list for precision) and report the LLM's decomposition consistency.","section":"Section 2.1 / Overall design assumption"}],"minor_comments":[{"comment":"The phrase 'can is implied' should read 'is implied'.","section":"Section 2.2, Stage 2"},{"comment":"There are typographical and formatting errors: 'evaluation methods, such as Pyramid ... or FactScore ... offer a more granular approach' contains an orphaned quotation mark and stray punctuation ('Harnly et al., 2005)„ '); these should be cleaned up.","section":"Section 1"},{"comment":"The threshold-tuning procedure (30 equally spaced increments on the dev set) is described, but the actual chosen threshold values for each metric are not reported; reporting them would make the comparison reproducible.","section":"Section 3.2"},{"comment":"The prompt example in Figure 5 contains a JSON syntax error: the third entry in 'predicted_fact_accuracy' is preceded by a missing comma after the previous object's closing brace. Since this prompt is central to the method, the one-shot example should be syntactically correct.","section":"Appendix A, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible and useful metric, and the authors have made a genuine attempt to validate with human annotations, but the evidence supporting the headline claims is thinner than the text suggests: a 400-pair binary comparison without significance tests, a 36-pair fact-level validation without inter-annotator information, and a gold-conditioned decomposition that could inflate precision. These are fixable with re-analysis and a protocol change. I would not reject the paper on scope; it is a good fit for a CS/AI venue. The main risk is that the central claim of 'superior correlation' may not survive a bootstrap or McNemar test, so the revision should either provide those tests or soften the claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful idea—evaluate intent extraction by decomposing gold and predicted intents into atomic facts and doing bidirectional entailment—but the main empirical claim is weakened by a design choice that biases the metric, and by thin statistics. I'd send it to review, but with the expectation of substantial revision.\n\nThe new part is the application of fact-level decomposition to UI intent comparison, with both precision and recall. That is a natural fit and fills a real gap: lexical and sentence-level NLI metrics miss fine-grained factual mismatches like wrong dates or missing business-class. The protocol is clearly described, and the comparison against BLEU, ROUGE, NLI, etc. is appropriate. The fact-level validation, while small, is a good idea.\n\nThe soft spot is in Stage 1 of the assessment prompt. The LLM decomposes the predicted intent while seeing the gold facts and is told to do it 'in accordance with' that list. That can filter out predicted facts that have no gold counterpart, inflating precision. The 'prerequisite counts as covered' rule for recall also loosens the comparison. The paper doesn't test an independent decomposition, so we don't know how much of the reported F1 (0.722) and Kappa (0.508) comes from this lenient fact recovery rather than genuine semantic agreement. This is load-bearing, not a footnote.\n\nThe statistics are also thin. Table 1 has no confidence intervals or significance tests across the 400 pairs; the F1 gap over NLI (0.722 vs 0.614) could easily be noise. The fact-level correlation (0.781, p<0.001) rests on 36 pairs with no inter-annotator agreement reported. And the LLM judge is unnamed, with no code or data release, which makes the result hard to reproduce.\n\nThe citation pattern is fine; the paper fairly builds on FactScore, Pyramid, and NLI work.\n\nBottom line: the core idea is coherent and worth pursuing, but the current evidence doesn't support the headline claim as stated. The metric's bias needs to be addressed—either by neutralizing the decomposition or by an ablation comparing gold-conditioned vs. independent factorization—and the statistics need to be tightened. This is a solid candidate for peer review after that. I would not cite it in its current form, but I'd watch the revised version.","headline":"Useful idea, but the gold-conditioned decomposition biases precision and the stats are too thin to support the headline.","tokens_in":8164,"tokens_out":2755,"would_cite":false,"duration_ms":25053,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bi-Fact is an automated evaluation metric that decomposes gold and predicted intents into atomic facts, checks entailment in both directions, and reports precision, recall, and F1; the paper shows this fact-level bidirectional approach…","keywords":["Bi-Fact","intent extraction","UI trajectories","atomic facts","bidirectional entailment","fact-level evaluation","LLM evaluation","GUI intent"],"falsifier":"Collect a set of intent pairs where human judges agree the two intents would achieve the same outcome but under Bi-Fact some gold fact is not entailed by the predicted intent (or vice versa); if such mismatches are common and Bi-Fact labels the pair as a non-match, that would show the entailment-of-facts equivalence does not track the human notion of task equivalence. A concrete version: take the 36 manually annotated pairs and check explicitly for cases where humans call the intents equivalent yet Bi-Fact's recall or precision is below 1.","tokens_in":7130,"feed_emoji":"🧩","tokens_out":7500,"duration_ms":58497,"temperature":0.7,"pith_summary":"The paper proposes Bi-Fact, an automated metric for evaluating how well a system extracts user intent from GUI interaction trajectories. Rather than comparing whole sentences, Bi-Fact splits each gold and predicted intent into atomic facts—single indivisible pieces such as \"destination is Paris\"—and checks in both directions whether each fact is entailed by the other intent. The paper argues this gives separate precision and recall, so a prediction that gets some facts right and others wrong is scored accordingly. On a benchmark of 400 gold-predicted intent pairs, Bi-Fact achieves the highest F1 (0.722) and kappa (0.508) among the compared metrics; on 36 manually fact-annotated pairs, its F1 correlates with human scores at Pearson 0.781. If the result holds, Bi-Fact gives intent-extraction research a fine-grained, automated evaluation that tracks human judgments more closely than existing measures.","feed_headline":"Bi-Fact beats prior metrics on UI intent evaluation","feed_subtitle":"Fact-by-fact bidirectional comparison aligns with human judgments better than lexical or NLI baselines.","key_machinery":"The load-bearing object is the atomic fact plus the bidirectional entailment comparison. An atomic fact is defined as a single indivisible piece of information—an action, an object, or a property—and intents are decomposed into such facts. Bi-Fact then tests support in each direction: \"is this gold fact entailed by the predicted intent as a whole?\" and \"is this predicted fact entailed by the gold intent as a whole?\" The ratio of supported gold facts gives recall, the ratio of supported predicted facts gives precision, and their harmonic mean gives F1. A large language model performs both the decomposition (with the gold decomposition fixed beforehand) and the entailment judgments, each fact labeled as covered (C) or missing (M) with a short reasoning string.","core_discovery":"The central discovery is that intent equivalence in the UI setting is best judged at the level of atomic facts rather than at the level of whole sentences. Bi-Fact operationalizes this by decomposing both intents into facts and running a bidirectional textual-entailment check: every gold fact must be implied by the predicted intent (recall), and every predicted fact must be implied by the gold intent (precision), with F1 as the summary. The paper reports that this fact-level bidirectional design outperforms lexical, embedding, and NLI-based baselines on binary intent-match (F1 0.722 vs. next best 0.669), and that its automatic fact-level scores strongly correlate with human fact-level annotations (Pearson 0.781). The implied message is that fine-grained factual completeness and correctness, not global text similarity, is what makes an extracted intent useful.","pith_inferences":["The bidirectional fact-entailment design could transfer to other structured prediction tasks where precision and recall of attributes matter, such as slot filling or information extraction, even outside UI trajectories.","Bi-Fact's scores inherit the quality of the underlying LLM's entailment reasoning; as LLM judgment improves or changes, the metric's agreement with humans would shift, so a version that calibrates entailment confidence rather than using a binary label might be more stable.","The 36-pair manual fact-level validation is small, so a natural next check is a larger fact-level annotation study; if it reproduces a Pearson correlation near 0.78, the fact-level claims would be on firmer ground.","Because the gold intents are factorized once and fixed, the metric is stable across predicted intents, but this also means errors in the gold factorization propagate into all subsequent scores."],"forward_implications":["Intent-extraction models can be ranked by fine-grained precision and recall, not just by whole-sentence agreement, so partial credit is given for correctly extracted facts.","Downstream systems can use Bi-Fact's per-fact labels to identify which specific facts are missing or wrong in a prediction.","Because Bi-Fact is fully automated with an LLM, it can be applied at scale to new model outputs without additional human annotation.","The high correlation with manual fact-level annotations suggests Bi-Fact can serve as a proxy for human fact-level scoring when manual annotation is expensive.","The comparison results imply that lexical-overlap metrics (BLEU, ROUGE, METEOR) and sentence-level NLI are insufficient for this task."],"supporting_citations":[{"why":"Supplies the Intent-Match dataset (100 examples, 400 gold-predicted pairs) and the AutoRater baseline against which Bi-Fact is compared.","marker":"(Berkovitch et al., 2024)"},{"why":"Mind2Web is one of the two source datasets from which the intent-match examples are sampled.","marker":"(Deng et al., 2024)"},{"why":"AitW is the other source dataset providing intent-match examples.","marker":"(Rawles et al., 2024)"},{"why":"BLEU serves as a lexical-overlap baseline metric in the comparison.","marker":"(Papineni et al., 2002)"},{"why":"ROUGE serves as a lexical-overlap baseline metric in the comparison.","marker":"(Lin, 2004)"},{"why":"METEOR serves as a lexical-overlap baseline metric in the comparison.","marker":"(Banerjee and Lavie, 2005)"},{"why":"Sentence-T5 cosine similarity is the embedding-based baseline that achieves the second-best F1 (0.669) on intent-match.","marker":"(Ni et al., 2021)"},{"why":"Provides the NLI-based baseline (TRUE) that is the second-best on fact-level correlation (0.517).","marker":"(Honovich et al., 2022)"},{"why":"FactScore is the precision-oriented fact-level evaluation method that Bi-Fact extends toward a bidirectional precision-recall design.","marker":"(Min et al., 2023)"},{"why":"The Pyramid method establishes the recall-oriented fact-level evaluation lineage that Bi-Fact builds on.","marker":"(Nenkova and Passonneau, 2004)"}],"fun_headline_variants":["Bi-Fact beats baselines on UI intent evaluation","Fact-level bidirectional scoring for UI intents","Atomic facts, not sentences, judge UI intents","Bi-Fact flips UI intent metrics with fact checks","UI intent precision/recall via bidirectional entailment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's central assumption is that two intents describe the same task exactly when every atomic fact of one is entailed by the other as a whole, so if the fact decomposition or the entailment judgments miss implicit user intentions, the scores will drift from true intent-match quality.","fun_headline_variants_meta":{"raw":{"variants":["Bi-Fact beats baselines on UI intent evaluation","Fact-level bidirectional scoring for UI intents","Atomic facts, not sentences, judge UI intents","Bi-Fact flips UI intent metrics with fact checks","UI intent precision/recall via bidirectional entailment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":1040,"prompt_tokens":757,"completion_tokens":283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":373,"completion_tokens_details":{"reasoning_tokens":208}},"tokens_in":373,"tokens_out":283,"duration_ms":3225,"temperature":1.0,"reasoning_tokens":208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:24:32.628167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of intent pairs where human judges agree the two intents would achieve the same outcome but under Bi-Fact some gold fact is not entailed by the predicted intent (or vice versa); if such mismatches are common and Bi-Fact labels the pair as a non-match, that would show the entailment-of-facts equivalence does not track the human notion of task equivalence. A concrete version: take the 36 manually annotated pairs and check explicitly for cases where humans call the intents equivalent yet Bi-Fact's recall or precision is below 1.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ROUGE serves as a lexical-overlap baseline metric in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Pyramid method establishes the recall-oriented fact-level evaluation lineage that Bi-Fact builds on."}],"review_version":1}