Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories

Anatoly Efros; Danielle Cohen; Ido Dagan; Noam Kahlon; Sapir Caduri; Yoni Halpern

REVIEW 4 major objections 4 minor 22 references

Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bi-Fact is an automated evaluation metric that decomposes gold and predicted intents into atomic facts, checks entailment in both directions, and reports precision, recall, and F1; the paper shows this fact-level bidirectional approach…

desk verdict Useful idea, but the gold-conditioned decomposition biases precision and the stats are too thin to support the headline. read the letter →

arxiv 2502.13149 v3 pith:SJBMGG44 submitted 2025-02-11 cs.AI

Sapir Caduri , Anatoly Efros , Noam Kahlon , Danielle Cohen , Yoni Halpern , Ido Dagan This is my paper · ORCID

classification cs.AI

keywords Bi-FactintentextractionUItrajectoriesatomicfactsbidirectionalentailmentfact-levelevaluationLLMGUI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Bi-Fact, an automated metric for evaluating how well a system extracts user intent from GUI interaction trajectories. Rather than comparing whole sentences, Bi-Fact splits each gold and predicted intent into atomic facts—single indivisible pieces such as "destination is Paris"—and checks in both directions whether each fact is entailed by the other intent. The paper argues this gives separate precision and recall, so a prediction that gets some facts right and others wrong is scored accordingly. On a benchmark of 400 gold-predicted intent pairs, Bi-Fact achieves the highest F1 (0.722) and kappa (0.508) among the compared metrics; on 36 manually fact-annotated pairs, its F1 correlates with human scores at Pearson 0.781. If the result holds, Bi-Fact gives intent-extraction research a fine-grained, automated evaluation that tracks human judgments more closely than existing measures.

What carries the argument

The load-bearing object is the atomic fact plus the bidirectional entailment comparison. An atomic fact is defined as a single indivisible piece of information—an action, an object, or a property—and intents are decomposed into such facts. Bi-Fact then tests support in each direction: "is this gold fact entailed by the predicted intent as a whole?" and "is this predicted fact entailed by the gold intent as a whole?" The ratio of supported gold facts gives recall, the ratio of supported predicted facts gives precision, and their harmonic mean gives F1. A large language model performs both the decomposition (with the gold decomposition fixed beforehand) and the entailment judgments, each fact labeled as covered (C) or missing (M) with a short reasoning string.

What would settle it

Collect a set of intent pairs where human judges agree the two intents would achieve the same outcome but under Bi-Fact some gold fact is not entailed by the predicted intent (or vice versa); if such mismatches are common and Bi-Fact labels the pair as a non-match, that would show the entailment-of-facts equivalence does not track the human notion of task equivalence. A concrete version: take the 36 manually annotated pairs and check explicitly for cases where humans call the intents equivalent yet Bi-Fact's recall or precision is below 1.

Watch

Extended reading notes

Core claim

The central discovery is that intent equivalence in the UI setting is best judged at the level of atomic facts rather than at the level of whole sentences. Bi-Fact operationalizes this by decomposing both intents into facts and running a bidirectional textual-entailment check: every gold fact must be implied by the predicted intent (recall), and every predicted fact must be implied by the gold intent (precision), with F1 as the summary. The paper reports that this fact-level bidirectional design outperforms lexical, embedding, and NLI-based baselines on binary intent-match (F1 0.722 vs. next best 0.669), and that its automatic fact-level scores strongly correlate with human fact-level annotations (Pearson 0.781). The implied message is that fine-grained factual completeness and correctness, not global text similarity, is what makes an extracted intent useful.

Load-bearing premise

The method's central assumption is that two intents describe the same task exactly when every atomic fact of one is entailed by the other as a whole, so if the fact decomposition or the entailment judgments miss implicit user intentions, the scores will drift from true intent-match quality.

Editorial extensions

If this is right

Intent-extraction models can be ranked by fine-grained precision and recall, not just by whole-sentence agreement, so partial credit is given for correctly extracted facts.
Downstream systems can use Bi-Fact's per-fact labels to identify which specific facts are missing or wrong in a prediction.
Because Bi-Fact is fully automated with an LLM, it can be applied at scale to new model outputs without additional human annotation.
The high correlation with manual fact-level annotations suggests Bi-Fact can serve as a proxy for human fact-level scoring when manual annotation is expensive.
The comparison results imply that lexical-overlap metrics (BLEU, ROUGE, METEOR) and sentence-level NLI are insufficient for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The bidirectional fact-entailment design could transfer to other structured prediction tasks where precision and recall of attributes matter, such as slot filling or information extraction, even outside UI trajectories.
Bi-Fact's scores inherit the quality of the underlying LLM's entailment reasoning; as LLM judgment improves or changes, the metric's agreement with humans would shift, so a version that calibrates entailment confidence rather than using a binary label might be more stable.
The 36-pair manual fact-level validation is small, so a natural next check is a larger fact-level annotation study; if it reproduces a Pearson correlation near 0.78, the fact-level claims would be on firmer ground.
Because the gold intents are factorized once and fixed, the metric is stable across predicted intents, but this also means errors in the gold factorization propagate into all subsequent scores.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper has a genuinely useful idea—evaluate intent extraction by decomposing gold and predicted intents into atomic facts and doing bidirectional entailment—but the main empirical claim is weakened by a design choice that biases the metric, and by thin statistics. I'd send it to review, but with the expectation of substantial revision.

The new part is the application of fact-level decomposition to UI intent comparison, with both precision and recall. That is a natural fit and fills a real gap: lexical and sentence-level NLI metrics miss fine-grained factual mismatches like wrong dates or missing business-class. The protocol is clearly described, and the comparison against BLEU, ROUGE, NLI, etc. is appropriate. The fact-level validation, while small, is a good idea.

The soft spot is in Stage 1 of the assessment prompt. The LLM decomposes the predicted intent while seeing the gold facts and is told to do it 'in accordance with' that list. That can filter out predicted facts that have no gold counterpart, inflating precision. The 'prerequisite counts as covered' rule for recall also loosens the comparison. The paper doesn't test an independent decomposition, so we don't know how much of the reported F1 (0.722) and Kappa (0.508) comes from this lenient fact recovery rather than genuine semantic agreement. This is load-bearing, not a footnote.

The statistics are also thin. Table 1 has no confidence intervals or significance tests across the 400 pairs; the F1 gap over NLI (0.722 vs 0.614) could easily be noise. The fact-level correlation (0.781, p<0.001) rests on 36 pairs with no inter-annotator agreement reported. And the LLM judge is unnamed, with no code or data release, which makes the result hard to reproduce.

The citation pattern is fine; the paper fairly builds on FactScore, Pyramid, and NLI work.

Bottom line: the core idea is coherent and worth pursuing, but the current evidence doesn't support the headline claim as stated. The metric's bias needs to be addressed—either by neutralizing the decomposition or by an ablation comparing gold-conditioned vs. independent factorization—and the statistics need to be tightened. This is a solid candidate for peer review after that. I would not cite it in its current form, but I'd watch the revised version.

Referee Report

4 major / 4 minor

Summary. The paper proposes Bi-Fact, a fact-level metric for evaluating predicted natural-language intents extracted from GUI trajectories. The method decomposes gold and predicted intents into atomic facts and uses an LLM to assess, bidirectionally, whether each fact of one intent is entailed by the other intent as a whole, yielding precision, recall, and F1. The metric is validated on 400 human-annotated gold–predicted intent pairs with binary task-equivalence judgments, where a binarized version of Bi-Fact achieves the highest F1 (0.722) and Cohen's kappa (0.508) over BLEU, ROUGE, METEOR, sentence T5 similarity, bidirectional NLI, and AutoRater; and on 36 manually factorized intent pairs, where Bi-Fact's F1 has a Pearson correlation of 0.781 with manual fact-level F1, outperforming NLI (0.517).

Significance. If valid, Bi-Fact would be a genuinely useful evaluation method for intent extraction from UI trajectories: it is more informative than sentence-level overlap or NLI because it produces fact-level feedback on which parts of an intent are correct or missing, and it is validated against human judgment in a task-specific, outcome-based sense of intent equivalence. Evidence for the claimed advantage is presented in the form of correlation with human annotations rather than only illustrative examples, and the checklists and prompts are included in the appendix, which strengthens reproducibility. The central weakness is that the evidence base is small (400 binary pairs, 36 fact-level pairs), no significance tests or confidence intervals are reported for Table 1, and the automatic gold-conditioned decomposition of predictions creates a risk that precision is inflated; the paper would need a stronger, more controlled validation to support the strongest claims of superiority.

major comments (4)

[Section 2.2, Stage 1 / Prompt in Figure 3] The automatic evaluation is vulnerable to a gold-conditioning bias in the decomposition of the predicted intent. In Stage 1, the predicted sentence is decomposed 'in accordance with' the fixed gold fact list, and the only worked example in Figures 3–5 shows the predicted facts being produced with the expert facts visible. This setup can suppress predicted facts that have no counterpart in the gold decomposition, so precision is computed over a filtered set rather than over all facts actually present in the prediction. Because the headline binary-intent-match F1 (0.722) and kappa (0.508) depend on a precision computed from this filtered set, the metric may be measuring fact recovery under a gold-aligned decomposition rather than faithful fact-level agreement. The 36-pair manual validation does not rule this out unless the manual factorization of predicted intents was performed independently of the gold facts; the paper does not state that it was, and no LLM-ablation or inter-annotator agreement is reported. The authors should either justify that the gold-conditioned decomposition does not systematically omit spurious predicted facts, or modify the protocol so that prediction decomposition is independent of the gold facts (e.g., decomposed once, before the gold facts are shown), and re-run the validation.
[Section 3.2, Table 1] Table 1 reports point estimates of precision, recall, F1, and kappa over 400 intent pairs, but no confidence intervals or significance tests are given for the comparison of Bi-Fact (F1 0.722, kappa 0.508) against the second-best metric (NLI: F1 0.614, kappa 0.467; T5-sim F1 0.669). With 400 binary decisions, the difference in F1 between 0.722 and 0.669 could easily be within sampling error, and the kappa differences are also modest. The claim that Bi-Fact 'notably achieves the highest F1 and Kappa ... indicating better agreement' needs either a paired significance test (e.g., McNemar or bootstrap with a defined resampling unit) or confidence intervals on the aggregate, and the thresholds should be acknowledged as selected on the dev set; otherwise the central claim of superiority over existing metrics is not statistically established.
[Section 3.3 and Section 3.1 'Fact-Level Data'] The fact-level validation rests on 36 intent pairs. The Pearson correlation of 0.781 (p < 0.001) is reported, but with n=36 and no confidence interval the precision of the correlation estimate is quite low, and the comparison against NLI (r=0.517) is likewise reported without an interval or a test of the difference between the two dependent correlations. More importantly, the manual annotations were produced by the authors using the same atomic-fact decomposition and entailment notion that defines Bi-Fact, so the validation is partly a self-consistency check of the protocol. The authors should report the manual annotation procedure in full (who annotated, whether predicted intents were factorized with or without the gold facts visible, whether multiple annotators labeled the entailment decisions, and inter-annotator agreement), and provide a confidence interval or bootstrap for the correlation.
[Section 2.1 / Overall design assumption] The core assumption that two intents are equivalent exactly when each atomic fact of one is entailed by the other as a whole, with 'atomic facts' as the representational units, is reasonable for the UI domain but is not independently established. The paper's example (book a business-class flight to Paris vs. book a flight to Paris) shows how fact-level scores can be more informative than a binary judge, but nothing in the 400-pair or 36-pair evaluations tests how robust the atomic-fact decomposition is to paraphrase variation or to implicit intent that is not lexically realized. The authors should at least discuss or ablate the decomposition granularity (e.g., compare against a baseline that uses gold facts only for recall and an independently produced fact list for precision) and report the LLM's decomposition consistency.

minor comments (4)

[Section 2.2, Stage 2] The phrase 'can is implied' should read 'is implied'.
[Section 1] There are typographical and formatting errors: 'evaluation methods, such as Pyramid ... or FactScore ... offer a more granular approach' contains an orphaned quotation mark and stray punctuation ('Harnly et al., 2005)„ '); these should be cleaned up.
[Section 3.2] The threshold-tuning procedure (30 equally spaced increments on the dev set) is described, but the actual chosen threshold values for each metric are not reported; reporting them would make the comparison reproducible.
[Appendix A, Figure 3] The prompt example in Figure 5 contains a JSON syntax error: the third entry in 'predicted_fact_accuracy' is preceded by a missing comma after the previous object's closing brace. Since this prompt is central to the method, the one-shot example should be syntactically correct.

Circularity Check

1 steps flagged · score 6.0 of 10

Bi-Fact's precision is partially circular: Stage 1 constructs the predicted fact set from the gold fact set, so the reported F1 and Kappa are not an independent evaluation of the predicted intent.

self definitional [Section 2.2, Stage 1 (Factoring the predicted intent) and Stage 3 (Assessing the predicted facts); Appendix A, Figure 3 prompt.]
"As an additional input, the LLM receives the decomposition of the gold intent (as fixed in Part 1), and is instructed to decompose the predicted intent in accordance with the given decomposition of the gold intent, to increase their consistency for comparison. ... Precision is then measured as the proportion of implied facts among the predicted facts."

The predicted fact set P is generated with the gold fact set G as a direct input and with an instruction to align P with G. Precision is defined as the fraction of P entailed by G. Because P is constructed to match G's decomposition, spurious predicted facts that have no gold counterpart are likely suppressed before precision is computed. The reported F1 (0.722) and Kappa (0.508), and the binarized intent-match decisions derived from them, therefore measure a gold-conditioned reconstruction rather than an independent evaluation of the predicted intent. The evaluation input is defined in terms of the reference standard it is supposed to assess.

full rationale

The central claim that Bi-Fact best agrees with human judgments rests on a pipeline whose Stage 1 feeds the gold fact decomposition into the LLM and instructs it to decompose the predicted intent in accordance with that gold decomposition. This makes the predicted fact set a function of the gold facts, so the subsequent precision and F1 scores are partially forced by construction rather than being an independent measurement of the predicted intent. The fact-level validation on 36 manually annotated pairs does not repair this, because the paper does not report whether the manual predicted-intent factorization was blind to gold facts; as written, the 0.781 correlation is compatible with the automatic method agreeing with annotations that share the same gold-conditioned bias. The threshold tuning on the dev set is standard practice and is not circular. There is no load-bearing self-citation or imported uniqueness theorem: the dataset from Berkovitch et al. (2024) and the AutoRater baseline are external comparison points, not arguments that force Bi-Fact's result. Because the headline scores are partly determined by the gold-conditioned construction of the predicted facts, the paper exhibits partial circularity rather than a fully independent derivation.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The main fitted quantity is the binarization threshold tuned on a 10% development split. The method itself makes two domain assumptions: atomic fact decomposition captures task-relevant meaning, and the LLM's entailment judgments are consistent with human judgments. These are validated only on 36 fact-level pairs, so the evidence for them is limited.

free parameters (1)

binarization threshold for Bi-Fact score = not reported; tuned over 30 increments on 10% dev split
Section 3.2: continuous scores are binarized using a threshold tuned on the development set; this parameter directly affects the F1 and kappa in Table 1.

assumptions (3)

domain assumption Atomic facts extracted from natural-language intents are the correct unit for comparing task equivalence.
Section 2.1 assumes decomposing intents into atomic facts and checking entailment measures whether the intended task outcome is the same.
domain assumption The LLM used as evaluator produces reliable and consistent factual decompositions and entailment judgments across runs.
The automatic Bi-Fact pipeline (Section 2.2) relies on an unspecified LLM for both factorization and entailment; no variance or model version is reported.
domain assumption Human annotations are accurate ground truth.
Section 3.1 treats the manually annotated binary judgments and fact-level annotations as gold labels; no inter-annotator agreement is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories." pith.science (2026). https://pith.science/paper/SJBMGG44

@misc{pith2026250213149,
  author       = {Pith},
  title        = {Pith review of: Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJBMGG44}},
  note         = {Machine review of arXiv:2502.13149}
}

read the original abstract

Evaluating intent extraction from GUIs demands accurate, fine-grained metrics. This paper introduces Bi-Fact, a novel method that decomposes intents into atomic facts and performs bidirectional comparisons to assess precision and recall. Experiments demonstrate Bi-Fact's superior correlation with human judgments compared to existing metrics, establishing a more robust evaluation framework for UI-driven intent understanding.

Figures

Figures reproduced from arXiv: 2502.13149 by the authors.

**Figure 2.** Example prompt for the fact decomposition task [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Assessment prompt part 1 - 3 step instructions [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Assessment prompt part 2 - Output Structure [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Assessment prompt part 3 - An Example (one-shot) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

2005
[4]

Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, and Ido Dagan. 2024. Identifying user goals from ui trajectories. arXiv preprint arXiv:2406.14314

arXiv 2024
[5]

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326

arXiv 2015
[6]

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177--190, Berlin, Heidelberg. Springer Berlin Heidelberg

2006
[7]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

2024
[8]

Yanjun Gao, Chen Sun, and Rebecca J Passonneau. 2019. Automated pyramid summarization evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

work page 2019

Show all 22 references

[9]

Aaron Harnly, Ani Nenkova, Rebecca Passonneau, and Owen Rambow. 2005. Automation of summary evaluation by the pyramid method. In Recent Advances in Natural Language Processing (RANLP), pages 226--232

2005
[10]

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. https://doi.org/10.18653/v1/2022.naacl-main.287 TRUE : Re-evaluating factual consistency evaluation . In Procee...

2022 doi
[11]

Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic macro mining from interaction traces at scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1--16

2024
[12]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL Workshop, pages 74--81

2004
[13]

Antonio Mart \' nez-Rojas, A Jim \'e nez-Ram \' rez, Jos \'e Gonzalez Enr \' quez, and Hajo A Reijers. 2024. A screenshot-based task mining framework for disclosing the drivers behind variable human actions. Information Systems, 121:102340

2024
[14]

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251

2023 arXiv
[15]

Ani Nenkova and Rebecca J Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the human language technology conference of the north american chapter of the association for computational linguistics: Hlt-naacl 2004, pages 145--152

2004
[16]

Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. 2021. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877

2021 arXiv
[17]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[18]

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2024. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36

2024
[19]

Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. 2024. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723

2024 arXiv
[20]

Qian Yang, Rebecca Passonneau, and Gerard De Melo. 2016. Peak: Pyramid evaluation via automated knowledge extraction. In Proceedings of the AAAI conference on artificial intelligence, volume 30

2016
[21]

Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. 2024. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256

2024 arXiv
[22]

Guanhua Zhang, Mohamed Ahmed, Zhiming Hu, and Andreas Bulling. 2024. Summact: Uncovering user intentions through interactive behaviour summarisation. arXiv preprint arXiv:2410.08356

2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

2005

[4] [4]

Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, and Ido Dagan. 2024. Identifying user goals from ui trajectories. arXiv preprint arXiv:2406.14314

arXiv 2024

[5] [5]

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326

arXiv 2015

[6] [6]

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177--190, Berlin, Heidelberg. Springer Berlin Heidelberg

2006

[7] [7]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

2024

[8] [8]

Yanjun Gao, Chen Sun, and Rebecca J Passonneau. 2019. Automated pyramid summarization evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

work page 2019