SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View

Hongru Liang; Peixin Qin; Wenqiang Lei; Yao Zhang; Yongjie Xiao

REVIEW 4 major objections 6 minor 25 references

SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No tested LLM operates an expert-level comprehension process; right answers often rest on wrong supporting sentences.

desk verdict A useful process-level evaluation benchmark with a clear central finding and one load-bearing gap: no human expert baseline, so the 'far from expert' claim is about a posited ideal rather than a measured one. read the letter →

arxiv 2506.05000 v1 pith:WSWFNNNW submitted 2025-06-05 cs.CL

Yongjie Xiao , Hongru Liang , Peixin Qin , Yao Zhang , Wenqiang Lei This is my paper

classification cs.CL

keywords readingcomprehensionlargelanguagemodelsprocesscognitiveevaluationfive-skilldecompositionsupportingsentencesbenchmarkconstructionlocalversusglobal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the right way to judge reading comprehension in language models is to evaluate the comprehension process itself, not just the final answer. It defines that process as five skills — locating, inferring, connecting, organizing, and selecting — arranged from local to global levels, and builds a 4,682-item test set to probe each one. Results from two open and two closed LLMs show that none operates an expert-level comprehension process: accuracy falls from about 93.55% at locating to 37.38% at inferring and 31.02% at interpreting. The paper also shows that models frequently answer correctly while selecting the wrong supporting sentences, and that supplying the right supporting sentences improves inferring accuracy by 8–13 percentage points. A sympathetic reader would take this as evidence that answer-only benchmarks overstate comprehension and that training should explicitly develop all five skills.

What carries the argument

The carrying mechanism is SCOP's five-skill decomposition of the comprehension process into locating (one supporting sentence), inferring (multiple supporting sentences), connecting (sentence cloze), organizing (subheading placement), and selecting (key-sentence extraction), grounded in a three-level cognitive taxonomy from local to global comprehension. Each skill gets a task whose golden labels are constructed by automatic pipelines — z-score semantic retrieval over declarative sentences for locating labels, syntactic decomposition plus retrieval for inferring labels — and a 4,682-sample test set spanning narrative and expository documents with span, multi-choice, and ordering answer styles. The evaluation compares two open and two closed LLMs at temperature zero on document-level exact-match accuracy for the process predictions.

What would settle it

Give the same 4,682 test items to expert human readers with the same five tasks and compare their process accuracy to the LLMs' scores. If human experts also fall far below the near-ceiling accuracy implied by the gold labels — for instance, disagreeing with a large share of the auto-generated supporting sentences — then the reported LLM-expert gap is an artifact of the labels rather than a property of the models.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that the comprehension process of LLMs can be decomposed into five cognitive skills, and that evaluating those skills reveals a large gap from expert-level comprehension: all tested LLMs are far from operating expert-level comprehension processes, they are better at local than global comprehension, and they are inconsistent — they can produce correct answers on the basis of incorrect supporting sentences. The paper further claims that a correct comprehension process causally helps: when the golden supporting sentences are provided, inferring accuracy rises by 8.03 to 13.21 percentage points across models. This is presented as evidence that matching answers alone cannot reliably judge comprehension and that improving LLMs requires developing all comprehension skills during training.

Load-bearing premise

The whole benchmark depends on the assumption that the automatically generated supporting-sentence labels and the filtered questions faithfully represent what expert readers would judge as the comprehension process; if those labels are noisy or the filtering removes ordinary comprehension questions, the scores measure the label generator rather than expert-like comprehension.

Editorial extensions

If this is right

Benchmarks that score only final answers overstate LLM comprehension; inconsistency scores between supporting sentences and correct answers range from 2.32% to 4.77% at the locating level and 12.80% to 22.50% at the inferring level.
Scaling does not automatically close the process gap: open-sourced Llama3.1-70B outperforms GPT-4o on inferring and Qwen2-72B outperforms GPT-4o on connecting, so larger closed models are not uniformly better at the comprehension process.
Training signal should be placed on intermediate comprehension skills: giving models the correct supporting sentences raises inferring accuracy by 8.03 to 13.21 percentage points.
Because more than 90% of source questions (33,827 of 37,023) were discarded as answerable from memory, contamination-aware filtering is a necessary part of any process-based evaluation.
The local-over-global gradient mirrors human reading patterns, suggesting that current next-token pretraining may cultivate local skills while leaving global integration weak.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the process decomposition is adopted as a training objective, one testable prediction is that curriculum training ordered locating → inferring → interpreting will transfer to held-out answer-only benchmarks better than equal-weight multitask training, because the paper observes that inferring builds on locating.
The inconsistency metric could be repurposed as a calibration or hallucination probe: models with high inconsistency at the inferring level may be the most likely to give confident but unsupported answers in open-domain settings.
The paper's finding that supplying golden supporting sentences boosts accuracy suggests a concrete extension: measuring whether retrieval-augmented generation, which supplies candidate supporting sentences, closes part of the global-comprehension gap.
Because 90% of source questions were dropped as memorized, one could estimate how much of the remaining benchmark score measures training-data novelty rather than skill by perturbing named entities in documents and re-measuring locating accuracy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — SCOP is a genuine attempt to move LLM reading evaluation from final answers to the process: five skills (locating, inferring, connecting, organizing, selecting), a documented pipeline that filters existing QA and summarization data, and an evaluation of four LLMs. The most useful empirical result is Table 4: models often produce the correct answer while failing to identify the supporting sentence, especially at the inferring level. That supports the paper's central claim that answer-accuracy alone is not enough, and the F1 checks for the pseudo-labeling steps (0.94/0.91 for locating, 0.85/0.96 for inferring) give the construction some measurable credibility. The code/data release is a plus.

The soft spots are real, and the stress-test note names the main one: expert-level is never measured. There is no human expert baseline on the SCOP tasks, so "all LLMs are far from expert-level" is a claim about a theoretical ideal, not about a measured reference. The 90% filtering (33,827 of 37,023 removed as memorizable) may leave a set that is hard and non-representative; without human performance data we don't know whether the surviving questions are reasonable comprehension questions or just retrieval-resistant leftovers. The gold supporting sentences are mostly automatic — z-score retrieval, syntactic decomposition, GPT-4o-mini for RACE — and the human check is 50 samples per dataset with a validity score (0.81, 73% agreement) that does not verify each gold sentence's necessity or sufficiency. So the headline numbers rest on labels that are plausible but not fully validated.

The local-better-than-global finding is plausible, but it is partly confounded with task difficulty and dataset differences: locating tasks are sentence-level, inferring and interpreting tasks are inherently harder and come from different sources. The inconsistency finding is the strongest piece, but it too is measured against the automatic labels, so it is somewhat circular.

Minor issues: some test cells are very small (MCTest n=72), and exact-match sentence metrics are brittle. Prompt sensitivity is acknowledged in the limitations.

My overall take: this is a solid benchmark paper with one load-bearing gap. It deserves peer review — send it to a serious venue with the expectation that the authors add an expert baseline and stronger label validation. The direction is right and the process-level evaluation axis is useful to the field.

Referee Report

4 major / 6 minor

Summary. The paper introduces SCOP, a benchmark and evaluation suite that measures five comprehension-process skills (locating, inferring, connecting, organizing, selecting) of LLMs instead of only comparing final answers. The authors construct 4,682 test items from 12 datasets, filter out questions that can be answered without the document, and evaluate four LLMs. They report that all models are far from an expert-level comprehension process, that LLMs are better at local than global comprehension, and that models often reach correct answers while failing to identify supporting sentences. A public GitHub repository is provided.

Significance. If its measurement validity is established, SCOP would be a useful complement to answer-based benchmarks, with a well-documented data construction pipeline, public resources, and a thought-provoking finding about the dissociation between correct answers and supporting-sentence identification. The paper ships detailed construction details, pseudo-label F1 checks, and a small human validation study. However, the central claims currently outrun the evidence: there is no expert baseline, the gold labels are largely automatically generated, and the filtering step removes over 90% of source questions. With an expert baseline and label-quality validation, the benchmark could become a credible process-oriented evaluation tool.

major comments (4)

[§4.2 and §3.6] The central claim that 'all LLMs are far from operating expert-level comprehension processes' is not empirically calibrated. No human expert baseline is run on the SCOP tasks; the 'expert-level' standard is posited from cognitive theory and operationalized by the authors' own definitions. The human evaluation in §3.6 (50 samples per dataset, average validity 0.81, agreement 73%) checks whether a sample matches the task definition, not whether the gold supporting sentences are correct or whether human experts would answer these questions at ceiling. The claim therefore conflates 'below the authors' operationalization of expert process' with 'far from expert-level.' I recommend adding a human expert study on a representative sample of SCOP items, reporting per-skill accuracy and inconsistency scores, and using those numbers to calibrate the 'expert-level' threshold.
[§3.6] The filtering step removes 33,827 of 37,023 source questions (over 90%) because Llama3.1-70B and GPT-4o can answer them without the document. The surviving 4,682 items are therefore selected to be non-memorized by these two LLMs, but there is no evidence that they are representative of the comprehension process or that human experts would also be unable to answer the removed items from memory. This selection may systematically bias the benchmark toward artificially hard questions and inflate the local-vs-global gap if, for example, memorized local facts are easier to answer without the document. The authors should either provide an analysis of the filtered-out questions (e.g., human performance on a sample, skill distribution) or temper the generalizability claims accordingly.
[§3.1–§3.2 and Eq. (1)] The gold supporting-sentence labels are largely automatically generated: z-score retrieval with Eq. (1) and dataset-specific thresholds for locating; syntactic decomposition with a correlation threshold of 0.84 and GPT-4o-mini/BM25 retrieval for inferring. The F1 checks reported (0.94/0.91 for locating pseudo-labels, 0.85/0.96 for inferring supporting sentences) validate the retrieval procedures against existing annotations on small datasets, but they do not validate the final SCOP gold labels on the actual test items. The 0.81 validity score in §3.6 is an aggregate sample-level check, not a sentence-level label-quality measure. If the gold labels contain errors, the skill accuracies in Table 3 and the inconsistency scores in Table 4 (correct answer but wrong supporting sentence) become difficult to interpret. I recommend reporting per-dataset sentence-level human agreement on the gold supporting sentences, including the RACE subset where no annotated gold exists.
[§4.3, Figure 4] The framework validation relies on correlations of LLM performances across datasets (Figure 4). This is an internal consistency check and cannot establish that the five skills correspond to the expert comprehension process posited in §2. The correlations could arise from shared difficulty or document properties. To support the claim that SCOP measures expert-level comprehension skills, the paper needs external validation, e.g., human experts labeling which skill each question tests, or a human performance profile that matches the theory's prediction (local > global).

minor comments (6)

[§3.2] The text says the correlation threshold for subquestion pruning is 0.84, but footnote 4 says the threshold is 0.8; please reconcile the inconsistency.
[§4.2] The statement 'where open-sourced models generally outperform closed-sourced ones' appears to be backwards; typical evaluations show closed-sourced models outperform open-sourced ones, and the surrounding discussion implies this reversal.
[Table 3] The column header 'Orgnazing' is a typo and should read 'Organizing'.
[§3.6] Describing the 33,827 filtered questions as 'noise samples' is misleading; they are merely answerable without the document by the two filtering models, and may still be valid comprehension questions.
[§4.3] The phrase 'LLMs may be "slacking off"' is informal; consider replacing it with a neutral description such as 'using shortcuts that bypass the supporting-sentence identification process.'
[Table 3 and §4.2] Test-set sizes per cell are small (e.g., MCTest 72, ClimateCentral 108, SourceSum 143) and no confidence intervals or significance tests are reported; the paper should at least note the absence of uncertainty estimates when comparing model performances.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SCOP's findings are empirical measurements on a purpose-built benchmark; no equation reduces to its input and no load-bearing self-citation chain is present.

full rationale

SCOP's central claims are empirical accuracy measurements on a constructed benchmark, not derivations from fitted values or from the authors' prior results. The five-skill taxonomy is imported from external cognitive-science citations (Krathwohl 2002; Afflerbach et al. 2015; Kintsch and van Dijk 1978), and the paper does not invoke any uniqueness theorem or prior work by the same authors as load-bearing evidence. Gold labels for the locating skill are anchored in annotated spans in SQuAD v2.0 and NewsQA or in human multiple-choice annotations in MCTest; the z-score retrieval in Eq. 1 is used only to filter noisy questions and is itself validated against annotated spans (F1 0.94 on SQuAD v2.0 and 0.91 on NewsQA). For RACE inferring labels, the paper requires agreement between GPT-4o-mini retrieval and an independent semantic-similarity retrieval, so the labels are not generated by the four main evaluated models. GPT-4o is used as one of the two filtering models in Section 3.6 and is also evaluated in Table 3, but the filter removes questions answerable without the document; this makes the benchmark harder but does not force the reported low skill accuracies or the inconsistency scores. The statement that all LLMs are 'far from operating expert-level comprehension processes' is an interpretation of the measured accuracies and could in principle have come out differently; it is not logically entailed by the task definitions. Likewise, the local-vs-global ordering (93.55% locating, 37.38% inferring, 31.02% interpreting) is an empirical observation even though the skill names carry a local-to-global ordering; the numbers themselves are measured rather than deduced. The 90% filtering step reduces data contamination and does not encode the target conclusion. The paper's own limitation statement in Section 7 acknowledges that the five skills interact in complex ways and that confirming their relationships is difficult, which is a validity caveat rather than a circular reduction. No parameter is fitted and then renamed as a prediction, and no equation in the paper is equivalent by construction to an input. I therefore find no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a postulated taxonomy of comprehension skills, on treating supporting-sentence identification as the process measurement, and on aggressive filtering of source data. The free parameters are hand-set thresholds in the data construction pipeline. No external human expert baseline is provided, so the validity of the whole measurement depends on these assumptions.

free parameters (5)

z-score threshold for locating candidates = 1.0 (SQuAD v2.0), 3.0 (NewsQA)
Hand-set in Section 3.1 to decide which sentences count as supporting candidates; changes which questions enter the locating set.
subquestion redundancy correlation threshold = 0.84 (main text), 0.8 (footnote 4)
Section 3.2: candidates with correlation above the threshold are removed as repetitive; threshold choice affects inferring question identification.
distractor similarity threshold for connecting = cosine similarity < 0.6
Section 3.3: top-2 sampled candidate sentences with cosine similarity below 0.6 are kept as distractions; controls task difficulty.
minimum document length for locating = more than 4 sentences
Appendix A.4: shorter documents are excluded, which affects sample composition.
minimum subheading length for organizing = at least 4 words
Section 3.4: subheadings must have at least four words to serve as segmentation constraints.

assumptions (4)

domain assumption The five skills and three levels (locating, inferring, connecting, organizing, selecting) are the requisite skills of the comprehension process.
Adopted from cognitive and educational theory (Krathwohl 2002; Afflerbach et al. 2015) without independent validation on this benchmark; the paper's framework rests on this taxonomy.
domain assumption Identifying supporting sentences is a valid proxy for the comprehension process.
Section 3.1 and Section 4.2: locating and inferring scores are computed only from predicted supporting sentences, so the claim about comprehension process depends on this equivalence.
domain assumption Filtering out questions answerable without the document removes shortcuts without biasing the benchmark.
Section 3.6: 33,827 of 37,023 samples are discarded as memorized; the paper assumes the remaining 4,682 are the valid comprehension questions.
domain assumption Syntactic decomposition of questions into subquestions identifies genuine multi-sentence inferring questions.
Section 3.2: the Berkeley Neural Parser and correlation filtering assume that question structure tracks the number of supporting sentences needed.

invented entities (1)

SCOP five-skill comprehension taxonomy
purpose: Defines the measured comprehension process and the benchmark's scoring dimensions.
No external validation that these five skills are necessary or sufficient for expert comprehension; the only checks are a 50-sample-per-dataset human validity score (0.81) and inter-task correlations on the same models that generated the measurements.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View." pith.science (2026). https://pith.science/paper/WSWFNNNW

@misc{pith2026250605000,
  author       = {Pith},
  title        = {Pith review of: SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSWFNNNW}},
  note         = {Machine review of arXiv:2506.05000}
}

read the original abstract

Despite the great potential of large language models(LLMs) in machine comprehension, it is still disturbing to fully count on them in real-world scenarios. This is probably because there is no rational explanation for whether the comprehension process of LLMs is aligned with that of experts. In this paper, we propose SCOP to carefully examine how LLMs perform during the comprehension process from a cognitive view. Specifically, it is equipped with a systematical definition of five requisite skills during the comprehension process, a strict framework to construct testing data for these skills, and a detailed analysis of advanced open-sourced and closed-sourced LLMs using the testing data. With SCOP, we find that it is still challenging for LLMs to perform an expert-level comprehension process. Even so, we notice that LLMs share some similarities with experts, e.g., performing better at comprehending local information than global information. Further analysis reveals that LLMs can be somewhat unreliable -- they might reach correct answers through flawed comprehension processes. Based on SCOP, we suggest that one direction for improving LLMs is to focus more on the comprehension process, ensuring all comprehension skills are thoroughly developed during training.

Figures

Figures reproduced from arXiv: 2506.05000 by the authors.

**Figure 2.** The syntax trees of an inferring question (up [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Average accuracy of LLMs with and without [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Inter-task performance correlation. 5) Tasks at the interpreting level correlate with both locating and inferring skills, indicating that global comprehension relies on the skills required for local comprehension. These correlations across comprehension process skills …

**Figure 5.** Figure 5: The two solutions for identifying locating [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 22 canonical work pages

[1]

There’s lots of information about their daytime activities, but nothing about what they do at night,because we don’t have the equipment

work page
[2]

InProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7839–7859

To test machine comprehension, start by defin- ing comprehension. InProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7839–7859. Sarah H Eason, Lindsay F Goldberg, Katherine M Young, Megan C Geist, and Laurie E Cutting. 2012. Reader–text interactions: How differential text and question types influence cogniti...

arXiv 2012
[3]

In this work we aim at addressing these limitations and make the following contributions:

work page
[4]

A related task called discourse segmentation

work page
[5]

Naturally these results do not imply that hierarchical models should be disregarded

work page
[6]

Operation: Mindcrime

But this new “scope” will make really clear pictures, even on moonless nights, so we will be able to find out more about the ducks. Table 14: Example of an inferring question in RACE Document: Klaus Meine || Klaus Meine (born 25 May 1948) is a German vocalist, best known as the lead singer of the hard rock band Scorpions. He and guitarist Rudolf Schenker ...

work page 1948
[7]

Lance King (born November 23, 1962) is an American heavy metal vocalist specializing in melodic rock progressive and power metal

work page 1962
[8]

L”. Near the center of the “L

She is best known as former main vocalist of the South Korean girl group Kara. Table 15: Example of an inferring question in HotpotQA Document: West DeLand, Florida || West DeLand is a census-designated place (CDP) in V olusia County, Florida, United States. The population was 3,535 at the 2010 census. Kendall Green, Pompano Beach, Florida || Kendall Gree...

work page 2010

Show all 25 references

[9]

Hopefully , in few hours , we should be able to confirm the exact number of missing -LRB- people -RRB- ,

Since fewer than two of these sentences overlap with the ones chosen by the LLM, we exclude this sample from the inference questions. This case shows that the supporting sentences extracted by LLMs are influenced by their background knowl- edge. Therefore, we incorporate seman...
[10]

Born in Ocala, Florida, Hampton is the youngest of five children of a dentist father and a professional model mother
[11]

As of the 2013 census, its population, estimated by the United States Census Bureau, was 57,468, making it the 45th most populated city in Florida

Ocala is a city located in Northern Florida. As of the 2013 census, its population, estimated by the United States Census Bureau, was 57,468, making it the 45th most populated city in Florida. Table 16: Example of an inferring question in MusiQue C.3 Connecting Document: Confi...

2013
[12]

Annuities are rarely a good idea at the age 35 because of withdrawal restrictions

Find your advantages Answer: 2, 1, 4, 3, 6 Table 17: Example of a connecting data in SCDE Document: Text segmentation is a traditional NLP task that breaks up text into constituents, according to predefined requirements. It can be applied to documents, in which case the object...
[13]

Our experiments showed that all of our models improve the current state-of-the-art
[14]

These units, or segments, can be any structure of interest, such as paragraphs or sections
[17]

In Figure 1 we show one ex-ample of document segmentation from Wikipedia, on which the task is typically evaluated
[18]

Just cause you have a nice car doesn’t make you a good driver. Don’t let your son drive your Porsche!

Multiple neural approaches have been recently proposed for document and discourse segmentation. Answer: 2, 6, 4, 7, 3 Table 18: Example of a locating question in ACL OCL C.4 Organizing Document: (0) This summer’s relentless record heat has stuck around into fall. (1) The plane...

2003
[19]

That’s one lesson to take away from a video posted by YouTube user Richard Stewart showing a Porsche Cayman flying out of control as it speeds from a green light on Prince Edward Island in Canada
[20]

KHOU reports that police have not made the identity of the driver public but have said that a 31-year-old driver was cited for the crash, leaving the car a totaled as it was towed away
[21]

feelings

Finally the car comes to a halt, a total wreck waiting for the tow truck . Table 21: Example of a selecting data in sourcesum Document: Sentiment is personal; the same sentiment can be expressed in various ways and the same expression might carry distinct polarities across dif...
[22]

As a result, personalized sentiment analysis requires effective exploitation of users’ own opinionated data and efficient execution of model updates across all users
[23]

To address these challenges, we propose to build personalized sentiment classification models via shared model adaptation
[24]

The shared global model adaptation makes personalized model estimation no longer independent, such that regularity is formed across individualized learning tasks
[25]

supporting_sentence

We empirically evaluated the proposed solution on two large collections of reviews, i.e., Amazon and Yelp reviews. Table 22: Example of a selecting data in ACL OCL D Prompts for five skill evaluation D.1 Locating Type Prompt EQA Answer the provided question based on the given ...

1994
[2020]

InProceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5726–5735

Starc: Structured annotations for reading com- prehension. InProceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5726–5735. David Caplan and Nancy Hildebrandt. 1988.Disorders of syntactic comprehension. MIT Press. Patricia L Carrell...

1988 arXiv
[2021]

multiple

Benchmarking machine reading comprehen- sion: A psychological perspective. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1592–1612. Yoshi Suhara and Dimitris Alikaniotis. 2024. Source identification in abst...

2012 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

There’s lots of information about their daytime activities, but nothing about what they do at night,because we don’t have the equipment

work page

[2] [2]

InProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7839–7859

To test machine comprehension, start by defin- ing comprehension. InProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7839–7859. Sarah H Eason, Lindsay F Goldberg, Katherine M Young, Megan C Geist, and Laurie E Cutting. 2012. Reader–text interactions: How differential text and question types influence cogniti...

arXiv 2012

[3] [3]

In this work we aim at addressing these limitations and make the following contributions:

work page

[4] [4]

A related task called discourse segmentation

work page

[5] [5]

Naturally these results do not imply that hierarchical models should be disregarded

work page

[6] [6]

Operation: Mindcrime

But this new “scope” will make really clear pictures, even on moonless nights, so we will be able to find out more about the ducks. Table 14: Example of an inferring question in RACE Document: Klaus Meine || Klaus Meine (born 25 May 1948) is a German vocalist, best known as the lead singer of the hard rock band Scorpions. He and guitarist Rudolf Schenker ...

work page 1948

[7] [7]

Lance King (born November 23, 1962) is an American heavy metal vocalist specializing in melodic rock progressive and power metal

work page 1962

[8] [8]

L”. Near the center of the “L

She is best known as former main vocalist of the South Korean girl group Kara. Table 15: Example of an inferring question in HotpotQA Document: West DeLand, Florida || West DeLand is a census-designated place (CDP) in V olusia County, Florida, United States. The population was 3,535 at the 2010 census. Kendall Green, Pompano Beach, Florida || Kendall Gree...

work page 2010