{"id":"0f06ec81-3d5a-421b-a0a6-5d54aba95b7e","arxiv_id":"2505.03481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-stage hotel-review summarizer that predicts the target summary's sentence embedding to select three source sentences, then generates the description abstractively, slightly outperforming 2018-era baselines on USEG.","lead":"This paper proposes USESUM, a two-stage system that first selects three hotel review sentences using Universal Sentence Encoder embeddings and then generates a one-sentence description with a Transformer. On the USEG dataset it reports small gains over older extractive baselines, but those gains are not significance-tested and a human evaluation shows only 60 percent grammatical correctness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of embedding targets for content selection rests on an unvalidated angle-to-information monotonicity, and the end-to-end comparison is confounded by training the abstractive model on embedding-selected oracle sentences.","rationale":"I agree with the reader that the weakest assumption is the semantic validity of USE angle as an information-gain measure. The paper's own Sec. 6.5 limitation is decisive evidence that this assumption is at least questionable: the authors note that same angles can correspond to very different locations in embedding space and that cosine similarity can reward averaging, which is precisely what the BASELINE does. Since the extractive model's loss is a difference of arcus cosines to the target, and inference selects sentences by cosine to the predicted embedding, the entire contribution hinges on this monotonicity. The paper provides no auxiliary validation (e.g., correlation between embedding proximity and human judgments or oracle informativeness) for the USEG domain. The additional confound I identified is that the abstractive component is trained on oracle sentences selected by the same embedding-angle criterion used by USESUM. A 'common abstractive model' is only fair if it has not been tuned to the USESUM selection distribution; as described, the other extractive systems are evaluated under distribution shift. This does not disprove the central claim, but it weakens the inference from Table 2 to the abstract's general statement. The paper's use of 'prove' is an overstatement: the evidence is a single dataset, no statistical tests, and the effect sizes are small. A conditional acceptance remains appropriate: the central idea is interesting, the code is released, but the verdict should require the above control experiments and a significance analysis before the claim is accepted.","tokens_in":10193,"tokens_out":5840,"duration_ms":57560,"concrete_test":"Train the identical extractive architecture with a word-level sentence-selection target (e.g., NEUSUM-style binary labels derived from ROUGE alignment) in place of the embedding-regression loss, keeping the downstream abstractive model and candidate-selection heuristics fixed; if USESUM no longer beats this probability-target extractor on BLEU/ROUGE-L/METEOR, the claimed advantage of the embedding target is not established. Additionally, train a second abstractive model on oracle sentences selected by ROUGE overlap rather than by embedding angle, and compare end-to-end quality; if ROUGE-oracle selections yield equal or better summaries, the embedding-based selection criterion is not necessary. On the same USEG training split, compute the rank correlation between cosine-to-target and human-provided informativeness scores; a weak or negative correlation would confirm that Eq.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that regressing on summary sentence embeddings (USESUM) yields better content selection than predicting sentence-selection probabilities on low-overlap corpora. The mechanism is the extractive training objective in Eq. 11, loss_i = acos(hat y_{i-1}, target) - acos(hat y_i, target), and the selection rule in Sec. 4.3: pick the three input sentences whose USE embeddings have the smallest angle to the predicted summary embedding. This requires that reducing angle to the target embedding be a monotone proxy for accumulating summary-relevant information. The paper does not validate this monotonicity, and Sec. 6.5 concedes that angle similarity can be misleading and can 'overly promote' predictions that simply average document vectors (the BASELINE). If angle-to-target can be reduced by averaging rather than by identifying informative sentences, the extractive signal is not a reliable content-selection objective, and the third of summaries with no aligned review content (Sec. 3) provides no check on this. The empirical comparison is also not a clean test: the abstractive model was trained on three sentences selected by embedding angle to the target summary (Sec. 4.3), so feeding the same model with the outputs of NEUSUM, BERTSUM, etc. may disadvantage those systems through distribution shift. The Table 2 BLEU/ROUGE-L/METEOR advantage of USESUM could therefore reflect the abstractive model's training selection rather than a generally better intermediate objective. The abstract's 'prove' claim is unsupported: no proof is given, and no significance tests are reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes USESUM, a two-stage summarisation system for hotel review data (USEG). In the first stage, an extractive LSTM with a gated update rule is trained to predict the Universal Sentence Encoder (USE) embedding of the target summary sentence, using as loss the change in arccos distance between the predicted embedding and the target embedding (Eq. 11). At inference, the three input sentences whose USE embeddings have the smallest angle to the predicted embedding are selected. In the second stage, an abstractive Transformer (customised OpenNMT with copy attention, focal loss, and beam-search entity promotion) generates the final summary from these three sentences. The paper compares USESUM against BASELINE, BERTSUM, NEUSUM, REFRESH, and BOTTOM-UP on USEG, reporting that USESUM achieves the highest BLEU, ROUGE-L, and METEOR in the end-to-end experiment (Table 2), with a human evaluation showing higher semantic coverage but much lower grammaticality than BOTTOM-UP. The authors argue that predicting sentence-level embeddings is a better intermediate objective than predicting sentence-selection probabilities when source and target have low lexical overlap.","tokens_in":10482,"tokens_out":5777,"duration_ms":55814,"significance":"If the results are correct, the idea of using pre-trained sentence embeddings as an intermediate regression target for content selection is a plausible and potentially useful contribution for summarisation tasks with large, loosely aligned inputs. The paper ships code and models, uses a real-world public dataset, and includes both automatic and human evaluation. The main scientific claim—that embedding prediction outperforms word-level selection probabilities for weakly aligned corpora—is interesting and testable. However, the current empirical support is not conclusive because of a design confound in the end-to-end comparison, the lack of statistical testing on very small margins, and the absence of direct validation of the angle-to-information monotonicity that the learning objective assumes. The strengths of code release and a non-standard dataset are appreciated, but the load-bearing evidence needs substantial strengthening before the central claim can be accepted.","major_comments":[{"comment":"The abstractive Transformer is trained on the three sentences selected by angle similarity to the target embedding (the oracle selection described in Section 4.3), but the end-to-end evaluation in Table 2 feeds this same model with sentences selected by NEUSUM, BERTSUM, REFRESH, and BASELINE. Those inputs come from a different distribution than the training data, so the observed BLEU, ROUGE-L, and METEOR advantage of USESUM may reflect the abstractive model's specialization to its training distribution rather than a general superiority of the embedding-prediction objective. A fair comparison would train a separate abstractive model for each extractive system, or train one abstractive model on a mixture of selections from all systems so that the extractive component is the only variable.","section":"Section 4.3 and Section 6.2"},{"comment":"The training signal in Eq. (11) assumes that decreasing the angle between the predicted embedding and the target embedding is a monotone proxy for accumulating summary-relevant information. Section 6.5 concedes that angle similarity can be misleading and can 'overly promote' the BASELINE, which simply averages document vectors. This concession directly undermines the central mechanism unless the monotonicity is validated on the USEG domain; the paper provides no oracle-style experiment showing that sentences selected by minimal angle to the target embedding yield higher end-to-end ROUGE than sentences selected by other scoring functions. The reader's strongest concern is that the extractive signal may reward embedding averaging rather than content selection.","section":"Section 4.2.1 and Section 6.5"},{"comment":"No significance tests, confidence intervals, or error bars are reported anywhere in the results. The BLEU difference between USESUM (0.0225) and NEUSUM (0.0208) is small in absolute terms, and the human evaluation covers only 100 outputs per system, with USESUM at 60% grammaticality versus 91% for BOTTOM-UP. Given the small automatic-metric margins and the manually tuned beam-search factors, the paper's claim of consistent superiority is not statistically supported. The authors should report variance or significance tests and discuss the grammar gap in the human evaluation more prominently.","section":"Table 2 and Section 6.3"},{"comment":"BERTSUM is restricted to 512 input tokens while the other systems process up to 800 sentences, so its low performance is partly an artifact of the input limit rather than a comparison of summarisation quality. The comparison should either adapt BERTSUM to handle the full input (as is done for NEUSUM) or report results on a subset of documents for which all models receive the same input length; otherwise the baseline is not on equal footing.","section":"Section 5, BERTSUM paragraph"},{"comment":"The extractive evaluation in Table 1 reports only the first selected sentence, whereas the abstractive component in Section 4.3 uses the top three sentences. To support the claim that the embedding-based selector is better for the end-to-end task, the paper should compare the quality of the three-sentence selections across extractors (e.g., ROUGE-L or METEOR of the selected set against the target summary) and show that the USESUM-selected set is indeed more informative. Without this, the connection between the extractive objective and the end-to-end result remains indirect.","section":"Section 6.1"}],"minor_comments":[{"comment":"There are several typos and notation inconsistencies: 'METOR' for METEOR in Section 5, 'NEOSUM' for NEUSUM in Table 2, 'week alignment' for weak alignment in Section 4.2.1, 'challange' in Section 3, and '0,5115' with a comma decimal in Table 2.","section":"Throughout"},{"comment":"The word 'prove' is too strong for an empirical study; 'demonstrate' or 'provide evidence that' would be more appropriate, since the paper presents experimental results rather than a mathematical proof.","section":"Abstract"},{"comment":"Dataset statistics are missing: the number of hotels, the train/validation/test split sizes, and the average number of reviews and sentences per hotel should be reported for reproducibility and context.","section":"Section 3"},{"comment":"The hyperparameters of the abstractive Transformer (number of layers, hidden sizes, training steps, batch size, learning rate) are not fully specified; only some features are described, which makes replication difficult.","section":"Section 4.3"},{"comment":"The candidate-selection heuristic that compares each candidate sentence embedding with the embedding of the three input sentences (or the whole document for BOTTOM-UP) should be described precisely, as it affects the reported scores and may interact with the USE embedding space in a way that favors USESUM.","section":"Section 6.2"},{"comment":"The reference 'Lew, 2004' in Section 3 should be 'Lin, 2004' for ROUGE, and the reference for Yang (2016) is a technical report without a formal venue; please check the citation format.","section":"References"},{"comment":"Figure 1 is referenced in Section 4 but is not visible in the manuscript text; please ensure the figure is included and referenced correctly.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the code release is a genuine strength, but the end-to-end comparison is confounded by the abstractive model being trained on USESUM-style oracle selections while evaluated on other extractors' outputs. This is a load-bearing issue that cannot be fixed by local edits alone; it requires at least a new experimental design. The unvalidated angle-to-information monotonicity is the second load-bearing concern. I would be willing to see a revised version that (a) trains separate abstractive models per extractor or a mixed-selection model, (b) reports significance tests and confidence intervals, and (c) includes a direct oracle experiment validating the embedding-space assumption. If those are not feasible, the paper's central claim should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper has a genuinely new technical idea—training the extractive selector to regress on the target summary's sentence embedding under an incremental angle loss—and it ships code and data. But the end-to-end comparison is too confounded to support the headline claim, and the 'prove' in the abstract is not earned.\n\nWhat is new and right: USESUM's extractor predicts a vector in Universal Sentence Encoder space and selects the three input sentences closest to it; the loss in Eq. 11 is the change in arc-cosine distance to the target at each step. I'm not aware of that objective in the summarization literature. The USEG dataset is well chosen for this problem because the low overlap between reviews and single-sentence summaries is exactly where word-level alignment fails. The authors also deserve credit for releasing the implementation and results, and for reporting the extractive and end-to-end stages separately.\n\nWhere it gets soft: the evidence base is thin. The baselines are from 2018, the metric gaps are small—BLEU 0.0225 vs 0.0208, ROUGE-L 0.1479 vs 0.1427—and there are no significance tests or confidence intervals. Human evaluation on 100 outputs per system shows USESUM at 60% grammatical versus 91% for BOTTOM-UP, which makes the improvement in automatic metrics less reassuring. More important is the confounding the stress-test note flagged: the abstractive Transformer was trained on sentences selected by the same embedding-angle rule USESUM uses. Feeding that model with the outputs of NEUSUM or BERTSUM is a distribution mismatch, so part of USESUM's gain could be that its inputs match training, not that the selection objective is superior. The paper does not address this.\n\nThe deeper worry about the angle-to-information monotonicity is also legitimate. The model's entire learning signal is the change in angle to the target embedding, yet the paper itself concedes in Section 6.5 that cosine similarity can be misleading and can 'overly promote' systems that simply average document vectors. No direct validation that decreasing angle corresponds to selecting more informative sentences is provided. That softens the central claim, though it doesn't sink the engineering result.\n\nThe abstract's 'prove' should be softened to 'demonstrate on a single dataset' or similar. The authors already state that adapting to other datasets requires major architectural changes, so the claim is narrow by their own admission.\n\nOverall, this is a coherent, honest engineering paper with a novel intermediate objective and a useful dataset. It is not a proof, and the empirical support is not yet solid enough to accept as-is. A serious referee should see it, but the revision needs significance testing, modern baselines, and an abstractive training setup that does not advantage USESUM by construction.\n\nI'd send it to review with a request for major revision; I wouldn't cite it in its current form unless I were working on this exact dataset.","headline":"Genuinely novel embedding-targeted extractor, but the end-to-end comparison is confounded and the 'prove' claim overreaches.","tokens_in":11053,"tokens_out":2444,"would_cite":false,"duration_ms":22437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Predicting a summary's sentence embedding, rather than word-level selection probabilities, improves end-to-end hotel review summarisation when source and target text are only loosely aligned.","keywords":["abstractive summarization","extractive summarization","sentence embeddings","Universal Sentence Encoder","content selection","weakly aligned corpora","hotel review summarization","end-to-end summarization"],"falsifier":"Train the same end-to-end architecture on a held-out split of the hotel-review dataset, but replace the embedding-regression objective with a sentence-selection probability objective over the same Universal Sentence Encoder embeddings; if the probability-based selector matches or beats USESUM's BLEU, ROUGE-L, and METEOR scores, the paper's central claim that embedding prediction is the better intermediate target is refuted.","tokens_in":9957,"feed_emoji":"🏨","tokens_out":9673,"duration_ms":83944,"temperature":0.7,"pith_summary":"The paper proposes USESUM, a two-stage system that first selects three sentences from a long set of hotel reviews and then generates a one-sentence summary with an abstractive Transformer. Its central claim is that the extractive selector should be trained to predict the sentence embedding of the target summary, and that this intermediate target works better for loosely aligned corpora than predicting sentence-selection probabilities. On the paper's hotel-review dataset, where only about a third of summaries overlap lexically with the reviews, USESUM outperforms the compared extractive-plus-abstractive systems on BLEU, ROUGE-L, and METEOR, and receives the highest human scores for semantic coverage. The result matters because it offers a way to train content selection without word-level alignment labels, a common bottleneck when summarising opinion and review text.","feed_headline":"Sentence embeddings beat word-level sentence selection","feed_subtitle":"On hotel reviews with low word overlap, this beat baselines on BLEU, ROUGE-L, and METEOR.","key_machinery":"The load-bearing mechanism is the angle-drop objective built on a Maximal Marginal Relevance-style update. At step $i$, the model keeps a running estimate $\\hat{y}_{i-1}$ of the summary embedding; after reading input embedding $x_i$, it updates to $\\hat{y}_i = \\hat{y}_{i-1} - \\mathrm{score}\\cdot(\\hat{y}_{i-1} - x_i)$, and the training loss is $\\mathrm{acos}(\\hat{y}_{i-1},\\mathrm{target}) - \\mathrm{acos}(\\hat{y}_i,\\mathrm{target})$, the reduction in arccosine distance to the target. This objective is what lets weakly aligned training pairs still teach sentence ranking: a sentence that moves the predicted embedding closer to the target, even if it is not identical to it, receives positive gradient. The abstractive Transformer with copy attention then turns the three selected sentences into the final summary.","core_discovery":"The paper argues and demonstrates that the angle between pre-trained sentence embeddings can serve as the training signal for extractive content selection in an end-to-end summariser. In USESUM, a bi-directional LSTM reads sentence embeddings produced by the Universal Sentence Encoder, an LSTM-style decoder updates a predicted summary embedding by moving it toward each candidate sentence proportionally to a learned score, and the loss at each step is the reduction in arcus-cosine distance to the target summary embedding. The extractor then chooses the three input sentences whose embeddings are closest to the final predicted embedding and passes them to a Transformer abstractive model. The paper's experiments on hotel-review data show that this embedding-prediction objective leads to better end-to-end summaries than word-level sentence-selection probabilities used by the compared extractive systems, and the human evaluation shows the generated summaries more often cover the target semantics.","pith_inferences":["If the mechanism is real, a natural extension is to make the number of selected sentences learnt or adaptive rather than fixed at three; the current system hard-codes three based on the LEAD-3 heuristic.","The paper's observation that cosine similarity rewards averaging suggests that a diversity-aware or coverage-aware variant of the angle-drop objective might close the gap to the BOTTOM-UP system's cosine score while keeping the semantic advantage.","A direct test of the central claim would be to replace the Universal Sentence Encoder with a different sentence-embedding model; if the advantage over word-level selection disappears, the effect is encoder-specific rather than a general property of embedding targets.","For multilingual review summarisation, language-agnostic sentence encoders could make the same weakly aligned training strategy applicable without parallel sentence-level data."],"forward_implications":["The comparison on the hotel-review data shows that an embedding-prediction extractor can outperform word-level sentence-selection extractors in end-to-end BLEU, ROUGE-L, and METEOR when source and target have low lexical overlap.","Weakly aligned source-target pairs, which cannot be used to train a sentence-selection classifier, can still contribute training signal to the embedding-regression extractor.","The extractor reduces the input to three sentence embeddings, making the abstractive stage independent of the original document length, up to 800 sentences in the dataset.","The paper's human evaluation shows that the improvement in automatic metrics is accompanied by higher semantic coverage, not merely higher lexical overlap.","The beam-search named-entity promotion step improves all compared end-to-end systems on the dataset."],"supporting_citations":[{"why":"Supplies the Universal Sentence Encoder embeddings used as the shared space for input sentences, predicted summary vectors, and cosine-similarity evaluation.","marker":"(Cer et al., 2018)"},{"why":"Provides the Maximal Marginal Relevance formulation that motivates the angle-based information-gain scoring function.","marker":"(Carbonell and Goldstein, 1998)"},{"why":"Defines the bottom-up two-stage abstractive baseline and the idea of decoupling content selection from generation.","marker":"(Gehrmann et al., 2018)"},{"why":"Provides the NEUSUM sentence-selection baseline that the paper compares against in extractive and end-to-end experiments.","marker":"(Zhou et al., 2018)"},{"why":"Provides the BERTSUM extractive baseline whose 512-token limit restricts its performance on long review inputs.","marker":"(Liu, 2019)"},{"why":"Provides the REFRESH reinforcement-learning extractive baseline.","marker":"(Narayan et al., 2018)"},{"why":"Defines ROUGE, the metric used to measure the low lexical overlap that makes only a third of summaries usable for training selection models.","marker":"(Lew, 2004)"},{"why":"Provides the Transformer architecture used as the abstractive summariser in the end-to-end system.","marker":"(Vaswani et al., 2017)"},{"why":"Supplies the copy-attention mechanism that lets the abstractive model reproduce input words.","marker":"(Gu et al., 2016)"},{"why":"Provides focal loss, used as the loss function of the abstractive Transformer instead of standard cross-entropy.","marker":"(Lin et al., 2017)"}],"fun_headline_variants":["Sentence embeddings as training target for summarisation","Embedding prediction boosts end-to-end summarisation","Summarise by predicting sentence embeddings","Angle between embeddings selects summary sentences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the angle between Universal Sentence Encoder embeddings measures how much of the target summary's content a source sentence carries, so that reducing that angle toward a predicted summary embedding selects sentences that the abstractive model can use.","fun_headline_variants_meta":{"raw":{"variants":["Sentence embeddings as training target for summarisation","Embedding prediction boosts end-to-end summarisation","Summarise by predicting sentence embeddings","Angle between embeddings selects summary sentences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000136,"raw_usage":{"total_tokens":1086,"prompt_tokens":827,"completion_tokens":259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":207}},"tokens_in":443,"tokens_out":259,"duration_ms":3244,"temperature":1.0,"reasoning_tokens":207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:50:06.658783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same end-to-end architecture on a held-out split of the hotel-review dataset, but replace the embedding-regression objective with a sentence-selection probability objective over the same Universal Sentence Encoder embeddings; if the probability-based selector matches or beats USESUM's BLEU, ROUGE-L, and METEOR scores, the paper's central claim that embedding prediction is the better intermediate target is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Maximal Marginal Relevance formulation that motivates the angle-based information-gain scoring function."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the bottom-up two-stage abstractive baseline and the idea of decoupling content selection from generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NEUSUM sentence-selection baseline that the paper compares against in extractive and end-to-end experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the REFRESH reinforcement-learning extractive baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ROUGE, the metric used to measure the low lexical overlap that makes only a third of summaries usable for training selection models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer architecture used as the abstractive summariser in the end-to-end system."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides focal loss, used as the loss function of the abstractive Transformer instead of standard cross-entropy."}],"review_version":1}