{"id":"5478d00b-25c6-4863-aa31-db54c6b004c1","arxiv_id":"2607.20078","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"First Romanian lexical complexity/simplification resource with baseline systems; a dictionary-and-grammar hybrid, DexFlex, beats LLM prompting on most ranking and coverage metrics.","lead":"This paper builds the first Romanian dataset for lexical complexity and simplification, with 3,921 annotated word-in-context samples and human-ranked replacement candidates. It compares several simplification approaches and finds a dictionary- and grammar-based system, DexFlex, generally beating LLM prompting on ranking and coverage metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's DexFlex advantage lacks a declared train/test split for the LCP ranker and any uncertainty quantification; without these, the central comparison is not yet supported.","rationale":"The reader's verdict (CONDITIONAL) is appropriate: the dataset resource is valuable and honestly presented, but the system-comparison claim needs stronger support. My concern goes beyond the reader's identified WSD weakness to a more direct methodological gap: the LS evaluation protocol for DexFlex is underspecified with respect to training data. Since the LCP ranker is trained on the same datasets that contain the HT test sentences, leakage is possible but not confirmed. The paper should disclose the split and provide uncertainty estimates. This does not overturn the CONDITIONAL verdict; it sharpens the specific condition required for acceptance. I marked agreement as 'partial' because the reader's weakest_assumption focuses on WSD generalizability, while my concern is about the internal validity of the reported comparison itself.","tokens_in":14367,"tokens_out":5273,"duration_ms":51827,"concrete_test":"Run a held-out re-evaluation: train the DexFlex LCP ranker exclusively on WT+RoLCP (or use grouped 10-fold splits of HT as in Table 2), recompute all MAP/Potential columns in Table 3, and add bootstrap 95% confidence intervals for each metric. If DexFlex no longer significantly beats RoLlama/GPT-4o, the headline comparison should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'DexFlex consistently outperforms all other approaches across MAP and Potential metrics' (Table 3) rests on an evaluation protocol that is underspecified in Section 4. DexFlex's candidate ranking uses 'the LCP pipeline' (Section 4, Lexical Simplification), but the paper never states whether that LCP ranker was trained on the HT subset that constitutes the RoLS test set. The LCP section reports grouped 10-fold cross-validation (Table 2), but no analogous split is described for the substitution-ranking task. If the Ridge regressor saw the HT sentences (or their LCP labels) during training, the MAP/Potential comparisons for DexFlex are effectively trained-on-test, and the reported margins over RoLlama/GPT-4o would not reflect genuine system performance. Additionally, no confidence intervals or significance tests are reported for any Table 3 entry, so even without leakage, the 'consistently outperforms' claim is not statistically supported (e.g., MAP@1: 0.41 vs. 0.40 for RoLlama). The under-validated WSD component (Appendix A) further compounds this, but the immediate threat is the missing evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RALS, a Romanian resource combining lexical complexity prediction (LCP) annotations for 3,921 word-in-context samples (HT, WT, and RoLCP subsets) with a lexical substitution dataset (RoLS) for the HT subset. The authors propose a Bradley–Terry ranking procedure to order substitution candidates from pairwise simplicity judgments, evaluate a Ridge-regressor LCP baseline under grouped cross-validation and cross-dataset transfer, and compare several simplification systems — Apertus-8B, RoLlama-8B, GPT-4o, a fine-tuned Romanian BERT, and the dictionary-based DexFlex — on MAP, Potential, and accuracy metrics. The central claim is that DexFlex consistently outperforms all other approaches on MAP and Potential metrics, and that this constitutes a strong non-LLM baseline for Romanian lexical simplification.","tokens_in":14682,"tokens_out":5671,"duration_ms":54016,"significance":"If the evaluation is sound, the resource contribution is valuable: Romanian currently lacks LCP/LS datasets, and the construction methodology — frequency sampling, sense-clustering for sentence selection, grouped cross-validation, and human-ranked substitution gold data — is careful and well contextualized against shared-task results. The paper also ships open code and data, and its Limitations section honestly discloses the moderate inter-annotator agreement, the HT-only scope of RoLS, and the unevaluated WSD component. The claim that a transparent dictionary-based pipeline can beat prompt-based LLMs on ranking and coverage metrics is interesting and practically relevant for low-resource settings. However, the headline comparison is not yet fully supported because the evaluation protocol for DexFlex's LCP-based ranker is underspecified and no uncertainty quantification is provided.","major_comments":[{"comment":"The DexFlex row uses \"the LCP pipeline ... to rank the candidate synonyms,\" but the manuscript does not state how that LCP ranker was trained for the Lexical Simplification evaluation. The LCP experiments in Table 2 use grouped 10-fold cross-validation; no analogous split is described for the candidate-ranking protocol. If the Ridge regressor was fitted on HT LCP labels (the same sentences that define the RoLS test set), the DexFlex comparisons are trained-on-test. Please specify the training data for this ranker, whether any HT or RoLCP labels were seen, and the exact split/grouping used in computing Table 3.","section":"Section 4, Table 3"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported for any MAP/Potential/ACC entry. The lead over RoLlama is only 0.01 at MAP@1 (0.41 vs 0.40); without uncertainty estimates the phrase \"consistently outperforms\" is not supported. Please report bootstrap or per-sample paired statistics, clustering by the 190 HT sentences.","section":"Section 4, Table 3"},{"comment":"The WSD component is central to DexFlex, yet the appendix states \"we did not run exhaustive word-sense-disambiguation evaluation.\" The reliability of the nearest-neighbor sense matching is asserted only through \"summary evaluations.\" Because the MAP/Potential superiority of DexFlex depends on retrieving the correct sense from dexonline, this is not a peripheral limitation. Please provide a quantitative WSD evaluation on a representative sample (including polysemous/rare words), report coverage of dexonline entries, and discuss failure cases.","section":"Appendix A"}],"minor_comments":[{"comment":"Reference formatting is inconsistent: \"Codrut, et al., 2024\" should be normalized, \"Horacio Saggio\" in the Štajner et al. (2024) reference should be \"Horacio Saggion,\" and \"DellFlex\" in the Limitations section should be \"DexFlex.\"","section":"References"},{"comment":"The definitions of MAP@N, Potential@N, and Accuracy@N@top_gold_1 are imprecise. MAP@N is not simply \"precision\" but mean average precision over candidate lists; Potential@N and the \"top_gold_1\" notation should be defined with formulas or a precise reference.","section":"Section 4, Evaluation"},{"comment":"The phrase \"first text simplification system for Romanian\" overstates the contribution: the implemented and evaluated component is lexical simplification (candidate substitution), not full sentence-level text simplification. Consider qualifying the claim.","section":"Abstract / Conclusions"},{"comment":"The sentence \"The final list is the union of both sets of suggestions, verified by a third annotator\" would benefit from a statement of how disagreements with the third annotator were resolved and how the union was pruned.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The resource contribution is solid and the data construction is thoughtful. The main issue is that the central comparison (DexFlex vs LLMs) is not yet supported because the evaluation protocol for the LCP ranker used in Table 3 is underspecified and no confidence intervals are given. This is fixable within revision: clarify the training/split for DexFlex's ranker, add uncertainty quantification, and strengthen the WSD validation. I do not see evidence that the result is fabricated; the paper is more likely under-specified than irreparably flawed. I recommend major revision, not rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: this paper gives Romanian NLP a real resource that didn't exist — 3,921 LCP annotations across three subsets, plus a human-ranked substitution gold standard for the HT subset, with code and data released. The construction is careful: frequency-based sampling, sense-clustering for sentence selection, grouped 10-fold CV, and cross-dataset checks. The pairwise Bradley-Terry ranking approach is a reasonable upgrade over frequency-based ranking, and the paper is honest about its limits, including the WSD component that was not exhaustively evaluated and the fact that RoLS only covers HT.\n\nThe soft spots are where the evaluation of DexFlex is concerned. The paper says the LCP pipeline is used to rank DexFlex's candidate synonyms, but it never states whether that ranker was trained on the HT data that constitutes the RoLS test set. Features like dependency-child count and sentence-position booleans are contextual, so if the ranker saw the HT sentences, the comparison is at least partially trained-on-test. That alone would justify a conditional, but there's more: no confidence intervals or significance tests for any Table 3 entry, and the strongest margin (MAP@1 0.41 vs 0.40 for RoLlama) is noise-level. The claim that DexFlex 'consistently outperforms' is overstated relative to the evidence.\n\nThat said, the resource contribution stands on its own, and the cross-dataset generalization results in Table 2 are informative. The paper's own limitation section acknowledges the gaps, which is a good sign. The right fix is not rejection but a revision that spells out the exact train/test split for the LCP ranker, reports bootstrap intervals or paired tests, and ideally adds a small WSD evaluation for DexFlex. I'd send this to peer review.","headline":"First Romanian LCP/LS dataset plus a hybrid system that looks competitive, but the system comparison needs a declared split and uncertainty bounds before the 'consistently outperforms' claim holds.","tokens_in":15129,"tokens_out":2457,"would_cite":true,"duration_ms":24755,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a transparent, dictionary-based pipeline outperforms large language models on Romanian lexical simplification, and releases the first joint complexity-prediction and simplification dataset for the language.","keywords":["Romanian lexical simplification","lexical complexity prediction","low-resource NLP","dictionary-based simplification","Bradley-Terry ranking","word-sense disambiguation","multilingual simplification","DexFlex"],"falsifier":"A direct test would take a set of polysemous Romanian words in context, run DexFlex's retrieval, and compare the selected synonyms against human gold senses; if retrieval frequently selects synonyms from the wrong sense, the MAP/Potential advantage would likely collapse on unrestricted text. Additionally, expanding the RoLS candidate collection to the RoLCP original-text samples and re-evaluating DexFlex would show whether the reported scores transfer beyond the human-translated genre.","tokens_in":14316,"feed_emoji":"📖","tokens_out":4979,"duration_ms":41723,"temperature":0.7,"pith_summary":"This paper introduces RALS, the first Romanian dataset that brings together lexical complexity prediction (3,921 word-in-context annotations across three subsets) and lexical simplification (human-ranked substitution candidates for a human-translated subset). The authors argue that a transparent, dictionary-grounded pipeline (DexFlex) that combines part-of-speech tagging, contextual BERT embedding retrieval, and morphological inflection outperforms prompt-based large language models at ranking simplification suggestions, as measured by mean average precision and coverage. They also report that handcrafted-feature complexity predictors for Romanian perform on par with other low-resource languages, and that translating English complexity annotations into Romanian shifts the complexity distribution. If this holds, low-resource languages without existing simplification resources can build practical simplification systems without relying on expensive or opaque LLM APIs.","feed_headline":"Dictionary pipeline beats LLMs on Romanian simplification","feed_subtitle":"First joint complexity-plus-simplification dataset for Romanian, with a rule-based system topping prompt-based models.","key_machinery":"DexFlex is the carrying mechanism: it uses part-of-speech and morphological analysis of the target word, retrieves synonym candidates from a Romanian dictionary by matching the sentence's BERT contextual embedding to cached embeddings of dictionary example sentences via approximate nearest-neighbour search, then inflects the chosen synonyms to match the sentence context in number, gender, and person. The ranking gold standard comes from a Bradley–Terry model applied to human pairwise simplicity judgments, replacing the usual frequency-based candidate ranking.","core_discovery":"The paper's central claim is that DexFlex, a quasi-rule-based Lexical Simplification system, achieves the best ranking and coverage scores for Romanian when compared against open-weight and closed-source LLMs and a fine-tuned BERT model. Its novelty lies in combining a morphological tagger, a dictionary of synonyms with cached example-sentence embeddings, and approximate nearest-neighbour retrieval to select context-appropriate synonyms, followed by rule-based inflection and a handcrafted-feature complexity re-ranker. The authors further show that the three LCP datasets they release — human-translated, word-translated, and original Romanian text — reach Pearson correlations of 0.56, 0.68, an","pith_inferences":["If the DexFlex advantage generalizes beyond the human-translated subset to unrestricted Romanian texts, it implies that a curated dictionary plus a small amount of human pairwise judgment may deliver simplification quality comparable to expensive LLM APIs in other under-resourced languages — a testable claim for each new language.","The paper's data show LLMs still edge out DexFlex on accuracy at the top gold candidate, suggesting a hybrid that uses LLM suggestions but re-ranks them with a dictionary-grounded complexity model could outperform either alone; the paper does not test this combination.","Because simplification candidates were only collected for the human-translated subset, the LS benchmark covers a narrow genre; expanding RoLS to original-text RoLCP sentences would reveal whether the ranking methodology survives varied text types and whether DexFlex retains its edge."],"forward_implications":["If the claim holds, Romanian text simplification can proceed without relying on closed LLMs, supporting privacy-preserving applications in health, education, and government.","The released dataset enables training and evaluation of LCP models for Romanian, with published baseline correlations for comparison.","The pairwise ranking methodology can be reused to build simplification gold standards for other under-resourced languages.","DexFlex's competitive performance suggests that dictionary- and rule-based systems should be considered strong baselines in multilingual simplification shared tasks.","The finding that cross-lingual complexity transfer distorts annotations cautions against using translated data as a proxy for original-language complexity."],"fun_headline_variants":["Rule-based system beats LLMs on Romanian simplification","DexFlex tops LLMs for Romanian lexical simplification","New Romanian dataset: rule-based wins over LLMs","Romanian simplification: no LLMs needed, rule-based wins","First Romanian LS dataset, with rule-based beating LLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the assumption that matching a sentence's contextual BERT embedding to the nearest cached example embedding in the dictionary reliably identifies the intended word sense and yields suitable synonyms; the authors acknowledge they did not run exhaustive word-sense-disambiguation evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Rule-based system beats LLMs on Romanian simplification","DexFlex tops LLMs for Romanian lexical simplification","New Romanian dataset: rule-based wins over LLMs","Romanian simplification: no LLMs needed, rule-based wins","First Romanian LS dataset, with rule-based beating LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":985,"prompt_tokens":602,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":346,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":346,"tokens_out":383,"duration_ms":74904,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:50:28.134423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would take a set of polysemous Romanian words in context, run DexFlex's retrieval, and compare the selected synonyms against human gold senses; if retrieval frequently selects synonyms from the wrong sense, the MAP/Potential advantage would likely collapse on unrestricted text. Additionally, expanding the RoLS candidate collection to the RoLCP original-text samples and re-evaluating DexFlex would show whether the reported scores transfer beyond the human-translated genre.","supporting_citations":[],"review_version":1}