{"id":"4b2ca7d8-adf2-4692-b8dd-08403bb8098c","arxiv_id":"2505.05016","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM performance at applying social choice-based group aggregation strategies declines as the number of ratings in the prompt grows, and in-context learning recovers much of the loss.","lead":"This paper tests how well small language models can apply social choice rules (like additive utility or approval voting) to recommend items for a group, finding that accuracy drops once the prompt contains more than about 100 ratings. It also shows that adding three worked examples (in-context learning) largely restores accuracy, while changing the format of the rating table can help or hurt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The overlap-based accuracy metric lacks a chance baseline and is not checked for output-length shifts in the ICL condition, so the headline ICL gain is not yet protected from a tie-list confound.","rationale":"The reader's weakest assumption is precisely the overlap metric without a chance baseline, and this remains the most load-bearing concern. My reading sharpens it by tracing a concrete mechanism through the ICL condition: the few-shot examples contain gold outputs that may be long tie lists, so ICL could teach the model to emit longer recommendation lists, inflating overlap-based accuracy. This does not overturn the paper's central complexity-decline result, because longer gold lists at high complexity would bias the metric upward, making the observed decline conservative. However, it does cast doubt on the magnitude and the word 'significantly' attached to the ICL improvement until output-length statistics and stricter metrics are reported. The paper's own Section 5.3 check covers only the baseline Mistral condition, not the ICL condition, so the gap is real. Since the reader already returned CONDITIONAL and this concern strengthens that verdict without changing it, no adjustment is needed.","tokens_in":13927,"tokens_out":6718,"duration_ms":73535,"concrete_test":"On the 245 50-item scenarios used in RQ2, recompute Phi4 scores under stricter metrics: exact match with the full gold_label set, precision = |LLM_out ∩ gold|/|LLM_out|, and a random-choice baseline using gold_label lengths. Also record the number of items returned by Phi4 for baseline vs ICL prompts and test whether list length increases under ICL (e.g., paired Wilcoxon or bootstrap). If the ICL advantage and the complexity decline persist under exact-match/precision and with unchanged list length, the central claims stand; if the gain attenuates or list length increases, the reported accuracy improvement is at least partly an artifact of the overlap metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the overlap-based correctness definition in Section 3.3: an LLM output is correct if it shares any item with the gold_label, and gold_label tie lists grow with complexity for LMS and MPL. Correctness can therefore be obtained by chance or by emitting longer lists, so every accuracy comparison in RQ1-RQ3 is vulnerable to an uncontrolled output-length/tie-list confound. The paper's only response, Section 5.3, documents over-recommending in the baseline condition and finds that only Mistral over-recommends; it does not report output list lengths for the ICL, explanation, domain-cue, or formatting conditions, nor does it provide a chance baseline. Since one central claim is that ICL 'significantly' raises accuracy at high complexity (0.57 to 0.85 in Section 4.3.2), and ICL examples are built from gold outputs that can themselves be long tie lists, the measured gain could partly reflect longer outputs rather than better strategy application. The RQ1 decline is probably conservative with respect to this confound because chance overlap should rise with complexity, but the ICL improvement and cross-strategy/cross-model comparisons are not yet protected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how well locally run LLMs can apply social choice-based aggregation strategies (ADD, APP, LMS, MPL) for group recommendations as the number of ratings in the prompt grows, and whether prompt modifications (explanations, in-context learning, domain cues) or data formatting improve accuracy. Using 1,000 synthetic groups, four quantized LLMs, and an overlap-based accuracy metric, the authors report that accuracy declines beyond roughly 100 ratings, that in-context learning (ICL) improves accuracy at high complexity, that explanations and domain cues do not, and that the data format matters. The authors provide a companion GitHub repository with code for group generation, strategy implementations, and prompts.","tokens_in":14138,"tokens_out":6616,"duration_ms":61841,"significance":"If the findings hold, the paper makes a useful contribution by identifying group complexity as a factor that should be controlled in LLM-based group recommender evaluation, and by providing early evidence on prompt robustness and the viability of small, locally run models. The systematic variation of complexity and the use of quantized open-weight models are practical strengths, and the public code supports reproducibility. The main claims are interesting and timely, but the evaluation metric and statistical reporting currently leave the quantitative conclusions under-supported, so the significance of the work will depend on the authors' ability to address these concerns.","major_comments":[{"comment":"The overlap-based correctness definition in Section 3.3 makes the headline results vulnerable to a tie-list/output-length confound. Because LMS and MPL produce long tie lists at high complexity, and the LLM output is scored as correct if it overlaps the gold list, models can earn credit by chance or by emitting longer lists. Section 5.3 reports output-list lengths only for the baseline condition (Table 7), not for the ICL, explanation, domain-cue, or formatting conditions. Since the ICL examples (Section 3.4.2) are constructed from gold outputs, which can themselves be long tie lists, the reported ICL improvement (0.57 to 0.85 in Section 4.3.2) may partially reflect longer outputs rather than better strategy application. The authors should provide a chance baseline (e.g., expected overlap under random item selection conditioned on list length) and report output-list-length statistics for all conditions, ideally alongside a stricter exact-match metric.","section":"§3.3, §4.3.2, §5.3"},{"comment":"The paper uses the word 'significantly' in the abstract and in Section 4.3.2 ('significantly improved the model's ability') to describe the ICL result, but no statistical test, confidence interval, or standard error is reported anywhere. Similarly, the RQ1 accuracy decline (Figure 2), the RQ2 condition comparisons, and the RQ3 formatting differences (Table 5) are presented without uncertainty quantification. The authors should add appropriate inferential statistics, such as McNemar's test for paired scenarios or bootstrap confidence intervals, to support the existence and magnitude of these differences.","section":"§4, §4.3.2, Table 5"},{"comment":"The RQ2 prompt-condition analysis is performed with a single model (Phi4) on a subset of 245 scenarios, yet the abstract and conclusions make general claims: 'we showed that In-Context Learning (ICL) can significantly increase the performance at higher degrees of group complexity' and that other prompt modifications 'did not impact accuracy.' These claims are not established across models; they are at best properties of Phi4 under the tested conditions. The authors should either restrict the claims to Phi4 or validate the main prompt-condition effects on at least one additional model.","section":"§3.4, §4.3"}],"minor_comments":[{"comment":"The text says ratings are generated 'between 0 and 10' while Table 2 states 'Ratings are on a scale of 1 to 10'; please align the description with the code and example.","section":"§3.1"},{"comment":"The model list contains a typo: 'Mistal' should be 'Mistral'; the same typo appears near Table 7.","section":"§3.2.2"},{"comment":"The sentence 'Gemma, Llama and Mistral underperfomed at lower group complexity compared toPhi' contains a typo ('underperfomed') and a missing space; also, the claim about lower-complexity underperformance could be more specific.","section":"§4.2"},{"comment":"The baseline accuracy of 0.57 for Phi4 at higher complexity appears to be the average across the 100/200/400 complexity levels; please state this explicitly, since Table 5 reports per-level values of 0.74, 0.53, and 0.40 for the same baseline format.","section":"§4.3"},{"comment":"The limitations section does not mention that the RQ2 and RQ3 comparisons are based on a single model (Phi4); adding this to the limitations would help calibrate the scope of the prompt-condition findings.","section":"§5.5"},{"comment":"References [9] and [10] cite the same paper; please deduplicate and renumber.","section":"References"},{"comment":"The sentence 'This results illustrated once more the extent that group complexity affects...' is grammatically incorrect; consider revising to 'This result illustrates once more the extent to which group complexity affects...'.","section":"§4.5"}],"recommendation":"major_revision","confidential_remarks":"This is a compact empirical study with a useful practical message, and the authors have been transparent about their soft accuracy metric and single-model prompt-condition experiments. The missing chance baseline and the absence of significance tests are the key hurdles; if the authors can provide the requested analyses, the paper could be made suitable for the venue. I would caution against letting the ICL claim be stated as a general result without supporting evidence from more than one model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the Waterschoot et al. paper on LLMs and social-choice aggregation for group recommendations. The useful core: they define group complexity as total ratings in the prompt, vary it from 10 to 400, and show that across four local LLMs accuracy at applying ADD/APP/LMS/MPL declines once you pass about 100 ratings. That is a simple, sensible evaluation dimension that the GRS-LLM literature has largely ignored, and it deserves to be controlled in future work. They also ship code and data, and they are transparent about the softness of their accuracy metric. The consistent trend across models gives me confidence the RQ1 finding is real.\n\nThe soft spots are mainly in the metric and the RQ2 conditions. The overlap-based correctness definition counts an output as correct if it shares any item with the gold tie list. Since LMS and MPL produce longer tie lists at high complexity, chance overlap rises with complexity. That makes the RQ1 decline conservative, which is fine. But it also means the absolute accuracies and the cross-strategy and cross-model comparisons are hard to interpret without a chance baseline. The more serious issue is the ICL result: accuracy jumps from 0.57 to 0.85, and the ICL examples are built from gold outputs that can themselves be long tie lists. If the model simply learns to output more items, part of that gain is an artifact. Section 5.3 only checks over-recommending in the baseline condition and finds only Mistral does it; it doesn't report output lengths for the ICL condition. That needs to be measured. Also, RQ2 and RQ3 use only Phi4, and there are no significance tests or error bars anywhere, so the word 'significant' for ICL is doing work the stats don't support. These are fixable, not fatal.\n\nIs the central argument sound? Mostly yes. The complexity effect survives the confound in the conservative direction, and the formatting effect (per-item JSON beats per-user JSON) is a useful practical note. The paper just overreaches with the ICL claim and with the absolute accuracy numbers.\n\nWho should read this: anyone evaluating LLM-based group recommenders, and researchers studying context-length effects on LLM reasoning. It's a solid empirical starting point, not a definitive one. I'd send it to peer review and ask for a revised version with a chance baseline and per-condition output-length checks. I would not desk reject it.","headline":"A useful, honestly-reported empirical result on how group complexity degrades LLM aggregation accuracy, but the headline ICL gain needs protection from an output-length confound.","tokens_in":14652,"tokens_out":2844,"would_cite":true,"duration_ms":28017,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based group recommenders break down once the prompt contains more than about 100 ratings, and in-context learning largely repairs the breakdown.","keywords":["Large Language Models","Group Recommender Systems","Social choice-based aggregation strategies","In-context learning","Group complexity","Prompt formatting","Zero-shot evaluation"],"falsifier":"Re-run the 1,000-scenario evaluation with strict set equality (the model must return exactly the gold-label list, or at least exactly the winning set) and with a random-overlap chance baseline conditioned on tie-list length; if accuracy no longer declines past 100 ratings, or if the model ranking changes, the complexity-deterioration claim is an artifact of the soft metric.","tokens_in":13733,"feed_emoji":"🤖","tokens_out":5909,"duration_ms":50774,"temperature":0.7,"pith_summary":"The paper tests whether LLMs can act as social-choice aggregators for group recommendations: given a matrix of ratings from several users and a specified strategy (sum scores, approval voting, least misery, or most pleasure), the model must name the winning item. Across 1,000 synthetic groups of 2–8 members and 5–50 items, accuracy stayed high until the product of members and items—what the authors call group complexity—passed about 100, then declined for all four included models. Showing three worked examples in the prompt (in-context learning) raised accuracy on large groups from 0.57 to 0.85 for Phi4; asking for explanations or adding movie titles did not help. Results also changed with data format: per-item JSON lists worked better than per-user JSON or a dataframe. If the claim holds, evaluations of LLM group recommenders should vary group complexity instead of testing only small matrices.","feed_headline":"LLM group picks fail past 100 ratings; examples restore accuracy","feed_subtitle":"In-context examples lift accuracy 28 points on large rating sets; prompt format also matters.","key_machinery":"The load-bearing objects are the four social-choice aggregation strategies—Additive Utilitarian (highest sum), Approval Voting (most ratings above a threshold), Least Misery (highest of the lowest per-item ratings), and Most Pleasure (highest individual rating)—each inserted into the prompt as a social-choice explanation instructing the model how to combine the group's rating matrix. Group complexity is defined as the product of group size and item count, i.e. the number of ratings in the prompt. The evaluation machinery is an overlap-based accuracy score: an LLM output is correct if it shares at least one item with the strategy's gold-label tie list, a soft criterion the paper acknowledges. In-context learning is the intervention that carries the positive result: three full worked group scenarios with correct outputs are prepended to the prompt, raising high-complexity accuracy by 28 percentage points for the best model.","core_discovery":"The central discovery is a scaling breakdown: LLMs can apply four standard social-choice aggregation strategies (Additive Utilitarian, Approval Voting, Least Misery, Most Pleasure) almost perfectly up to a group complexity of about 80–100 total ratings, but accuracy falls once the prompt contains more than roughly 100 ratings. The deterioration is not identical across models—Phi4 held near-perfect accuracy to 80 and stayed best overall, Mistral peaked at complexity 100–200, and Llama and Gemma declined steadily. The paper further claims that in-context learning with three worked examples is the intervention that works: at complexities of 100, 200, and 400 ratings, Phi4's accuracy rose from 0.57 to 0.85, while requesting explanations or adding real-world domain cues did not significantly improve accuracy. Data formatting changes the outcome too: per-item JSON (0.74/0.53/0.40 across 100/200/400 ratings) outperformed per-user JSON (0.59/0.24/0.16) and a dataframe (0.58/0.45/0.40). The authors conclude that group complexity and prompt format belong in the evaluation pipeline of LLM-based group recommenders, and that small locally run models suffice under favorable conditions.","pith_inferences":["If the overlap-accuracy criterion is replaced by strict exact match, the reported accuracy numbers would drop substantially—especially for Least Misery and Most Pleasure, which generate long tie lists—so the magnitude of the complexity effect should be re-estimated before using the 100-rating threshold as a design rule.","An implicit testable implication is that a random-chance baseline, conditioned on gold tie-list length, should be reported alongside accuracy; without it, part of the apparent decline at high complexity could be a property of tie proliferation rather than of the model.","The in-context learning result suggests a broader recipe: rather than merely instructing with the strategy definition, injecting worked executions of the aggregation rule may be an inexpensive way to push the complexity ceiling of LLM-based group recommenders further, potentially beyond 400 ratings with more or tailored examples.","The format effect hints that token-level readability, not just information content, shapes aggregation accuracy; one could test this directly by comparing JSON variants with identical content but different key orderings or spacing."],"forward_implications":["Any evaluation of LLM-based group recommenders that only tests small rating matrices will overstate real-world capability; the paper argues that group complexity should be manipulated in future evaluation pipelines.","Prompt engineering matters at high complexity: adding in-context examples is an effective fix, while explanation prompting and domain cues are not.","Data format is a confound: per-item JSON consistently outperforms per-user JSON and dataframe rendering, so format choices need to be reported and tuned.","Smaller local models (7–14B parameters) can apply aggregation strategies accurately under good conditions, supporting privacy-preserving and cost-effective deployment.","Ranking ability also degrades with complexity: NDCG@10 for Phi4 falls from 0.97 at 100 ratings to 0.74 at 400 ratings, so the effect is not limited to picking a single winner."],"supporting_citations":[{"why":"Introduces social choice-based aggregation strategies for group recommender systems, which constitute the task the LLMs are prompted to perform.","marker":"[22]"},{"why":"Supplies the consensus/majority/borderline taxonomy and the definitions of ADD, APP, LMS, and MPL used to generate gold labels.","marker":"[31]"},{"why":"Provides the social choice-based explanation phrasing and group-scenario generation approach that the prompts are built from.","marker":"[2]"},{"why":"Co-introduces the social choice-based explanations that are inserted into prompts to instruct the model on the aggregation procedure.","marker":"[26]"},{"why":"Provides the crowdsourcing pipeline for group explanations that the prompt formulation draws on.","marker":"[16]"},{"why":"Defines in-context learning and few-shot prompting, the mechanism behind the paper's main performance-boosting intervention.","marker":"[7]"},{"why":"Prior LLM-generated group recommendation evaluation that the paper positions itself against by testing explicit application of aggregation strategies.","marker":"[33]"},{"why":"Supplies the MovieLens movie titles used as real-world domain cues in the RQ2 condition.","marker":"[12]"},{"why":"Gives the theoretical basis for the NDCG metric used in the ranked-recommendation analysis.","marker":"[35]"}],"fun_headline_variants":["LLM group recs degrade past 100 ratings; few-shot examples fix it","Group complexity breaks LLM recommenders; in-context examples recover 28 pts","Formatting group reviews matters: per-item beats per-user for LLM picks","Small LLMs handle group recs up to 100 ratings; beyond that, errors soar","In-context examples rescue LLM group picks as rating sets grow past 100"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that overlap with the strategy's gold-label tie list is a fair measure of correctness; because Least Misery and Most Pleasure produce long tie lists at high complexity, a model can be counted correct by luck or by over-recommending, and no chance baseline is reported.","fun_headline_variants_meta":{"raw":{"variants":["LLM group recs degrade past 100 ratings; few-shot examples fix it","Group complexity breaks LLM recommenders; in-context examples recover 28 pts","Formatting group reviews matters: per-item beats per-user for LLM picks","Small LLMs handle group recs up to 100 ratings; beyond that, errors soar","In-context examples rescue LLM group picks as rating sets grow past 100"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000802,"raw_usage":{"total_tokens":3588,"prompt_tokens":1072,"completion_tokens":2516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2409}},"tokens_in":688,"tokens_out":2516,"duration_ms":15829,"temperature":1.0,"reasoning_tokens":2409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:14:46.950445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1,000-scenario evaluation with strict set equality (the model must return exactly the gold-label list, or at least exactly the winning set) and with a random-overlap chance baseline conditioned on tie-list length; if accuracy no longer declines past 100 ratings, or if the model ranking changes, the complexity-deterioration claim is an artifact of the soft metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MovieLens movie titles used as real-world domain cues in the RQ2 condition."},{"cited_title":"ACM Comput","cited_arxiv_id":null,"evidence_quote":"Gives the theoretical basis for the NDCG metric used in the ranked-recommendation analysis."}],"review_version":1}