{"id":"c68101cd-5a0f-47b3-843c-edd8fce991ff","arxiv_id":"2501.10313","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Open-source Llama models fail to overcome popularity bias in third-party library recommendations, with low recall across all six tested configurations.","lead":"This paper tests whether open-source LLMs can avoid recommending popular but irrelevant Java libraries, using prompt tuning, fine-tuning, and a popularity penalty. The study finds that accuracy remains low and popularity bias persists across all tested configurations, even though diversity improves.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low recall is likely an artifact of evaluation mismatch: IST3 forbids recommending popular libraries while recall is scored against ground-truth dependencies that are predominantly popular, capping achievable recall by construction.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the 'avoid popular libraries' instruction (IST3) is evaluated against a ground truth composed of libraries actually used by projects, which are likely popular. My stress-test confirms this is the single most consequential weakness because the paper's core claim—low recall means LLMs cannot address popularity bias—depends on recall being a meaningful measure under that instruction. A concrete ceiling analysis would settle whether the recall values are forced by prompt design rather than model incapability. The paper also has secondary issues (unspecified N, no uncertainty bounds, an apparent C3/C4 naming slip in Section V), but the ground-truth/instruction mismatch is primary. Because the conditional verdict already requires addressing this concern, I do not change the reader's verdict, but I emphasize that without the ceiling computation or an IST3-ablation the negative conclusion is not yet supported.","tokens_in":12801,"tokens_out":3434,"duration_ms":36504,"concrete_test":"Compute the achievable-recall ceiling for each held-out project: the fraction of its ground-truth libraries that are not in the popular set targeted by IST3, i.e., |G_nonpopular|/|G_total|, and compare the observed recall in Table III to this ceiling. Then rerun configurations C4 and C5 with IST3 removed from the prompt and report recall. If observed recall is close to the ceiling, or rises sharply when IST3 is removed, the low-recall evidence for 'LLMs cannot address popularity bias' collapses and the conclusion must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim—that open-source LLMs cannot address popularity bias because recall remains 0.08–0.20 across all configurations—rests on an unexamined alignment between the anti-popularity instruction and the evaluation metric. Section III-B defines IST3 as 'Avoid popular libraries,' and Listing 1 shows IST1–IST6 are present in every configuration, including the baseline C1. Section IV-A defines R@N as the number of recommended items matching the ground-truth divided by the total number of ground-truth items, where the ground truth is the set of libraries actually used by held-out projects. Those projects very likely depend on popular libraries (junit, slf4j, log4j, guava are the most frequent in Fig. 3). Under IST3, the model is explicitly told not to recommend such libraries, so the maximum obtainable recall is only the fraction of ground-truth libraries that are not in the excluded popular set. The paper never reports this ceiling, nor does it report N, the popularity threshold, or the composition of the ground-truth set. Without that bound, a recall of 0.20 cannot be interpreted as the model failing to address popularity bias; it may simply indicate that the instruction succeeds in steering the model away from the ground truth. The conclusion is also in tension with the paper's own EPC results (C6 reaches 60%) and coverage improvements, which suggest some bias mitigation does occur. The load-bearing assumption is therefore that recall against a popularity-saturated ground truth is a valid measure of bias mitigation under an explicit 'avoid popular' instruction; this assumption is not justified and likely forces part of the reported negative result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether open-source LLMs (Llama-2-7b-chat, Llama-2-13b-chat, Llama-3-8b-instruct) can mitigate popularity bias in third-party library (TPL) recommendation. The authors design an ablation study with six configurations (C1-C6) that combine zero-shot/few-shot prompting, conversation history, LoRA fine-tuning, and a popularity penalty mechanism. They evaluate with P@N, R@N, F1, Coverage@N, and Expected Popularity Complement (EPC). The main finding is that recall remains low (0.08-0.20) across all configurations, leading the authors to conclude that the considered LLMs cannot address popularity bias in TPL recommenders, while fine-tuning and penalty mechanisms improve diversity. A replication package is provided.","tokens_in":13123,"tokens_out":5196,"duration_ms":47834,"significance":"If the central negative result were methodologically airtight, the paper would be a useful counterpoint to optimism about LLM-based TPL recommenders, and the ablation design would help researchers avoid ineffective mitigation strategies. The choice of a reproducible, openly available setup, the inclusion of diversity-oriented metrics (Coverage@N, EPC), and the public replication package are genuine strengths. However, the current evaluation has several load-bearing gaps: the value of N is never stated, the ground-truth construction is not described, the interaction between the anti-popularity instruction (IST3) and the recall metric is unexamined, and the reported numbers lack variance estimates. These gaps prevent the paper from supporting its main claim in its present form, though the claim is potentially salvageable with additional analysis.","major_comments":[{"comment":"N is never specified for any of P@N, R@N, F1, Coverage@N, or EPC@N. Without a fixed N, the values in Table III cannot be interpreted or compared across configurations, and the study is not reproducible. Please state N explicitly and justify the choice, or report results for a range of N values.","section":"Section IV-A, Eqs. (2)-(6)"},{"comment":"IST3 (\"Avoid popular libraries\") is present in every prompt, including the baseline C1 (Listing 1), while R@N (Eq. 3) is scored against ground-truth libraries actually used by held-out projects, which Fig. 3 shows are predominantly popular (junit, slf4j, log4j, guava). This creates a ceiling on obtainable recall: the model is instructed not to recommend the very libraries that constitute a large share of the ground truth. The paper never reports this ceiling, the fraction of ground-truth items in the excluded popular set, or the composition of the ground-truth set, so recall values of 0.08-0.20 cannot be unambiguously interpreted as \"LLMs cannot address popularity bias.\" The paper's own EPC result (C6 reaches 60% vs. 15% for C1) also suggests that some mitigation is occurring. Please quantify the ceiling and either remove IST3 from the baseline or analyze recall on the non-popular subset of the ground truth.","section":"Section III-B and Section IV-A"},{"comment":"The text identifies \"C3\" as the best configuration and describes it as \"the Llama-3-8b-instruct model ... with only the few-shots technique,\" but Table I defines C3 as Llama-2-13b-chat with few-shots + history and C4 as Llama-3-8b-instruct with few-shots. This mislabeling makes the discussion of which technique drives the improvement, and the comparison with C5/C6, untrustworthy. Please correct the configuration numbering or the table.","section":"Section V-A, Table I and Table III"},{"comment":"Eq. (1) defines the penalty score, but the paper never describes how this score is applied to the generation process (e.g., logit adjustment, re-ranking, filtering, or sampling) nor how the \"Popularity Rank\" from Maven is computed and aligned with the dataset used for evaluation. Without this information, the C5 and C6 results are not reproducible, and the lack of an effect on recall cannot be attributed to the mechanism itself.","section":"Section III-D"},{"comment":"No confidence intervals, error bars, or statistical tests are reported for any metric. Several comparisons used to support the negative conclusion are small (e.g., C5 R@N=0.16 vs. C6 R@N=0.17; C5 EPC=10% vs. C1 EPC=15%), so it is unclear whether the observed differences are reliable. Please report variance across repeated runs or apply an appropriate significance test, and state the number of held-out projects in the ground-truth set.","section":"Table III"}],"minor_comments":[{"comment":"The paper uses \"LoRa\" where \"LoRA\" is standard, and \"dicussed\" should be \"discussed.\"","section":"Section III-C and Section III-D"},{"comment":"The text refers to \"the prompts defined in Section III-D,\" but the prompt templates and instructions are presented in Section III-B; Section III-D covers the penalty mechanism. Please fix the cross-reference.","section":"Section V-A"},{"comment":"The paper uses \"few-shots\" in most places but \"few-shot\" in Section III-B; please standardize the terminology.","section":"Section III-B"},{"comment":"The example output contains \"org.jsonwebtoken:jwt-simple\" and \"com.nimbusds:oauth2,\" which appear to be malformed or incomplete Maven coordinates; please verify or correct the example.","section":"Listing 4"},{"comment":"In Eq. (6), REC_r(p) is described as reflecting the \"popularity of library at position r,\" but the notation is not formally introduced and is easy to confuse with R@N. Please define all symbols either in the text or in a notation table.","section":"Section IV-A, Eq. (6)"},{"comment":"Table II lists fine-tuning hyperparameters, but configurations C1-C4 do not use fine-tuning. Please clarify which configurations actually use these settings, since the current layout suggests they apply to all rows of the ablation study.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a preliminary negative result with a useful ablation design and a public replication package. The most important issue is that the central conclusion may be an artifact of the IST3 instruction interacting with a popularity-heavy ground truth; this is fixable by reporting the recall ceiling and analyzing recall on subsets of the ground truth. The missing N, the undetailed penalty mechanism, and the absence of statistical support are also addressable in revision. If these points are handled, the paper could make a useful negative-result contribution to the RSSE community; in its current form, however, the evidence does not yet support the claimed universal inability of open-source LLMs to address popularity bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first empirical look at whether open-source LLMs can mitigate popularity bias in third-party library recommendations, and the six-configuration ablation (Llama 2/3, zero/few-shot, LoRA, penalty) is a sensible way to ask the question. Second, the central negative result—that LLMs cannot address the bias because recall stays between 0.08 and 0.20—is not supported by the experiment as reported. The stress-test note is right: IST3 (\"Avoid popular libraries\") is present in every prompt, including the baseline, while recall is scored against a ground truth of libraries actually used by held-out projects, which are presumably mostly popular. The paper never reports the composition of that ground truth or the ceiling IST3 imposes on recall. Without that, a recall of 0.20 may simply mean the instruction worked: the model avoided the libraries it was told to avoid, and those libraries were in the ground truth. The paper's own EPC numbers (C6 at 60%) and coverage improvements point the other way, so the abstract's conclusion is in tension with its own evidence.\n\nOther soft spots are more mundane but still load-bearing: N in R@N/P@N is never specified; the penalty formula in Eq. 1 is never connected to the generation process (is it a re-ranking step, a logit adjustment, what?); Section V-A calls C3 the best configuration when Table III clearly shows C4; and there are no error bars or significance tests, so even relative ordering is fragile. To its credit, the paper publishes a replication package, uses a known dataset, and the ablation logic is transparent. The contradiction between the recall narrative and the EPC/diversity results is exactly the kind of thing a careful revision could fix.\n\nThe question is real and the novelty claim is legitimate, so this deserves a serious referee rather than a desk reject. As it stands, though, I would not accept it: the headline finding is an artifact of evaluation mismatch until proven otherwise. A revision needs a configuration without IST3, documented N and ground-truth statistics, and a concrete description of how the penalty affects recommendations. For the RSSE and debiasing subfield, this could become useful negative evidence once it is actually negative.","headline":"The first LLM-based TPL debiasing study, but its central negative claim is undercut by an instruction/metric mismatch that caps recall by construction.","tokens_in":13652,"tokens_out":2455,"would_cite":false,"duration_ms":24774,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs cannot overcome popularity bias in third-party library recommenders.","keywords":["popularity bias","third-party library recommendation","large language models","recommender systems for software engineering","long-tail effect","fine-tuning","ablation study","Llama"],"falsifier":"Find the held-out Java projects whose ground-truth dependencies include popular libraries such as junit or slf4j, rerun configurations C4 and C6 with instruction IST3 (avoid popular libraries) removed but all other settings identical, and compare recall. If recall rises substantially toward the level of a non-LLM recommender such as LibSeek on the same split, the paper's conclusion that LLMs cannot address popularity bias would be weakened, because the low recall would be an artifact of the instruction conflicting with the ground truth.","tokens_in":12641,"feed_emoji":"🤖","tokens_out":6254,"duration_ms":54432,"temperature":0.7,"pith_summary":"This paper asks whether open-source large language models can counteract popularity bias when recommending third-party libraries to Java developers. Popularity bias is the tendency of recommenders to keep suggesting the same well-known libraries, such as junit or log4j, while ignoring less-used libraries that may fit the project better. The authors run an ablation study across six configurations that combine three Llama models, zero-shot and few-shot prompts, LoRA fine-tuning, and a popularity penalty score. Their central finding is that none of these configurations fixes the problem: recall stays between 0.08 and 0.20 in every configuration, even though precision, catalog coverage, and diversity improve. The practical consequence, if the result holds, is that prompting or lightly fine-tuning an LLM is not enough, and TPL recommenders need different mechanisms such as retrieval-augmented generation, post-hoc debiasing, or explicit user feedback.","feed_headline":"LLMs fail to fix popularity bias in library recommenders","feed_subtitle":"Six Llama configurations kept recall between 0.08 and 0.20, even with fine-tuning and a popularity penalty.","key_machinery":"The argument is carried by an ablation study with six configurations, built on the CrossRec dataset of Java projects and their dependencies. Three variables are manipulated: prompt strategy (zero-shot, few-shot, few-shot with history), model (Llama-2-7b-chat, Llama-2-13b-chat, Llama-3-8b-instruct), and two debiasing modules. The first debiasing module is parameter-efficient fine-tuning via LoRA (rank 16, $\\alpha$ 32, dropout 0.05) applied only in configuration C6. The second is a popularity penalty score defined as $1/(\\text{popularity rank}+1)$, computed from Maven usage counts, which is used to lower the likelihood of recommending frequently used libraries. The outcome is measured with precision, recall, F1, catalog coverage, and Expected Popularity Complement (EPC), and the pattern across configurations is what supports the negative conclusion.","core_discovery":"On the paper's own terms, the discovery is a negative result: the considered open-source LLMs cannot address popularity bias in third-party library recommendation. Across the six ablation configurations (C1–C6), recall never exceeds 0.20, with the best recall achieved by Llama-3-8b-instruct with few-shot prompting alone (C4), while the configuration that adds both fine-tuning and the popularity penalty (C6) reaches only 0.17 recall but raises catalog coverage to 55% and EPC to 60%. The authors interpret this as evidence that the long-tail effect persists in LLM-generated recommendations and that fine-tuning and post-processing penalties mainly increase diversity rather than relevance. They therefore conclude that substantially different mitigation strategies are needed.","pith_inferences":["The negative result is probably stronger than the paper states in one way and weaker in another: the penalty mechanism's formula $1/(\\text{rank}+1)$ is applied as a post-hoc reweighting that never changes the model's internal predictions, so a failure to reach high recall with it does not rule out better constrained-decoding or re-ranking approaches.","A direct testable extension would be to swap the popularity penalty for a calibrated log-popularity term or a learned re-ranker; if recall then approaches a traditional recommender's level, the bottleneck would be the debiasing method, not the LLM.","The same evaluation design could be applied to code-specific open models such as CodeLlama or CodeMistral, which the authors mention as future work; a comparative result there would show whether the failure is generic to LLMs or specific to chat-tuned general models.","The instruction-based avoidance of popular libraries may trade accuracy for novelty in a way that the metrics do not disentangle; a user study asking developers whether the diverse recommendations are actually useful would be the natural follow-up."],"forward_implications":["Fine-tuning and the popularity penalty do not rescue recall; the best configuration reaches only 0.20 recall, so developers cannot rely on these cheap interventions alone.","The debiasing modules do improve diversity: catalog coverage rises from 26% in the baseline to 55% in C6, and EPC from 15% to 60%, so the modules are not inert.","Because the gains in precision and diversity do not translate into recall, the long-tail effect in TPL recommendations remains unresolved for open-source LLMs.","The authors' proposed next steps—retrieval-augmented generation, post-processing debiasing algorithms, and explicit user feedback—are the directions that would need to be tested next."],"supporting_citations":[{"why":"Supplies the dataset of Java projects and third-party library usage that the ablation study evaluates on.","marker":"[11]"},{"why":"Defines the Llama architecture and chat models whose recommendations are measured across the six configurations.","marker":"[28]"},{"why":"Provides evidence that LLMs suffer from long-tailed code distributions, motivating the research question.","marker":"[26]"},{"why":"Shows that traditional TPL recommenders still fail to handle popularity bias, giving the baseline this study extends to LLMs.","marker":"[19]"},{"why":"Supplies the LoRA parameter-efficient fine-tuning method used in configuration C6.","marker":"[37]"},{"why":"Inspires the popularity penalty mechanism as a re-weighting strategy to promote less popular libraries.","marker":"[9]"}],"fun_headline_variants":["LLMs can't shake popularity bias in library picks","Popularity bias persists in LLM library recommendations","Fine-tuning only diversifies, doesn't de-bias LLM library picks","Six LLM setups all fail to beat popularity bias","LLMs can't fix library recommendation popularity bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that asking the model to avoid popular libraries (instruction IST3) is compatible with a ground truth made of the libraries that held-out projects actually use; when popular libraries appear in that ground truth, the instruction itself suppresses recall, so the measured failure may be partly forced by the experiment rather than by the model's inability to address popularity bias.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't shake popularity bias in library picks","Popularity bias persists in LLM library recommendations","Fine-tuning only diversifies, doesn't de-bias LLM library picks","Six LLM setups all fail to beat popularity bias","LLMs can't fix library recommendation popularity bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3119,"prompt_tokens":914,"completion_tokens":2205,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2126}},"tokens_in":530,"tokens_out":2205,"duration_ms":15425,"temperature":1.0,"reasoning_tokens":2126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:13:12.585861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find the held-out Java projects whose ground-truth dependencies include popular libraries such as junit or slf4j, rerun configurations C4 and C6 with instruction IST3 (avoid popular libraries) removed but all other settings identical, and compare recall. If recall rises substantially toward the level of a non-LLM recommender such as LibSeek on the same split, the paper's conclusion that LLMs cannot address popularity bias would be weakened, because the low recall would be an artifact of the instruction conflicting with the ground truth.","supporting_citations":[{"cited_title":"CrossRec: Supporting Software Developers by Recommending Third-party Libraries,","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of Java projects and third-party library usage that the ablation study evaluates on."},{"cited_title":"Dealing with popularity bias in recommender systems for third-party libraries: How far are we?","cited_arxiv_id":null,"evidence_quote":"Shows that traditional TPL recommenders still fail to handle popularity bias, giving the baseline this study extends to LLMs."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA parameter-efficient fine-tuning method used in configuration C6."},{"cited_title":"Diversified third-party library prediction for mobile app development,","cited_arxiv_id":null,"evidence_quote":"Inspires the popularity penalty mechanism as a re-weighting strategy to promote less popular libraries."}],"review_version":1}