{"id":"ac5d7a93-5ee7-4a67-bae3-6cb7536954b1","arxiv_id":"1908.04122","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Publication venue, past citations of the author team, paper length, reference count, and reference recency are the strongest predictors of citations to software engineering journal papers.","lead":"This paper analyzes 25,113 software engineering journal papers to see which paper, journal, and author characteristics best predict citations. It is the first multivariate study of citation factors in software engineering, and most results echo patterns already known in other fields.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'most influential factors' claim and the paper's advice assume the measured predictors are not proxies for unmeasured paper quality; the authors cannot rule out that venue, length, and references merely track quality, yet the Abstract presents these as actionable causes.","rationale":"The reader's weakest assumption identified the same load-bearing concern: omitted paper quality confounds the predictor-citation associations, and the paper's prescriptive advice over-interprets correlational findings. My review confirms this concern is the most serious threat to the central claim. In good faith, I note the paper has independent support: the negative binomial and quantile regressions are appropriate for count data, the dataset is large and transparently described, the authors acknowledge data limitations (e.g., TOSEM missing pages, author disambiguation), and they explicitly caveat the halo-effect ambiguity in Section 7. These strengths mean the descriptive finding (these factors are associated with citations) is reasonably supported. The load-bearing weakness is the step from association to actionable advice, and the abstract's phrasing 'most influential factors' implies a causal ranking that the design cannot establish. The proposed concrete test—augmenting the model with external rigor scores or using within-author-team variation—would directly assess whether the coefficients are robust to a quality proxy. Because the reader already conditioned the verdict on this issue, my stress-test does not move the verdict; UNCHANGED is appropriate. The paper should be accepted only with the condition that the causal-framed advice be softened or explicitly labeled as associational pending such a robustness check.","tokens_in":34471,"tokens_out":7344,"duration_ms":79574,"concrete_test":"Re-estimate the complete negative binomial model (Table 16) on the subset of papers that overlap with Molléri et al.'s 718 SE papers (ref. [45]) that have independent research-rigor scores, adding the rigor score as a covariate. If the coefficients on venue dummies, log(Pages), References, or Price change by more than 20% or lose significance, the 'most influential factors' claim is confounded by unmeasured quality. If the overlap is insufficient, run a within-author-team fixed-effects model comparing papers by the same author team; substantial attenuation of the log(Pages) and References coefficients would indicate the associations are artifacts of author quality rather than direct influences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim and the Section 7 advice require that the estimated associations between venue, page count, reference count/recency, and citations are not confounded by unmeasured paper quality. Section 3.1 explicitly excludes 'subjective' factors (quality, novelty, study design) for lack of data, so the complete models (Tables 16-18) control only for objective proxies. If higher-quality work is longer, better referenced, and placed in higher-impact venues, then the positive coefficients on these predictors overstate their independent 'influence,' and the advice to 'write longer papers' or 'aim for high-profile venues' misattributes what actually belongs to quality. The authors acknowledge this uncertainty for venue and past citations ('we cannot say whether this is due to paper quality or halo effect,' Section 7), yet the Abstract and Conclusions still present these as actionable recommendations. Additionally, the venue/impact-factor analysis is partly circular: a journal's impact factor is derived from the citations of its papers, so regressing citations on venue partly regresses citations on journal-level aggregates of the same outcome. The paper correctly frames its RQs as exploratory and descriptive, but the prescriptive phrasing of the conclusions goes beyond what the observational design can support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies factors associated with citation counts of software engineering (SE) journal papers. Using a Scopus dataset of 25,113 papers from 16 SE journals published between 1970 and 2018, the authors fit negative binomial regression models (for the mean) and quantile regression models (for the median) to examine paper age, publication venue, journal impact-factor metrics, author-team characteristics, and paper properties such as length, reference count, reference recency, paper type, and title length. The headline findings are that venue, author team's past citations, paper length, number of references, and recency of references are the most influential factors, and the paper closes with prescriptive advice for researchers. The results are compared with two prior reviews of citation factors in other fields.","tokens_in":34765,"tokens_out":2981,"duration_ms":31412,"significance":"If the results hold, this would be a useful contribution as the first multivariate study of citation determinants specifically for SE journals, with a large corpus and two complementary regression approaches. The paper also provides a systematic comparison with earlier reviews in Table 23, and the authors are candid about several limitations, including missing TOSEM page data and the difficulty of disambiguating author profiles. The public availability of the analysis scripts is a further strength. However, the observational design and the construction of some predictors substantially limit the strength of the causal and prescriptive claims, and several technical issues in the impact-factor analysis and model comparison need to be resolved before the headline conclusions can be accepted at face value.","major_comments":[{"comment":"The impact-factor models are circular in an important sense. Each journal's CiteScore, SNIP, and SJR values are computed from citations to papers in that journal, including the very papers in the dataset, and the authors assign the same average score to all papers of a journal for all years (Section 4.2 and Table 8). Regressing individual citation counts on these journal-level averages therefore introduces a mechanical correlation: the outcome of a paper contributes to the predictor assigned to that same paper. This affects the 'impact factor is a strong predictor' conclusion and the complete models in Tables 17 and 19. The dummy-variable venue models (Tables 16 and 18) are not subject to this problem and give broadly similar conclusions, which mitigates the concern, but the impact-factor results should either be re-derived with lagged or leave-one-out impact factors, or explicitly presented as descriptive only, with the circularity acknowledged in the interpretation.","section":"Section 4.2, Tables 9, 10, 17, 19"},{"comment":"The complete regression models in Tables 16 and 18 control only for objective proxies and explicitly exclude 'subjective' factors such as paper quality, novelty, and study design for lack of data. The paper's headline claim that venue, author team's past citations, paper length, and reference characteristics are 'most influential' and the Section 7 advice ('aim for high-profile venues', 'write longer papers', 'build a high-quality author team') attribute causal or quasi-causal influence to variables that may merely track unmeasured paper quality. The authors do note in Section 7 that they cannot distinguish quality from halo effects, but the abstract and conclusions nevertheless present these as actionable recommendations. The manuscript should reframe the conclusions as associations, add a sensitivity discussion of likely omitted-variable directions, or both.","section":"Section 3.1, Table 1, Section 7"},{"comment":"AIC is used throughout to compare quantile regression models, including the RQ1 and RQ5 model-selection decisions (e.g., AIC values reported in Sections 4.1 and 4.2). However, quantile regression is estimated by minimizing a sum of absolute deviations and does not have a likelihood function in the usual sense, so the stated formula AIC = 2k - 2ln(L) is not directly applicable. The authors should justify the use of AIC for quantile regression models, provide the exact quantity they computed, or replace it with a comparison criterion that is valid for quantile regression, such as cross-validated prediction error for the median.","section":"Section 3.4 and Tables 4-19"},{"comment":"In the complete quantile regression model, TOSEM has a coefficient of 33.93 with a standard error of 8.08, based on only 199 papers after excluding papers with missing page counts; the authors themselves note that this subset is biased toward older papers and makes the coefficient volatile. This large and unstable coefficient is nonetheless used in Section 5.2 and Table 23 as evidence about venue rankings. Given the acknowledged data problem, the TOSEM-specific numerical results in the complete quantile model should not be interpreted as a reliable venue effect, and the venue-ranking discussion should be correspondingly qualified.","section":"Section 4.5, Table 18"}],"minor_comments":[{"comment":"The sentence 'negative binomial regression is better as it does not require transforming the independent variables to meet the requirement of the normal distribution that is precursory of independent variables in linear multiple regression' is unclear and should be rewritten; count regression does not require normality of the outcome or predictors.","section":"Section 3.3, paragraph on negative binomial regression"},{"comment":"The caption reads 'Coefficient of binomial negative regression'; this should be 'negative binomial regression'.","section":"Table 6 caption"},{"comment":"The text says 'we compared tree metrics from Scopus'; this should be 'three metrics'.","section":"Section 5.2, first paragraph"},{"comment":"The subsection heading 'Paper (venue) Type' is confusing; the variable is the Scopus publication type, not a venue type, and the heading should be changed accordingly.","section":"Section 4.4"},{"comment":"The paper removes different numbers of papers for missing author, affiliation, country, and page-count data (e.g., 975, 1,282, 910), and AIC values are compared across models with different sample sizes at some points, but this is not consistently flagged. The authors should state the effective sample size for every model whose AIC is compared.","section":"Section 4.3 and 4.4"},{"comment":"The 'TP' column reports the number of models offering support, but the counting rule is not fully transparent; for example, author count is significant in some but not all models and the table entry '3 / 1' should be explained in the notes with reference to the exact models.","section":"Table 23"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid dataset and a clear multivariate approach, and I do not think the issues are fatal. The circularity of the impact-factor models and the questionable use of AIC for quantile regression are fixable with targeted reanalysis or reframing, and the omitted-variable problem can be addressed by softening the causal language. I would encourage the editor to send this back for revision rather than reject. One additional editorial concern: the 'first study in SE' novelty claim is somewhat strong given that the authors themselves cite prior SE bibliometric work [20, 45]; the novelty is better framed as the first multivariate regression-based analysis rather than the first study of multiple factors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Mäntylä and Garousi paper. It is what it says: the first SE-specific multivariate analysis of citation predictors, using 25k journal papers and two regression approaches—negative binomial for the mean, quantile for the median. The statistical work is mostly careful: age gets a nonlinear treatment, models are compared with AIC, and the authors are open about Scopus problems (author disambiguation, missing page counts for TOSEM). They also make their analysis scripts available. The genuinely new result is the sign flip: once the author team's past citations are in the model, the number of past papers turns negative. That is worth reporting and is well-flagged.\n\nI part ways with the paper on the \"most influential factors\" claim and on the advice in Section 7. The coefficients in Tables 16/18 are not a formal variable-importance analysis, so \"most influential\" asks the method to do more than it does. And because the models exclude exactly the factors that could be doing the real work—paper quality, novelty, study design—advising people to aim for high-profile venues and write longer papers attributes to proxies what may just be correlated with quality. The authors do admit they cannot separate quality from halo for venue and past citations, but the Abstract and Conclusions still present the findings as actionable. That overstates an observational study with omitted confounders.\n\nThe circularity concern about the impact-factor models is real but contained: using journal-level average scores to predict individual citations builds in a mechanical link. The dummy-variable venue models avoid that and give the same broad conclusions, so the central qualitative finding holds up. The TOSEM data problem is real but well-documented and does not threaten the headline.\n\nNet: this is a competent, transparent paper that deserves a serious referee. It is a useful data point for anyone working on research evaluation in SE or doing comparative bibliometrics. I would recommend major revision rather than rejection—tone down \"most influential,\" move the quality-confounding caveat into the abstract, and keep the negative productivity result prominent. I would already cite it for that result; with revisions it becomes the default citation-factor reference for the field.","headline":"First multivariate citation-factor study for SE with a real counterintuitive finding, but the advice section outruns the observational design.","tokens_in":35221,"tokens_out":4752,"would_cite":true,"duration_ms":50279,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The measurable predictors of a software engineering paper's citation count are venue, the team's past citations, length, reference count, and reference recency.","keywords":["citations","software engineering","bibliometrics","negative binomial regression","quantile regression","Price index","publication venue","citation factors"],"falsifier":"Re-run the complete regression on a subset of software engineering papers independently rated for research rigor using a published rubric; if adding that rigor score drives the coefficients for venue, log page count, reference count, and Price index to near zero, then the paper's reported associations are confounded by unmeasured quality.","tokens_in":34297,"feed_emoji":"📊","tokens_out":4813,"duration_ms":49730,"temperature":0.7,"pith_summary":"This paper aims to establish which measurable features of a software engineering journal paper predict how many citations it will receive, using all 25,113 papers published in 16 major SE journals between 1970 and 2018. It claims to be the first SE study to use multivariate regression rather than isolated pairwise comparisons, fitting both negative binomial regression for mean citations and quantile regression for median citations. The authors report that publication venue, the author team's past citations, paper length, the number of references, and the recency of references are the most influential factors, with paper age controlled throughout. They also report a counterintuitive result: once past citations are accounted for, an author team's total number of past papers is negatively associated with citations, meaning a short but highly cited track record predicts higher citations than a long publication list. A sympathetic reader would care because the findings turn the vague sense that extrinsic factors matter into a ranked, quantified list that could inform how research evaluations weight citations.","feed_headline":"Venue and track record drive software paper citations","feed_subtitle":"First multivariate study of 25,113 papers shows which factors predict citations—and which don't.","key_machinery":"The analytical engine is a pair of count-data regressions fitted to the full corpus with a common age control. Negative binomial regression models the expected mean citation count, and quantile regression at the median models the expected median, which matters because the distribution is skewed (mean 22.5 citations, median 6). Predictors are added stepwise and retained only when the Akaike Information Criterion improves, giving a principled ranking of factors. The Price index, defined as the share of a paper's references published in the five years before the paper, operationalizes reference recency and emerges as one of the strongest paper-level predictors.","core_discovery":"The central claim is that citation counts of software engineering journal papers can be modeled jointly by paper age, venue, author-team history, and paper characteristics, and that the dominant predictors are venue, author team's past citations, log-transformed page count, number of references, and reference recency as measured by the Price index. Across the individual and complete models, for both mean and median citations, these factors are statistically significant at p < 0.0001, while author-team size loses significance in the complete mean model once venue and paper factors are added. A distinctive finding is the sign reversal of author-team productivity: with past citations in the model, more past papers is associated with fewer expected citations. The paper frames its contribution as the first robust multivariate analysis of citation factors in software engineering, extending earlier SE work that had treated factors in isolation.","pith_inferences":["Because the dataset contains no direct measure of paper quality, the advice to write longer papers or aim for high-profile venues likely overstates causal control; a natural experiment such as a submission lottery or a page-limit discontinuity would be needed to separate author and venue effects from content quality.","The negative coefficient on past papers suggests a selective 'best work' signaling effect rather than a pure productivity effect; a testable extension would compare first-authored versus team-authored papers to see whether the association is an aggregation artifact.","The venue results imply that evaluators comparing SE papers across venues should normalize or weight citation counts by venue, an idea the authors mention but do not develop.","The Price index result may be partly circular: papers with recency-heavy reference lists tend to sit in fast-moving subfields, and controlling for publication age may not fully separate citing recent work from being in a fast-moving field."],"forward_implications":["Authors publishing in lower-impact venues face an uphill battle: venue alone shifts expected citations substantially even after controlling for age and other factors.","A short but highly cited publication track record predicts higher future citations than a long list of past papers with the same citation count.","Comprehensiveness pays: longer papers with more references and fresher references receive more citations, consistent with reciprocity and thorough literature coverage.","The standard two-year impact-factor window is ill-suited to software engineering, since citation accumulation continues for roughly 15 years for highly cited papers.","Mean and median citation models largely agree on the major factors, so the conclusions are not an artifact of the highly skewed citation distribution."],"supporting_citations":[{"why":"Supplies the comprehensive taxonomy of 28 citation factors and the prior evidence counts used for comparison.","marker":"[53]"},{"why":"Provides the prior multivariate analysis in other fields, including the Price index finding and the comparison baseline.","marker":"[47]"},{"why":"The earlier SE study relating citations to research rigor, used as a contrast for the present multivariate approach.","marker":"[45]"},{"why":"The prior SE bibliometric study this work extends, supplying the venue list and per-venue citation baselines.","marker":"[20]"},{"why":"Earlier SE bibliometric dataset and comparative average citation counts used to benchmark the present results.","marker":"[21]"},{"why":"Defines negative binomial regression, the primary method for modeling mean citation counts.","marker":"[27]"},{"why":"Defines quantile regression, the method used to model median citations.","marker":"[39]"},{"why":"Defines the Price index, the measure of reference recency that is a key predictor.","marker":"[48]"},{"why":"Provides the Akaike Information Criterion used to decide which predictors enter the final models.","marker":"[4]"}],"fun_headline_variants":["First multivariate study: venue and author history drive citations","25k SE papers show venue and team history are citation keys","Venue and author track record dominate SE citation counts","Venue and past citations predict SE paper citations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that no unmeasured property like genuine technical quality is driving both the visible predictors (venue, length, references) and the citation counts, since the dataset contains no direct measure of paper quality and the authors explicitly exclude quality and novelty from the factors considered.","fun_headline_variants_meta":{"raw":{"variants":["First multivariate study: venue and author history drive citations","25k SE papers show venue and team history are citation keys","Venue and author track record dominate SE citation counts","Venue and past citations predict SE paper citations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2740,"prompt_tokens":919,"completion_tokens":1821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1757}},"tokens_in":535,"tokens_out":1821,"duration_ms":13853,"temperature":1.0,"reasoning_tokens":1757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:50:39.146352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the complete regression on a subset of software engineering papers independently rated for research rigor using a published rubric; if adding that rigor score drives the coefficients for venue, log page count, reference count, and Price index to near zero, then the paper's reported associations are confounded by unmeasured quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the comprehensive taxonomy of 28 citation factors and the prior evidence counts used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior multivariate analysis in other fields, including the Price index finding and the comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier SE study relating citations to research rigor, used as a contrast for the present multivariate approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior SE bibliometric study this work extends, supplying the venue list and per-venue citation baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier SE bibliometric dataset and comparative average citation counts used to benchmark the present results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines negative binomial regression, the primary method for modeling mean citation counts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines quantile regression, the method used to model median citations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Price index, the measure of reference recency that is a key predictor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Akaike Information Criterion used to decide which predictors enter the final models."}],"review_version":1}