{"id":"1d22a8e4-e3ee-4299-bd01-afe431200dfd","arxiv_id":"1908.04219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The most popular GitHub projects promote themselves mainly through Twitter, user meetings, and blogs, and successful Hacker News posts coincide with an increase in GitHub stars.","lead":"This study examined how the 100 most popular GitHub projects promote themselves, finding that Twitter, user meetings, and blogs are the most common channels. It also reports that successful posts on Hacker News are followed by a rise in the number of GitHub stars a project receives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hacker News causal effect is confounded: successful posts are selected post-hoc and mostly coincide with releases that independently drive stars.","rationale":"The reader's weakest assumption correctly identifies the possibility of concurrent events masking the Hacker News effect. My stress-test sharpens that concern into a concrete mechanism: the definition of 'successful posts' by upvotes, combined with the paper's own categorization of those posts, means the treatment is not Hacker News exposure but a major project event that would likely produce star gains by itself. This is a load-bearing threat to the central finding in Section 3.4, not a minor caveat. However, the reader already issued a CONDITIONAL verdict with medium confidence and explicitly called out the confounder assumption as the weakest point, so my analysis does not shift the verdict; it reinforces the same conditionality. The paper's descriptive results about promotion channels in Sections 3.1-3.3 remain useful and credible, and the Hacker News analysis is honestly presented as a before/after observation rather than a controlled experiment. The missing data release also limits independent verification, but the primary concern is statistical identification, not data availability. I would keep the conditional verdict and require either a matched-control analysis or an explicit acknowledgment that the Hacker News effect cannot be separated from release effects.","tokens_in":5661,"tokens_out":2525,"duration_ms":30741,"concrete_test":"For each successful Hacker News post categorized as announcing a new release, identify the associated GitHub release timestamp. Build a control set of releases in the same project or in comparably sized projects with similar language, age, and prior star velocity, matched within a 60-day window, that were either not posted on Hacker News or whose Hacker News post did not reach the top 10% by upvotes. Compare the median star gain in the three days following the release. If control releases show an increase of similar magnitude to the 138-star median reported for successful Hacker News posts, then the Hacker News-specific effect claimed in Section 3.4 is not supported. Alternatively, fit a regression discontinuity at the Hacker News post timestamp that includes a release-day indicator; if the release indicator absorbs the effect, the causal claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The causal claim in Section 3.4 rests on a before/after star comparison around successful Hacker News posts, but successful posts are defined post-hoc as the top 10% by upvotes (at least 132). This selection mechanic makes the comparison endogenous: posts that become successful are disproportionately about major project milestones. The paper's own categorization supports this: 44.9% of successful posts announce new releases, 16.5% announce first releases, 10.6% highlight new features, and 1.6% announce open-sourcing. These events cause star increases regardless of where they are posted, so the observed median increase from 74 to 138 stars in Figure 8 may reflect release-day momentum or coordinated announcements rather than Hacker News promotion. The analysis also pools all successful posts across projects and uses a one-tailed Mann-Whitney U test on the before/after distributions, which ignores non-independence: many projects have multiple successful posts, and the three-day windows for posts from the same project can overlap. Without a control group of comparable releases not posted on Hacker News, or a difference-in-differences design around the exact post time, the central claim that Hacker News promotes open source projects is not identifiable from these data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a manual study of promotion channels used by 100 top-starred GitHub repositories, compares them with 100 repositories randomly selected from the top-5,000, and measures the association between successful Hacker News posts and GitHub star growth. The authors find that Twitter, user meetings, and blogs are the most common channels, that popular projects use these channels more often than the random sample, and that median stars gained in the three days after successful Hacker News posts (138) exceed those gained in the three days before (74). They conclude with practical recommendations for open source project maintainers.","tokens_in":5914,"tokens_out":4296,"duration_ms":45088,"significance":"If taken as a descriptive measurement study, the paper is a useful addition to the empirical software engineering literature: the channel taxonomy is concrete, the top-100 versus random-sample comparison is a sensible design, and the Hacker News effect-size estimate is transparently reported. However, the causal claim that Hacker News has a positive impact on project popularity rests on a before/after comparison without a control group, and the paper's own post-category data show that successful posts predominantly announce releases. The manuscript therefore needs a substantially strengthened causal analysis, or a reinterpretation as associational evidence, before the central RQ3 conclusion is supported.","major_comments":[{"comment":"The before/after comparison in Figure 8 does not identify a causal effect of Hacker News because 'successful' posts are defined post hoc as the top 10% of posts by upvotes (at least 132 upvotes), and those posts overwhelmingly coincide with major project events: 44.9% announce new releases, 16.5% first releases, 10.6% new features, and 1.6% open-sourcing. Such events plausibly drive GitHub stars independently of whether they are discussed on Hacker News, so the median increase from 74 to 138 stars may reflect release-day momentum rather than promotion via Hacker News. The analysis should include a control group of comparable releases or announcements not posted on Hacker News, a difference-in-differences specification, or project-level fixed effects that absorb time-invariant project popularity; without such controls, the abstract's and Section 5's 'Hacker News promotes' statements should be downgraded to associational language.","section":"Section 3.4, Fig. 8"},{"comment":"The statistical test treats each successful post as an independent observation, but projects contribute multiple posts and the three-day windows around posts from the same project can overlap, violating the independence assumption and likely inflating statistical significance. The authors should account for clustering by project (e.g., a mixed-effects model, a per-project paired test, or a block bootstrap) and report the number of distinct projects contributing to Figure 8.","section":"Section 3.4, Mann-Whitney U test"}],"minor_comments":[{"comment":"Because the promotion-channel coding is manual and performed by a single author, the paper should report an inter-rater reliability check or at least state the coding rules for ambiguous cases, such as when a README links to a Twitter account that is neither official nor verified.","section":"Section 2"},{"comment":"The phrase 'first three days before and after the publication date' should be made precise; if star counts are timestamped per hour, specify whether the windows are exact 72-hour intervals or inclusive calendar days, since this affects the count and the comparison.","section":"Section 3.4"},{"comment":"The boxplot appears truncated at 600 stars while the text reports a median of 138; clarify the y-axis treatment of outliers so that the visual scale does not mislead readers.","section":"Section 3.4, Fig. 8"},{"comment":"Typos should be corrected: 'appple/swift' (Section 3.4), 'torvalds/linuxhas' (Section 3.2), and 'Maleej' (Section 4, should be 'Maalej').","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The descriptive contribution is the core strength of the paper. The RQ3 causal claim is not identified from the current before/after design; either substantially strengthen the identification or reframe the conclusion as associational. I would not reject on the current evidence, but the abstract and Section 5 overstate what the data can support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look, but not for the reason it thinks. The real value is the descriptive census: 100 top GitHub projects manually inspected for promotion channels, with Twitter at 56, user meetings at 41, blogs at 38, plus a randomized comparison showing popular projects use channels more heavily and that the channel rankings are strongly correlated (rho = 0.904). That part is straightforward, reproducible in principle, and new enough to be useful to maintainers and to MSR researchers.\n\nThe weak spot is Section 3.4. The claim that Hacker News has an impact on popularity rests on a before/after star comparison around the top 10% of posts, selected post hoc as successful (at least 132 upvotes). Those posts are disproportionately about new releases (44.9%), first releases (16.5%), new features (10.6%), and open-sourcing (1.6%). Events like that drive stars whether or not they appear on Hacker News, and without a control group of releases not posted there, or a difference-in-differences design, the observed median jump from 74 to 138 stars cannot be attributed to Hacker News. The analysis also pools multiple posts from the same projects and uses a one-tailed Mann-Whitney U test on non-independent observations; the medium Cliff's delta does not fix that. This is not a minor caveat—it is the paper's most actionable claim, and it is not identifiable from these data.\n\nElsewhere I have no serious reservations. The manual channel classification is thoughtful; they correctly exclude Q&A and discussion forums from promotion, and the distinction between events and user meetings is reasonable. The comparison between top and random projects is simple but the right kind of evidence. No shared data or replication package is mentioned, which is a small sustainability problem for a study that depends on manual labels.\n\nCitation pattern is fine. The self-citation to their earlier popularity-factors work is routine background, not a load-bearing dependency.\n\nWho gets value: open source maintainers thinking about promotion, and MSR people who want a baseline descriptive study. The paper deserves a serious referee but should not be accepted as is; Section 3.4 needs reanalysis or an explicit reframing as association. I would send it out.","headline":"Useful descriptive census of promotion channels for top GitHub projects, but Section 3.4's Hacker News effect is confounded by release-driven star gains and post-hoc success selection.","tokens_in":6289,"tokens_out":2261,"would_cite":true,"duration_ms":24377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that popular GitHub projects promote themselves mainly through Twitter, user meetings, and blogs, and that successful Hacker News posts coincide with a doubling of the median GitHub stars gained in the following three…","keywords":["open source promotion","GitHub","Hacker News","Twitter","user meetings","blogging","software popularity","social media"],"falsifier":"Repeat the Hacker News measurement on a control set of GitHub projects with similar pre-post star trajectories but without any successful Hacker News post; if the control projects show the same median growth from 74 to 138 stars in the same calendar windows, the claimed promotional effect would be contradicted. Alternatively, remove from the analysis any post immediately followed by a project release or a concurrent major announcement, and check whether the 74-to-138 gap survives.","tokens_in":5514,"feed_emoji":"📣","tokens_out":3539,"duration_ms":35619,"temperature":0.7,"pith_summary":"The paper investigates how developers promote open source projects by manually inspecting the top 100 most-starred GitHub repositories. It claims that Twitter, user meetings, and blogs are the most common promotion channels, used by 56, 41, and 38 projects respectively, and that popular projects use these channels far more than a random sample. It also claims that successful Hacker News posts—the top 10 percent by upvotes—are associated with a median increase in stars gained over three days from 74 to 138. A sympathetic reading is that promotion is a real and measurable part of open source project management, not an afterthought. The paper concludes with practical recommendations for project leaders to invest in Twitter, user meetings, blogs, and news aggregator sites.","feed_headline":"Hacker News success doubles GitHub projects' star gains","feed_subtitle":"Twitter, user meetings, and blogs dominate; top Hacker News posts lift median three-day star gains from 74 to 138.","key_machinery":"The machinery is a manually curated taxonomy of promotion channels—blogs, events, user meetings, Twitter, Facebook, Google+, newsletters, and RSS feeds—built by inspecting READMEs and project websites, plus a before-and-after comparison around Hacker News posts. User meetings are operationalized through Meetup topic groups, and Hacker News data come from its public API. The load-bearing comparison is the star-gain measurement in the three days before versus after each successful post, where 'successful' means a post in the top 10 percent by upvotes. That comparison is what converts the descriptive channel survey into a claim about promotion impact.","core_discovery":"The central discovery is a descriptive map of promotion practice among the most popular GitHub projects, together with quantitative evidence that successful promotion on Hacker News coincides with increased popularity. On the paper's own terms, the most common channels are Twitter (56 projects), user meetings (41), and blogs (38), with events and RSS feeds tied at 33; Facebook and Google+ are rare. Popular projects use more channels than a random sample, while the ranking of channels is strongly correlated across the two groups. For Hacker News, the paper finds 3,019 posts referencing 96 of the 100 projects, but attention is highly skewed: the top 10 percent of posts receive at least 132 upvotes. Comparing the three days before and after such successful posts, the median number of stars gained rises from 74 to 138, a difference the paper reports as statistically significant under a one-tailed Mann-Whitney U test with medium effect size.","pith_inferences":["The paper leaves causality open: its before-and-after design cannot rule out that a project's own momentum or a coinciding release caused both the Hacker News success and the star increase. A natural extension would compare projects with successful Hacker News posts against matched projects without such posts in the same calendar window.","The finding implies that open source managers should not expect average Hacker News posts to drive growth, since the distribution is extremely skewed; the practical question becomes how to craft the rare post that reaches the front page, which the paper does not address.","The channel taxonomy could be extended to measure not just presence but sustained activity, for example linking blog post frequency or Twitter cadence to star growth over months, which would tell managers whether consistency matters more than occasional spikes.","The paper's method could be applied to other aggregator communities such as Reddit to test whether the Hacker News effect is specific to that site or generalizes to any high-traffic technology forum."],"forward_implications":["If the paper is right, promotion is a distinguishing practice of successful open source projects, with two thirds of top projects using at least one channel and more than half using at least two.","Twitter is the channel most worth investing in: it is the most common, and 83.9 percent of projects with a Twitter account keep it active.","User meetings are a substantial promotion mechanism, led by torvalds/linux with 2,261 Meetup groups across 96 countries.","Successful Hacker News posts coincide with a doubling of median three-day star gains, so a single high-attention post can matter more than routine social media activity.","Hacker News outcomes are highly unequal: only 10 percent of posts attract substantial attention, so project leaders should treat front-page success as rare rather than typical.","The ranking of channels in popular projects strongly correlates with the ranking in a random sample, suggesting that popularity changes the intensity of promotion more than the choice of channels."],"supporting_citations":[{"why":"Cited for the premise that attracting participants and contributors increases the long-term success chances of open source projects.","marker":"[1]"},{"why":"Cited to justify GitHub stars as the measure of project popularity used throughout the study.","marker":"[3]"},{"why":"Provides the prior study of open source marketing and communication strategies that the paper contrasts with its own channel survey.","marker":"[5]"},{"why":"Supplies the prior finding that developers post links on news aggregators mainly to promote their own work, motivating the Hacker News analysis.","marker":"[10]"}],"fun_headline_variants":["Successful Hacker News posts double GitHub star gains","Top Hacker News posts lift GitHub stars from 74 to 138","Hacker News bumps GitHub stars; Twitter, blogs top channels","How devs promote open source: Twitter, meetups, blogs, Hacker News","Open source promotion: Hacker News success doubles star gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The before-and-after star comparison assumes that no other event—such as a new release or other media coverage—coincides with the successful Hacker News posts, so the observed star increase can be attributed to the posts themselves.","fun_headline_variants_meta":{"raw":{"variants":["Successful Hacker News posts double GitHub star gains","Top Hacker News posts lift GitHub stars from 74 to 138","Hacker News bumps GitHub stars; Twitter, blogs top channels","How devs promote open source: Twitter, meetups, blogs, Hacker News","Open source promotion: Hacker News success doubles star gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2950,"prompt_tokens":860,"completion_tokens":2090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":2001}},"tokens_in":476,"tokens_out":2090,"duration_ms":13702,"temperature":1.0,"reasoning_tokens":2001,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:46:48.051717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the Hacker News measurement on a control set of GitHub projects with similar pre-post star trajectories but without any successful Hacker News post; if the control projects show the same median growth from 74 to 138 stars in the same calendar windows, the claimed promotional effect would be contradicted. Alternatively, remove from the analysis any post immediately followed by a project release or a concurrent major announcement, and check whether the 74-to-138 gap survives.","supporting_citations":[{"cited_title":"Comino, F","cited_arxiv_id":null,"evidence_quote":"Cited for the premise that attracting participants and contributors increases the long-term success chances of open source projects."},{"cited_title":"Borges, A","cited_arxiv_id":null,"evidence_quote":"Cited to justify GitHub stars as the measure of project popularity used throughout the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior study of open source marketing and communication strategies that the paper contrasts with its own channel survey."},{"cited_title":"Aniche, C","cited_arxiv_id":null,"evidence_quote":"Supplies the prior finding that developers post links on news aggregators mainly to promote their own work, motivating the Hacker News analysis."}],"review_version":1}