{"id":"138bc595-19d9-480a-a3e0-98011af15bc7","arxiv_id":"1908.07471","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A games-with-a-purpose system in which crowd workers and a simulated annealing algorithm co-design network layouts outperforms automated layout algorithms on cyclic biological signaling networks.","lead":"Flud turns the task of drawing biological network diagrams into an online game where non-expert players and a simulated-annealing algorithm take turns improving a shared layout. On three cell-signaling networks, the combined human-algorithm approach produced higher-scoring layouts than four automated layout programs, especially for networks with many feedback loops.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DP path-counting recurrence in Appendix 1, as written, returns zero for every layout, so the headline score comparisons are not reproducible until the recurrence is corrected or the code is released.","rationale":"The reader's weakest assumption targeted the biological validity of the downward-pointing-path metric. That is a legitimate concern, but I found a more immediate, internal problem: the metric itself, as specified in Appendix 1, cannot produce the reported numbers. If the recurrence in the paper is what was implemented, every DP count is zero and all score-based comparisons collapse. If the paper contains a typographical omission, the results may still be sound, but the text does not provide a reproducible definition, and Section 6.2.1 explicitly concedes that the normalization is a poor approximation for cyclic networks. Because the DP criterion dominates the weighted total score, this uncertainty propagates to the central claim about hybrid approaches outperforming automated baselines on G2 and G3. The issue is not an attack on the authors; it is a request for a precise, checkable specification or a code release. Until the recurrence is corrected or the scoring implementation is made available, the empirical claims cannot be independently verified, so I would move the reader's CONDITIONAL verdict to UNVERDICTED rather than REJECT, since a corrected recurrence or released code could restore confidence.","tokens_in":31644,"tokens_out":8789,"duration_ms":91196,"concrete_test":"Implement the DP recurrence exactly as stated in Appendix 1 on a toy DAG that contains at least one directed path, including a single-edge path (e.g., s→a→t and s→t). Under the printed recurrence and base case, the output is zero for every node, so π(G)=0. Then ask the authors to provide the code used to compute the raw DP counts in Figure 11 and run that code on the same toy graph. If the code matches the printed recurrence, the DP counts and all headline score comparisons are invalid; if the code differs, the paper must state the corrected recurrence and release the scoring code so the results can be reproduced.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The headline comparisons are driven by the total score, and because the downward-pointing-path (DP) criterion receives priority 400 versus 3/1 for the other four criteria (Section 4.5), DP is the load-bearing component of that score. As printed in Appendix 1, the DP computation cannot count anything: the recurrence π(v)=Σ_{(v,u) downward} π(u) with base case π(v)=0 for nodes with no downward outgoing edges yields π(v)=0 for every node, because the downward-edge subgraph is acyclic and the induction bottoms out at zero. Hence π(G)=0 for every layout. This contradicts the nonzero DP counts in Figure 11 and makes the score rankings in Figures 9–13 unreproducible from the text. The intended recurrence presumably adds a +1 or uses a target base case, but the paper does not specify it. Section 6.2.1 compounds the problem by admitting that the normalizing denominator ρ(G) is a poor approximation for cyclic networks, so even after correcting the recursion the DP score is an author-defined proxy with an arbitrary weight. The central claim therefore leans on a metric that is not unambiguously defined and is not independently validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Flud, a game-with-a-purpose for laying out biological signaling networks, along with a hybrid approach in which crowd workers and simulated annealing alternate sessions and build on each other's layouts. Layout quality is measured by a weighted sum of five criteria, including a new downward-pointing-paths (DP) criterion that is given priority weight 400. The authors report two experiments on three signaling networks with varying numbers of cycles: Experiment 1 compares priority-ordered versus random assignment of criterion-specific modes, and Experiment 2 compares Crowd, Crowd-Random, and three Crowd-SA hybrids against SA, Dig-Cola, IPSEP-Cola, and a spring-electrical layout. The central claims are that Crowd and Hybrid approaches clearly outperform automated baselines on the cyclic networks G2 and G3, that Crowd-SA100 achieves a better rate of score improvement than Crowd or SA, and that clue-based suggestions help players improve criterion-specific scores.","tokens_in":31877,"tokens_out":5936,"duration_ms":64948,"significance":"If the results hold, the paper makes a useful empirical contribution to mixed-initiative and human-computation visualization: it demonstrates a concrete, implementable hybrid loop between non-expert crowd workers and simulated annealing for a domain-specific layout objective, and it provides a system (Flud) with publicly described scoring and gameplay mechanics. Strengths include the use of raw downward-pointing-path counts as a corroborating outcome alongside the weighted score, repeated game sequences per network, parameter searches for the automated baselines, and qualitative layout comparisons. The main limitations are that the DP recurrence as printed is internally inconsistent (yielding zero for every layout), no significance tests or confidence intervals are reported, only three networks are used, and the biological meaningfulness of the DP proxy is asserted rather than validated.","major_comments":[{"comment":"The downward-pointing-paths recurrence as printed cannot produce the reported results. The recurrence π(v) = Σ_{(v,u) downward} π(u) with base case π(v) = 0 for every node with no downward outgoing edge forces π(v) = 0 for all v by induction on the acyclic downward-edge subgraph, so π(G) = 0 and DP(G) = 0 for every layout. This contradicts the nonzero DP counts in Figure 11 and makes the total-score rankings in Figures 9–13 unreproducible from the text. Please state the intended base case and summation convention (e.g., whether a sink contributes 1 and whether single-edge paths count), define π(G) and ρ(G) unambiguously, and ideally release the scoring code so the reported numbers can be checked.","section":"Appendix 1, Section 3.3"},{"comment":"The central comparison relies on medians, distributions, and bar aggregates without significance tests, confidence intervals, or per-condition sample sizes. Because Experiment 2 stopped recruitment once total gameplay exceeded 24 hours (Section 4.5.2), the number of crowd sessions and SA runs can differ across methods and networks. The large effect sizes for G2 and G3 are encouraging, but the statement that Crowd and Hybrid approaches 'clearly outperform' automated methods needs statistical support: report n per condition and provide bootstrap confidence intervals or permutation tests for the total-score and raw-DP-count comparisons.","section":"Section 5.2, Figures 10–13"},{"comment":"The rate-of-improvement comparison may be confounded by unequal time horizons. The SA baseline was run for 24 hours (Section 4.5.2), whereas each hybrid SA segment was about 15 minutes and each crowd session up to one hour; average improvement per minute over a 24-hour run is not directly comparable to per-minute improvement over short sessions because SA's early high-rate phase is diluted by long later phases. Please compare methods over matched time budgets or report per-interval rates and learning curves, not only the aggregate per-minute averages.","section":"Section 5.2, Figure 13"},{"comment":"The headline metric is dominated by the downward-pointing-paths criterion (priority 400 versus 3 and 1 for the other criteria), and the paper itself states that the normalizing denominator ρ(G) is a poor approximation for cyclic networks. The network-specific normalization does not invalidate within-network method comparisons, but it does mean that the quantitative rankings are driven by an author-defined proxy whose biological validity is asserted rather than measured. The raw DP counts in Figure 11 partially ground the claim, yet they are subject to the same recurrence issue as the normalized scores. Please either validate the DP proxy against biologist judgments or explicitly scope the conclusion to 'higher scores on Flud's stated objective,' and show that the G2/G3 conclusions are robust to the DP weight (e.g., wDP = 4, 40, 400).","section":"Section 6.2.1, Section 4.5"}],"minor_comments":[{"comment":"The definition of DP(G) uses ρ(v) where ρ(G) is intended, and the notation for π(v) versus π(G) should be clarified so it is clear that π(G) is the sum over relevant start nodes.","section":"Section 3.3"},{"comment":"In the recurrence description, 'outgoing neighbors of v that have smaller y-coordinate than u' appears to contain a typo; it should be 'smaller y-coordinate than v' or should otherwise specify the coordinate convention for downward edges.","section":"Appendix 1"},{"comment":"There is a typo in 'xcur r ent', and the sign convention for Δs in the acceptance probability e^{−Δs/T} should be stated explicitly.","section":"Section 3.4"},{"comment":"The text contains the typo 'arequester' at the start of Section 3.","section":"Section 3.1"},{"comment":"The bonus formula uses b and starget without prior definition; please define these variables in the text before the equation.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a human-computation/visualization venue and the empirical system is a genuine contribution. My recommendation of major revision is driven by the unreproducible DP recurrence, which is load-bearing for every score comparison, and by the absence of inferential statistics for the headline claims; both are fixable within the manuscript's scope. I do not see a circularity problem beyond the standard issue of benchmarking on the optimized objective, provided the DP definition is corrected and the biological-proxy claim is appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a real HCI systems paper with a substantial empirical contribution, but the appendix as printed contains a load-bearing flaw. The recurrence for counting downward-pointing paths defines π(v) as the sum of π(u) over downward outgoing successors, with base case 0 for nodes that have no downward outgoing edges. On the acyclic subgraph of downward edges, that computes zero for every node. The nonzero counts in Figure 11 therefore cannot be reproduced from the text. This is likely a typo—missing a +1 or a target-node base case—but it matters because the DP criterion receives weight 400, making it the dominant term in the total score comparisons.\n\nWhat is genuinely new: Flud is a GWAP for network layout with a novel domain-specific criterion (maximize downward-pointing paths), and the paper tests a true mixed-initiative design where crowd workers and simulated annealing alternate. The empirical scale is impressive—about 2,000 MTurk workers across three networks—and the system design is thoughtful (criterion-specific modes, clues, real-time scoring). The core claim, that crowd and hybrid approaches beat automated baselines on cyclic signaling networks, is plausible: the effect sizes are large, and the raw downward-path counts corroborate the total-score results, so the main comparison likely holds up.\n\nWhere it is soft: first, the recurrence problem above. Second, no significance tests or confidence intervals appear anywhere, and only three networks were used, so inferential strength is limited. Third, the biological meaningfulness of the downward-path proxy is asserted rather than validated; the paper itself admits the normalization is a poor approximation for cyclic networks. These are secondary to the reproducibility issue but worth weighing.\n\nWho this is for: researchers in mixed-initiative crowdsourcing, graph drawing, and biological visualization. They should read it, but with the caveat that the published math is, as written, wrong. I would accept it for peer review—the idea and the empirical work merit referee time—and I would ask the authors to release code or at minimum correct the recurrence before acceptance.\n\nMy recommendation: engage, but require fixes.","headline":"A serious GWAP study with a plausible core result, but the printed DP recurrence in the appendix returns zero for every layout, so the headline numbers are unreproducible until the authors correct the text and release code.","tokens_in":32423,"tokens_out":2764,"would_cite":false,"duration_ms":30483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flud claims that alternating novice crowd workers with a high-temperature simulated annealing algorithm produces higher-scoring layouts of cyclic signaling networks than state-of-the-art automated layout tools, and that the hybrid…","keywords":["crowdsourcing","human computation","graph drawing","games with a purpose","simulated annealing","biological networks","mixed-initiative systems","network visualization"],"falsifier":"A pre-registered study in which biologists rank layouts from Crowd-SA100, plain simulated annealing, and Dig-Cola for readability of signal flow, with the DP score hidden and other aesthetics roughly matched, would settle whether the metric the paper optimizes tracks what biologists find useful; if readers show no preference for high-DP layouts, the claimed advantage is an advantage on an author-defined score rather than on biological readability.","tokens_in":31446,"feed_emoji":"🧬","tokens_out":6623,"duration_ms":60633,"temperature":0.7,"pith_summary":"This paper claims that recruiting non-expert crowd workers to arrange the nodes of a biological signaling network, and alternating their sessions with runs of a simulated annealing optimizer, produces layouts that score higher than layouts from state-of-the-art automatic algorithms, especially on networks dense with feedback cycles where automated methods get stuck. The authors built Flud, a browser game with a purpose that turns layout into a scored puzzle, and tested it on three protein signaling networks using about 2,000 crowd workers. On the two cycle-heavy networks, the crowd-only and hybrid strategies clearly outscored four automated baselines, and the hybrid Crowd-SA100 variant improved the layout score faster per minute than crowd workers alone. The practical point is that biologists routinely hand-tune automatic layouts, and Flud asks whether untrained crowds can do that tuning at scale.","feed_headline":"Crowds plus simulated annealing out-lay automated network tools","feed_subtitle":"On cycle-heavy signaling networks, the hybrid beats algorithms and even crowd-only play.","key_machinery":"The load-bearing mechanism is the alternating hybrid protocol: a sequence of game sessions in which each crowd worker starts from the best layout so far and is assigned one criterion-specific mode with an algorithmically generated clue, and a simulated annealing schedule begins each of its sessions from the best human layout. The temperature variant matters: SA100 starts at a high temperature ($T_0 = 100$) and makes large, non-local node jumps, which is what lets it reorient edges for the downward-pointing-path criterion after humans have roughly placed them, while low-temperature SA20 only makes local adjustments and behaves differently on distance-based criteria. The paper's central scoring object is the normalized downward-path ratio $\\mathrm{DP}(G) = \\pi(G)/\\rho(G)$, the number of downward-pointing paths divided by the total number of directed paths from sources to targets, combined into an overall weighted score with the four aesthetic criteria.","core_discovery":"Flud's central claim is that a mixed-initiative loop, in which humans move nodes guided by criterion-specific clues and simulated annealing runs then start from the human-produced layout, can escape the local optima that trap fully automatic layout algorithms on cyclic signaling networks. The paper evaluates this with a weighted score combining five criteria, the dominant one being the number of downward-pointing paths from receptor nodes at the top to transcription-factor targets at the bottom, where an edge counts as downward if it descends at least 15 degrees. On networks G2 and G3, which contain thousands to hundreds of thousands of simple cycles, the crowd and hybrid approaches clearly outperformed simulated annealing, Dig-Cola, IPSEP-Cola, and a spring-electrical force model on both total score and number of downward paths, and Crowd-SA100 achieved a better rate of score improvement per minute than the crowd alone. The authors also report that moving a node that Flud highlighted in a clue increased the criterion score on average, while moving a non-clue node decreased it, evidence that the algorithmic suggestions are what let novices make productive moves.","pith_inferences":["Because the 15-degree threshold and the DP normalization are implementation choices rather than validated perceptual thresholds, the measured margins over the baselines could shift under a different but equally defensible definition of downward; a sensitivity analysis over that angle parameter would show how brittle the headline result is.","The hybrid recipe likely generalizes to any cooperative layout task with a cheap scoring function: high-temperature annealing serves as a non-local move generator between human sessions, a pattern the paper itself gestures toward for circuit and interior-design layouts.","The DP upper bound $\\rho(G)$ counts all directed source-to-target paths and is loose on cyclic networks; a tighter bound would rescale the DP scores for G2 and G3 and might narrow the apparent crowd advantage, though probably not reverse it.","If volunteers engage more deeply than paid crowd workers, as the paper's discussion suggests, the measured crowd advantage may be a lower bound on what a motivated player community could achieve."],"forward_implications":["On signaling networks with many feedback cycles, prioritizing the downward-pointing-path criterion is exactly where automated algorithms collapse and where crowd and hybrid methods win, so biologists working with cyclic networks are the natural users of this approach.","Crowd-SA100 improves the total score faster per minute than crowd-only play or pure simulated annealing, making the hybrid the efficient choice when crowd time is budgeted.","Clue-guided moves improve per-criterion scores on average while non-clue moves hurt them, so the algorithmically generated suggestions carry real weight in the reported gains.","Assigning criterion-specific modes in priority order beats random assignment, giving a concrete design rule for future game-with-a-purpose layout systems."],"supporting_citations":[{"why":"Supplies the simulated annealing method that is both the main automated baseline and the algorithmic half of the hybrid approach.","marker":"[17]"},{"why":"Dig-Cola, the directed hierarchical layout algorithm that the crowd and hybrid approaches must beat on the three test networks.","marker":"[20]"},{"why":"IPSEP-Cola, an automated baseline that enforces directed-edge placement constraints for information flow.","marker":"[21]"},{"why":"Evidence that layouts produced by non-expert users can match automated ones, motivating the crowd-only strategy.","marker":"[22]"},{"why":"Supplies the fine-tuning procedure applied to the best layout at the end of every approach.","marker":"[36]"},{"why":"The spring-electrical force-directed model (sfdp), a third automated baseline run with a parameter search.","marker":"[41]"},{"why":"Prior work showing non-expert crowds can design biological network layouts, which Flud extends with real-time scoring, clues, and hybrid collaboration.","marker":"[63]"}],"fun_headline_variants":["Flud: Crowd and algorithm team up to beat network layout","Hybrid crowd-algorithm layout wins on cycle-heavy networks","Game with algorithmic hints outplays solo algorithms","Simulated annealing plus crowds crack tangled networks","Flud's mixed-initiative beats purely automated layout"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's measure of a biologically meaningful layout is the number of downward-pointing paths, but this proxy is never validated against actual biologists reading signaling flow, and the paper itself concedes the normalization is a poor approximation on cyclic networks.","fun_headline_variants_meta":{"raw":{"variants":["Flud: Crowd and algorithm team up to beat network layout","Hybrid crowd-algorithm layout wins on cycle-heavy networks","Game with algorithmic hints outplays solo algorithms","Simulated annealing plus crowds crack tangled networks","Flud's mixed-initiative beats purely automated layout"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1404,"prompt_tokens":967,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":360}},"tokens_in":583,"tokens_out":437,"duration_ms":5250,"temperature":1.0,"reasoning_tokens":360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:18:36.090180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A pre-registered study in which biologists rank layouts from Crowd-SA100, plain simulated annealing, and Dig-Cola for readability of signal flow, with the DP score hidden and other aesthetics roughly matched, would settle whether the metric the paper optimizes tracks what biologists find useful; if readers show no preference for high-DP layouts, the claimed advantage is an advantage on an author-defined score rather than on biological readability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the simulated annealing method that is both the main automated baseline and the algorithmic half of the hybrid approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IPSEP-Cola, an automated baseline that enforces directed-edge placement constraints for information flow."},{"cited_title":"Robertson, and Chris North","cited_arxiv_id":null,"evidence_quote":"Evidence that layouts produced by non-expert users can match automated ones, motivating the crowd-only strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fine-tuning procedure applied to the best layout at the end of every approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The spring-electrical force-directed model (sfdp), a third automated baseline run with a parameter search."}],"review_version":1}