{"id":"c09e6251-1469-4a8a-9f8e-81d33fc83aaa","arxiv_id":"2605.28334","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"CSI meta-scaffold unifies five LLM agent harnesses; a blackboard multi-agent system solves 19/33 cybench challenges (57.6%) versus 15/33 for the best single scaffold.","lead":"The paper introduces CSI, a meta-scaffold unifying heterogeneous LLM-driven cybersecurity agent harnesses under one orchestration layer with a blackboard for collaboration. A smart generalist might read it to learn whether combining structurally different agent designs can measurably outperform any single harness on benchmark tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader's weakest assumption correctly flags the external validity issue for any benchmark claim. Because the paper's argument is purely empirical and makes no stronger theoretical assertion, that assumption is the primary limit on confidence rather than an internal inconsistency or calculation error. With full text available the verdict remains UNVERDICTED pending reproducibility checks; no adjustment is warranted.","tokens_in":1843,"tokens_out":304,"duration_ms":21957,"concrete_test":"Re-run the five scaffolds plus blackboard on the same 33 challenges with independent random seeds (or temperature >0 if applicable) and report solve counts with 95% CI; if the blackboard advantage over the best single scaffold falls below 2 solves, the 27% relative gain is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (blackboard at 19/33 vs. best single scaffold at 15/33) rests on a fixed-model benchmark across five harnesses on the Cybench set. The reported union (17/33) and blackboard (19/33) numbers are internally consistent with the stated heterogeneity premise; no hidden assumption in the arithmetic or comparison is required for the headline delta to follow from the counts given. Full-text details on blackboard protocol, run variance, or exact per-challenge attribution would be needed to surface a technical flaw, but none is evident from the supplied argument structure.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces CSI, a meta-scaffold unifying heterogeneous LLM-driven agent harnesses for cybersecurity tasks. It benchmarks five scaffolds (CSI::Claude, CSI::Codex, CSI::GCAI, CSI::Mistral, CSI::CAI) on the 33 Cybench challenges with fixed model alias2-mini. Key results: best single scaffold solves 15/33 (45.5%), four-scaffold union solves 17/33 (51.5%), and blackboard multi-agent architecture solves 19/33 (57.6%), achieving a 27% relative gain over the best single scaffold, 25% faster (20.2h vs 26.8h) at comparable cost ($5,480 vs $5,122). The central claim is that no single scaffold dominates and that structurally heterogeneous scaffolds combined via blackboard yield highest coverage.","tokens_in":1948,"tokens_out":637,"duration_ms":23546,"significance":"If the empirical results hold under scrutiny, the work demonstrates that multi-harness orchestration leveraging scaffold heterogeneity can improve coverage on cybersecurity benchmarks without added cost, providing a concrete step toward Cybersecurity SuperIntelligence. It merits credit for using a public benchmark suite and reporting concrete solve counts, timing, and cost metrics. However, the absence of detailed methods substantially limits verifiability and immediate impact.","major_comments":[{"comment":"Results section (and abstract): The manuscript reports concrete performance claims including 19/33 solves for the blackboard vs. 15/33 for the best single scaffold (CSI::Claude), but provides no experimental protocol, run parameters, success criteria for Cybench challenges, timeout handling, per-challenge attribution, or any statistical tests/error analysis/controls. This is load-bearing for the central claim of a 27% relative gain, as the numbers cannot be reproduced or assessed for robustness without these details.","section":"Results section"},{"comment":"Blackboard architecture description: The paper states that the blackboard enables parallel execution and exchange of intermediate findings to produce non-redundant solves, but does not specify the exact orchestration rules, conflict resolution, or how scaffold outputs are integrated on the shared substrate. This detail is required to evaluate whether the reported 19/33 count follows from the heterogeneity premise or from unstated implementation choices.","section":"Blackboard architecture"}],"minor_comments":[{"comment":"Abstract: The phrasing 'the four-scaffold union solves 17/33 (51.5%), with the fifth (CSI::Mistral, 10/33) contributing one exclusive solve' could be clarified to explicitly state whether the union includes all five or only four, to avoid ambiguity in interpreting the incremental contribution.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The soundness rating is low primarily due to unavailable methods; if the full manuscript contains a detailed protocol section not visible in the provided abstract, this would alter the assessment. The work fits the journal scope as an empirical systems paper in cybersecurity AI."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We agree that additional methodological details are required for reproducibility and will revise the manuscript to address both major comments. Point-by-point responses follow.","responses":[{"response":"We acknowledge that the current version lacks a complete experimental protocol, which limits verifiability of the reported solve counts. In the revised manuscript we will add a dedicated experimental setup subsection (and update the abstract) that specifies: (i) exact run parameters and model configurations for alias2-mini across all five scaffolds, (ii) Cybench success criteria and verification procedure, (iii) timeout and retry handling, (iv) per-challenge solve attribution table, and (v) any statistical controls or error analysis performed. These additions will allow independent reproduction of the 15/33, 17/33, and 19/33 figures while leaving the empirical claims unchanged.","revision_made":"yes","referee_comment":"[Results section] Results section (and abstract): The manuscript reports concrete performance claims including 19/33 solves for the blackboard vs. 15/33 for the best single scaffold (CSI::Claude), but provides no experimental protocol, run parameters, success criteria for Cybench challenges, timeout handling, per-challenge attribution, or any statistical tests/error analysis/controls. This is load-bearing for the central claim of a 27% relative gain, as the numbers cannot be reproduced or assessed for robustness without these details."},{"response":"We agree that the orchestration mechanics must be stated explicitly. The revised manuscript will expand the blackboard architecture section to describe: (i) the precise rules governing parallel execution of the five scaffold-specialised agents, (ii) the protocol for posting and reading intermediate findings on the shared substrate, (iii) conflict-resolution logic (priority weighting by per-scaffold historical accuracy plus consensus fallback), and (iv) the integration step that produces the final non-redundant solve set. This will make clear that the additional solves arise from scaffold heterogeneity rather than hidden implementation details.","revision_made":"yes","referee_comment":"[Blackboard architecture] Blackboard architecture description: The paper states that the blackboard enables parallel execution and exchange of intermediate findings to produce non-redundant solves, but does not specify the exact orchestration rules, conflict resolution, or how scaffold outputs are integrated on the shared substrate. This detail is required to evaluate whether the reported 19/33 count follows from the heterogeneity premise or from unstated implementation choices."}],"tokens_in":1599,"tokens_out":538,"duration_ms":28752,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core result is that no single scaffold wins across the board and that a blackboard letting five different ones share intermediate results solves 19 of the 33 Cybench challenges while the best individual one solves 15. It also reports the blackboard run finishing in 20.2 hours versus 26.8 for the top single scaffold at roughly the same cost. That is a concrete, measurable demonstration that structural diversity helps coverage.\n\nThe implementation itself is the main new piece: a meta-layer that can plug in any LLM-driven harness, run them together, and use the blackboard as shared memory. The numbers line up internally—the union of four scaffolds already reaches 17, and the fifth adds one more exclusive solve—so the arithmetic supports the claim that heterogeneity matters.\n\nThe main limitation is the test set. Thirty-three challenges is small, and the abstract gives no variance across runs, no statistical tests, and no breakdown of which problems each scaffold actually solved. Without those details it is hard to judge how much the 27 % relative gain would hold up on a broader or different set of tasks. Keeping the underlying model fixed is reasonable for isolating the scaffold effect, but it also leaves open whether stronger models would shrink the gap.\n\nThis is useful reading for anyone building or evaluating LLM agents for cybersecurity. It supplies a working architecture and head-to-head numbers rather than just another single-scaffold loop. The work is grounded enough in its own benchmark to warrant peer review; the referee can ask for the missing protocol details and a larger test suite.","headline":"The blackboard meta-scaffold delivers a clear lift on Cybench (19/33 vs 15/33) by running heterogeneous harnesses in parallel, but the evaluation stays narrow and lacks controls.","tokens_in":2467,"tokens_out":401,"would_cite":false,"duration_ms":16231,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A blackboard that lets different AI scaffolds share findings solves more cybersecurity challenges than any one scaffold alone.","keywords":["cybersecurity","AI agents","multi-agent systems","blackboard architecture","LLM scaffolds","agent harnesses","meta-scaffold"],"falsifier":"Repeating the benchmark on a fresh collection of cybersecurity tasks outside the cybench set and finding that the blackboard no longer exceeds the best single scaffold.","tokens_in":2771,"feed_emoji":"🛡️","tokens_out":614,"duration_ms":20610,"temperature":0.7,"pith_summary":"The paper sets out to determine the best execution harness for LLM-based cybersecurity agents and concludes that no individual scaffold works best across all tasks. It introduces CSI, a meta-scaffold that runs five structurally different agent harnesses in parallel and lets them exchange results on a shared blackboard. On the 33 cybench challenges the blackboard combination reaches 19 solves while the strongest single scaffold reaches only 15, and it does so faster at similar cost. A reader would care because current cybersecurity AI work is converging on single iterative loops, yet the results indicate that deliberate heterogeneity plus shared memory produces measurable gains in coverage.","feed_headline":"Blackboard multi-agent setup solves 19 of 33 cyber challenges","feed_subtitle":"Combining five heterogeneous scaffolds via shared substrate beats best single harness by 27 percent relative gain","key_machinery":"CSI's blackboard-based multi-agent architecture, in which scaffold-specialised agents run in parallel and exchange intermediate findings via a shared substrate.","core_discovery":"No single scaffold is the best harness; the combination of structurally heterogeneous scaffolds inside a blackboard-based multi-agent architecture produces the highest coverage, solving 19 of 33 cybench challenges versus 15 of 33 for the strongest individual scaffold at 25 percent less time and comparable cost.","pith_inferences":["The same blackboard pattern could be tested on real-world incident response logs rather than benchmark challenges.","Adding further scaffolds or refining the sharing rules on the blackboard might increase the number of unique solves beyond 19.","The result suggests that progress toward more capable cybersecurity AI may depend more on orchestration diversity than on improving any one harness."],"forward_implications":["Union of four scaffolds already reaches 17 solves, with the fifth adding one exclusive solve.","Blackboard use yields a 27 percent relative gain over the best individual scaffold.","No scaffold dominates every challenge type, so coverage improves only when heterogeneous designs are combined.","The blackboard approach maintains comparable cost while reducing total runtime by about 25 percent."],"fun_headline_variants":["Blackboard combines heterogeneous scaffolds for 19 of 33 cybench solves","No single scaffold leads as blackboard combo hits 19 of 33 tasks","Heterogeneous scaffolds via blackboard reach 57.6 percent coverage","Multi scaffold blackboard beats best individual on 33 cyber challenges"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 33 cybench challenges form a representative sample of cybersecurity tasks and the five scaffolds are different enough that parallel execution and blackboard sharing produce non-redundant solves.","fun_headline_variants_meta":{"raw":{"variants":["Blackboard combines heterogeneous scaffolds for 19 of 33 cybench solves","No single scaffold leads as blackboard combo hits 19 of 33 tasks","Heterogeneous scaffolds via blackboard reach 57.6 percent coverage","Multi scaffold blackboard beats best individual on 33 cyber challenges"]},"model":"grok-4.3","cost_usd":0.004595,"raw_usage":{"total_tokens":2315,"prompt_tokens":739,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":45949500,"prompt_tokens_details":{"text_tokens":739,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1501,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":739,"tokens_out":75,"duration_ms":12070,"temperature":1.0,"reasoning_tokens":1501,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T11:57:50.609117+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Repeating the benchmark on a fresh collection of cybersecurity tasks outside the cybench set and finding that the blackboard no longer exceeds the best single scaffold.","supporting_citations":[],"review_version":1}