{"id":"82a51902-4735-4342-b625-b186a711e0a3","arxiv_id":"2607.03158","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LaTeX-style pseudocode helps most under sparse algorithm specs, but with complete details format effects shrink or vanish by model, so authors should write implementation rules explicitly instead of relying on surface format.","lead":"This paper tests whether the written format of an algorithm description changes how accurately LLMs implement it on the first try. It finds format can matter when details are sparse, but the safer fix is to state interfaces, steps, numerical rules, and edge cases explicitly rather than betting on one syntax.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Format effects rest on difference-comparison averages that exclude most GPT-5.4 mini complete-info runs and may not support a general author writing recommendation.","rationale":"The reader correctly flags external validity (five compact functions, no repair, three models) as the main limit and lands on CONDITIONAL. That concern is real and already acknowledged in §5.3. The more load-bearing internal issue for the *strongest claim as stated* is the difference-comparison filter: the headline LaTeX effect and complete-info ranking are averages over the subset of matched groups where formats already disagree, which systematically down-weights the strongest model under complete information (0/50 differences). The paper is transparent about this (Section 4.1, Table 3), so it is not a hidden error, but it means the format-ranking language in the abstract and strongest claim is more fragile than the content-first writing recommendation. The concrete test—unconditional re-average including zero-difference comparisons—would settle whether the ranking survives or whether only the ‘state details explicitly’ half remains. That keeps the verdict CONDITIONAL rather than REJECT: the matched design and hidden-test construction are sound; the claim needs tighter scoping of when format rankings are reported. Agreement with the reader is partial because external validity is important, but the difference-comparison conditioning is the more direct threat to the format-effect half of the strongest claim.","tokens_in":14609,"tokens_out":704,"duration_ms":6316,"concrete_test":"Recompute Table 4 and Figure 1 using all 50 fixed comparisons per model (not only difference comparisons), reporting mean format effect and 95% bootstrap CIs with zero-difference comparisons included as effect 0. If LaTeX’s pooled effect falls below ~0.01 or loses the top rank, the format-ranking half of the strongest claim should be demoted and the abstract should lead only with the content-explicit recommendation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim packages two results: (1) under core information LaTeX has the largest average format effect (+0.037, mean hidden fraction 0.384 over difference comparisons), and (2) therefore authors should state interface/steps/rules/boundaries explicitly rather than rely on surface format. The load-bearing soft spot is that the pooled format ranking is computed only over difference comparisons (Section 4.1, Tables 4–5, Figures 1–3). Under complete information GPT-5.4 mini contributes 0 of 50 fixed comparisons to that ranking (Table 3), so the complete-info ordering is entirely from Gemma 3 4B and Llama 3.2 3B. Even under core information, GPT-5.4 mini differs in only 21/50 comparisons. The writing recommendation is content-first and cautious, but the paper still leads with a format ranking (LaTeX largest average effect) that is conditional on the subset of runs where formats already disagree. If the unconditional mean over all 50 fixed comparisons per setting (including the many zero-difference GPT runs) shrinks the LaTeX effect toward zero, or if the ranking flips, the format-effect half of the claim is overstated relative to the evidence that actually supports the content-first recommendation.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"This paper studies whether the surface format of an algorithm specification changes first-pass LLM implementation accuracy when the target function and tests are held fixed. It compares seven formats (prose, LaTeX algorithm-style pseudocode, PDF-like extracted pseudocode, Markdown fields, YAML-like and JSON-like specs, and Python code stubs) on five compact ML functions, three models, and four information settings, for 4,020 generations. Evaluation uses public and hidden tests that probe tie-breaking, shapes, numerical rules, return structure, and invalid-input behavior. Under core information, LaTeX pseudocode has the largest average format effect among difference comparisons, with YAML-like specs and prose close behind; under complete information, GPT-5.4 mini shows no format differences while the two smaller models still do. Code stubs do not consistently improve hidden-test correctness. The authors conclude that authors should state interface, steps, numerical rules, and boundary behavior explicitly rather than rely on a particular surface format.","tokens_in":14993,"tokens_out":1422,"duration_ms":20286,"significance":"The problem is timely and well-motivated by paper-to-code and research-reproduction benchmarks. The main methodological contribution is a paired design that isolates representation format while holding model, task, information setting, prompt wrapper, and tests fixed, with a large sample (4,020 implementations), bootstrap intervals, pairwise win shares, prompt-length diagnostics, and failure-category analysis. The findings are carefully conditional rather than a single format ranking, and the content-first writing recommendation is more robust than any claim that one syntax is universally best. If the results hold under broader tasks and models, the paper gives concrete, actionable guidance for how algorithm descriptions should be written when LLM implementation is a use case, which is a useful contribution to empirical software engineering and ML research practice.","major_comments":[{"comment":"Section 4.1 and Tables 4–5: the primary format rankings are computed only over “difference comparisons,” which exclude fixed comparisons where all seven formats obtain the same hidden-test fraction. Under complete information this removes all 50 GPT-5.4 mini comparisons (Table 3), so the complete-info ordering is entirely from Gemma 3 4B and Llama 3.2 3B. The paper states this, but the abstract and §4.3 still lead with “LaTeX … largest average format effect” without an equally prominent unconditional mean over all fixed comparisons. Please report unconditional format effects (and CIs) for all 50 comparisons per model/setting alongside the conditional summaries, so readers can see whether the LaTeX/YAML advantages shrink toward zero when zero-difference runs are included.","section":"Section 4.1, Tables 4–5, Table 3"},{"comment":"Section 5.2 and the abstract writing recommendation: the evidence is first-pass, no-repair generation on five function-level tasks. Section 5.3 already notes that full-system reproduction and repair-with-logs are out of scope, but the recommendation is phrased as general author practice for “algorithm specifications” that “may be used as inputs to LLM-based implementation.” Please tighten the claim language so the recommendation is explicitly scoped to first-pass implementation of well-defined functions under the tested information regimes, or add at least one analysis that speaks to transfer (e.g., multi-attempt repair, or a larger multi-function task) if a broader claim is retained.","section":"Section 5.2, Abstract, Section 5.3"},{"comment":"Section 4.3 / Table 4: under core information, LaTeX leads by hidden-test fraction and average format effect, but code stub has the highest strict all-hidden-tests-passed rate (0.163 vs 0.102 for LaTeX). The paper notes the discrepancy but still treats hidden-test fraction as the primary score without justifying why partial-test success is the right decision criterion for a writing recommendation. A short sensitivity analysis ranking formats by strict pass rate (and by public-to-hidden gap from Table 9) would show whether the format ordering is metric-dependent in a way that affects the recommendation.","section":"Section 4.3, Table 4, Table 9"}],"minor_comments":[{"comment":"Several figures appear to have corrupted labels (e.g., “fract on,” “algor thm,” “Y AML,” “Markdo-n”) in Figures 1–6. These look like rendering/OCR artifacts and should be regenerated before publication.","section":"Figures 1–6"},{"comment":"Section 3.2 and §4.4: the PDF-like format is a controlled linearization, not real PDF/OCR extraction. This is stated, but the complete-info result that PDF-like leads among difference comparisons is easy to over-read as advice about paper PDFs. A one-sentence caution in the results paragraph (not only in limitations) would help.","section":"Section 4.4, Section 5.3"},{"comment":"Section 4.7 field-removal: non-monotone effects and the confounded prompt-length change are acknowledged, but the experiment is still listed among the four main settings. Consider moving the strongest interpretation to “sensitivity to prompt composition” in the main text and de-emphasizing field-level causal claims.","section":"Section 4.7"},{"comment":"Related work is appropriate but brief on structured specification / intermediate representation work outside pure prompt-format studies. A short pointer to prior work on structured task specs for code generation would situate the YAML/JSON conditions more clearly.","section":"Section 1.2"},{"comment":"Table 1 and §3.1: the hidden-test design is a strength; making the test suites or a subset available as supplementary material would improve reproducibility of the evaluation claims.","section":"Section 3.1, Table 1"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical SE paper with a careful paired design; not a methods breakthrough, but the conditional findings and content-first recommendation are useful. The main risk is over-reading the LaTeX ranking from difference comparisons. Minor revision is enough if unconditional results and scoped claim language are added. Fit is good for an empirical software-engineering / ML-systems venue; less so for a pure ML theory venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is not a format winner. It is that first-pass hidden-test correctness can move with surface form when the prompt is thin, and that once interfaces, numerical rules, and edge cases are stated, a stronger model stops caring about format. That is the result worth remembering.\n\nWhat is new is the paired design: same task, same tests, same model, same information budget, only the written form changes, across 4,020 generations. They separate all comparisons from difference comparisons, check prompt-length correlations (small, opposite signs), and score with hidden tests that hit tie-breaking, shapes, stability, and invalid inputs. Related work is used cleanly as motivation, not as a ranking substitute. The writing recommendation is content-first rather than format-dogmatic, which matches their own complete-info GPT result.\n\nSoft spots are real but proportional. The pooled LaTeX lead is over difference comparisons; under complete information GPT contributes zero of fifty fixed comparisons to that ranking, so the complete-info ordering is only the two small models. The PDF-like condition is a controlled linearization, not real extraction. Tasks are five compact functions, one shot, no repair. Those limits are mostly flagged in §5.3; they do not break the internal measurements, but they do mean the author-facing advice should stay scoped to first-pass function-level specs.\n\nMath and stats are light and appropriate: bootstrap intervals, pairwise win shares, failure categories from test names. Citations look standard for paper-to-code and code eval. No circularity problem.\n\nThis is for people who write algorithm specs, build paper-to-code benchmarks, or care about prompt structure in coding models. I would send it to peer review. Engage if that is your lane; treat the format ranking as conditional and the content checklist as the durable part.","headline":"Solid matched experiment on format vs content for LLM algorithm coding; the content-first writing advice holds, while the LaTeX ranking is conditional and the external scope is narrow.","tokens_in":15570,"tokens_out":465,"would_cite":true,"duration_ms":5090,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"The written format of an algorithm specification changes first-pass language-model implementation accuracy, but only until authors make the interface, steps, numerical rules, and boundary cases fully explicit.","keywords":["algorithm specification","language models","code generation","pseudocode","paper-to-code","hidden tests","machine learning algorithms","prompt format"],"falsifier":"Re-run the same matched seven-format comparison on multi-file system-level tasks with repair allowed from failure logs; if format effects vanish or reverse once models can iterate, the writing recommendation overstates the importance of first-pass surface format.","tokens_in":15499,"feed_emoji":"📝","tokens_out":1019,"duration_ms":22473,"temperature":0.7,"pith_summary":"This paper asks whether the surface form of an algorithm description—ordinary prose, LaTeX-style pseudocode, PDF-like extracted steps, Markdown fields, YAML-like or JSON-like specifications, or a Python code stub—changes how accurately language models implement machine learning functions on the first try. Across five compact tasks, three models, and 4,020 generated implementations, matched comparisons that hold the target function and tests fixed show that format often matters when the prompt carries only core algorithm information, with LaTeX algorithm-style pseudocode posting the largest average format effect and YAML-like specifications and ordinary prose close behind. Once configuration values, numerical rules, and invalid-input behavior are also stated, the strongest model shows no format differences at all, while the smaller models still do. Code stubs that fix the function signature do not consistently improve hidden-test correctness. The practical message is that authors should write the implementation-critical details explicitly rather than counting on any particular surface syntax to carry them.","feed_headline":"LaTeX pseudocode leads first-pass LLM algorithm coding","feed_subtitle":"Complete specs erase format gaps for the strongest model; state rules and edge cases, not just syntax.","key_machinery":"The fixed comparison unit: within each model, task, information setting, prompt instruction, and repeat, only the representation format changes while the target function and hidden tests stay fixed. Format effect is a format's hidden-test fraction minus the seven-format average in that same comparison, with secondary summaries restricted to difference comparisons where at least two formats disagree.","core_discovery":"In matched comparisons that vary only the written representation of the same algorithm, format affects first-pass hidden-test correctness under core information—LaTeX algorithm-style pseudocode has the largest average format effect, with YAML-like specifications and ordinary prose close behind—while under complete information the strongest model shows zero format differences across all fixed comparisons and the smaller models still differ. Code stubs state the signature but do not by themselves supply the computational rules needed for hidden-test success. Authors should therefore state the interface, computation steps, numerical rules, and boundary-case behavior explicitly instead of relyin","pith_inferences":["Venues that encourage reproducible machine learning could usefully publish lightweight algorithm-spec checklists for reusable functions, independent of which syntax authors prefer.","If multi-turn repair from failure logs becomes the default workflow, first-pass format effects may shrink and the value of public tests may shift from prompt text to interactive debugging.","Real PDF extraction and OCR pipelines, which add layout noise beyond the paper's controlled linearization of pseudocode, could reverse the ranking of PDF-like formats under complete information.","The performance gap between YAML-like and JSON-like field formats, despite similar named fields, suggests training-data familiarity with surface syntax may matter beyond mere field structure."],"forward_implications":["Authors writing for LLM-assisted implementation should put interface, shapes, axes, numerical rules, tie-breaking, and invalid-input behavior in the paper or supplement rather than leave them implicit.","Relying on a code stub alone is insufficient: an empty body does not transmit the computation rules that hidden tests check.","Stronger models can erase format sensitivity once the specification is complete, so complete content matters more than any fixed format ranking for capable models.","Supplementary material can hold signatures, shapes, examples, and public tests while the main text keeps readable LaTeX algorithm-style pseudocode.","Paper-to-code evaluations that ignore representation format may conflate missing content with surface-form effects."],"fun_headline_variants":["LaTeX pseudocode tops first-pass LLM algorithm accuracy","Format shapes LLM coding until specs are complete","LaTeX, YAML, prose beat other algorithm formats for LLMs","Code stubs alone fail hidden tests for ML algorithms","State edge cases explicitly: format alone won't save LLMs"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The findings transfer from single-pass generation with no repair on five small function-level machine learning tasks to how authors should write algorithms for real paper-to-code workflows.","fun_headline_variants_meta":{"raw":{"variants":["LaTeX pseudocode tops first-pass LLM algorithm accuracy","Format shapes LLM coding until specs are complete","LaTeX, YAML, prose beat other algorithm formats for LLMs","Code stubs alone fail hidden tests for ML algorithms","State edge cases explicitly: format alone won't save LLMs"]},"model":"grok-4.5","effort":"low","cost_usd":0.00442,"raw_usage":{"total_tokens":1348,"prompt_tokens":821,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":44200000,"prompt_tokens_details":{"text_tokens":821,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":464,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":821,"tokens_out":63,"duration_ms":3786,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T04:27:29.444338+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same matched seven-format comparison on multi-file system-level tasks with repair allowed from failure logs; if format effects vanish or reverse once models can iterate, the writing recommendation overstates the importance of first-pass surface format.","supporting_citations":[],"review_version":1}