{"id":"c8efc9dc-1ea3-4ee1-80b1-c490c057bac3","arxiv_id":"2608.05234","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PPDL adds factor-based probabilistic inference to a prompt programming language, turning multi-step LLM flows into distributions over execution traces with pluggable inference engines.","lead":"PPDL is a new programming language that wraps LLM and tool calls in a probabilistic framework, letting developers score each step of a flow and then switch between inference strategies such as majority voting, importance sampling, and sequential Monte Carlo without rewriting the flow. The paper also shows the language at work on a Rocq theorem-proving agent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MiniF2F case study's stated SMC mechanism is not implemented: Figure 24 applies a constant factor -1 on any proof error, while §4.2 says particle scores use the number of errors, so the reported SMC advantage lacks its claimed cause.","rationale":"The reader's weakest_assumption listed two issues: uncalibrated factor scores and the Rocq code/prose mismatch. I agree with the second as the more load-bearing concern. Uncalibrated factors are explicitly acknowledged in the Limitations section and framed as future work, so they weaken the uncertainty-quantification claim but do not contradict the paper's internal logic. The Rocq mismatch is different: it is an inconsistency between the algorithm described in §4.2 and Appendix E.6 on one hand and the shipped program in Figure 24 on the other. The case study is the paper's principal demonstration that SMC's resampling at factor points improves a long, multi-step flow. If the factor is constant across all failed proofs, SMC's resampling cannot be prioritizing better partial proofs, so the reported accuracy gain is unexplained by the stated mechanism. This does not invalidate the language design or the formal semantics, but it means the strongest empirical support for the central claim is suspect until the code and prose are reconciled. Because the issue is concrete, localized, and fixable by either changing the factor to count errors or correcting the prose, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":30471,"tokens_out":9672,"duration_ms":114067,"concrete_test":"Run the MiniF2F-Rocq experiment in three configurations under the same 1M-token budget and same particle counts k = 5, 10, 20, 40: (a) as shipped, with `factor:-1` in Figure 24; (b) with that factor removed (or set to 0), so all failed attempts are unweighted; (c) with `factor` set to minus the number of distinct verifier errors in `result['error']`. If (a) and (b) both reproduce the +7.3 SMC-over-IS delta, the reported advantage is not caused by error-count scoring and the prose must be corrected. If (c) gives a materially larger delta than (a), the missing error-count mechanism is real and the shipped code does not implement §4.2 as described.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that PPDL lets users write one flow and then compare inference engines whose factors steer execution toward better traces. The theorem-proving case study (§4.2) is the main empirical evidence that SMC's resampling of factor-scored states is more than a reimplementation of importance sampling. That evidence is undercut by a concrete code/prose mismatch. Section 4.2 states: 'SMC@k: ... use the number of errors at each attempt to score the particles and prioritize the most promising proofs.' Appendix E.6 repeats: 'This scoring mechanism allows SMC to prioritize particles with fewer errors.' But Figure 24, lines 30–47, applies a constant `factor:-1` in the `case:Error` branch, regardless of how many errors the Rocq verifier reported. The error string is fed back as prompt text (lines 40–46), but it is never counted and never enters `pdl score`. With identical weights on every failed attempt, SMC resampling at each factor point is uniform resampling with replacement, which cannot prioritize fewer-error proofs; in expectation it leaves the particle population distribution unchanged. Therefore the +7.0/+7.3 SMC-over-IS deltas in Table 3 cannot be attributed to the mechanism the paper describes. The result may still be reproducible, but it would then come from some other property of the implementation (e.g., early-termination or token-accounting details) that the paper does not identify. Even granting the separate calibration caveat about factor scales, this is an internal inconsistency: either the shipped code does not implement the stated scoring, or the prose and Appendix E.6 misdescribe the implementation. The formal semantics of Section 3 appears coherent as a weighted-sampler framework; the load-bearing problem is that the one deep-flow experiment meant to validate the SMC factor mechanism does not, as written, use the mechanism the text claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PPDL, an extension of the prompt programming language PDL with a single probabilistic primitive, `factor`. This turns an LLM-based flow into a probabilistic program whose execution yields a weighted distribution over traces and output values. The authors formalize a weighted-sampler semantics, define three inference engines (majority voting, importance sampling, and sequential Monte Carlo) as approximations of an ideal semantics, and describe a parallelized interpreter. The empirical evaluation compares these engines across five benchmarks (GSM8k, Math500, MBPP, LiveCodeBench, FEVER) with six LLMs, and presents a theorem-proving case study on MiniF2F-Rocq. The central claims are that PPDL is the first probabilistic programming language for LLM-based flows and that it lets developers quantify and propagate uncertainty and compare inference scaling strategies without modifying flow logic.","tokens_in":30768,"tokens_out":9317,"duration_ms":87577,"significance":"If the central claims hold, PPDL is a genuinely useful contribution: it brings the decoupling of program logic from inference strategy, familiar from classic PPLs, to LLM and tool flows. The formal semantics in Section 3 is a solid basis for implementation and further reasoning, and the open-source release is a concrete strength. The evaluation is broad and the same programs are used across inference engines, which supports the claim of orthogonality. However, the significance is substantially tempered by two issues: the theorem-proving case study's stated SMC mechanism is not implemented in the provided code, and the uncertainty-quantification claim relies on uncalibrated factor scores, which the paper itself acknowledges as future work.","major_comments":[{"comment":"The description of the SMC@k strategy in §4.2 and Appendix E.6 is inconsistent with the implementation in Figure 24: the text says that the number of errors at each attempt scores the particles, but Line 39 of Figure 24 applies a constant factor of -1 for any failed verification, regardless of the number of errors in result['error']. Consequently, all failed particles receive identical weight at each resampling point, making the resampling step uniform with replacement and incapable of prioritizing fewer-error proofs; the +7.0 to +7.3 SMC-over-IS deltas in Table 3 therefore cannot be attributed to the described mechanism. The authors should either implement error-count-based scoring or revise the text and reinterpret the results.","section":"§4.2, Table 3, Figure 24"},{"comment":"The abstract and introduction claim that PPDL enables developers to 'quantify and propagate uncertainty,' but the distribution returned by PPDL is a function of user-supplied factor scores, which the Limitations section acknowledges are uncalibrated. Without calibration or an analysis of how the posterior probabilities relate to empirical frequencies, the numerical probabilities produced by PPDL (e.g., Table 1) are not validated as uncertainty estimates. The authors should either temper the uncertainty-quantification claim or provide calibration experiments.","section":"Abstract, §3, §6 Limitations"}],"minor_comments":[{"comment":"The scoring formula `score = log(exp(lpt)/(exp(lpt)+exp(lpf)))` is a log-softmax, but the function `utils.llm_judge` is not defined in the paper; consider adding its definition or a reference to the released code.","section":"§2, Eq. (1)"},{"comment":"In Figure 8, the `Categorical` constructor is passed log-scores directly; if `Categorical` expects unnormalized weights, the scores should be exponentiated. This pseudocode should be aligned with the formal semantics.","section":"Appendix B, Figure 8"},{"comment":"The notation `D_p,S` and `D_v,S'` in the SMC rule is confusing because the subscripts mix programs and values; consider using a clearer notation to distinguish distributions over states from distributions over values.","section":"§3.2"},{"comment":"The third row of the right-hand panel of Table 1 appears to have a truncated probability and a missing count; please fix the typesetting.","section":"Table 1"},{"comment":"Since all comparisons use only 5 particles and 3 runs, many differences in Table 2 fall within one standard deviation; the conclusions about which algorithm is best for each task should be phrased as observations rather than statistically significant findings.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is the code/prose mismatch in the theorem-proving case study. If the authors can either change the code to score by error count or re-describe the algorithm, the paper is likely publishable. The calibration issue is acknowledged and can be addressed by tempering the uncertainty claim. The manuscript is within the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PPDL is a real step forward for LLM flow programming. The combination of PDL's declarative flow syntax with a single factor primitive and pluggable IS/SMC inference engines is new, and the weighted-sampler semantics in Section 3 gives it a clean formal core. The implementation is open source, the benchmark across three engines and multiple models is sensible, and the admission that factors are uncalibrated is honest. I agree with the reader that the central framework is not circular: no parameters are fit to the benchmark answers, and the same programs run under all engines.\n\nThe soft spots are concentrated in the theorem-proving case study. Section 4.2 and Appendix E.6 say SMC scores particles by the number of proof errors at each attempt. The shipped code in Figure 24 does something different: it applies a constant factor of -1 for any proof error, and only feeds the error string back into the prompt. With identical weights on every failed attempt, SMC resampling is uniform, so the +7 point SMC-over-IS gains in Table 3 cannot be attributed to the counting mechanism the text describes. This is a real mismatch, not a nitpick, and it needs to be fixed one way or the other: either change the code to actually count errors, or rewrite the prose. The result might still hold through early termination or token accounting, but the paper doesn't identify that.\n\nThe other concern is calibration. The factor scores from LLM judges and linter warnings are summed on an arbitrary scale, and the paper acknowledges calibration as future work. That's acceptable for a language paper, but it means the 'uncertainty quantification' language in the abstract should be tempered. The distribution PPDL returns is a factor-reweighted posterior, not a calibrated probability.\n\nOverall, this is a useful paper for people building agentic workflows and for researchers comparing inference scaling strategies. It deserves a serious referee and likely acceptance after major revision. I would send it to review, but I'd insist the MiniF2F mechanism mismatch be resolved before publication.","headline":"A genuinely useful language contribution undercut by a case-study code/prose mismatch that needs fixing before publication.","tokens_in":31396,"tokens_out":2467,"would_cite":true,"duration_ms":24605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A flow written once in PPDL compiles to a distribution over outputs that voting, importance sampling, and SMC can all explore.","keywords":["PPDL","probabilistic programming","LLM-based flows","inference scaling","sequential Monte Carlo","prompt programming","factor scoring","uncertainty quantification"],"falsifier":"Take one PPDL benchmark flow, keep the program fixed, and re-scale a single factor source — for instance, multiply every flake8-derived penalty by a constant or change the LLM judge's score formula — then re-run the inference engines. If the normalized probabilities, the top-output ranking, or the relative ordering of majority voting, IS, and SMC changes materially under such re-scaling, the factor combination itself, not the flow logic, is carrying the result, and the distribution PPDL returns is not a well-defined object until calibration is settled.","tokens_in":30265,"feed_emoji":"🎲","tokens_out":10345,"duration_ms":93291,"temperature":0.7,"pith_summary":"PPDL is the paper's attempt to bring probabilistic programming to flows of large language model and tool calls. The authors claim that one flow specification, written with the two classic probabilistic constructs — sample, which an LLM call already supplies, and factor, which the language adds — yields a distribution over output values whose quality can be estimated by pluggable inference engines. That decoupling is the point: majority voting, importance sampling, and sequential Monte Carlo become runtime options rather than hand-written search loops inside the program, and the user gets a confidence-bearing distribution instead of a single answer. A formal semantics defines the ideal output distribution as a normalized weighted sum over all execution traces, and experiments across five benchmarks plus a theorem-proving agent for Rocq show accuracy gains and engine-dependent trade-offs. If the claim holds, multi-step LLM applications become testable as probabilistic programs, with uncertainty propagated and reported rather than guessed.","feed_headline":"Adding one primitive makes LLM app flows probabilistic","feed_subtitle":"With a factor statement scoring each trace, one PPDL program runs voting, importance sampling, or SMC unchanged.","key_machinery":"The load-bearing mechanism is the factor block together with the weighted-sampler reduction rules. A factor evaluates its expression and increments the environment variable pdl score, so every completed trace carries weight $\\exp(\\texttt{pdl score})$; the model rule samples from an LLM distribution, and the function-call rule propagates pdl context and pdl score so that scoring accumulates across calls. These rules define both the single-trace sampler and the ideal distribution, and the factor points double as resampling points for sequential Monte Carlo, which is what lets computation be redirected toward promising partial traces instead of only reweighting finished ones.","core_discovery":"The central discovery, on the paper's own terms, is that the two primitives that define probabilistic programming are already nearly present in LLM flows and can be completed with one construct. PDL supplies sample in the form of a model call; PPDL adds factor, a statement that adds a real-valued score to the current trace's log-weight. The ideal semantics then normalizes the exponentials of trace scores over all possible executions to define a categorical distribution over output values, and the language's operational semantics is a weighted sampler that any approximate inference engine can drive. The same program can therefore be executed once for a single answer or under majority voting, importance sampling, or sequential Monte Carlo, with resampling occurring exactly at factor points, and the experiments show that this lets users pick the best scaling strategy per task and model — with SMC outperforming IS in the Rocq proof-repair loop where intermediate verifier feedback carries useful signal.","pith_inferences":["The meaningfulness of the returned distribution rests on an unstated calibration assumption: scores from an LLM judge's log-odds, a linter's warning count, and a proof checker's errors are added as if on one scale, so the reported probabilities should not be read as calibrated confidences until factor calibration is validated.","The paper's own LiveCodeBench numbers already hint at the limits of that assumption — importance sampling underperforms majority voting for gpt-oss-20b — so a natural extension is to test whether reweighting the factor scales changes the engine rankings.","In the Rocq case study, the stated mechanism (scoring particles by the number of errors per attempt) is not what the provided program implements: Figure 24 applies a constant factor of -1 to every failed attempt, so the reported SMC advantage is evidence for resampling with a binary score, not for error-count scoring.","The same factor-based interface could generalize to process reward models and other learned verifiers with no language change, since factor is agnostic to where its score comes from."],"forward_implications":["One flow specification supports every inference-scaling strategy the runtime offers, so comparing majority voting, importance sampling, and SMC across tasks and models becomes a configuration change rather than a rewrite.","The output of a PPDL program is a distribution, giving end users a visible confidence signal and letting developers pick the probability-maximal answer rather than the majority answer, which can win even when it comes from a minority of particles.","SMC's resample-at-factor design makes it the preferred engine for long, deep flows with informative intermediate constraints, as in the Rocq theorem-proving agent where SMC@40 solves 95 of 244 problems versus 87.7 for IS@40 at the same token budget.","Because the semantics defines the target distribution precisely, any new inference algorithm can be added as a runtime plug-in and evaluated by how well it approximates that target.","Hard and soft constraints from heterogeneous sources — LLM judges, linters, verifiers — are unified under a single factor construct, so flows can mix cheap rule-based checks with expensive model-based judges."],"supporting_citations":[{"why":"The declarative prompt programming language PDL that PPDL extends; supplies the YAML program style, model calls, parsers, and the sample construct.","marker":"[Vaziri et al., 2024]"},{"why":"The overview of probabilistic programming whose core constructs (sample and factor) PPDL adopts as its framing.","marker":"[Gordon et al., 2014]"},{"why":"The Stan language, cited as the source of the accumulated log-probability convention PPDL follows for pdl score.","marker":"[Carpenter et al., 2017]"},{"why":"Foundational sequential Monte Carlo theory behind the resampling engine PPDL implements at factor points.","marker":"[Del Moral et al., 2006]"},{"why":"LLM-as-a-judge methodology used to score factor constraints in the benchmarks and the limitation discussion.","marker":"[Zheng et al., 2023]"},{"why":"Introduces the verifier concept and the GSM8k dataset; one of the benchmarks PPDL is evaluated on.","marker":"[Cobbe et al., 2021]"},{"why":"Cited in the limitations as the source of the claim that inference scaling with imperfect verifiers has limits, motivating factor calibration as future work.","marker":"[Stroebl et al., 2026]"},{"why":"Rollout Roulette, the closest SMC-over-LLM alternative that PPDL distinguishes itself from by supporting multi-call flows and general constraints.","marker":"[Puri et al., 2025]"},{"why":"Self-consistency; the majority-voting baseline that PPDL's engine suite includes as a special case with uniform weights.","marker":"[Wang et al., 2023]"}],"fun_headline_variants":["One factor construct turns LLM flows into probabilistic programs","LLM flows become probabilistic with a single factor statement","Add factor, get probabilistic LLM flows with tunable inference","PPDL: one primitive makes LLM flows fully probabilistic","Probabilistic LLM flows with a single factor construct"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything the language promises depends on the assumption that the numbers different constraints write into factor — an LLM judge's log-odds, a linter's warnings, a proof checker's errors — are on a common scale, so that adding them and normalizing produces a probability distribution that actually tracks how good a trace is; the paper does not calibrate these scales and lists doing so as future work.","fun_headline_variants_meta":{"raw":{"variants":["One factor construct turns LLM flows into probabilistic programs","LLM flows become probabilistic with a single factor statement","Add factor, get probabilistic LLM flows with tunable inference","PPDL: one primitive makes LLM flows fully probabilistic","Probabilistic LLM flows with a single factor construct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2532,"prompt_tokens":841,"completion_tokens":1691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":457,"tokens_out":1691,"duration_ms":10316,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:34:41.503265+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one PPDL benchmark flow, keep the program fixed, and re-scale a single factor source — for instance, multiply every flake8-derived penalty by a constant or change the LLM judge's score formula — then re-run the inference engines. If the normalized probabilities, the top-output ranking, or the relative ordering of majority voting, IS, and SMC changes materially under such re-scaling, the factor combination itself, not the flow logic, is carrying the result, and the distribution PPDL returns is not a well-defined object until calibration is settled.","supporting_citations":[],"review_version":1}