{"id":"e5952e46-6e0c-4481-a3fc-0b48a64f6c99","arxiv_id":"2608.09187","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"A recoverable LLM translation system holds a 64-character validation window, resumes from boundary-safe prefixes, and passes 38 conformance tests in a sanitized artifact.","lead":"A streaming translation system can return a syntactically valid response that is empty, truncated, or full of source text, even when the API call succeeds. This report describes a recovery protocol that waits through a 64-character check before showing text, resumes from safe boundaries after interruptions, and clearly marks machine fallbacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conformance counts are not derivable from the paper's protocol description; key rules (boundary-safe prefix alignment, paragraph-retention ratio) are underspecified, so the passing tests do not demonstrate conformance to a checkable published spec.","rationale":"The reader's condition to re-run the artifact is necessary but not sufficient. The deeper problem is that the protocol's key detectors and boundary-safe derivation are described in prose without formal definitions, so even a successful test run would not tie the results to the published specification. I agree with the reader's overall CONDITIONAL verdict but would add a second condition: formalize the underspecified rules (or reference the artifact's code as the specification) before the conformance counts can be taken as evidence. The paper is honest and does not overclaim external validity; the concern is strictly about the internal claim that the tests check the published control flow. If the artifact is run and the spec is formalized, the central claim would be supported. Therefore the verdict remains CONDITIONAL.","tokens_in":9559,"tokens_out":10152,"duration_ms":96698,"concrete_test":"Publish a formal definition of (a) the boundary-safe prefix algorithm (sentence alignment, punctuation rule, source-sentence bound) and (b) the paragraph-retention and character-ratio computations. Then have a third party implement the protocol from Sections 4–6 alone and run the 38 public tests. If the third-party implementation reproduces the same expected counts (31 retained characters, 235 contained characters, 14 labels), the specification is sufficient; otherwise the conformance claim is underdetermined.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the artifact passes 38 tests and that these results are 'executable checks of the published control flow.' That claim requires the published control flow to be specified precisely enough for an independent reader to derive the expected test outcomes. It is not. Section 5.3 states that a boundary-safe prefix retains complete paragraphs and 'complete sentence-like units ending in punctuation, bounded by the number of complete source sentences,' but does not define how such units are aligned to source sentences or what punctuation rule is used; Table 3's '31 characters retained' and '4/4 expected boundary matches' are therefore not reproducible from the prose. Section 4's omission guard rejects output when 'paragraph retention is below 0.70 and the output-to-source character ratio is below 0.15,' without defining how paragraph retention is measured (exact match, fuzzy match, positional?) or whether the character ratio uses raw or normalized text. The multilingual residue check in Section 4 similarly gives qualitative descriptions ('English residue requires a sufficiently long run') without thresholds. Because these rules are underspecified, the tests' expected labels encode the author's private interpretation rather than an independently checkable specification. The 38 passing tests establish the artifact's internal consistency, but not the abstract's claim that they validate the published control flow. The reader's concern about hand-authored cases is a separate external-validity issue; this concern is internal and more fundamental: conformance cannot be assessed if the spec is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a failure-recovery protocol for streaming long-form machine translation. The system buffers the first 64 characters before release, validates assembled output against prompt-echo, source-residue, and omission heuristics, uses typed stream events to distinguish replacement from continuation, retains boundary-safe prefixes after interruption, and routes retries through a stable model order with a shared deadline before entering a provenance-marked machine-fallback path. The manuscript reports a sanitized companion artifact with 38 passing tests and fixed conformance cases, reporting numbers of contained invalid characters, retained boundary-safe characters, and satisfied attempt/event/provenance invariants. The claimed contribution is operational: the tests are described as executable checks of the published control flow, while translation quality and failure prevalence are explicitly out of scope.","tokens_in":9728,"tokens_out":5695,"duration_ms":50139,"significance":"If the protocol works as described, it addresses a real and under-reported problem: provider-level success can hide application-level failures in long-form translation. The paper's clear separation of buffered, visible, boundary-safe, and committed text, and its typed event design for replacement versus continuation, are sensible engineering contributions. The authors are unusually honest about the evaluation's scope: the fixed cases are hand-authored, the counts are software-test results rather than statistical samples, and no live-provider or quality evaluation is claimed. The main strength is the explicit, bounded routing policy with provenance marking, which makes recovery behavior observable. The significance is limited by the fact that the protocol's core rules are not specified precisely enough to be independently implemented or checked, and by the absence of any external validation; the current evidence supports internal consistency, not the stronger claim of conformance to a published spec.","major_comments":[{"comment":"The boundary-safe retention rule is not specified precisely enough for an independent reader to derive the reported results. The text states that a boundary-safe prefix retains 'complete sentence-like units ending in punctuation, bounded by the number of complete source sentences,' but it does not define the punctuation set, the alignment between output sentence-like units and source sentences, or the positional paragraph matching used. The '4/4 expected boundary matches; 31 characters retained' entry in Table 3 therefore cannot be reproduced from the published description, and the expected labels in the fixed cases encode the author's interpretation rather than an externally checkable specification. This directly affects the abstract's claim that the results are 'executable checks of the published control flow.'","section":"§5.3, Table 3"},{"comment":"The omission and residue checks are described qualitatively, which prevents verification of the completion-guard results. Section 4 defines the omission guard in terms of 'paragraph retention is below 0.70' and 'output-to-source character ratio is below 0.15,' but it does not state how paragraph retention is measured (exact string match, positional alignment, fuzzy matching) or whether the character ratio uses raw or normalized text. The residue checks are similarly underspecified: 'English residue requires a sufficiently long run,' 'Japanese uses kana,' and 'Korean uses Hangul density' are not accompanied by thresholds or algorithms. As a result, the '0 false acceptances; 0 false rejections' outcome for the 14 completion-guard cases in Table 3 is not independently checkable.","section":"§4, Table 3"},{"comment":"The conformance evidence is self-referential with respect to the published specification. Section 7.2 correctly states that the fixed cases are hand-authored and use neither live provider calls nor production traces, and Section 11 notes that the counts are 'correlated software tests rather than statistical samples.' However, given the specification gaps identified in the two previous comments, the conformance tests establish that the artifact is internally consistent with its own test expectations, not that it conforms to a spec a third party could check. The abstract and Section 1 should either supply the missing algorithmic definitions so that the expected outcomes are derivable, or weaken the claim from 'executable checks of the published control flow' to 'executable checks of the authors' implementation.'","section":"Abstract, §7.2, §11"}],"minor_comments":[{"comment":"The heading 'Routing, F allback, and Progress Preservation' contains a spacing error; it should read 'Routing, Fallback, and Progress Preservation.'","section":"§6 heading"},{"comment":"The manuscript does not provide an artifact availability statement, such as a persistent URL, a checksum, or a command to run the 38 public tests; adding one would make the conformance claims reproducible.","section":"§7.1"},{"comment":"The phrase 'At the recorded snapshot' is vague; it would be clearer to say 'In the reported configuration' or to give the date/version of the recorded deployment.","section":"§5.1"},{"comment":"Table 2 would benefit from a column indicating which detections are deterministic protocol checks versus implementation-specific heuristics; the caption mentions this distinction but the rows do not carry it through.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a systems/engineering report with a deliberately narrow evaluation. The central issue is the gap between the abstract's claim that the 38 passing tests are 'executable checks of the published control flow' and the actual level of specification in Sections 4 and 5; this is fixable by adding pseudocode or formal invariants for the boundary-safe alignment, paragraph-retention measurement, and residue heuristics. The author's explicit acknowledgment of the evaluation's limitations is a strength. If the journal's scope prioritizes empirical NLP advances, the contribution may be modest, but as a design-and-implementation case study it is coherent and potentially useful to practitioners building similar guarded streaming systems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid engineering report, better than most. The author has built a recovery protocol for long-form streaming translation that solves a real problem: an API can return a syntactically valid stream that is empty, truncated, filtered, source-echoing, or interrupted after useful text. The design—64-character leading hold window, typed replace/restart events, boundary-safe sentence/paragraph retention, stable model rotation with at-most-once attempts, layered machine fallback with provenance markers—is a genuine combination I haven't seen in the routing/abstention/serving literature. The paper also deserves credit for scoping its evidence honestly: Section 7.2 calls the fixed cases 'internal conformance evidence' and Section 11 lists the missing live-provider and quality evaluations explicitly.\n\nBut there is a real soft spot, and the stress-test note lands. The conformance numbers in Table 3 are not derivable from the protocol description. Section 4 says an output is rejected for omission when 'paragraph retention is below 0.70 and the output-to-source character ratio is below 0.15,' without defining how either quantity is measured. Section 5.3 says a boundary-safe prefix keeps 'complete sentence-like units ending in punctuation, bounded by the number of complete source sentences,' but gives no alignment or punctuation rule. So the '31 characters retained' and '4/4 expected boundary matches' are properties of the author's private interpretation, not of a checkable spec. The artifact is the only place the rules live, which makes the headline claim—'executable checks of the published control flow'—stronger than the published control flow supports.\n\nThis is not a fatal flaw. The recovery decisions are plausible, the design lessons in Section 8 (e.g., why sentence rescue was limited, why at-most-once routing replaced a fixed model pair) are instructive, and the paper does not overclaim quality or prevalence. But a serious referee should ask for either a pseudocode-level specification of the boundary and omission heuristics or a pointer to unambiguous code, and a softened abstract.\n\nWho is this for? Practitioners building streaming LLM applications with long inputs, not MT researchers. It deserves peer review because the design is new and the artifact is a concrete starting point for replication. My recommendation: send it out, but with the expectation of a revision that makes the conformance claims checkable.","headline":"A useful, honest engineering design for recoverable streaming translation, undermined by conformance claims that outrun the precision of the published protocol.","tokens_in":10403,"tokens_out":3855,"would_cite":true,"duration_ms":34410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-form translation needs a recovery protocol because API success is not translation success.","keywords":["long-form translation","recoverable LLM translation","streaming guard","boundary-safe resume","failure-aware routing","machine translation fallback","provenance","conformance testing"],"falsifier":"Replay the artifact against a corpus of naturally occurring provider streams and observe whether any invalid prefix becomes visible before the guard fires, whether any omitted output passes the paragraph-retention and character-ratio thresholds, and whether retry counts exceed the documented bounds.","tokens_in":1540,"feed_emoji":"🌐","tokens_out":1932,"duration_ms":60070,"temperature":0.7,"pith_summary":"The paper claims that a long-form translation request can succeed at the API layer and still produce an unusable result, such as empty output, truncation, filtered content, source or prompt echo, or a stream interrupted after valid text. It argues that release, recovery, and fallback should be separate decisions governed by typed stream events, not by request status alone. The proposed protocol holds the first 64 characters before showing them, validates assembled output, retains only structurally aligned paragraph or sentence prefixes after interruption, rotates through a stable bounded model order, and marks machine-fallback provenance. A sanitized companion artifact with 38 public tests and fixed cases is offered as executable evidence that the control flow behaves as specified.","feed_headline":"64-character buffer keeps bad translations unseen","feed_subtitle":"Recoverable translation holds output through a guard, resumes at safe boundaries, and marks fallback provenance.","key_machinery":"The load-bearing mechanism is the guarded streaming state machine with typed server-sent events. A leading-window guard buffers the first 64 characters and runs prompt-marker and source-residue checks before flushing, while a completion validator checks finish reason, non-emptiness, prompt markers, residue, and a conservative omission heuristic. A boundary-safe prefix, built by retaining complete paragraphs plus sentence-like units that align to available source units, defines what may survive an interruption. The at-most-once model rotation with a shared handler deadline bounds recovery before a provenance-marked machine fallback runs.","core_discovery":"The central discovery is that generating text cannot be the same as committing it. The paper models translation output through four states: buffered, visible, boundary-safe, and committed. It shows how an orchestrator can decide whether to show, retain, retry, or route text while recovery is still in progress. The key claim is that holding a short leading window before any visible release catches early prompt echo and source-heavy prefixes, while typed events such as restart and replace let the client distinguish a fresh attempt from a correction of current text. After an interruption, only a boundary-safe prefix derived from complete paragraphs and punctuation-complete sentence-like units aligned to the source may survive.","pith_inferences":["Beyond the paper: the 64-character window is a deployment compromise, so the same architecture could adapt the window size dynamically to expected latency, source-language residue risk, or observed provider behavior.","Beyond the paper: the omission heuristic thresholds (paragraph retention below 0.70 and output-to-source character ratio below 0.15) could be benchmarked as a tunable detector on naturally occurring outputs, which the paper explicitly leaves unmeasured.","Beyond the paper: the paper's own proposed next step is selective recovery as a learned policy that chooses accept, retry, route, fall back, or abstain, which could be compared against the fixed policy on translation quality, latency, cost, and release risk.","Beyond the paper: because boundary-safe alignment is structural rather than semantic, a retained sentence that aligns positionally but translates incorrectly could still pass, a residual risk the paper itself flags."],"forward_implications":["If a provider returns a syntactically valid stream of source echo or hidden instructions, the leading window prevents any of the invalid prefix from becoming visible until whole-output validation can act.","Interrupted work is not lost entirely: retries resume from a boundary-safe prefix rather than restarting the document, preserving valid text while discarding structurally incomplete fragments.","Recovery is bounded by a stable model order, an at-most-once rule per path, and a shared deadline, so latency and cost cannot grow without limit.","Machine fallback output is provenance-marked and all-or-nothing, so a timeout cannot be converted into a plausible-looking half translation.","The fixed conformance cases are executable checks of release, retention, attempt, event, and provenance behavior, but they do not estimate translation quality, detector recall, or failure prevalence."],"supporting_citations":[{"why":"Establishes document-level context effects in machine translation, motivating chunked document-level progress and context-aware recovery.","marker":"[10]"},{"why":"Documents length issues in document-level machine translation, motivating truncation and large-omission checks.","marker":"[12]"},{"why":"Shows how context can be incorporated into machine translation metrics, supporting the adequacy-sensitive evaluation stance.","marker":"[17]"},{"why":"Provides a broad evaluation of LLMs for translation, framing the adequacy-sensitive context for LLM-specific failure modes.","marker":"[6]"},{"why":"Examines refinement in literary translation, supporting the claim that adequacy rather than fluency alone matters for LLM output.","marker":"[15]"},{"why":"Provides a benchmark for hallucination and omission detection in machine translation, informing the design space of the omission guard.","marker":"[3]"},{"why":"Shows infrastructure-level recovery of generation state, which the paper distinguishes from application-level release safety.","marker":"[14]"},{"why":"Proposes learned routing of LLM queries, which the paper contrasts with its fixed stable-order and fallback policy.","marker":"[9]"}],"fun_headline_variants":["Translation recovery: hold output, validate, then reveal","Buffered translation catches bad prefixes before they show","LLM translation: retry safely with a 64-char guard","Don't show translation until it passes the buffer test","Recoverable LLM translation with typed stream events"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The reported numbers are properties of a small hand-written test set that uses neither live provider calls nor production traces, so the protocol's success in the tests transfers to operation only if real provider event sequences resemble those cases.","fun_headline_variants_meta":{"raw":{"variants":["Translation recovery: hold output, validate, then reveal","Buffered translation catches bad prefixes before they show","LLM translation: retry safely with a 64-char guard","Don't show translation until it passes the buffer test","Recoverable LLM translation with typed stream events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1338,"prompt_tokens":874,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":490,"tokens_out":464,"duration_ms":4442,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:48:46.315450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay the artifact against a corpus of naturally occurring provider streams and observe whether any invalid prefix becomes visible before the guard fires, whether any omitted output passes the paragraph-retention and character-ratio thresholds, and whether retry counts exceed the documented bounds.","supporting_citations":[{"cited_title":"Document-level machine translation with large-scale pub- lic parallel corpora","cited_arxiv_id":null,"evidence_quote":"Establishes document-level context effects in machine translation, motivating chunked document-level progress and context-aware recovery."},{"cited_title":"Inves- tigating length issues in document-level machine transla- tion","cited_arxiv_id":null,"evidence_quote":"Documents length issues in document-level machine translation, motivating truncation and large-omission checks."},{"cited_title":"Embarrassingly easy document- level MT metrics: How to convert any pretrained metric into a document-level metric","cited_arxiv_id":null,"evidence_quote":"Shows how context can be incorporated into machine translation metrics, supporting the adequacy-sensitive evaluation stance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Examines refinement in literary translation, supporting the claim that adequacy rather than fluency alone matters for LLM output."},{"cited_title":"Costa-juss` a","cited_arxiv_id":null,"evidence_quote":"Provides a benchmark for hallucination and omission detection in machine translation, informing the design space of the omission guard."}],"review_version":1}