{"id":"76138c79-dc6b-47d8-a17e-202be89557ad","arxiv_id":"2506.14715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Procedural Knowledge Libraries are proposed as versioned, executable records of research processes, with a lens- and patch-based architecture for Jupyter workflows.","lead":"This paper proposes Procedural Knowledge Libraries, a framework for recording the full process behind research, including failed attempts and decisions, not just final results. It sketches a Jupyter-friendly architecture using lenses, patch-based versioning, and metadata querying to make that process reusable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic tags from AST/import pattern matching are asserted to capture researcher intent, yet §5.5 concedes goals are implicit and §3.1 defines completeness against 'original intent'; retrieval and debugging rest on an unvalidated, arguably circular ground truth.","rationale":"I agree with the reader's weakest-assumption identification: semantic tagging is the load-bearing link between raw code traces and the claimed procedural knowledge. The reversibility of lenses is also unproven, but it is secondary; even a perfectly reversible lens system would not distinguish PKLs from ordinary version control without reliable intent-carrying tags. The paper is a conceptual proposal, and the reader's CONDITIONAL verdict remains appropriate. My stress-test does not move that verdict, but it sharpens the condition: the authors should release the prototype, supply a ground-truth evaluation of tag accuracy against author-provided intent, and either operationalize the 'original intent' criterion in §3.1 or explicitly reframe semantic tags as optional human annotations rather than automatic intent recovery. Credit is due for the coherent synthesis of lenses, patch storage, and notebook provenance, and for citing relevant prior work; the concern is about an unvalidated and internally tension-filled premise, not about the overall framing.","tokens_in":13248,"tokens_out":6834,"duration_ms":66875,"concrete_test":"Run the classification pipeline described in §5.6 (or the claimed prototype) on a held-out set of notebooks with per-cell ground-truth intent annotations supplied by the original authors, e.g., by extending the Rule et al. 2018 notebook corpus with intent labels. Compute precision and recall of the inferred semantic tags against these labels, and separately have two domain-qualified interpreters reconstruct procedures from a PKL generated by the system and compare their reconstructed intent to the ground truth. If tag agreement is at chance or the interpreters diverge, the completeness definition of §3.1 is not operationalizable and the retrieval/debugging model loses its foundation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PKLs provide 'executable, version-controlled records that contextualize each step'—depends on semantic tags that stand in for researcher intent. Section 5.6 says a classification engine labels each unit using ASTs, import resolution, and pattern matching; Section 5.5 acknowledges that in exploratory notebooks 'goals often evolve and remain implicit' and that the system only stores tags such as 'load data', 'train model', 'plot results'. These are operation-type descriptors, not intent: the same cell `plt.plot(x, y)` can be an exploratory check, a final figure, or a debugging artifact, and the engine has no described basis for distinguishing them. Section 3.1 defines a PKL artifact as complete if a 'domain-qualified interpreter' can reconstruct the procedure 'in a way that aligns with its original intent,' but original intent is not recorded independently; it is proxied by the very tags whose reliability is at issue, making completeness circular. The paper's only concrete extraction code (§5.6.1) collects imports and function definitions; it does not produce the claimed semantic tags, and no evaluation or ground-truth comparison is provided. If tags do not reliably reconstruct intent, retrieval by intent and the debugging scenario in §6.5.2—which claims to surface 'meaningful differences' such as a removed hyperparameter tuning step—cannot deliver the promised contextualization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Procedural Knowledge Libraries (PKLs) as a framework for capturing the full research process—not just final outputs—by storing executable, version-controlled, semantically annotated records of procedural steps. It defines a lens-based transformation model for selective extraction/abstraction of workflow units, a patch-based storage schema using unified diffs, and a retrieval layer with semantic tags and Lamport timestamps. The architecture is described for Jupyter notebooks and generalized to other environments (scripts, pipelines, IDEs). The central claim is that PKLs provide a practical blueprint for preserving procedural knowledge so that researchers can share reasoning, discarded approaches, and intermediate analyses, enabling earlier collaboration and reproducibility.","tokens_in":13478,"tokens_out":4010,"duration_ms":35375,"significance":"If the architecture were fully implemented and validated, PKLs would address a real and timely problem: the loss of tacit procedural knowledge in computational research. The paper brings together established concepts (lenses, unified diffs, logical clocks, semantic tagging) and offers concrete artifacts—a database schema, example lens templates, and code snippets—that make the proposal more actionable than a purely abstract position piece. The writing is clear and the related work is relevant, including a nice connection to model diffing. However, the paper currently reads as a design proposal with several central guarantees asserted rather than demonstrated. The load-bearing issues are the reliability of semantic tags as a proxy for intent, the circularity of the completeness definition, and the unproven reversibility of lens templates. These can be addressed by adding validation, or by scoping the claims to operation-type classification and a weaker notion of completeness.","major_comments":[{"comment":"The classification engine in §5.6 labels units with semantic tags such as 'data loading' or 'model training' via ASTs, import resolution, and pattern matching, but §5.5 explicitly concedes that in exploratory notebooks 'goals often evolve and remain implicit.' The same code cell (e.g., plt.plot(x, y)) can be an exploratory check, a final figure, or a debugging artifact, and the described engine has no basis for distinguishing these. Because retrieval (§6.5) and the debugging scenario (§6.5.2) rely on these tags to surface 'meaningful differences,' the central claim of contextualization is unsupported unless the tags are validated against a ground truth of intent or the claims are explicitly reduced to operation-type classification rather than intent recovery.","section":"§5.6, §5.5"},{"comment":"Completeness in §3.1 is defined as a domain-qualified interpreter being able to reconstruct the procedure 'in a way that aligns with its original intent.' However, original intent is not recorded independently; it is proxied by the semantic tags whose reliability is at issue in §5.6. This makes the definition circular and leaves no empirical way to falsify a claim of completeness. Please specify an independent ground-truth definition of intent (e.g., explicit author annotations, a structured goal specification, or an oracle based on execution outcomes) and state how reconstruction would be measured against it.","section":"§3.1"},{"comment":"The paper asserts that lens templates 'preserve reversibility, ensuring that any representational transformation (or view) applied to a procedure can also be undone without loss of information or procedural integrity.' This is a strong guarantee, but it is not proven. The example template's InversePatchTemplate (a deep-merge strategy) is not shown to invert the extraction pattern in general, and the composition code in §6.1 only checks schema compatibility, not round-trip correctness. The claim needs either a formal proof for the proposed lens algebra (for both atomic and composed lenses) or a substantial weakening, ideally accompanied by round-trip tests on representative procedures.","section":"§6.1"},{"comment":"The code snippets do not constitute a working prototype as implied by the 'practical blueprint' claim in §13. The §5.6.1 classifier extracts only imports and function definitions, not the semantic tags used later; the §6.4.2 apply_lens function calls an undefined generate_patch_from_template and the database insert references variables (view_id, parameters, view_path) that are not clearly defined in the surrounding code. No evaluation is presented against real notebooks or workflows. To support the central claim, the manuscript needs at least a minimal validation (e.g., a small case study, a pilot implementation, or an illustrative run on several notebooks) or an explicit downgrade of the contribution to a design proposal.","section":"§5.6.1, §6.4.2"}],"minor_comments":[{"comment":"The regex for function definitions appears to contain a typo: \\s*\\$( should presumably be \\s*\\(.","section":"§5.6.1"},{"comment":"The text contains a typo: 'strcutured' should be 'structured'.","section":"§6"},{"comment":"The phrase 'the two probing methods are compliments of each other' should be 'complementary to each other'.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"This is a vision/design paper. The architecture is coherent and the topic is timely, but the current version overclaims validation for the 'practical blueprint' contribution. The central issues (tag reliability, circular completeness, lens reversibility) are fixable within the manuscript's scope by adding a small evaluation and carefully scoping the claims. If the journal prefers fully validated systems papers, it might not be a strong fit; otherwise, major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a coherent position piece: it proposes Procedural Knowledge Libraries as a lens-based, patch-backed way to preserve the process behind computational research, and it is honestly framed as a blueprint, not an evaluation. The integration of bidirectional lenses, unified diffs, and semantic metadata for Jupyter-style workflows is genuinely new as a synthesis, and the storage schema and lens composition rules are sensible. Credit where due: the author engages seriously with relevant work (episodic memory, model diffing, existing provenance tools) and the Jupyter examples are concrete enough to follow.\n\nThe soft spots are in the load-bearing parts. Semantic tags are supposed to express researcher intent ('type: plotting, intent: exploratory'), but the classification engine actually described in §5.6 parses imports and function definitions—it does not infer why a cell exists. The author acknowledges in §5.5 that goals in exploratory notebooks are often implicit. So the retrieval and debugging scenario in §6.5.2, which promises 'meaningful differences' like a removed hyperparameter step, has no demonstrated basis. Completeness, defined in §3.1 as reconstruction 'in a way that aligns with its original intent,' suffers from a similar problem: intent is not recorded independently, so you cannot test the definition. This is a real gap, not a nitpick, but it's also the kind of gap that a serious revision could address by releasing a prototype, running a user study, or formally bounding what tags can and cannot capture.\n\nThe paper also overclaims in §9 where it says a prototype exists—no code is released or evaluated, so that sentence needs tempering. The complexity analysis (O(n log v) retrieval) is a sketch, not a proof, and is fine as such.\n\nWho gets value: tool builders and digital-library people who want a vocabulary and a checklist for what a process-level capture system should look like. It's a useful conversation starter, not a working system. I'd send it to peer review, but expect heavy revision: the authors need to either ship the prototype or narrow the claims to a research agenda. A referee who pushes on the tagging-intent link would be doing the field a service.\n\nRecommendation: consider for publication as a position/vision paper with major revisions, not as an architecture paper with working code.","headline":"A coherent position piece with a genuinely new synthesis, but the semantic tagging and completeness definitions carry more weight than the paper can yet support.","tokens_in":14029,"tokens_out":3204,"would_cite":false,"duration_ms":55040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Procedural Knowledge Libraries would preserve the full arc of scientific inquiry—hypotheses, failures, decisions, and iterations—as executable, version-controlled records, built on lens transformations, patch-based storage, and semantic…","keywords":["procedural knowledge libraries","executable research memory","lens-based transformations","semantic retrieval","patch-based versioning","Jupyter notebooks","research reproducibility","tacit knowledge preservation"],"falsifier":"Take a sample of notebooks whose authors have annotated each cell's purpose, encode them as PKL artifacts, and ask independent domain-qualified researchers to reconstruct the original procedure from the artifact alone. If their reconstructed intentions match the authors' annotations no better than chance—or if reconstruction fidelity does not exceed what readers get from the raw notebook—the central claim collapses.","tokens_in":12959,"feed_emoji":"📚","tokens_out":8278,"duration_ms":76435,"temperature":0.7,"pith_summary":"The paper argues that the most valuable part of research—the reasoning, failed attempts, and decisions behind results—is routinely lost, so it proposes Procedural Knowledge Libraries (PKLs), frameworks for capturing the full arc of inquiry as executable, versioned records. The central claim is that PKLs provide executable, version-controlled records that contextualize each step of a research process, and that an architecture based on lens transformations, patch-based storage, and semantic retrieval is a practical blueprint for building them. If this is right, sharing a Jupyter notebook could mean sharing not just outputs but the reasoning, discarded approaches, and intermediate analyses that shaped them, enabling earlier collaboration and clearer methodological accountability. A sympathetic reader would care because this turns tacit expertise into explicit, queryable, and reusable units rather than leaving it in notebooks, emails, or memory.","feed_headline":"Capture every research step, including the dead ends","feed_subtitle":"A lens-based framework turns Jupyter notebooks into queryable records of decisions, failed attempts, and intent.","key_machinery":"The central object is the lens: a bidirectional transformation that isolates, extracts, or reinterprets procedural components and is implemented as a parameterized patch template with an inverse, so any view can be undone without loss. A classification engine labels each procedural unit (a notebook cell, function, or task) with semantic tags using abstract syntax trees, import resolution, and pattern matching; Lamport timestamps record causal order; unified diffs store changes; and a lightweight metadata database indexes procedures, lenses, and views. Lenses compose when the output schema of one matches the input schema of the next, which is what makes complex queries such as 'extract hyperparameters, then visualize as a table' possible.","core_discovery":"On the paper's own terms, the central claim is that Procedural Knowledge Libraries (PKLs) can preserve the full arc of scientific inquiry—hypotheses, failures, decisions, and iterations—as executable, version-controlled records, and that a workable blueprint is the lens-based architecture proposed here. A PKL artifact is complete not when it contains every detail but when a domain-qualified interpreter can reconstruct the procedure in a way that aligns with its original intent. The paper contributes a definition of PKLs, a lens-based abstraction for selectively encoding and transforming procedural units, and an architecture with procedural storage, patch templates, and query sketches. The motivating contrast is that today's tools record outputs and parameters but not the reasoning that produced them; PKLs aim to make that reasoning a first-class, queryable object.","pith_inferences":["The paper treats lenses as tools for views and diffs; one implicit next step is to use lens inverses for reconciliation, so that divergent copies of a procedure can be merged by inverting each side's transformations—extending the standard three-way merge idea in a reversible setting.","A testable corollary of the completeness definition: reconstruction fidelity between the original author and a domain-qualified reader becomes a measurable quantity, and one could use it to compare PKL artifacts against raw notebooks as documentation formats.","The same framework could be turned on the research process itself: a PKL of how a paper was written would make peer review and replication audits richer than code-and-data deposits alone, by exposing which experiments were tried and dropped."],"forward_implications":["A researcher using Jupyter notebooks could share not just final outputs but the reasoning, discarded approaches, and intermediate analyses that informed them, letting collaborators engage before work is finished.","Failed steps can be stored with annotations like “intent: model training” and “outcome: failed convergence,” so future readers can see what was attempted and why it was abandoned.","A temporal lens can diff two versions and return semantic differences—for example, a removed hyperparameter-tuning step—rather than raw line edits, streamlining root-cause debugging.","Because lenses are reversible and composable, the same procedure can be summarized, transformed, or exported through different views without losing the original, supporting reuse across contexts.","At the level of model oversight, PKLs complement weight-level model diffing by capturing shifts in intent, strategy, or framing."],"supporting_citations":[{"why":"defines lenses as bidirectional transformations between a database and a view of it, the basis for reversible procedural views","marker":"[19, 22]"},{"why":"provides combinators for bidirectional tree transformations, grounding lens composition and the view-update problem","marker":"[20]"},{"why":"supplies Lamport timestamps used to record the causal order of procedural units","marker":"[28]"},{"why":"defines patches as structured deltas, the basis for semantic patch templates","marker":"[25]"},{"why":"documents AST-based parsing used by the semantic classification engine","marker":"[7]"},{"why":"documents why shared code and numerical outputs are rarely enough for reproducibility, motivating the PKL approach","marker":"[35]"},{"why":"describes a model family whose reported scaling results came without released methods, motivating procedural archaeology","marker":"[33]"},{"why":"shows how goals in exploratory notebooks evolve and remain implicit, motivating semantic intent tags","marker":"[38]"}],"fun_headline_variants":["Executable memory for your full research journey","Don't just save results—save the reasoning","Turn Jupyter notebooks into queryable decision logs","Preserve the failures that led to your success","Capture the arc of science, failures and all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on whether the meaning of a research step can be reliably read off from its code: automatic semantic tags must capture what the researcher actually intended, because completeness, retrieval, and debugging all assume they do.","fun_headline_variants_meta":{"raw":{"variants":["Executable memory for your full research journey","Don't just save results—save the reasoning","Turn Jupyter notebooks into queryable decision logs","Preserve the failures that led to your success","Capture the arc of science, failures and all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2794,"prompt_tokens":834,"completion_tokens":1960,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":1890}},"tokens_in":450,"tokens_out":1960,"duration_ms":14414,"temperature":1.0,"reasoning_tokens":1890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:10:31.659951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of notebooks whose authors have annotated each cell's purpose, encode them as PKL artifacts, and ask independent domain-qualified researchers to reconstruct the original procedure from the artifact alone. If their reconstructed intentions match the authors' annotations no better than chance—or if reconstruction fidelity does not exceed what readers get from the raw notebook—the central claim collapses.","supporting_citations":[],"review_version":1}