{"id":"64ddcfa0-94fc-4cae-add9-13357cb8a24d","arxiv_id":"2611.16986","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Primer provides an open-source, LLM-agnostic multi-agent system for educators to author adaptive educational simulations through a guided interview-to-deployment workflow.","lead":"Primer is an open-source AI framework that helps teachers create custom educational simulations by combining their own expertise with generative AI. The system uses a multi-agent pipeline to interview educators, design scenarios, and produce playable roleplay or teachback experiences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"LLM-agnosticism is asserted but never tested; the tool-heavy prompts and strict JSON requirements may not work with local models, undermining the governance flexibility claim.","rationale":"The reader's verdict was CONDITIONAL, focused on the absence of evidence for pedagogical effectiveness. I agree that is a real limitation, but it is explicitly acknowledged by the authors ('they do not establish its effectiveness as a learning intervention') and future work. The more load-bearing unacknowledged claim is LLM-agnosticism, which underpins the paper's governance flexibility value proposition and is stated without any cross-model testing. If the system only works with frontier models, the claim that it can be used under varied governance regimes (with local models) fails, weakening the contribution. This concern does not overturn the paper's core architectural contribution—the open-source design and prompts are inspectable and adaptable—so the verdict remains CONDITIONAL, with the added condition that model portability be demonstrated. The reader did not identify this specific gap, though they noted the lack of effectiveness evidence; hence partial agreement.","tokens_in":45251,"tokens_out":4416,"duration_ms":32117,"concrete_test":"Run Primer's full authoring pipeline (intake, design, casting, build) with a representative open-weight model (e.g., Llama 3.1 70B or 405B) and with a non-OpenAI commercial model (e.g., Claude or Gemini) on the same scripted educator scenario. Measure: (1) whether the pipeline reaches the handoff phase without human intervention, (2) whether the Phases Configuration artifact is valid JSON with the required keys and no extra top-level keys, (3) whether the resulting experience actually runs with mentor-practice-feedback phases. Compare completion rates and any qualitative failure modes against the OpenAI baseline. If the open-weight model fails on JSON strictness, tool-call sequence, or agent handoff, the LLM-agnosticism claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims Primer is 'LLM-agnostic' and 'allow[s] the use of various AI models, including local and open source models,' and the paper frames this as enabling implementation under varied governance regimes. Yet the only model used in testing is OpenAI's gpt-realtime-1.5 for voice (Section 'Voice, Pedagogy, and Risks'), and no evidence is presented that the authoring or runtime agents function with any other model. The system's prompts require reliable tool calling (artifact, sessionEvent, delegation, checklist) and the Design Agent must emit a Phases Configuration that 'must parse as JSON by itself' with strict constraints. These are exactly the behaviors where model capability varies substantially; frontier models may comply, local open-weight models often fail at structured tool use or multi-step instruction following. The paper's own reference to Jurenka et al. notes that models produce uneven outputs without scaffolding. Since LLM-agnosticism is a stated value proposition (governance flexibility), it is a load-bearing empirical claim. Without cross-model testing, the claim is unsupported, and the paper's stated benefit of being deployable under varied legal regimes is not established. This is distinct from the acknowledged lack of effectiveness evidence; it is an unacknowledged assumption about model portability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents Primer, an open-source AI system that uses a multi-agent pipeline to help educators create adaptive educational simulations. The system elicits educator intent through an interview, produces a design document for review, and then generates a playable simulation with mentor, practice, and feedback phases. The paper describes the architecture, prompts, and formative testing with educators that identified usability issues. The authors state that the system is LLM-agnostic and can use local/open-source models, and they position the work as a design contribution with an open-source release. The paper explicitly acknowledges that the formative testing does not establish effectiveness as a learning intervention.","tokens_in":48702,"tokens_out":1469,"duration_ms":28922,"significance":"If the system works as described, it could lower the barrier for educators to create practice simulations, enabling more varied and adaptive practice opportunities. The open-source release and detailed prompts allow replication and extension by the research community. However, the current evidence is limited to formative usability testing with educators playing the role of students; effectiveness in terms of student learning outcomes is not established. The main contribution is the systematic architecture and its detailed documentation, not empirical validation. The unsupported claim of LLM-agnosticism weakens the governance-flexibility value proposition, which is a central stated benefit.","major_comments":[{"comment":"The paper claims Primer is 'LLM-agnostic' and can use local/open-source models, enabling implementation under varied governance regimes. However, the only model used in testing is OpenAI's gpt-realtime-1.5. The prompts in Appendix B require reliable tool calling (artifact, sessionEvent, delegation, checklist) and the Design Agent must emit a Phases Configuration that 'must parse as JSON by itself' with strict constraints. These are exactly the capabilities that vary substantially across models; local open-weight models often struggle with structured tool use and multi-step instruction following. Without cross-model testing, the claim of LLM-agnosticism is unsupported. The authors should either provide evidence that the system functions with at least one alternative model (e.g., a local open-source model) or explicitly qualify the claim to reflect that it has only been tested with OpenAI models.","section":"Abstract; Section 'Voice, Pedagogy, and Risks'; Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The formative testing provides useful usability feedback, but the evidence that the system 'was capturing their intent' rests on three positive quotes from educators. This is anecdotal. The paper would benefit from a more systematic evaluation of the authoring pipeline's output, such as having independent raters compare design documents to educator intent or assessing the quality of generated simulations against pedagogical criteria. The authors acknowledge that effectiveness requires formal research, but the claim that Primer 'helps educators create adaptive educational simulations' would be strengthened by evidence beyond usability. The authors should either provide such evidence or frame the contribution more narrowly as a system design and prototype, with evaluation of output quality left to future work.","section":"Section 'Testing and Development with Educators'","authors_acknowledge":"That requires formal research with students and educators under classroom conditions.","authors_acknowledge_p":0.67}],"minor_comments":[{"comment":"Add a sentence explicitly stating that cross-model testing has not been conducted and that the LLM-agnostic claim is currently a design goal rather than an empirically supported property.","section":"Limitations and Future Work","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The section notes that educators played the role of students, but this should be highlighted earlier in the section to avoid any overinterpretation of the results.","section":"Section 'Testing and Development with Educators'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Consider adding a summary table of the agents and their roles to improve readability and quick reference.","section":"Primer Overview","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The reference for OpenAI is incomplete (OpenAI, n.d.); provide full documentation details or a direct URL.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper references 'Appendix D: A Paper That Builds Itself,' but this appendix is not present in the provided text. Ensure the appendix is included or remove the reference.","section":"Discussion: Building Amid a Shifting Frontier","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Primer is an open-source AI system that lets educators create adaptive simulations by simply describing their goals to an AI interviewer.","keywords":["educational simulations","AI authoring","multi-agent systems","educator-in-the-loop","generative AI in education","adaptive practice","open-source AI","instructional design"],"falsifier":"A controlled study comparing student learning outcomes from Primer-generated simulations against equally scripted non-AI simulations (or against traditional instruction) would test the central claim. If students using Primer simulations show no significant improvement on transfer assessments compared to students who received equivalent non-AI instruction, the claim that Primer produces effective adaptive practice would be undermined.","tokens_in":51232,"feed_emoji":"🎓","tokens_out":1374,"duration_ms":36421,"temperature":0.7,"pith_summary":"Primer is an open-source, LLM-agnostic AI system that enables educators to create adaptive educational simulations through a structured multi-agent authoring pipeline. The system treats simulation creation as a guided instructional design process: an Intake Agent interviews the educator about goals, student context, and materials; a Design Agent translates that into a structured plan; and after educator review, specialized agents assemble the playable experience. At runtime, students encounter a three-phase structure (mentor preparation, practice application, feedback reflection) with agents that maintain character roles while tracking pedagogical goals. The paper argues for AI-human complementarity: the educator supplies local instructional judgment, and AI provides adaptive variation within that frame. If the approach succeeds, it could make high-quality practice simulations widely accessible to educators without requiring technical expertise.","feed_headline":"Open-source AI system turns educator intent into playable simulations","feed_subtitle":"Primer uses a multi-agent pipeline to translate teacher goals into a three-phase learning experience.","key_machinery":"The multi-agent authoring pipeline centered on the Intake Agent and the design document as a boundary object. The Intake Agent conducts a structured interview that elicits educator expertise and maintains a planning journal, then delegates to the Design Agent which produces a design document that serves as the shared reference for all downstream agents. The three-phase runtime structure (Mentor, Practice, Feedback) with dual-role agents that simultaneously embody characters and track pedagogical goals is the mechanism that translates educator intent into adaptive practice.","core_discovery":"Primer is an open-source AI framework that makes simulation authoring accessible to any educator by using a multi-agent pipeline to translate instructional intent into a playable experience. The system's Intake Agent conducts a structured interview to elicit the educator's learning goals, student context, prior knowledge, common misconceptions, and available materials. This conversation is transformed into a design document that functions as a boundary object: the educator reviews it as a lesson plan, while downstream agents (Casting Director, Image Generation Agent, Experience Builder) interpret it as a build specification. At runtime, students interact with three agents (Mentor, Practice, Feedback) that simultaneously embody characters and track learning objectives, adapting challenge based on student performance. The central innovation is the artifact-centered workflow: educator intent travels through the system as inspectable artifacts (transcripts, design documents, journals, checklists) that maintain alignment across the authoring and runtime phases.","pith_inferences":["The most significant barrier to adoption may be organizational rather than technical: educators need dedicated time to author simulations, trust the output, and integrate them into existing curricula, and the paper's formative testing with educators playing as students does not address classroom workflow or student learning outcomes.","The system's pedagogical design is grounded in specific learning theories (backward design, scaffolding, retrieval practice, desirable difficulties), but the paper does not test whether these theories hold in AI-mediated practice contexts; the effectiveness of Primer depends on empirical validation that is not yet conducted.","The explicit limitation that practice variation unfolds within a single encounter (not across varied cases) means the system has not yet implemented one of the key mechanisms the authors identify as important for durable learning: varied practice across different contexts."],"forward_implications":["Educators without technical or game-design expertise can create adaptive simulations tailored to their specific students, context, and learning objectives.","The open-source release of the full system, including prompts and orchestration code, enables researchers and developers to inspect, modify, and improve the architecture, supporting community-driven development of AI authoring tools.","The artifact-centered workflow provides a design pattern for maintaining alignment with user intent across multi-step generative processes, applicable beyond simulations to other educator-in-the-loop AI systems.","The architecture points toward a scalable model where AI handles adaptive variation while educators focus on instructional judgment, potentially increasing the availability of high-quality practice opportunities across educational settings."],"supporting_citations":[{"why":"Establishes the pedagogical value of simulations that Primer aims to make accessible.","marker":"Chernikova et al., 2020"},{"why":"Provides evidence for retrieval practice that informs Primer's simulation design.","marker":"Agarwal et al., 2021"},{"why":"Supports the claim that generative AI without guardrails can harm learning, motivating Primer's structured approach.","marker":"Bastani et al., 2025"},{"why":"Supports the need for scaffolding and educator oversight of AI outputs.","marker":"Jurenka et al., 2024"},{"why":"Provides prior work on AI agents for educational practice that Primer builds on.","marker":"Mollick et al., 2024"},{"why":"Provides the backward design framework used in Primer's intake process.","marker":"McTighe & Thomas, 2003"},{"why":"Provides learning science foundation for the simulation design.","marker":"National Research Council, 2000"},{"why":"For scaffolding theory used in the design.","marker":"van de Pol et al., 2010"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the multi-agent orchestration and prompt engineering it describes will reliably produce simulations that are pedagogically sound and aligned with educator intent, without evidence from controlled studies of student learning outcomes or classroom deployment.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7.3e-05,"raw_usage":{"cost":7.30002e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.30002e-05,"upstream_inference_prompt_cost":7.26362e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":1005,"prompt_tokens":1003,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1003,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":2306,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T07:41:00.355132+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A controlled study comparing student learning outcomes from Primer-generated simulations against equally scripted non-AI simulations (or against traditional instruction) would test the central claim. If students using Primer simulations show no significant improvement on transfer assessments compared to students who received equivalent non-AI instruction, the claim that Primer produces effective adaptive practice would be undermined.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}