{"id":"2cbb6f46-808e-45c9-95f9-efdd553144e9","arxiv_id":"2501.11774","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A case study reports that stakeholders in Petrobras and Americanas projects largely accepted PUC-Rio's Lean R&D process, though the evidence is self-reported and lacks a baseline.","lead":"This industrial experience paper reports how a university team applied its own Lean R&D process to innovation projects with Petrobras and Americanas, based on 57 survey responses. It offers a practical example of structuring fast, outcome-focused industry-academia collaboration, but its evidence is mostly stakeholder perception without a comparison baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness claim rests on successes that passed the process's own go/no-go checkpoints, with no data on failed or filtered projects; self-reported acceptance alone cannot support 'transforming ideas into outcomes.'","rationale":"The reader's weakest assumption—self-report and in-group bias—is real, but the more load-bearing structural defect is selection/survivorship bias. The process's checkpoints are explicitly designed to stop unpromising ideas; reported successes are therefore conditional on having survived those checkpoints. Without counting failed inceptions or failed MVPs, the paper cannot support the causal-sounding claim that Lean R&D transforms ideas into outcomes. That is not an ad hominem charge; it is an internal design property of the process that the paper itself describes. The reader also correctly notes the absence of a baseline, and the paper acknowledges this in Section 5. My concrete test asks for attrition data and independent outcome protocols, which would adjudicate between 'Lean R&D enables filtering' and 'Lean R&D causes transformation.' Until then, the evidence supports a conditional acceptance as an experience report, not as evidence of comparative effectiveness. I therefore recommend no change to the reader's CONDITIONAL verdict; the concern strengthens the condition, not the rejection.","tokens_in":7570,"tokens_out":2052,"duration_ms":23168,"concrete_test":"Download the linked Zenodo dataset (doi:10.5281/zenodo.8317454) and reconstruct the full project pipeline: count Lean Inceptions started per partner, MVP outlines rejected at checkpoint 1, projects stopped at checkpoints 2–4, and MVPs that reached transition. If the dataset contains no failed projects, request attrition counts from the authors; if attrition is nonzero, recompute the effectiveness claim with failures in the denominator. Additionally, locate the continuous-experimentation data behind the Smart Tocha energy claim and the 1:20 ROI calculation; if no measurement protocol exists, the headline claim should be downgraded from 'effectiveness' to 'perceived suitability.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Lean R&D is effective at transforming ideas into meaningful business outcomes—rests on reported outcomes of completed projects and 57 survey responses. Both pillars are structurally incapable of supporting an effectiveness reading, and the reason is more specific than courtesy bias.\n\nFirst, selection. Lean R&D's design (Section 2) uses checkpoints to kill unpromising MVPs: at the first checkpoint, the steering committee can reject an MVP outline and trigger fail-fast. Reported successes are therefore conditional on passing these internal filters. The paper gives no denominator: no count of Lean Inceptions conducted, MVP outlines rejected, projects stopped at later checkpoints, or MVPs that never reached transition. Without that denominator, the observed successes cannot distinguish 'Lean R&D transforms ideas' from 'Lean R&D filters ideas until a success remains.' The claimed fail-fast property cannot be evaluated without failed cases, and 'effectiveness' is overstated if failures occurred but are unreported.\n\nSecond, outcome measurement. The energy-savings and report-efficiency statements are attributed to 'continuous experimentation measurements' (Section 4.1), and Section 6 cites an ROI exceeding 1:20 calculated by the company. No measurement protocol, sample size, confidence interval, or calculation method is provided. The only linked data (Zenodo) are questionnaires; the business outcomes are not independently auditable from the manuscript.\n\nThird, the questionnaire is the authors' own instrument, administered to participants in projects run by the authors. The paper itself concedes 'the absence of a formal baseline or direct comparison to alternative frameworks' (Section 5). Agreement on Likert items about suitability and TAM constructs is consistent with goodwill, career interest, or social desirability; it cannot establish comparative effectiveness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This industrial experience paper reports on the application of Lean R&D, an agile research-and-development approach developed at PUC-Rio, to industry-academia collaboration projects with Petrobras (oil and gas) and Americanas (retail). The paper describes the Lean R&D phases and checkpoints, reports project outcomes including patent applications, energy savings, and an ROI estimate, and analyzes 57 questionnaire responses from team members, managers, and sponsors assessing phase suitability and overall acceptance, largely using Technology Acceptance Model statements. The abstract claims that the findings highlight Lean R&D's effectiveness in transforming ideas into meaningful business outcomes, while the discussion acknowledges the absence of a formal baseline or direct comparison to alternative frameworks.","tokens_in":7821,"tokens_out":2637,"duration_ms":29437,"significance":"If the claims were fully supported, the paper would provide useful practitioner guidance on structuring fast-moving industry-academia innovation projects. The paper's strengths are its concrete description of a real process, two different application contexts (experienced R&D staff at Petrobras and student teams at Americanas), an openly linked questionnaire data set, and explicit attention to stakeholder feedback and improvement opportunities. The reported stakeholder acceptance and the successful delivery of MVP-based projects are valuable experience data for the software engineering community. However, the evidence as presented supports claims about perceived suitability and acceptance, not the stronger effectiveness claim made in the abstract and conclusion.","major_comments":[{"comment":"The central claim that Lean R&D is 'effective in transforming ideas into meaningful business outcomes' is stronger than the evidence presented. Section 5 explicitly acknowledges the absence of a formal baseline or direct comparison to alternative frameworks, and the outcome evidence is a combination of company-reported figures and self-reported perceptions from participants in the authors' own projects. The abstract and conclusion should be revised to state the narrower claim that stakeholders perceived Lean R&D as suitable and accepted it, and that the approach was associated with the reported project deliveries, rather than asserting demonstrated effectiveness.","section":"Abstract and Section 5"},{"comment":"The quantitative business outcomes—energy savings comparable to a city of 20,000 inhabitants per refinery, faster equipment-inspection report completion, and an ROI exceeding 1:20—are presented without any measurement protocol, sample size, confidence interval, or calculation method. The ROI in Section 6 is stated to be 'calculated by the company,' and no public report or data artifact supporting these numbers is linked. Since these figures are the main objective support for the 'business outcomes' claim, the paper must either provide the measurement details or a reference to an auditable source, or clearly label each figure as an unaudited company claim.","section":"Section 4.1"},{"comment":"The reported successes are conditional on passing Lean R&D's internal checkpoints, which can reject MVP outlines, technical feasibility results, MVP readiness, and business-hypothesis validation. The paper provides no denominator: there is no count of Lean Inceptions conducted, MVP outlines rejected, projects stopped at later checkpoints, or MVPs that failed to transition. Without these filtered cases, the observed successes cannot be attributed to Lean R&D 'transforming ideas' rather than to the process selecting ideas that already had high potential or whose failures were not reported. Reporting the number and outcomes of filtered projects is necessary to support the fail-fast property and the effectiveness claim.","section":"Section 2 and Section 3"},{"comment":"The evaluation instrument was designed by the same authors who developed Lean R&D and was administered to participants in projects run by those authors; the questionnaire data are therefore partly an internal assessment, as the reader's report notes. Although the paper mentions the lack of a baseline, it does not state which conclusions remain robust under this constraint. The paper should add an explicit sentence in the discussion clarifying that the acceptance and suitability findings reflect the perceptions of participating stakeholders and do not constitute an independent effectiveness evaluation.","section":"Section 4.3 and Section 5"}],"minor_comments":[{"comment":"The phrase 'joined MVP vision' should be 'joint MVP vision,' and the sentence beginning 'It is noteworthy...' contains a missing breaking space between 'to' and 'sustain'.","section":"Section 4.1"},{"comment":"The statement 'their ROI ... exceeds the IAC investment by a factor of more than 1:20' is ambiguous: specify whether the ROI is annual or project-total, which of the two Petrobras projects it covers, and how the company computed it.","section":"Section 6"},{"comment":"The figures report Likert-scale results, but the captions do not state the exact question wording or the number of respondents per item; adding a small table with response counts and the full statement text would improve reproducibility and readability.","section":"Figures 2 through 7"},{"comment":"The description of the checkpoints would benefit from a short numbered list mapping each checkpoint to the specific decision and the criteria used, to make the go/no-go logic easier for practitioners to apply.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"This is a reasonable industrial experience report that could become publishable if the authors revise the central claim from 'effectiveness' to 'perceived suitability and acceptance,' and if they either provide auditability for the quantitative outcomes or clearly label them as unaudited company reports. The main risks for the editor are that the abstract's effectiveness claim may oversell the evidence and that the lack of a failure denominator undermines the fail-fast narrative. In the current form, the paper is better positioned as a lessons-learned report than as a validation of Lean R&D."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper is a solid industrial experience report, but the abstract overreaches. The new material is two recent Lean R&D applications—Petrobras AI projects and Americanas student teams—plus a 57-person perception survey and some company-reported outcomes. That's a legitimate extension of the authors' earlier work, and they make the questionnaire data openly available on Zenodo. Credit where due: the process description is clear, and the limitations section explicitly acknowledges the absence of a baseline or comparative framework.\n\nThe soft spots are real and concentrated in the central effectiveness claim. First, selection: Lean R&D uses checkpoints to kill weak MVPs, yet the paper gives no denominator—no count of Lean Inceptions run, MVPs rejected, or projects stopped at later gates. The reported successes therefore come from projects that already passed internal filters. That means \"fail fast\" can't be evaluated and \"transforms ideas into outcomes\" is overstated. Second, outcome measurement: the energy savings and 1:20 ROI are company-calculated with no protocol, sample size, or confidence interval, so they're not independently auditable from the manuscript. Third, self-evaluation: the authors designed the approach and administered the questionnaire to participants in their own projects. The survey supports stakeholder acceptance, but not comparative effectiveness.\n\nThese aren't trivial nits; they directly affect what the paper can claim. However, the paper is honest about its limits, and the acceptance data is what it is: a clear signal that stakeholders found the process suitable. That is a legitimate, if modest, contribution. The paper would be more accurate if the abstract said \"stakeholders accepted Lean R&D and teams delivered MVPs\" rather than \"effectiveness in transforming ideas into meaningful business outcomes.\"\n\nWho should read it: anyone designing industry-academia R&D collaboration processes or studying agile research frameworks. It's a useful data point, not a definitive one.\n\nRecommendation: it deserves a serious referee. An experienced reviewer could push for a proper denominator and outcome metrics, and the paper could be improved with a revised claim. I wouldn't desk reject it.","headline":"Useful industrial experience report, but the abstract overclaims effectiveness beyond what a self-selected acceptance survey can support.","tokens_in":8405,"tokens_out":2447,"would_cite":false,"duration_ms":25003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lean R&D is suitable for industry-academia innovation projects, on the evidence of 57 stakeholder responses and delivered MVPs.","keywords":["Lean R&D","industry-academia collaboration","agile research and development","minimal viable product","continuous experimentation","innovation projects","technology acceptance","software engineering"],"falsifier":"A matched comparison study in which new industry-academia projects are randomly assigned to Lean R&D or to a conventional plan-driven R&D contract, with independent evaluators who do not know which process was used, measuring time-to-MVP, business outcomes, and stakeholder acceptance, would settle whether the reported suitability is specific to Lean R&D or common to well-resourced collaborations.","tokens_in":7417,"feed_emoji":"📈","tokens_out":6413,"duration_ms":58927,"temperature":0.7,"pith_summary":"This industrial experience paper argues that Lean R&D, a structured agile process for research and development projects, is well suited to industry-academia collaboration because it delivers testable minimum viable products in fast cycles. The authors base this on two sets of collaborations, one in oil and gas and one in retail, where MVPs were delivered on schedule and, in the oil and gas case, produced patent applications and an estimated return above twenty times the project investment. Responses from 57 team members, managers, and sponsors largely endorsed the suitability of the process's phases and the approach overall, while flagging that the method takes effort to learn and could better support describing business hypotheses. If these self-reports reflect genuine effectiveness, the process offers a transferable option for university-industry innovation projects that need short-term business impact.","feed_headline":"Lean R&D wins over industry and academic teams in 57-person survey","feed_subtitle":"A survey of 57 participants, plus patents and a 20-to-1 return, backs the process for fast, impactful innovation.","key_machinery":"Lean R&D is the central object: a phased process that starts with a Lean Inception co-creation workshop to define an MVP and business hypotheses, followed by parallel technical feasibility and conception work, Scrum-based agile development, and a transition phase using continuous experimentation. Four steering-committee checkpoints sit between these phases and decide whether the project proceeds, fails fast, or is handed over, making the checkpoint decision the main waste-reduction mechanism. A dedicated research team supports the development team on technical feasibility, which is what lets the process address research uncertainties early.","core_discovery":"The paper's central claim is that Lean R&D succeeds as a framework for industry-academia R&D by structuring innovation around four checkpoints and five phases: Lean Inception, technical feasibility, conception, agile development, and transition with continuous experimentation. Each checkpoint lets a steering committee kill weak ideas early or approve progression, which the authors identify as the mechanism for avoiding wasted effort on unpromising projects. The evidence presented is a set of 57 questionnaire responses indicating agreement with the suitability of each phase and with statements of perceived usefulness, ease of use, and intention to adopt, plus delivered MVPs that in one case were loaded into a scaled agile release train and in another were integrated into an industrial inspection system. The paper positions Lean R&D as complementary to larger-scale delivery frameworks rather than a replacement for them.","pith_inferences":["If the acceptance pattern is real, the strongest untested consequence is that Lean R&D's value is independent of the two partner companies' particular conditions; testing the same process with smaller or less innovation-mature firms would reveal that.","Because the paper's evidence is self-report from projects run by the process's designers, a fair reading treats these results as feasibility evidence; a head-to-head comparison with a plan-driven R&D contract would be needed to claim superiority.","The student-team experience suggests an additional benefit the paper leaves implicit: Lean R&D can double as a talent-development pipeline, training students in business hypothesis testing and MVP delivery while producing industry value.","One testable extension would be to instrument the checkpoints themselves, measuring how often steering committees actually stop or redirect weak ideas, since the waste-avoidance claim rests on those decisions being binding."],"forward_implications":["Lean R&D can take an idea from conception to a deployed MVP in roughly six months per cycle, a pace that fits industry partners' short-term innovation windows.","The four checkpoints give sponsors an explicit mechanism to stop unpromising efforts before further investment, which the authors identify as a way to avoid waste.","The approach works with both experienced R&D professionals and less experienced students, provided mentoring and training are in place.","MVPs produced by Lean R&D can feed into larger delivery pipelines: in the reported oil and gas case an MVP was rolled into a scaled agile release train for multi-site deployment.","Measurable business outcomes are achievable from the process, including international patent applications, an inventor award, and a company-calculated return exceeding 20 times the R&D investment."],"supporting_citations":[{"why":"Supplies the Lean Inception ideation workshop that starts the process.","marker":"[1]"},{"why":"Supplies continuous software engineering principles behind the fast-paced delivery design.","marker":"[2]"},{"why":"Previous full description of Lean R&D, the approach whose suitability is assessed here.","marker":"[6]"},{"why":"Source of the lean production concepts used to design waste-minimising checkpoints.","marker":"[9]"},{"why":"Source of the minimum-viable-product and business-hypothesis testing concepts.","marker":"[10]"},{"why":"Earlier case study that gave initial feasibility indications for Lean R&D.","marker":"[12]"},{"why":"Framework of industry-academia collaboration success factors used to contextualise the results.","marker":"[14]"},{"why":"Patent application for the refinery torch AI solution, an outcome of the process.","marker":"[7]"},{"why":"Patent application for the inspection-report AI solution, an outcome of the process.","marker":"[8]"}],"fun_headline_variants":["Lean R&D framework wins over 57 industry-academia participants","Survey of 57 endorses Lean R&D for fast, impactful innovation","Lean R&D: five phases, four checkpoints, 57 backers","Industry-academia teams rate Lean R&D highly in 57-person study","Lean R&D earns endorsement from 57 in industry-academia projects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 57 questionnaire responses and the companies' reported business outcomes reflect genuine effectiveness, even though the respondents took part in projects run and evaluated by the same team that designed the process, and no comparison baseline was used.","fun_headline_variants_meta":{"raw":{"variants":["Lean R&D framework wins over 57 industry-academia participants","Survey of 57 endorses Lean R&D for fast, impactful innovation","Lean R&D: five phases, four checkpoints, 57 backers","Industry-academia teams rate Lean R&D highly in 57-person study","Lean R&D earns endorsement from 57 in industry-academia projects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1720,"prompt_tokens":829,"completion_tokens":891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":797}},"tokens_in":445,"tokens_out":891,"duration_ms":8420,"temperature":1.0,"reasoning_tokens":797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:52:31.854053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched comparison study in which new industry-academia projects are randomly assigned to Lean R&D or to a conventional plan-driven R&D contract, with independent evaluators who do not know which process was used, measuring time-to-MVP, business outcomes, and stakeholder acceptance, would settle whether the reported suitability is specific to Lean R&D or common to well-resourced collaborations.","supporting_citations":[{"cited_title":"Editora Caroli (2018)","cited_arxiv_id":null,"evidence_quote":"Supplies the Lean Inception ideation workshop that starts the process."},{"cited_title":"Journal of Systems and Software123, 176–189 (2017)","cited_arxiv_id":null,"evidence_quote":"Supplies continuous software engineering principles behind the fast-paced delivery design."},{"cited_title":"In: Product-Focused Software Process Improvement: 21st International Conference, Experiences Applying Lean R&D 15 PROFES 2020, Turin, Italy, November 25–27, 2020, Proceedings 21","cited_arxiv_id":null,"evidence_quote":"Previous full description of Lean R&D, the approach whose suitability is assessed here."},{"cited_title":"CRc Press (2011)","cited_arxiv_id":null,"evidence_quote":"Source of the lean production concepts used to design waste-minimising checkpoints."},{"cited_title":"Crown Currency (2011)","cited_arxiv_id":null,"evidence_quote":"Source of the minimum-viable-product and business-hypothesis testing concepts."},{"cited_title":"In: ICEIS (2)","cited_arxiv_id":null,"evidence_quote":"Earlier case study that gave initial feasibility indications for Lean R&D."},{"cited_title":"IEEE software29(2), 67–73 (2011)","cited_arxiv_id":null,"evidence_quote":"Framework of industry-academia collaboration success factors used to contextualise the results."},{"cited_title":"17/890,539","cited_arxiv_id":null,"evidence_quote":"Patent application for the refinery torch AI solution, an outcome of the process."},{"cited_title":"18/061,068","cited_arxiv_id":null,"evidence_quote":"Patent application for the inspection-report AI solution, an outcome of the process."}],"review_version":1}