Pith. sign in

REVIEW 4 major objections 4 minor 14 references

Experiences Applying Lean R&D in Industry-Academia Collaboration Projects

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Lean R&D is suitable for industry-academia innovation projects, on the evidence of 57 stakeholder responses and delivered MVPs.

desk verdict Useful industrial experience report, but the abstract overclaims effectiveness beyond what a self-selected acceptance survey can support. read the letter →

arxiv 2501.11774 v1 pith:FFH4Y2K6 submitted 2025-01-20 cs.SE

Marcos Kalinowski , Lucas Romao , Ariane Rodrigues , Clarissa Barbosa , Hugo Villamizar , Simone D.J. Barbosa , Helio Lopes This is my paper · ORCID
classification cs.SE
keywords LeanR&Dindustry-academiacollaborationagileresearchanddevelopmentminimalviableproductcontinuousexperimentationinnovationprojectstechnologyacceptancesoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This industrial experience paper argues that Lean R&D, a structured agile process for research and development projects, is well suited to industry-academia collaboration because it delivers testable minimum viable products in fast cycles. The authors base this on two sets of collaborations, one in oil and gas and one in retail, where MVPs were delivered on schedule and, in the oil and gas case, produced patent applications and an estimated return above twenty times the project investment. Responses from 57 team members, managers, and sponsors largely endorsed the suitability of the process's phases and the approach overall, while flagging that the method takes effort to learn and could better support describing business hypotheses. If these self-reports reflect genuine effectiveness, the process offers a transferable option for university-industry innovation projects that need short-term business impact.

What carries the argument

Lean R&D is the central object: a phased process that starts with a Lean Inception co-creation workshop to define an MVP and business hypotheses, followed by parallel technical feasibility and conception work, Scrum-based agile development, and a transition phase using continuous experimentation. Four steering-committee checkpoints sit between these phases and decide whether the project proceeds, fails fast, or is handed over, making the checkpoint decision the main waste-reduction mechanism. A dedicated research team supports the development team on technical feasibility, which is what lets the process address research uncertainties early.

What would settle it

A matched comparison study in which new industry-academia projects are randomly assigned to Lean R&D or to a conventional plan-driven R&D contract, with independent evaluators who do not know which process was used, measuring time-to-MVP, business outcomes, and stakeholder acceptance, would settle whether the reported suitability is specific to Lean R&D or common to well-resourced collaborations.

Watch

Extended reading notes

Core claim

The paper's central claim is that Lean R&D succeeds as a framework for industry-academia R&D by structuring innovation around four checkpoints and five phases: Lean Inception, technical feasibility, conception, agile development, and transition with continuous experimentation. Each checkpoint lets a steering committee kill weak ideas early or approve progression, which the authors identify as the mechanism for avoiding wasted effort on unpromising projects. The evidence presented is a set of 57 questionnaire responses indicating agreement with the suitability of each phase and with statements of perceived usefulness, ease of use, and intention to adopt, plus delivered MVPs that in one case were loaded into a scaled agile release train and in another were integrated into an industrial inspection system. The paper positions Lean R&D as complementary to larger-scale delivery frameworks rather than a replacement for them.

Load-bearing premise

The load-bearing premise is that the 57 questionnaire responses and the companies' reported business outcomes reflect genuine effectiveness, even though the respondents took part in projects run and evaluated by the same team that designed the process, and no comparison baseline was used.

Editorial extensions

If this is right

  • Lean R&D can take an idea from conception to a deployed MVP in roughly six months per cycle, a pace that fits industry partners' short-term innovation windows.
  • The four checkpoints give sponsors an explicit mechanism to stop unpromising efforts before further investment, which the authors identify as a way to avoid waste.
  • The approach works with both experienced R&D professionals and less experienced students, provided mentoring and training are in place.
  • MVPs produced by Lean R&D can feed into larger delivery pipelines: in the reported oil and gas case an MVP was rolled into a scaled agile release train for multi-site deployment.
  • Measurable business outcomes are achievable from the process, including international patent applications, an inventor award, and a company-calculated return exceeding 20 times the R&D investment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the acceptance pattern is real, the strongest untested consequence is that Lean R&D's value is independent of the two partner companies' particular conditions; testing the same process with smaller or less innovation-mature firms would reveal that.
  • Because the paper's evidence is self-report from projects run by the process's designers, a fair reading treats these results as feasibility evidence; a head-to-head comparison with a plan-driven R&D contract would be needed to claim superiority.
  • The student-team experience suggests an additional benefit the paper leaves implicit: Lean R&D can double as a talent-development pipeline, training students in business hypothesis testing and MVP delivery while producing industry value.
  • One testable extension would be to instrument the checkpoints themselves, measuring how often steering committees actually stop or redirect weak ideas, since the waste-avoidance claim rests on those decisions being binding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This industrial experience paper reports on the application of Lean R&D, an agile research-and-development approach developed at PUC-Rio, to industry-academia collaboration projects with Petrobras (oil and gas) and Americanas (retail). The paper describes the Lean R&D phases and checkpoints, reports project outcomes including patent applications, energy savings, and an ROI estimate, and analyzes 57 questionnaire responses from team members, managers, and sponsors assessing phase suitability and overall acceptance, largely using Technology Acceptance Model statements. The abstract claims that the findings highlight Lean R&D's effectiveness in transforming ideas into meaningful business outcomes, while the discussion acknowledges the absence of a formal baseline or direct comparison to alternative frameworks.

Significance. If the claims were fully supported, the paper would provide useful practitioner guidance on structuring fast-moving industry-academia innovation projects. The paper's strengths are its concrete description of a real process, two different application contexts (experienced R&D staff at Petrobras and student teams at Americanas), an openly linked questionnaire data set, and explicit attention to stakeholder feedback and improvement opportunities. The reported stakeholder acceptance and the successful delivery of MVP-based projects are valuable experience data for the software engineering community. However, the evidence as presented supports claims about perceived suitability and acceptance, not the stronger effectiveness claim made in the abstract and conclusion.

major comments (4)
  1. [Abstract and Section 5] The central claim that Lean R&D is 'effective in transforming ideas into meaningful business outcomes' is stronger than the evidence presented. Section 5 explicitly acknowledges the absence of a formal baseline or direct comparison to alternative frameworks, and the outcome evidence is a combination of company-reported figures and self-reported perceptions from participants in the authors' own projects. The abstract and conclusion should be revised to state the narrower claim that stakeholders perceived Lean R&D as suitable and accepted it, and that the approach was associated with the reported project deliveries, rather than asserting demonstrated effectiveness.
  2. [Section 4.1] The quantitative business outcomes—energy savings comparable to a city of 20,000 inhabitants per refinery, faster equipment-inspection report completion, and an ROI exceeding 1:20—are presented without any measurement protocol, sample size, confidence interval, or calculation method. The ROI in Section 6 is stated to be 'calculated by the company,' and no public report or data artifact supporting these numbers is linked. Since these figures are the main objective support for the 'business outcomes' claim, the paper must either provide the measurement details or a reference to an auditable source, or clearly label each figure as an unaudited company claim.
  3. [Section 2 and Section 3] The reported successes are conditional on passing Lean R&D's internal checkpoints, which can reject MVP outlines, technical feasibility results, MVP readiness, and business-hypothesis validation. The paper provides no denominator: there is no count of Lean Inceptions conducted, MVP outlines rejected, projects stopped at later checkpoints, or MVPs that failed to transition. Without these filtered cases, the observed successes cannot be attributed to Lean R&D 'transforming ideas' rather than to the process selecting ideas that already had high potential or whose failures were not reported. Reporting the number and outcomes of filtered projects is necessary to support the fail-fast property and the effectiveness claim.
  4. [Section 4.3 and Section 5] The evaluation instrument was designed by the same authors who developed Lean R&D and was administered to participants in projects run by those authors; the questionnaire data are therefore partly an internal assessment, as the reader's report notes. Although the paper mentions the lack of a baseline, it does not state which conclusions remain robust under this constraint. The paper should add an explicit sentence in the discussion clarifying that the acceptance and suitability findings reflect the perceptions of participating stakeholders and do not constitute an independent effectiveness evaluation.
minor comments (4)
  1. [Section 4.1] The phrase 'joined MVP vision' should be 'joint MVP vision,' and the sentence beginning 'It is noteworthy...' contains a missing breaking space between 'to' and 'sustain'.
  2. [Section 6] The statement 'their ROI ... exceeds the IAC investment by a factor of more than 1:20' is ambiguous: specify whether the ROI is annual or project-total, which of the two Petrobras projects it covers, and how the company computed it.
  3. [Figures 2 through 7] The figures report Likert-scale results, but the captions do not state the exact question wording or the number of respondents per item; adding a small table with response counts and the full statement text would improve reproducibility and readability.
  4. [Section 2] The description of the checkpoints would benefit from a short numbered list mapping each checkpoint to the specific decision and the criteria used, to make the go/no-go logic easier for practitioners to apply.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the effectiveness claim rests on independently reported project outcomes and a standard technology-acceptance survey, not on a definitional or fitted reduction.

full rationale

The paper does not derive its central claim from its own assumptions by construction. Lean R&D is introduced as a designed approach referencing external sources for Lean, Lean Startup, and continuous software engineering, with self-citations only supplying background detail on the approach. The reported evidence consists of two kinds: project outcomes (patents, a company-calculated ROI, a company LinkedIn statement) and 57 questionnaire responses using the Technology Acceptance Model. Neither outcome is defined as the survey result, and no parameter is fitted to a subset of data and then renamed as a prediction. The first checkpoint does filter projects before they proceed, so successes are conditioned on passing internal go/no-go gates; however, the paper does not define effectiveness as passing those gates, and the patent, ROI, and company communications are external to the checkpoint criterion. This is a selection-threat to external validity, not a circularity of the kind where an equation or definition reduces to its own input. The authors explicitly acknowledge the absence of a formal baseline or comparison to alternative frameworks in Section 5, which weakens the strength of the causal claim but does not make the reasoning circular. Self-citations to prior Lean R&D papers are used for describing the method, not as the sole justification for the new empirical conclusion, and external references support the constituent ideas. The evaluation is not independent in the sense that the authors designed both the approach and the questionnaire, which is a legitimate bias concern; under the specified circularity taxonomy, however, that concern is an independence and reporting limitation rather than a demonstrated circular step. No specific equation, fitted parameter, or uniqueness theorem is invoked in a way that makes the conclusion equivalent to the input.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical parameters are fitted. The central evidence is qualitative. The assumptions above concern trust in self-report, attribution of outcomes, and generalizability. No new physical or conceptual entities are proposed; Lean R&D is a previously introduced framework.

assumptions (3)
  • domain assumption Self-reported Likert responses from project participants accurately represent their true perceptions of Lean R&D's suitability and acceptance.
    The survey is the primary evidence for acceptance; no observational or behavioral validation is provided. This enters in Section 4, where all suitability and acceptance conclusions are drawn from these responses.
  • domain assumption Company-reported outcomes (patents, ROI greater than 20:1, energy savings) are accurate and attributable to Lean R&D.
    These figures come from Petrobras and are not independently audited; causal attribution to the process is assumed in Sections 4 and 6.
  • domain assumption The two case companies are representative enough to support general claims about Lean R&D in industry-academia collaboration.
    Only two companies, both large Brazilian firms, are studied; the paper discusses broader suitability without a sampling frame in Sections 3 and 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Experiences Applying Lean R&D in Industry-Academia Collaboration Projects." pith.science (2026). https://pith.science/paper/FFH4Y2K6

@misc{pith2026250111774,
  author       = {Pith},
  title        = {Pith review of: Experiences Applying Lean R&D in Industry-Academia Collaboration Projects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFH4Y2K6}},
  note         = {Machine review of arXiv:2501.11774}
}
read the original abstract

Lean R&D has been used at PUC-Rio to foster industry-academia collaboration in innovation projects across multiple sectors. This industrial experience paper describes recent experiences and evaluation results from applying Lean R&D in partnership with Petrobras in the oil and gas sector and Americanas in retail. The findings highlight Lean R&D's effectiveness in transforming ideas into meaningful business outcomes. Based on responses from 57 participants - including team members, managers, and sponsors - the assessment indicates that stakeholders find the structured phases of Lean R&D well-suited to innovation projects and endorse the approach. Although acknowledging that successful collaboration relies on various factors, this industrial experience positions Lean R&D as a promising framework for industry-academia projects focused on achieving rapid, impactful results for industry partners.

Figures

Figures reproduced from arXiv: 2501.11774 by the authors.

Figure 1
Figure 1. Overview of the Lean R&D approach. The timeline is illustrative and reflects typical timeframes used within the ExACTa PUC-Rio laboratory [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Suitability of Lean R&D phases according to the ExACTa R&D team working with Petrobras. that the main purpose of Lean Inception is focused on outlining the MVP to deliver business value. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Suitability of Lean R&D phases according to the ExACTa FIT team working with Americanas. Finally, two participants disagreed on the importance of the checkpoints, and one partially disagreed. However, based on their justifications, they did not understand the purpose of these checkpoints. One of them mentioned that the checkpoints are insufficient to communicate all project-related problems. The other one mentioned … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Technology acceptance by the ExACTa R&D team members of the projects with Petrobras. Analyzing [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Technology acceptance by the business and technical managers and sponsors of Petrobras. was of the sponsor and on the ease of use. In fact, he was only involved in the checkpoints. The results for the ExACTa FIT team members can be seen in [PITH_FULL_IMAGE:figures/ful…
Figure 7
Figure 7. Figure 7: allows to observe that the managers and sponsors at Americanas also largely agreed with the TAM statements. One of the two managers more conservatively reported being neutral on the ease of use, and one of the two sponsors reported being neutral regarding improved prod…
Figure 6
Figure 6. Figure 6: Technology acceptance by the ExACTa FIT team members of the projects with Americanas. short-term, impactful results for industry. Stakeholders found Lean R&D’s struc￾ture conducive to rapid iteration and tangible business outcomes, with many af￾firming the approach as …
Figure 7
Figure 7. Figure 7: Technology acceptance by the business and technical managers and sponsors of Americanas. project oversight and facilitated communication between academic and industry teams. Additionally, strong support from company management and alignment with business strategy were …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

  1. [1]

    Editora Caroli (2018)

    Caroli, P.: Lean Inception: how to align people and build the right product. Editora Caroli (2018)

    work page 2018

  2. [2]

    Journal of Systems and Software123, 176–189 (2017)

    Fitzgerald,B.,Stol,K.J.:Continuoussoftwareengineering:Aroadmapandagenda. Journal of Systems and Software123, 176–189 (2017)

    work page 2017

  3. [3]

    IEEE software23(6), 88–95 (2006)

    Gorschek, T., Garre, P., Larsson, S., Wohlin, C.: A model for technology transfer in practice. IEEE software23(6), 88–95 (2006)

    work page 2006

  4. [4]

    In: Software Quality: Future Perspectives on Software Engineering Quality: 13th International Conference, SWQD 2021, Vienna, Austria, January 19–21, 2021, Pro- ceedings 13

    Gorschek, T., Mendez, D.: Solving problems or enabling problem-solving? from pu- rity in empirical software engineering to effective co-production (invited keynote). In: Software Quality: Future Perspectives on Software Engineering Quality: 13th International Conference, SWQD 2021, Vienna, Austria, January 19–21, 2021, Pro- ceedings 13. pp. 109–116. Sprin...

    work page 2021

  5. [5]

    In: 2020 46th Eu- romicro Conference on Software Engineering and Advanced Applications (SEAA)

    Kalinowski, M., Batista, S.T., Lopes, H., Barbosa, S., Poggi, M., Silva, T., Vil- lamizar,H.,Chueke,J.,Teixeira,B.,Pereira,J.A.,etal.:Towardsleanr&d:anagile research and development approach for digital transformation. In: 2020 46th Eu- romicro Conference on Software Engineering and Advanced Applications (SEAA). pp. 132–136. IEEE (2020)

    work page 2020

  6. [6]

    In: Product-Focused Software Process Improvement: 21st International Conference, Experiences Applying Lean R&D 15 PROFES 2020, Turin, Italy, November 25–27, 2020, Proceedings 21

    Kalinowski, M., Lopes, H., Teixeira, A.F., da Silva Cardoso, G., Kuramoto, A., Itagyba, B., Batista, S.T., Pereira, J.A., Silva, T., Warrak, J.A., et al.: Lean r&d: An agile research and development approach for digital transformation. In: Product-Focused Software Process Improvement: 21st International Conference, Experiences Applying Lean R&D 15 PROFES ...

    work page 2020

  7. [7]

    17/890,539

    Kuramoto, A.S.R., De Oliveira, A.D.C.M., Torres, P.H.L., Fernandes, W.P.D., Lopes, H.C.V., Schwaner, W.K., Paravidino, B.I., Pereira, C.S., Happ, P.N., Co- mandulli, S., et al.: Method of assessment of the quality of the burn of the gases in the flare and adjustment to the vapor flow rate in a continuous and constant way (2023), uS Patent App. 17/890,539

    work page 2023

  8. [8]

    18/061,068

    Kuramoto, A.S.R., Marques, S.A., Sant, H.M., Campos, A.B., Dos Reis, R.A.G., Etter, J.A.N., Moraes, P.R., Marques, A.T.A., Comandulli, S., Kalinowski, M., et al.: Method for detecting inaccuracies and gaps and for suggesting deterioration mechanisms and actions in inspection reports (2023), uS Patent App. 18/061,068

    work page 2023

Show all 14 references
  1. [9]

    CRc Press (2011)

    Monden, Y.: Toyota production system: an integrated approach to just-in-time. CRc Press (2011)

    2011

  2. [10]

    Crown Currency (2011)

    Ries, E.: The lean startup: How today’s entrepreneurs use continuous innovation to create radically successful businesses. Crown Currency (2011)

    2011

  3. [11]

    https:// scaledagileframework.com/ (2023), accessed: 2025-01-20

    Scaled Agile, Inc.: Scaled Agile Framework (SAFe). https:// scaledagileframework.com/ (2023), accessed: 2025-01-20

    2023

  4. [12]

    In: ICEIS (2)

    Teixeira, B.R., Ferreira, B., de Damasceno, A.L.B., Barbosa, S.D., Novello, C., Villamizar, H., Kalinowski, M., Silva, T., Chueke, J., Lopes, H., et al.: Lessons learned from a lean r&d project. In: ICEIS (2). pp. 345–352 (2021)

    2021

  5. [13]

    https://www.timeshighereducation.com/world-university-rankings/2023/ latin-america-university-rankings (2023), accessed: August 2023

    Times Higher Education: Latin America University Rankings 2023. https://www.timeshighereducation.com/world-university-rankings/2023/ latin-america-university-rankings (2023), accessed: August 2023

    2023

  6. [14]

    IEEE software29(2), 67–73 (2011)

    Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., Grahn, H., Henningsson, K., Kagstrom, S., Low, G., et al.: The success factors powering industry-academia collaboration. IEEE software29(2), 67–73 (2011)

    2011

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.