{"id":"5a5c1542-df16-43e2-894b-40c267e1376e","arxiv_id":"2411.09973","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A semi-structured literature review synthesizing six trustworthy AI requirements and their evaluation methods, plus cross-cutting research challenges.","lead":"This paper reviews how to define, build, and measure six requirements of trustworthy AI, from fairness to accountability. It consolidates a scattered field into one reference and lists research challenges that could shape future AI governance and evaluation.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first unified review' claim (Sec. 1) rests on a Scopus search that screens only the top 100 abstracts per requirement; a broader check could reveal a prior survey covering all six requirements.","rationale":"The reader identified the semi-structured literature selection in Sec. 2.3 as the weakest assumption, and I agree that the selection could bias the synthesis. The most load-bearing consequence is the paper's explicit claim to be the first unified treatment of all six requirements. If the search misses an existing survey that already covers all six with evaluation aspects, that central claim is false, even if the present review is still a useful synthesis. The concern is therefore not about internal consistency but about the adequacy of the evidence for a 'first' claim. The concrete test would settle this by checking whether such a prior survey exists. If none is found, the paper's claim stands and the reader's CONDITIONAL verdict is appropriate; if one is found, the novelty claim should be revised. Since the reader already conditioned on the literature-selection risk, the verdict should remain unchanged at CONDITIONAL, with the condition being that the authors verify the novelty claim via a more exhaustive search.","tokens_in":33495,"tokens_out":5511,"duration_ms":61121,"concrete_test":"Run a targeted search in Scopus and Google Scholar using combinations of the six requirement terms (e.g., 'human agency', 'fairness', 'transparency', 'robustness', 'privacy', 'accountability') with 'trustworthy AI' and 'survey' or 'review', and screen all retrieved abstracts (not just top 100). Also perform backward citation chasing from the six requirement definitions and from the surveys cited in Sec. 2.2. If any publication dated before November 2024 covers all six requirements with both implementation and evaluation discussion, the 'first' claim is undercut.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim (Sec. 1) is that it is the first to investigate all six requirements of trustworthy AI in a unified way, covering implementation and evaluation across the lifecycle. This claim depends on the completeness of the literature search in Sec. 2.3, which screens only the 100 most relevant Scopus abstracts per requirement and excludes over-specialized articles. Such a cutoff can systematically miss prior surveys that span multiple requirements but are not top-ranked in any single-requirement query, or that appear in venues outside the searched set. The subsequent snowballing is citation-biased and unlikely to recover works that are poorly cited. If a prior survey covering all six requirements with evaluation aspects exists, the novelty claim collapses. The reader's concern about representativeness is valid, but the load-bearing risk is specifically that the selection procedure is too weak to support a 'first' claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a literature review that synthesizes existing conceptualizations of trustworthy AI along six requirements: human agency and oversight, fairness and non-discrimination, transparency and explainability, robustness and accuracy, privacy and security, and accountability. For each requirement, the authors provide a definition, describe methods to establish and evaluate the requirement, and discuss requirement-specific research challenges. The review also identifies five overarching challenges across the requirements: interdisciplinary research, conceptual clarity, context-dependency, dynamics in evolving systems, and real-world investigations. The paper is based on a semi-structured literature search of Scopus and Google Scholar, with 183 papers included after screening the top 100 abstracts per requirement plus snowballing. The authors claim that this is the first work to investigate all six requirements in a unified way, with emphasis on implementation and evaluation across the whole AI lifecycle.","tokens_in":33636,"tokens_out":6504,"duration_ms":62352,"significance":"If its coverage is accepted, this review could serve as a useful reference for researchers and practitioners, bringing together technical, human-centered, and legal perspectives in one place. Its explicit mapping of all six requirements to the AI lifecycle, and its parallel structure of definition, establishment, evaluation, and open challenges, make it accessible to a broad audience. The paper also provides a transparent (though incomplete) description of its review methodology and acknowledges the inherently interdisciplinary nature of trustworthy AI. The authors deserve credit for including evaluation aspects, which several prior surveys omit, and for identifying recurring tensions such as trade-offs between fairness, accuracy, privacy, and explainability. The main value of the paper is as a synthesis; it does not introduce new methods or empirical results, and its central novelty claim needs stronger support.","major_comments":[{"comment":"The claim that 'our paper is the first to investigate all six requirements of trustworthy AI in a unified way' is not supported by the literature selection protocol described in Section 2.3. Screening only the 100 most relevant Scopus abstracts per requirement, followed by snowballing, can systematically miss multi-requirement surveys that are not top-ranked for any single requirement and are poorly cited. Since this novelty claim is a core part of the stated contribution, the authors should either conduct a targeted prior-art search for existing multi-requirement surveys that cover all six requirements and their evaluation, or soften the claim and explicitly note that the selection procedure was not designed to prove the absence of prior work.","section":"Section 1 (last paragraph) and Section 2.3"},{"comment":"The methodology is not reported at a level that permits reproducibility or an assessment of completeness. The authors do not provide the Scopus query strings, the date range of the search, the number of records retrieved and screened at each stage, or operationalized inclusion and exclusion criteria; the exclusion of articles with 'over-specialization' and 'limited contributions' is subjective. Without these details, the representativeness of the 183 selected papers cannot be judged, and the review is not reproducible. Please add a detailed protocol or explicitly label the work as a non-systematic scoping review with the corresponding limitations clearly stated.","section":"Section 2.3"},{"comment":"The evaluation methods for human agency and oversight are presented as a hierarchy of dependencies (AI literacy, system understandability, human oversight, human agency) without clear attribution to the reviewed literature. If this hierarchy is the authors' own synthesis, it should be explicitly identified as such, because the paper's contribution is a review rather than a new evaluation framework; if it is drawn from the literature, specific sources and the basis for the particular ordering should be provided.","section":"Section 3.1.3"}],"minor_comments":[{"comment":"The entry 'Accountaibility' contains a spelling error and should be corrected to 'Accountability'.","section":"Table 1"},{"comment":"The first bullet point attributes the human engagement patterns to 'Anders et al. (2022)', but the reference list contains 'Anderson and Fort (2022)' as the source for these patterns, and the same bullet later cites 'Anderson and Fort (2022)'; the in-text citation should be corrected accordingly.","section":"Section 3.1.2"},{"comment":"The citation 'Verma and Julia (2018)' should be 'Verma and Rubin (2018)', and the corresponding reference list entry should list Rubin as the second author.","section":"Section 3.2.3"},{"comment":"The phrase 'with respect to the AI-lifecycle (see Section 3)' should refer to Section 2, where the AI lifecycle is actually described.","section":"Section 3.3.2"},{"comment":"The sentence 'Tagiou et al. (2019) suggest a “a tool-supported framework...' contains a duplicated article and should be rephrased.","section":"Section 3.6.3"},{"comment":"The sentence 'Thus, it incorporated fairness in the training algorithms themselves' uses the past tense inconsistently with the surrounding present-tense description; it should read 'it incorporates fairness' or be rephrased.","section":"Section 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad scoping review that could be a useful reference, but the 'first unified review' claim is a risk point. I suggest that the editor also ask a reviewer with expertise in the AI ethics and fairness survey literature to check for any existing multi-requirement surveys that would undermine the novelty claim. Additionally, the paper's methodology section would benefit from a more detailed search protocol or a more modest framing, as the current description is too imprecise to establish comprehensiveness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the Kowald et al. review. It's a solid, well-organized survey of six trustworthy AI requirements — human agency, fairness, transparency, robustness, privacy, accountability — each with an agreed definition, ways to build it in, evaluation approaches, and open problems. The consistent structure makes it genuinely useful as a reference, especially for people coming into the field or for practitioners who need a map. The cross-cutting challenges at the end (interdisciplinarity, conceptual clarity, context-dependency, evolving systems, real-world evaluation) are a real contribution; they frame where the field is stuck.\n\nWhat's new is organizational, not empirical. There are no new results, which is fine for a review. The paper's claim to be 'the first to investigate all six requirements in a unified way' is the weak spot. The methodology — top 100 Scopus abstracts per requirement, plus snowballing — is not robust enough to support a 'first' claim. A broader search could well turn up prior surveys covering the same six, or very close to it. I don't think this sinks the paper, because the value of the synthesis doesn't actually depend on being first. But the authors should soften that claim or do a more systematic search. The section on human agency is the thinnest, but that's because that literature is less mature; the authors say as much.\n\nThe self-citations are present but not load-bearing; they're used as examples of state-of-the-art, which is fine. The paper is transparent about its selection limits, which I appreciate.\n\nVerdict: this deserves a serious peer review. It's not ground-breaking, but it's a competent, honest, and useful consolidation. I'd send it out with a request to moderate the novelty claim and to add a sentence acknowledging that a fuller systematic review could change the 'first' claim. For a reading group, it's a decent overview but not a research paper I'd assign for deep discussion.","headline":"A solid, useful survey of six trustworthy-AI requirements; the 'first unified review' claim is overstated but the synthesis stands on its own.","tokens_in":34213,"tokens_out":2691,"would_cite":true,"duration_ms":29242,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper maps all six requirements of trustworthy AI into one framework, pairing each with ways to build and test it.","keywords":["trustworthy AI","human agency and oversight","fairness and non-discrimination","transparency and explainability","robustness and accuracy","privacy and security","accountability","AI lifecycle evaluation"],"falsifier":"An independent systematic search that draws the full relevant literature rather than only the top-ranked abstracts per requirement and finds a substantial trustworthy-AI requirement or evaluation approach missing from the paper's synthesis, such as a maturing standard for safety or sustainability with established metrics, would show that the claimed unified coverage is incomplete.","tokens_in":33323,"feed_emoji":"🤖","tokens_out":5347,"duration_ms":53878,"temperature":0.7,"pith_summary":"This review paper sets out to organize the sprawling discussion of trustworthy AI around six requirements that recur across policy, ethics, and technical work: human agency and oversight, fairness and non-discrimination, transparency and explainability, robustness and accuracy, privacy and security, and accountability. For each requirement it gives a working definition, surveys methods for building systems that satisfy it, and reviews how to evaluate whether the requirement is met. Its main claim is that no previous review treats all six in a unified way across the full AI lifecycle, with both implementation and evaluation in scope. A sympathetic reader should care because the paper turns a fragmented field into a structured reference and a gap list for future research, and because the contrast it draws is sharp: technical requirements like robustness and accuracy have mature metrics, while human-centred requirements such as agency and accountability still lack settled evaluation schemes.","feed_headline":"One map links all six requirements of trustworthy AI","feed_subtitle":"The review pairs every requirement with evaluation methods and pinpoints where metrics are still missing.","key_machinery":"The organizing device is the six-requirement matrix over the AI lifecycle. Each requirement is treated through the same four-step template: definition, methods to establish it, evaluation methods, and open research challenges. The lifecycle framing from design through development to deployment is what lets the paper argue that trustworthiness can be damaged or repaired in any phase, and the requirement-by-requirement template is what makes the synthesis systematic rather than anecdotal.","core_discovery":"The paper's core claim is that trustworthiness of AI is not a single property but a set of six distinct requirements, each with its own definition, methods, evaluation toolkit, and open problems, and that treating them together is necessary because they interact and trade off. It organizes the field around four ethical principles from European guidelines—respect for human autonomy, fairness, explicability, and prevention of harm—and maps the six requirements onto the AI lifecycle (design, development, deployment). Its synthesis shows that evaluation maturity is uneven: accuracy and robustness can lean on established statistical metrics, transparency and explainability have a growing but contested set of evaluation properties, while fairness, human agency, and accountability are context-dependent and lack standard, legally robust measurement. The paper concludes by condensing the field's open problems into five overarching challenges: interdisciplinary research, conceptual clarity, context-dependency, dynamics in evolving systems, and real-world investigation.","pith_inferences":["The requirement-by-requirement template could be turned into an evaluation checklist or benchmark suite that scores a system on all six requirements, making the paper's qualitative comparison operational.","The identified interdependence between requirements suggests a multi-objective view of trustworthy AI: future work could treat fairness, robustness, privacy, and explainability as jointly optimised objectives with explicit trade-off surfaces.","Regulatory certification efforts could use the paper's gap list as a roadmap, prioritising the requirements where no standard measurement exists."],"forward_implications":["If the framework is right, a system cannot be certified as trustworthy by checking a single property; each of the six requirements must be considered at design, development, and deployment.","Evaluation practice should mix established quantitative metrics (accuracy, robustness, privacy attacks) with qualitative, context-specific methods for fairness, agency, and accountability.","Trade-offs between requirements, such as fairness versus accuracy or privacy versus explainability, become a design decision that must be documented and reviewed rather than an afterthought.","Composite AI systems and models that learn during deployment need continuous monitoring, because trustworthiness of parts does not guarantee trustworthiness of the whole.","Generative AI and large language models require new or transformed evaluation methods, since existing metrics were designed for simpler settings."],"supporting_citations":[{"why":"Supplies the four ethics principles and the definitional basis from which the six requirements are derived.","marker":"High-Level Expert Group on AI (2019)"},{"why":"Situates the EU ethics approach to trustworthy AI that underpins the paper's framing.","marker":"Smuha (2019)"},{"why":"Earlier review of requirements that the paper builds on and distinguishes itself from.","marker":"Kaur et al. (2021)"},{"why":"Represents the technical, reliability-oriented conception of trustworthy AI that the paper contrasts with human-centric requirements.","marker":"Floridi (2021)"},{"why":"Recent survey moving from principles to practices; serves as a baseline for the paper's claimed unified coverage.","marker":"Li et al. (2023)"},{"why":"Provides the AI lifecycle model that structures the paper's whole-lifecycle analysis.","marker":"Haakman et al. (2021)"},{"why":"Supplies the semi-structured literature review methodology used to select and synthesize the 183 sources.","marker":"Snyder (2019)"},{"why":"Defines the three-category evaluation scheme (application-grounded, human-grounded, functionally-grounded) used for explainability.","marker":"Doshi-Velez and Kim (2017)"},{"why":"Gives the actor-forum-account-consequences definition of accountability on which the accountability section is built.","marker":"Bovens (2007)"},{"why":"The ALTAI self-assessment checklist that the paper positions itself as complementing with evaluation-focused synthesis.","marker":"Ala-Pietilä et al. (2020)"}],"fun_headline_variants":["AI trust: six requirements, five big gaps","Trustworthy AI: mapping six requirements","Where AI trust lacks benchmarks: a review","Six requirements, uneven metrics: AI trust review","AI trust: from principles to evaluation gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's coverage claim rests on the assumption that screening the 100 most relevant abstracts per requirement and excluding over-specialised papers yields a representative picture of the field's definitions, evaluation methods, and challenges.","fun_headline_variants_meta":{"raw":{"variants":["AI trust: six requirements, five big gaps","Trustworthy AI: mapping six requirements","Where AI trust lacks benchmarks: a review","Six requirements, uneven metrics: AI trust review","AI trust: from principles to evaluation gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1200,"prompt_tokens":942,"completion_tokens":258,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":191}},"tokens_in":558,"tokens_out":258,"duration_ms":3176,"temperature":1.0,"reasoning_tokens":191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:05:25.378370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent systematic search that draws the full relevant literature rather than only the top-ranked abstracts per requirement and finds a substantial trustworthy-AI requirement or evaluation approach missing from the paper's synthesis, such as a maturing standard for safety or sustainability with established metrics, would show that the claimed unified coverage is incomplete.","supporting_citations":[],"review_version":1}