{"id":"434f2f2c-65dc-429f-9070-61cc55731adc","arxiv_id":"2506.14430","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Works-magnet is an open-source, public-facing system for correcting affiliation metadata in OpenAlex, with more than 71,000 correction requests logged so far.","lead":"This paper introduces Works-magnet, an open-source tool that lets researchers and librarians review and fix institution names attached to scientific papers in the OpenAlex database. It matters because clean, open metadata helps track and evaluate research without relying on commercial databases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The title-level claim that Works-magnet accelerates curation is not supported: no baseline, throughput, or accuracy data is reported for the 71,283 correction requests.","rationale":"Read in good faith, the paper is a short project report rather than a rigorous empirical study; its value is in open code and data and in the honest limitations section. The reader correctly flags missing evaluation. I considered the reader's weak assumption about OpenAlex propagation: while real and acknowledged in §5, it is not the most load-bearing because corrected data is independently released as an open dataset (Section 4), so the platform's value does not fully depend on OpenAlex accepting each request. The more load-bearing gap is the absence of any quantitative support for the 'accelerates' claim in the title and abstract. A raw count of 71,283 requested corrections is compatible with a platform that merely collects requests, or even one that slows curation via extra overhead. Without a baseline or throughput/latency/accuracy measure, the central causal claim is not internally established. This is not a dispute with external consensus; it is a missing link between the described workflow and the advertised acceleration. Since the reader already made the verdict CONDITIONAL on evaluation, my recommendation is UNCHANGED; the condition should be a quantitative evaluation in a revised version.","tokens_in":3239,"tokens_out":2948,"duration_ms":30741,"concrete_test":"Compute from the public GitHub issue API (https://github.com/dataesr/openalex-affiliations) and the corrections dataset: weekly closure counts and median latency from creation to closure for a fixed cohort of French-affiliation issues; then obtain the same metrics for a matched sample of OpenAlex affiliation-correction tickets filed before Works-magnet's launch, or run a controlled pilot where 50 known affiliation errors are corrected by curators using Works-magnet vs. OpenAlex's standard interface, recording time per error and verification accuracy. If Works-magnet does not reduce time-to-correction or increase corrections per hour, 'accelerates' should be weakened to 'facilitates with uncorroborated speed benefit.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Works-magnet accelerates metadata curation by making automated AI calculations visible and correctable. For that claim to hold, the tool must at least increase curation throughput or reduce latency relative to the existing OpenAlex correction workflow, and the logged 'corrections' must be genuine verified fixes rather than raw requests. Neither condition is tested. Section 5 reports only that 71,283 corrections had been requested, with 'a significant proportion already closed,' plus dashboards. There is no pre/post comparison, no per-curator throughput, no measure of time from request to closure, and no accuracy check on a sample of closed issues. Because the count conflates issue-creation with curation outcomes, it cannot by itself justify the word 'accelerates.' This is an internal-evidence gap, not a demand for external consensus: the same paper cites GitHub issues and an open dataset, so the missing evaluation is feasible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Works-magnet, an open-source web application developed by the French Ministry of Higher Education and Research to support human curation of bibliographic metadata in OpenAlex, with a focus on affiliation matching. The tool makes automated AI-generated corrections visible as GitHub issues, allows anyone to request or review corrections, and publishes the resulting data openly. The paper describes the challenges of open metadata curation, outlines the tool's design and workflow, reports that 71,283 correction requests had been filed as of writing with a significant proportion closed, and candidly lists limitations such as dependency on OpenAlex verification and limited staffing. The authors argue that Works-magnet accelerates metadata curation by putting humans in the loop and that the corrected data becomes open and reusable.","tokens_in":3519,"tokens_out":2997,"duration_ms":30871,"significance":"If the claimed acceleration and quality improvements are substantiated, Works-magnet would be a valuable contribution to open science infrastructure, addressing a real bottleneck in open bibliographic databases. The paper's strengths are its open-source availability (code on GitHub, data on an open data portal) and its transparent discussion of limitations. However, the central claim of acceleration is not supported by any measured baseline, throughput metric, or accuracy evaluation. The paper is essentially a systems description with no empirical validation, so its significance currently rests on the potential of the tool rather than demonstrated performance.","major_comments":[{"comment":"The title-level claim that Works-magnet 'accelerates' metadata curation is not supported by the evidence in Section 5. The only quantitative indicator is the statement that '71,283 corrections had been requested, with a significant proportion already closed.' This is a count of issue-creation events, not a measure of curation throughput or latency. To substantiate the acceleration claim, the authors should compare the time-to-closure or per-curator workload against the existing OpenAlex correction workflow (e.g., via the OpenAlex web form or API), and report a pre/post analysis or a baseline. Without such a comparison, the word 'accelerates' is unsubstantiated.","section":"5"},{"comment":"Section 5 acknowledges that delays in verification by OpenAlex can accumulate a backlog of corrections, but the paper reports no measurement of this backlog or of the rate at which corrections are actually applied to OpenAlex. Since the stated goal is improving open data quality and making corrected data open and reusable, the authors need to document how many of the 71,283 requested corrections have been accepted and propagated by OpenAlex, and over what time period. This is essential to distinguish the tool's activity from its real-world impact.","section":"5"},{"comment":"The paper does not report any accuracy or quality assessment of the corrections made through Works-magnet. Section 2 notes that automated matchers achieve 85–95% accuracy before human intervention, but there is no evaluation of whether the human corrections are themselves reliable. A straightforward test would be to take a random sample of closed correction requests, have independent experts judge whether the proposed affiliation changes are correct, and report precision. Such a check is feasible because the issue data is public, and it would directly support the claim of improving metadata quality.","section":"2"}],"minor_comments":[{"comment":"The sentence 'in a research entity can up to five or more supervisors' is grammatically incomplete; it likely should read 'a research entity can have up to five or more supervisors'.","section":"1"},{"comment":"Typographical errors 'NonThis' and 'Despitetechnical' should be corrected to 'This' and 'Despite technical' respectively.","section":"2"},{"comment":"The heading 'Code and data availibility' contains a spelling error ('availibility' should be 'availability'), and the text 'https://github.com/dataesr/openalex-affiliations/issuesandwithopendataset' lacks spaces between words.","section":"4"},{"comment":"The phrase 'As of recently' is vague; please provide a specific date or version for the 71,283 correction count.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of a project report than a conventional research article, and its central claim is not empirically supported. The authors should add an evaluation section with baseline comparisons and accuracy checks. If the journal publishes systems descriptions without such evaluations, the paper might be acceptable as is, but for this venue I recommend requiring the additional evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a short, honest description of a real, open-source curation platform for OpenAlex affiliation metadata. The novel piece is the transparent GitHub-issue-based correction loop and the public release of the corrected data. That is worth a look.\n\nWhat is actually new: prior work cited (L'Hôte and Jeangirard 2021; Duran-Silva et al. 2024) are automated matching tools. Works-magnet is an application that makes those predictions visible and correctable by humans, with corrections logged as public issues. The author is explicit that this is a human-in-the-loop system rather than a new algorithm. Code and data are linked, and the correction counts are verifiable. That is real evidence for the tool's existence.\n\nThe soft spots: the central claim that this 'accelerates' curation has no baseline, no throughput comparison, and no accuracy check on the 71,283 requested corrections. The count conflates requests with outcomes—'a significant proportion already closed' is not a number. So the word 'accelerates' is doing work the evidence does not support. This is not a load-bearing flaw in the sense of invalidating the tool; it is a gap between the title's promise and the reported evidence. The paper itself acknowledges verification backlogs and resource constraints, so it is not hiding its limits. A future version with a pre/post comparison or a sampled accuracy check on closed issues would make the claim credible.\n\nOverall: this is a project report, and a decent one. Readers interested in open science monitoring or national-level metadata curation will get value from it, mainly as a concrete example of an open correction workflow. The stress-test concern lands and should be taken seriously: the missing evaluation is feasible because the issues and data are public. The paper deserves a serious referee, but the referee should push for either softening the 'accelerates' language or adding the missing metrics.","headline":"A candid, useful project report on an open curation tool for OpenAlex affiliations; the 'accelerates' claim is plausible but unmeasured.","tokens_in":3887,"tokens_out":1692,"would_cite":true,"duration_ms":16599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Works-magnet accelerates open metadata curation by making automated affiliation matches visible and correctable, with every correction request published as reusable open data.","keywords":["open science","metadata curation","affiliation matching","human-in-the-loop","OpenAlex","ROR","open data","scholarly communication"],"falsifier":"Take a random sample of the correction issues in the public tracker and check, after a fixed period such as six months, whether the corresponding OpenAlex affiliation records have been updated to match the requested corrections; if nearly none are propagated, the claim that Works-magnet accelerates open metadata curation fails.","tokens_in":3046,"feed_emoji":"🧲","tokens_out":5194,"duration_ms":50649,"temperature":0.7,"pith_summary":"The paper introduces Works-magnet, an open-source platform from the French Ministry of Higher Education and Research that lets any user see the automated AI calculations behind bibliographic metadata, mainly affiliation assignments in OpenAlex, and request corrections. The author's claim is that making these automated predictions visible and correctable accelerates curation, because correction requests are tracked publicly and the results are published as open data instead of being locked inside a proprietary system. The paper reports that 71,283 corrections had been requested as of writing. This matters because it offers a concrete path toward replacing proprietary evaluation databases with open data that institutions can trust and reuse, while putting humans back in the loop for AI-produced metadata.","feed_headline":"Works-magnet exposes AI affiliation matches for public correction","feed_subtitle":"OpenAlex records get human-checked fixes; 71,283 corrections requested and released as reusable open data.","key_machinery":"The mechanism is the correction-request loop connecting OpenAlex records, a public issue tracker, and a published open dataset of corrections. Works-magnet exposes automated affiliation matches so that a human curator can confirm or challenge them; each challenge becomes a tracked issue and the resulting corrected data is released openly. This loop is what transforms imperfect machine-generated metadata, which the paper says is accurate in roughly 85 to 95 percent of cases before human review, into openly reusable, human-verified data. The platform also tracks correction status and contributor domains publicly, making the curation work measurable.","core_discovery":"The central claim is that Works-magnet is a functioning open environment for metadata curation: it takes OpenAlex affiliation records that were assigned by automated tools, displays them alongside the raw affiliation string, and lets users file a correction request that is publicly tracked and ultimately published as reusable open data. The paper argues this open paradigm reverses the usual proprietary curation loop—where corrected data strengthens dependence on the vendor—by using the public-sector workforce to improve open data, so the benefit accumulates to the whole research community. Evidence offered is operational: 71,283 correction requests logged, with a significant proportion already closed, and dashboards for individual institutions. This is a new application rather than a new algorithm: the novelty is the human-in-the-loop correction workflow wrapped around existing AI matching tools.","pith_inferences":["An implication the paper leaves implicit: the 71,283 figure counts correction requests, not accepted corrections, so the real-world impact will hinge on how many are actually merged into OpenAlex, a metric the paper does not report.","A testable extension: the same visible-AI-plus-human-correction-plus-open-publication loop could serve as a general pattern for any AI-generated structured data where errors are costly, though the paper only claims the bibliographic case.","If the open correction dataset grows, it could be used to benchmark affiliation matchers, not just train them, giving researchers a way to compare competing tools on real curated ground truth."],"forward_implications":["If the workflow works at scale, OpenAlex's French affiliation data would converge toward human-corrected accuracy, giving France's Open Science Monitor a fully open and reliable data source.","The published correction dataset becomes a reusable training resource for automated affiliation-matching models, potentially raising their pre-human accuracy above the current 85 to 95 percent range.","Public institutions and other national Open Science initiatives could copy the same open-correction model for datasets, software mentions, grants, and other metadata types.","Transparent public tracking of corrections would make metadata quality measurable per institution, exposing where more curation attention is needed."],"supporting_citations":[{"why":"Describes the dataESR affiliation matcher that Works-magnet builds upon; its predictions are the kind of automated calculation the platform exposes for correction.","marker":"L'Hôte and Jeangirard 2021"},{"why":"Presents the AffilGood matcher, cited as one of the third-party AI tools whose 85 to 95 percent accuracy before human intervention motivates the human-in-the-loop design.","marker":"Duran-Silva et al. 2024"},{"why":"Earlier note on machine learning in the French Open Science Barometer, referenced as the basis for the idea that curated corrections can train future AI curation models.","marker":"Jeangirard 2022"}],"fun_headline_variants":["Works-magnet opens AI metadata matches to public correction","Human corrections refine AI metadata in open science tool","Open tool lets researchers fix AI-assigned affiliations","Works-magnet turns AI metadata errors into public fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop's stated purpose—improving open data quality—depends on OpenAlex actually processing and propagating the correction requests, and the paper itself acknowledges that OpenAlex verification delays can accumulate a backlog.","fun_headline_variants_meta":{"raw":{"variants":["Works-magnet opens AI metadata matches to public correction","Human corrections refine AI metadata in open science tool","Open tool lets researchers fix AI-assigned affiliations","Works-magnet turns AI metadata errors into public fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1172,"prompt_tokens":851,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":467,"tokens_out":321,"duration_ms":3883,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:16:29.194752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the correction issues in the public tracker and check, after a fixed period such as six months, whether the corresponding OpenAlex affiliation records have been updated to match the requested corrections; if nearly none are propagated, the claim that Works-magnet accelerates open metadata curation fails.","supporting_citations":[],"review_version":1}