{"id":"5c46df5e-6747-4d81-9068-23b84684b357","arxiv_id":"2509.09470","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A web-navigating AI agent plus form-filling automation found India-affiliated papers across 586 conference papers with 100% recall and 99.4% accuracy.","lead":"AEGIS is an automated pipeline that reads conference paper lists, identifies papers whose authors work at Indian institutions, and automatically submits them to a nomination form. On 586 papers from five conferences it found all 37 qualifying papers, with three false submissions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% recall claim rests on an undocumented ground-truth label set; without label provenance, recall is unverifiable and potentially circular.","rationale":"The reader identified the same load-bearing weakness I find: the ground-truth labels for India affiliation are never described, so the headline recall figure cannot be independently checked. This is the most security-critical assumption because recall=1.00 is the paper's central quantitative claim, and the metric is only meaningful relative to a trustworthy label set. The paper's own pipeline is designed to populate the IKDD Premier Papers page, creating a concrete circularity risk: if the evaluation labels are derived from the pipeline's submissions, perfect recall is tautological. I considered other potential concerns: the admitted pagination limitation (footnote 3) limits generalization but does not invalidate the measured claim if the five test pages did not paginate; per-publisher prompt templates raise overfitting concerns but do not undermine the in-sample result; and the stochastic single-run issue is addressable but secondary. None of these is as load-bearing as the missing gold standard. The reader's CONDITIONAL verdict already captures this: the paper should be accepted only if the authors release the labeled dataset and protocol. My stress test does not move that verdict, so UNCHANGED is appropriate. I credit the paper for internally consistent numbers, a clear pipeline description, and an honest statement of the pagination limitation, but the central evaluation remains unverified without label provenance.","tokens_in":4810,"tokens_out":3748,"duration_ms":43533,"concrete_test":"Release the gold-standard label file and annotation protocol for all 586 papers (criteria, annotators, inter-annotator agreement, and handling of ambiguous affiliations), plus the raw HTML and AEGIS outputs. Independently recompute TP/FP/FN/FN from these artifacts. Then check whether the gold positives coincide exactly with the set of forms submitted by the RPA step; if they do, the evaluation is circular. This single check settles whether recall=1.00 is a measured result or an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.7 reports recall=1.00 and accuracy 0.99 across 586 papers, but never states how the ground-truth labels were obtained. There is no annotation protocol, no annotator count, no definition of 'India-affiliated' (institute address vs. nationality vs. affiliation string), and no statement that labels were fixed before running AEGIS. Because the pipeline's final step (Section 3.6) submits matches to the IKDD Premier Papers page, if the gold labels are simply the papers the pipeline itself submitted, or were extracted by the same LLM/author in a loop, then recall is definitionally 1.00 and the evaluation is circular. The 'Custom Data' row in Table 1 is especially suspect: 100 custom-curated papers with a 20% positive ratio are described only as a 'custom-curated collection,' with no source or labeling description. Without an independent gold standard, every metric in Table 1 is unverifiable. The arithmetic of Table 1 is internally consistent, so the issue is not calculation but label provenance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AEGIS, a pipeline that takes a conference proceedings URL, extracts paper metadata using the Agent-E LLM agent, identifies papers with at least one India-affiliated author, and automatically submits them to the IKDD Premier Papers page via Selenium-based RPA. The system is evaluated on 586 papers from five datasets (SIGKDD, ACL, NeurIPS, IEEE ICDM, and a custom collection), reporting perfect recall of 1.00 and near-perfect accuracy of 99.4%. The contribution is framed as a proof-of-concept demonstration of task-oriented AI agents for scholarly workflow automation.","tokens_in":5048,"tokens_out":2397,"duration_ms":28813,"significance":"The paper addresses a real and practically useful task: reducing manual effort in bibliographic discovery and targeted submission workflows. If the reported results are sound, the pipeline is a valuable demonstration for the digital-library and scholarly-communication communities. The authors are transparent about some limitations (e.g., the pagination caveat in Section 3.1) and implement reproducibility-oriented local caching. However, the central quantitative claims rest on an evaluation whose ground-truth labels are never described, which makes recall and accuracy numbers unverifiable. The fact that Agent-E is the product of Emergence AI, the employer of co-author Akkil, should also be disclosed as a competing interest, and it raises an additional objectivity concern for the evaluation.","major_comments":[{"comment":"The reported recall of 1.00 and the accuracy numbers depend entirely on the ground-truth labels for which of the 586 papers count as 'India-affiliated.' The paper never specifies how these labels were obtained: no annotation protocol, no number of annotators, no definition (affiliation address vs. nationality vs. institutional email), and no statement that labels were fixed independently of AEGIS's output. Because the pipeline's final step (Section 3.6) submits matches to the IKDD Premier Papers page, if the gold labels are simply the papers the pipeline itself submitted or otherwise derived from the pipeline's own output, then recall=1.00 is definitional. Please provide a documented, independently fixed labeling procedure or an external reference standard.","section":"Section 3.7, Table 1"},{"comment":"The 'Custom Data' row is load-bearing for the precision estimates (0.90) and the overall claim, yet it is described only as a 'custom-curated collection' with a 20% positive ratio. There is no mention of where these 100 papers came from, how they were chosen, or how their India-affiliation labels were established. Without this provenance, the row cannot be interpreted as a meaningful precision stress test.","section":"Table 1, 'Custom Data' row"},{"comment":"There is a small but concrete arithmetic discrepancy: pooling Table 1 gives (37+546)/586 = 583/586 = 99.5% accuracy, while the abstract reports 99.4%, which is actually the unweighted mean of the per-dataset accuracies. The paper should state explicitly which aggregate quantity is being reported. This is not a major computational error, but it signals imprecision in reporting that should be fixed alongside the label-provenance issue.","section":"Section 3.7 vs. Abstract"},{"comment":"The text notes that handling pagination is 'work in progress.' The evaluation implicitly assumes that every eligible paper appears on a single proceedings page. For the perfect-recall claim to be meaningful, the authors should explicitly state whether all five evaluated proceedings pages were single-page listings and, if not, how pagination was handled. As written, a reader cannot rule out that some false negatives were avoided only because the selected datasets happened to avoid pagination.","section":"Section 3.1, footnote 3"}],"minor_comments":[{"comment":"The sentence 'We created a library of prompt templates, each tailored to a specific conference series or publisher (one per library)' is confusing. Please clarify what 'one per library' refers to.","section":"Section 3.4"},{"comment":"Reference [1] (Agent-E) has a co-author who is also an author of this paper and is employed at Emergence AI. The close relationship between the system under evaluation and the authors' own institution/employer should be explicitly disclosed as a potential conflict of interest in the manuscript.","section":"References / Author affiliations"},{"comment":"The RPA stage's reliability is not evaluated beyond 'waits for a confirmation message.' It is unclear whether any submissions failed or were verified independently. A brief statement about RPA success/failure rates would strengthen the demonstration.","section":"Section 3.6"},{"comment":"Minor typos and stylistic issues exist, e.g., 'Robotic Process Automation' capitalization inconsistency, and the demo footnote/URL formatting in the ACM template. These do not affect the technical content.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a demo/system paper and the technical engineering is plausible, but the evaluation's central claim hinges on labels that are never described. This is a fixable issue if the authors can provide independent ground truth, but without it the reported numbers cannot be accepted. There is also a notable self-evaluation risk: the system is based on Agent-E, which is a product of the employer of one of the authors. Even if unintentional, this makes the undocumented label provenance more consequential. I recommend major revision, not rejection, because the concern is addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take.\n\nThe genuinely new thing here is the integration: an LLM browser agent (Agent-E) plus an RPA front end, with layout-aware link normalization and per-publisher prompt templates, that goes from a proceedings URL to a submitted nomination form. That is a real pipeline, not a toy. The evaluation on 586 papers from five sources is a genuine measurement against external pages, and the arithmetic of Table 1 is internally consistent. The authors also admit their main failure mode (ICDM table-of-contents false positive) and flag pagination as an open item. Credit where due.\n\nThe soft spot is the one the stress-test note hits: Section 3.7 never describes how the ground-truth labels were obtained. No annotation protocol, no definition of 'India-affiliated' (institute address? nationality? affiliation string?), no statement that labels were fixed before running AEGIS. The 'Custom Data' row is especially opaque. Because the pipeline's final step submits to the IKDD Premier Papers page, if the gold labels are simply the pipeline's own submissions, recall of 1.0 is definitionally true and the evaluation is circular. The paper gives no evidence against that. That is a load-bearing gap for a claim of perfect recall, though not for the system's existence.\n\nI also note the co-author is at Emergence AI, the company behind Agent-E, so the evaluation measures the authors' own system. That's not disqualifying, but it raises the burden. Add a single run of a stochastic system and per-publisher templates that match the test set, and you have a set of addressable issues.\n\nNone of this breaks the core claim as a proof-of-concept. The system clearly works on these pages. But the headline '100% recall' should not be taken as established until the dataset and label protocol are released. For a workshop demo, this is a solid submission; for a rigorous benchmark, it needs work.\n\nWho it's for: anyone building agentic workflows over semi-structured web data, and societies thinking about automating nominations. It deserves a serious referee — the system is real and the claims are testable. I'd recommend engaging, but conditioning acceptance on releasing code, the dataset, the label protocol, and ideally repeated runs and a baseline.\n\nWould I bring it to reading group? Maybe, as a case study in evaluation pitfalls.","headline":"A competent demo paper whose headline 100% recall is real-sounding but rests on a ground-truth protocol the paper never describes; worth engaging, but the evaluation needs transparency before the claim can be taken literally.","tokens_in":5565,"tokens_out":2275,"would_cite":false,"duration_ms":24949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AEGIS turns a proceedings URL into an automatic paper sweep and nomination submission, reporting 100% recall and 99.4% accuracy on 586 papers.","keywords":["task-oriented AI agents","scholarly workflow automation","information extraction","geographic affiliation detection","conference proceedings parsing","large language model agent","robotic process automation"],"falsifier":"Re-run the pipeline on the same 586 proceedings URLs using two independent human coders who label 'at least one author affiliated with an Indian institution' with no access to the system's output; if any labelled-positive paper is absent from the submitted list, the perfect-recall claim is false. A cheaper check is to reproduce the reported count: the table lists 37 true positives and 0 false negatives, so auditing the cached HTML files and submission logs for the five datasets should confirm exactly those numbers.","tokens_in":4661,"feed_emoji":"🤖","tokens_out":9983,"duration_ms":109567,"temperature":0.7,"pith_summary":"The paper is trying to show that one URL can drive a complete scholarly housekeeping task: find every paper in a conference proceedings with at least one India-affiliated author, and automatically submit each to a predefined nomination form. The authors built a pipeline that caches the rendered HTML, normalizes links according to page layout, invokes a browser-navigating language-model agent to read each paper and return its metadata, verifies the agent's output, and then uses browser automation to fill and submit the form. Their reported run over 586 papers from five conference datasets produced zero false negatives, perfect recall on every dataset, and near-perfect accuracy, with the only errors being a few false positives attributed to ambiguous affiliation strings. The stated purpose is to show that task-oriented AI agents can move from retrieving information to completing real-world actions, and the paper also notes that pagination of long paper lists remains an open item.","feed_headline":"AEGIS paper sweep returns 100% recall on 586 papers","feed_subtitle":"An AI agent reads five conference proceedings, finds every India-affiliated paper, and submits its own nomination forms.","key_machinery":"The load-bearing machinery is the combination of a layout-aware link normalizer and a browser-navigating language-model agent, with a verification layer and a browser-automation submitter at the two ends. The normalizer inspects the structure of the proceedings page and chooses one of two strategies: a regular-expression filter for flat continuous paper lists, or a DOM traversal that links each track heading to its block of paper links for track-based pages. This reduces heterogeneous HTML to one clean list of URLs. Each paper URL is then passed to a prompt-filled LLM agent that returns extracted metadata; the verification layer discards responses with empty author or institution lists, and","core_discovery":"On the paper's own account, the central discovery is that a structured pipeline—HTML parsing, layout-aware normalization, prompt-driven LLM reading, response verification, and robotic form submission—can perform a geographic filtering and nomination task with perfect recall. The evidence is the evaluation in Section 3.7: across 586 papers from five datasets, the system never missed a paper judged to have an India-affiliated author (recall 1.00), and its accuracy was 1.00 on three datasets, 0.99 on one, and 0.98 on the custom set; the abstract reports the blended figure of 99.4 per cent. The paper attributes the few false positives to ambiguous affiliation strings, giving the example of a 'Ta","pith_inferences":["The evaluation's ground-truth labels for India affiliation are never described in Section 3.7; if the reference list was taken from the same nomination page the pipeline is designed to populate, the perfect-recall figure could be circular. An independent hand-labeled audit of the 586 papers—or even of the cached HTML files the pipeline stores—is the most direct way to verify the claim.","The specific false-positive example (a 'Table of Contents' page) hints at a cheap improvement the paper does not explore: a pre-filter that rejects non-paper documents before the LLM agent is invoked would likely recover most of the lost precision.","The underlying URL-to-action recipe is more general than geographic filtering. Replacing the India-affiliation predicate with topic, funding-body, or author criteria, and replacing form submission with notification or database insertion, would turn this into a generic template for scholarly curation.","The stated pagination gap means the current claim of perfect recall covers only single-page listings; long multi-page proceedings are untested, so the perfect-recall result should not be assumed beyond the evaluated setting until pagination support lands."],"forward_implications":["A society that currently pays humans to collect geographically eligible papers from proceedings can instead supply a URL, and the system will return (and submit) the eligible list autonomously.","Because the prompt templates are per publisher, the architecture extends to other publishers and repositories as new templates are written; the paper lists that as future work.","If the recall claim holds, the hard requirement of the application—never missing a relevant paper—is met, and the remaining risk is precision, concentrated in ambiguous affiliation strings rather than in omitted papers.","The reported perfect recall suggests that for proceedings of the size tested (up to about 150 papers), scale is not a limiting factor for the pipeline."],"fun_headline_variants":["AI agent finds every India paper in 5 conferences—then submits forms","Robot reads 586 papers, nails every India match, files nominations","Agent-E sweeps 586 papers, 100% recall, then fills forms","Perfect recall: AI finds all 586 flagged papers and submits","From discovery to action: AI agent finds India papers, files forms"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the ground-truth labels saying which of the 586 papers count as India-affiliated are correct and were fixed independently of the pipeline's output; the paper never describes who labelled them or how, so if those labels came from the target nomination page itself, the perfect-recall claim would be circular rather than independent.","fun_headline_variants_meta":{"raw":{"variants":["AI agent finds every India paper in 5 conferences—then submits forms","Robot reads 586 papers, nails every India match, files nominations","Agent-E sweeps 586 papers, 100% recall, then fills forms","Perfect recall: AI finds all 586 flagged papers and submits","From discovery to action: AI agent finds India papers, files forms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3276,"prompt_tokens":686,"completion_tokens":2590,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":2496}},"tokens_in":430,"tokens_out":2590,"duration_ms":17383,"temperature":1.0,"reasoning_tokens":2496,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:02:18.099307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the same 586 proceedings URLs using two independent human coders who label 'at least one author affiliated with an Indian institution' with no access to the system's output; if any labelled-positive paper is absent from the submitted list, the perfect-recall claim is false. A cheaper check is to reproduce the reported count: the table lists 37 true positives and 0 false negatives, so auditing the cached HTML files and submission logs for the five datasets should confirm exactly those numbers.","supporting_citations":[],"review_version":1}