{"id":"144f117f-ecc8-4288-bf59-b80de3218ecc","arxiv_id":"2501.04455","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Few-shot Llama 2 7B and Mistral 7B models detect and classify dataset and software URLs in GitHub READMEs with low strict precision, and the paper introduces a benchmark of 811 repositories and 1,439 URLs.","lead":"The paper tests whether large language models can find and sort links to datasets and software inside GitHub README pages, using a few examples as guidance. It finds the models often miss links, invent fake ones, and misclassify them, so they are not yet reliable for automated knowledge-base building.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gold-standard annotations are the load-bearing assumption: DatasetLandingPage is defined too broadly, no inter-annotator agreement is reported, and if labels are noisy the low precision/recall may be an artifact rather than a true LLM limitation.","rationale":"In good faith, the paper is a clearly scoped empirical evaluation with a new, publicly available dataset and an honest negative result. The strongest claim, however, depends on the correctness of the manually constructed gold standard. My read of Section 2.3 is that the class definitions, particularly the DatasetLandingPage class, leave substantial room for subjective judgment: a software repository whose code generates a dataset is labeled DatasetLandingPage, while a source-code repository without such a connection is Software, and a file in a repository whose dataset is elsewhere is also DatasetLandingPage. Such distinctions are hard to apply consistently, and the paper reports no measure of annotation reliability. If the gold standard itself is noisy, the low strict/type precision and recall could be an artifact of ambiguous labels rather than evidence that LLMs are inadequate for the task. This is not an accusation of investigator carelessness; it is a missing piece of evidence that is standard for NLP annotation studies. The proposed re-annotation study would settle this directly: high kappa would strengthen the paper's negative conclusion, while low kappa would make the central claim unverdictable. I agree with the reader that this is the weakest assumption, though I focus more on annotation consistency than on the LCS matching procedure. The reader's additional concerns, such as the absent regex baseline and the broad 'off-the-shelf models' phrasing, are secondary and would not by themselves change my recommendation. The conditional verdict remains appropriate pending the reliability check.","tokens_in":8547,"tokens_out":3845,"duration_ms":40604,"concrete_test":"Randomly sample 150-200 URLs stratified by class from the 1,439 labeled URLs; have two researchers who did not do the original annotation re-label them using only the Section 2.3 class definitions, then compute Cohen's kappa overall and per class, and record how often the DatasetLandingPage vs Software boundary is judged ambiguous. If kappa is below 0.6 or the ambiguous cases exceed 20%, the gold standard is too noisy to support the paper's negative conclusion; if kappa is high, the original verdict stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim (off-the-shelf LLMs are inadequate for extracting and classifying dataset/software URLs at KG-grade precision and recall) is only as strong as the gold standard used to measure precision and recall. Section 2.3 defines four classes, but DatasetLandingPage is exceptionally broad: it includes index pages, software repositories that merely generate or download a dataset, and even a URL to a file in a GitHub repository where the dataset lives in another file. This makes the boundary between DatasetLandingPage and Software, and between DatasetLandingPage and DatasetDirectLink, a matter of annotator judgment. The paper reports no inter-annotator agreement, no adjudication protocol, and no annotation error analysis. If the 1,439 labels are applied inconsistently, especially in the largest class (678/1439 = 47%), then the reported strict/type precision (0.24-0.54) and recall (0.26-0.55) may reflect label noise more than LLM capability. The LCS bipartite matching in Section 2.4 adds a second potential distortion, but gold-standard reliability is the more fundamental load-bearing assumption. A regex or majority-class baseline would help contextualize the numbers, but without a trustworthy yardstick the article's central conclusion is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether open-weight 7B LLMs (Llama 2 7B, Mistral 7B, and their 4-bit quantized versions) can extract and classify dataset and software URLs from GitHub READMEs using few-shot prompts. The authors construct a gold standard of 1,439 URLs from 811 repositories with four classes, apply four prompt variants (static/dynamic examples, extraction+classification vs classification-only), and report precision and recall under strict, exact, partial, and type matching. They find modest classification performance, frequent parsing failures, and conclude that off-the-shelf models are inadequate for knowledge-graph-grade URL identification and classification.","tokens_in":8803,"tokens_out":4003,"duration_ms":37285,"significance":"If taken at face value, the paper provides a useful negative result and a publicly released manually annotated dataset for an underexplored NER variant in which the entities are URLs. Strengths include the manual annotation effort, the comparison of four model variants and four prompt settings, and the explicit analysis of output parsing failures. The evaluation is measured against an independent gold standard, and no model parameters are fitted to the test outcome, so the reported numbers are not circular. The contribution is weakened by the narrow model scope, the absence of reliability evidence for the gold standard, and the lack of statistical analysis; these gaps currently prevent the strong conclusion about 'off-the-shelf models' from being fully supported.","major_comments":[{"comment":"The definition of Dataset Landing Page is exceptionally broad: it includes index pages, software repositories that merely generate or download a dataset, and even a URL to a file in a GitHub repository where the dataset is stored in another file. Since 678 of the 1,439 URLs (47%) belong to this class, the boundary between Dataset Landing Page and Software, and between Dataset Landing Page and Dataset Direct Link, becomes a matter of annotator judgment. The paper reports no inter-annotator agreement, no adjudication protocol, and no annotation error analysis. If labels in this largest class are applied inconsistently, the strict and type precision/recall values in Tables 2 and 3, and hence the central claim that LLMs are inadequate, could be artifacts of label noise rather than genuine model limitations. Please report inter-annotator agreement on a subsample, an adjudication procedure, or a sensitivity analysis that shows the main conclusions are robust to reasonable label disagreements.","section":"Section 2.3 (Gold Standard Data)"},{"comment":"The evaluation treats unparsable outputs as invalid and, for zero-shot learning, the parsed ratio is so low (e.g., 18/733 for Llama 2 7b) that zero-shot results are omitted entirely. Consequently, the evidence for the claim that 'off-the-shelf models are inadequate' rests only on few-shot results from which 3.1% to 14.2% of outputs were discarded. No confidence intervals, error bars, or significance tests are reported for the differences between models or prompt variants, so it is unclear whether the observed differences in strict and type scores are meaningful. Please report zero-shot parsing-recovered results if they can be salvaged, provide uncertainty estimates for the main metric comparisons, or explicitly restrict the conclusion to the few-shot setting with a caveat about discarded outputs.","section":"Section 2.4-2.5 and Table 1"},{"comment":"The concluding sentence that 'off-the-shelf models are inadequate for addressing intricate tasks demanding high precision and recall' goes beyond the evidence, which covers only Llama 2 7B, Mistral 7B, and their 4-bit quantized versions. No larger open models or commercial LLMs are tested, and no quantitative comparison with a regular-expression or majority-class baseline is provided, even though the text states that regex-based methods achieve higher precision for URL detection. Please scope the conclusion to the tested model family and size, or add experiments with larger/commercial models and at least one non-LLM baseline to support the broader claim.","section":"Section 4 (Conclusion and Outlook)"}],"minor_comments":[{"comment":"The text mentions 'LLama 3 and Llama 3.1' but the experiments only use Llama 2; please clarify which models are actually evaluated and fix the typo 'LLama'.","section":"Section 2.2 (Large Language Models)"},{"comment":"The prompt instructions define class labels in camelCase (e.g., 'DatasetLandingPage') while the example output uses lowercase snake_case (e.g., 'dataset_landing_page'); please ensure the parser maps both formats consistently and state which canonical form is used in evaluation.","section":"Figure 1"},{"comment":"The observation that dynamic examples do not improve performance 'in contrast with recent literature [25]' is made without a statistical test; please add a caveat or a significance test before drawing this comparison.","section":"Section 3 (Results and Discussion)"},{"comment":"The abstract emphasizes zero-shot learning as part of the investigated approach, but zero-shot results are excluded from the reported evaluation due to parsing failures; please adjust the abstract or include the zero-shot parsing statistics to avoid overstating the scope.","section":"Abstract"},{"comment":"The sentence 'All the resources of this paper are available here' does not include an actual URL or repository identifier; please provide a link or DOI.","section":"Introduction & Background"},{"comment":"Reference [17] cites the Llama 1 paper 'Llama: Open and efficient foundation language models', but the text refers to Llama 2; please update the reference to the correct Llama 2 publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a genuinely useful negative result for a niche task and releases a manually annotated dataset, which is a valuable contribution for a workshop venue. However, the gold-standard reliability is the central load-bearing assumption and is not currently evidenced; this, together with the overgeneralized conclusion, justifies a major revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new things here are the dataset (811 repositories, 1,439 manually labeled URLs) and the first evaluation of few-shot open 7B LLMs on URL-as-entity detection and classification in READMEs. The paper is scoped, the resources are released, and the reporting of parsing failures is honest. That is real value for a niche but active problem in knowledge-graph population.\n\nWhat the paper does well: it separates extraction from classification, defines four evaluation modes (strict, exact, partial, type), and reports both. It also compares static and dynamic few-shot prompting and finds no clear win, which is a useful counterpoint to earlier claims about dynamic examples. The finding that Llama 2 7B and Mistral 7B, including quantized variants, get strict precision/recall around 0.3–0.5 on this task is credible for those models.\n\nThe soft spots are in proportion: the conclusion overreaches from two 7B open models to \"off-the-shelf models\" generally. No larger or commercial model is tested, so the claim should be narrowed. There is also no measured regex baseline, even though the paper says regex-based methods are better; that is an assertion, not a result. Zero-shot results are dropped because of low parsing rates, and unparsable outputs are scored as invalid, which mixes formatting failure with task failure. No error bars or significance tests are reported, and the numbers in Tables 2 and 3 are close enough in places that some bootstrapping would help.\n\nThe stress-test concern about gold-standard reliability is real but not fatal. The DatasetLandingPage class is broad, no inter-annotator agreement is reported, and no adjudication protocol is described. That could add label noise to strict and type scores. But the exact and partial URL extraction scores are high across models, and those do not depend on class labels. So the extraction half of the negative result probably survives label noise; the classification half is shakier. Because the dataset is published, reannotation or an IAA study is actually feasible, and I would ask for that in revision.\n\nWho gets value from this: researchers building KG pipelines from GitHub or other semi-structured sources, and people benchmarking small open LLMs on structured extraction. It is a legitimate empirical contribution, not a tech breakthrough. I would send it to peer review, with revisions: tighten the claim to the tested models, add a regex baseline, report IAA, and give error bars. It deserves a serious referee, and with those changes it would be a solid workshop or short-conference paper.","headline":"A narrow, honestly reported negative result on 7B LLMs for URL extraction from GitHub READMEs, with a useful new dataset; just don't buy the broader 'off-the-shelf models' conclusion.","tokens_in":9320,"tokens_out":2191,"would_cite":false,"duration_ms":24224,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Few-shot LLMs cannot reliably detect and classify dataset and software URLs in GitHub READMEs for knowledge graph construction.","keywords":["Named Entity Recognition","Large Language Models","Few-shot learning","Knowledge graph construction","URL classification","GitHub READMEs","Dataset detection","Software detection"],"falsifier":"If any unmodified off-the-shelf 7B-class LLM, using the paper's own prompt templates on the same 1,439-URL gold standard, achieved strict precision and strict recall both above 0.9 in the extraction-and-classification task, the central claim that off-the-shelf models are inadequate for this task would be refuted.","tokens_in":8382,"feed_emoji":"🤖","tokens_out":5527,"duration_ms":51742,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf large language models can find and classify the URLs in GitHub READMEs that point to datasets and software, a task needed to populate knowledge graphs from research repositories. Using few-shot prompting of Llama 2 7B, Mistral 7B, and their 4-bit quantized versions on 811 repositories with 1,439 manually annotated URLs, the authors report that URL detection is often accurate, but fine-grained classification into dataset direct links, dataset landing pages, software, and other is not. The best model reaches strict precision and recall around 0.5 in the joint extraction-and-classification task, and dynamic example selection does not help. The paper concludes that off-the-shelf models are inadequate for this high-precision, high-recall task, and that expectations for LLM-based knowledge graph population should be tempered.","feed_headline":"Few-shot LLMs miss and mislabel GitHub dataset URLs","feed_subtitle":"On 811 repositories, Llama 2 and Mistral find many URLs but too rarely type them correctly for reliable knowledge graphs.","key_machinery":"The study's machinery is a few-shot prompt template that casts the model as a human annotator, instructs it to emit a JSON array of URL/label pairs, and supplies four static or four dynamically selected examples; the four labels are Dataset Direct Link, Dataset Landing Page, Software, and Other. Evaluation follows the MUC-5 precision/recall schema applied under four matching modes—strict, exact, partial, and type—with a longest-common-substring bipartite matching step that aligns model output URLs to gold URLs. The control knob is the comparison between the joint extraction-and-classification task and a classification-only task that feeds the URL and its context to the model.","core_discovery":"On its own terms, the paper's finding is that a straightforward few-shot LLM pipeline—instructions plus four worked examples, with output parsed into JSON—is not yet a viable substitute for manual annotation or regex-based heuristics when the goal is a knowledge graph with reliable entity types. The models can reproduce or partially match a URL most of the time, with exact-match precision and recall often above 0.8, but the four-way type assignment collapses: strict precision for Llama 2 7B in the extraction-and-classification task is at best 0.371 and for Mistral 7B at best 0.519. The bottleneck is not seeing URLs but deciding what they are, particularly separating a dataset direct link from a dataset landing page and both from software. The paper further argues that richer context (classification-only prompts with the URL supplied) does not rescue the models, and that a simpler two-way dataset-versus-not decision is substantially more reliable.","pith_inferences":["A natural hybrid that follows from these numbers is to keep regex or link extractors for URL discovery and reserve the LLM for classifying already-extracted URLs, isolating the one subtask where models show partial ability.","The broad 'Dataset Landing Page' class, which includes folders, GitHub files, and software that generates a dataset, may be inflating confusion; a finer or hierarchical label set could change the measured ceiling.","Because quantization cost little on several settings and Mistral 7B beat Llama 2 7B, testing current 7–8B instruction-tuned models on the same 1,439-URL gold standard is a cheap, decisive extension.","The negative result is itself evidence for the field that prompt engineering alone is unlikely to close the gap, and that KG population should invest in task-specific training data or constrained decoding."],"forward_implications":["LLM outputs must be post-processed and matched to expected URLs before scoring, and output-format adherence alone costs 3–14% of responses; any production pipeline needs a structured-output wrapper or repair step.","Regex- or link-based extraction remains a stronger baseline for finding URLs in READMEs than generative extraction, since LLMs miss URLs and hallucinate non-existent ones.","A binary dataset-versus-not classification is a more realistic LLM use case than four-way typing, with Mistral reaching about 0.75 type-level recall in the classification-only setting.","Static and dynamic few-shot examples perform comparably in this task, so the extra cost of similarity-based example selection is not justified by the measured results.","Because the strict and type scores are far below exact scores, evaluation for KG population must report entity-type accuracy separately from boundary detection, not just URL overlap."],"supporting_citations":[{"why":"Supplies the Llama 2 model family used as one of the two base LLMs in the few-shot experiments.","marker":"[17]"},{"why":"Supplies Mistral 7B, the other base model, whose exact and type scores are the best in the study.","marker":"[18]"},{"why":"Provides the unarXive scholarly dataset from which the 811 GitHub repositories and their README URLs were seeded.","marker":"[22, 23]"},{"why":"Defines the MUC-5 precision/recall evaluation metrics that the strict, exact, partial, and type scores implement.","marker":"[24]"},{"why":"Gives the prior claim that dynamic few-shot examples can improve LLM annotation, which the paper's static-vs-dynamic comparison does not reproduce.","marker":"[25]"}],"fun_headline_variants":["LLMs see GitHub URLs but fumble dataset vs software labels","Few-shot LLMs detect resources but misclassify their types","Dataset extraction stumbles: LLMs can't tell data from code","On GitHub, LLMs find URLs but mislabel what they point to","Zero-shot promise fades: few-shot LLMs mislabel entity types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the manually assigned gold-standard labels for 1,439 URLs being correct and consistent, and on the longest-common-substring matching between model output and gold URLs faithfully representing what the model actually detected.","fun_headline_variants_meta":{"raw":{"variants":["LLMs see GitHub URLs but fumble dataset vs software labels","Few-shot LLMs detect resources but misclassify their types","Dataset extraction stumbles: LLMs can't tell data from code","On GitHub, LLMs find URLs but mislabel what they point to","Zero-shot promise fades: few-shot LLMs mislabel entity types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1475,"prompt_tokens":922,"completion_tokens":553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":538,"tokens_out":553,"duration_ms":5079,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:31:44.400749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If any unmodified off-the-shelf 7B-class LLM, using the paper's own prompt templates on the same 1,439-URL gold standard, achieved strict precision and strict recall both above 0.9 in the extraction-and-classification task, the central claim that off-the-shelf models are inadequate for this task would be refuted.","supporting_citations":[{"cited_title":"Chinchor, B","cited_arxiv_id":null,"evidence_quote":"Defines the MUC-5 precision/recall evaluation metrics that the strict, exact, partial, and type scores implement."}],"review_version":1}