{"id":"00db2201-dab3-4c3a-8061-e4273d66e594","arxiv_id":"2507.11102","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"KptLLM++ unifies keypoint semantic understanding, visual-prompt detection, and text-prompt detection in a single multimodal LLM, reporting SOTA accuracy on COCO, AP-10K, Human-Art, and other benchmarks.","lead":"This paper presents KptLLM++, a multimodal large language model that identifies and localizes object keypoints in images using text prompts, coordinate prompts, or reference examples. The authors report state-of-the-art results on several pose benchmarks and claim the model generalizes to unseen object categories.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'unseen' generalization claim is the weakest load-bearing point: AnimalPose categories are likely represented in the training pool (AnimalWeb/AP-10K/Animal Kingdom), so PCK@0.2=79.2 may not demonstrate zero-shot generalization; SOTA tables also lack protocol/error details.","rationale":"The reader's weakest_assumption identifies the same central flaw: the AnimalPose evaluation is advertised as 'unseen' but the training pool likely contains overlapping species, so the headline generalization number is not a clean test. I agree this is the most load-bearing concern because the paper's novelty is generic keypoint comprehension, and generalization to unseen categories is the evidence that the model is generic rather than dataset-specific. My own reading additionally notes that the SOTA tables are under-specified (no error bars, no stated protocol matching across methods, and several tables use the earlier KptLLM rather than KptLLM++), but those issues are secondary to the contaminated generalization test. The method itself is coherent: the identify-then-detect chain-of-thought is well described, the LoRA/DINOv2/text-decoding choices are plausible, and the failure-case analysis is honest. The concern is evidentiary, not architectural. A category-overlap audit plus per-category PCK would settle whether the 79.2 reflects true generalization or memorized familiar categories. If the audit shows overlap, the paper needs a new held-out split or a revised claim; if the audit shows no overlap, the generalization claim stands. Either way, the verdict should remain CONDITIONAL until that test is run and the protocol details are released.","tokens_in":23599,"tokens_out":3644,"duration_ms":41988,"concrete_test":"Audit category overlap between AnimalPose and every training dataset (AnimalWeb, AP-10K, Animal Kingdom, APT-36K, etc.) at the species/family level; then report PCK@0.2 separately for AnimalPose categories with and without training-set overlap. If the non-overlap categories show substantially lower PCK than 79.2, the generalization claim fails. As a further check, re-run Table 6 and Table 9 with released code and a stated common protocol (same GT boxes, resolution, no flip test) to confirm the small SOTA margins are reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.5's central claim ('KptLLM++ achieves the best performance, breaking the previously perceived performance upper bound for LLM-based approaches') rests on two legs: the multi-benchmark SOTA tables (Tables 6-12) and the 'unseen dataset' generalization result (Table 9, AnimalPose PCK@0.2=79.2). The second leg is the most load-bearing because it is what makes the model 'generic' rather than merely well-trained on familiar categories. Section 4.1.1(4) calls AnimalPose unseen, but the training pool explicitly includes AnimalWeb (334 species with facial keypoints), AP-10K (23 animal families/54 species), Animal Kingdom, and APT-36K. The five AnimalPose categories (dog, cat, cow, horse, sheep) are common domestic animals almost certainly present in those training sources; the paper provides no per-category breakdown or overlap analysis. If those categories were seen, PCK 79.2 measures familiar-category detection, not generalization to unseen keypoints, and the claimed advantage over X-Pose (73.4) is not a clean generalization comparison. Additionally, the SOTA tables lack matched-protocol details (GT box source, resolution, flip test, error bars), and Tables 1-4 report only the earlier KptLLM, so the unified KptLLM++ claim is not fully evidenced. The correct test is a category-overlap audit with per-category PCK and a leave-family-out re-split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KptLLM++, a multimodal large language model for generic keypoint comprehension, unifying three tasks: keypoint semantic understanding, visual-prompt-based keypoint detection, and textual-prompt-based keypoint detection. The model uses an identify-then-detect strategy, first generating semantic keypoint descriptions and then localizing keypoints via textual coordinate output, with a DINOv2 visual encoder and Vicuna-7B backbone. Training is scaled to over 500K samples across eight datasets. Experiments report state-of-the-art results on COCO, AP-10K, Human-Art, MacaquePose, Animal Kingdom, and CarFusion, plus a generalization result on AnimalPose (PCK@0.2=79.2 vs X-Pose's 73.4). Ablations on the earlier KptLLM model support design choices such as the identify-then-detect strategy and the prompt feature extractor.","tokens_in":23901,"tokens_out":5098,"duration_ms":58078,"significance":"If the results hold, this is a valuable step toward unified keypoint understanding, demonstrating that an MLLM can combine semantic interpretation with pixel-precision localization across diverse object categories. The paper's main strengths are the clearly described architecture, the large-scale training recipe, the textual-coordinate decoding simplification, and the explicit identify-then-detect mechanism. The claimed contributions are significant for the keypoint detection and vision-language communities. However, the evidence as currently presented is not yet sufficient to support the 'generic' and 'state-of-the-art' claims because the generalization experiment is not clean, several core results are reported for the predecessor model KptLLM rather than KptLLM++, and the SOTA comparisons lack protocol details.","major_comments":[{"comment":"The claim that AnimalPose is an 'unseen dataset' for evaluating generalization is not supported by the reported training pool. Section 3.7 and Section 4.1.1(3) list AnimalWeb (334 species with facial keypoints), AP-10K (23 animal families, 54 species), APT-36K, and Animal Kingdom among the eight joint-training datasets, and the five AnimalPose categories (dog, cat, cow, horse, sheep) are common domestic animals almost certainly represented in those sources. The paper provides no per-category breakdown, no category-overlap analysis, and no leave-family-out re-split. Consequently, the PCK@0.2=79.2 in Table 9 may measure familiar-category detection rather than zero-shot generalization to unseen keypoints, which is exactly the load-bearing premise of the 'generic' claim. This tension is reinforced by Section 4.7, which acknowledges that KptLLM++ 'struggles with generalizing to completely unseen keypoints.' The authors should provide a category-overlap audit, per-category PCK on AnimalPose, and a leave-family-out evaluation that excludes all animal families present in the training pool.","section":"Section 4.1.1(4) and Table 9"},{"comment":"The paper's title and abstract present KptLLM++ as the unified model, but the keypoint semantic understanding results (Table 1), visual-prompt-based detection results (Tables 2 and 3), and textual-prompt-based AP-10K results (Table 4) are all reported for the earlier KptLLM model, not for KptLLM++. KptLLM++ is evaluated only in Section 4.5's joint generic detection section. Since KptLLM uses a different decoding strategy (special-token regression head), a CLIP visual encoder, and a smaller training set, the unified three-task capability claimed for KptLLM++ is not directly evidenced. The paper should either report KptLLM++ results on MP-100 and AP-10K or clearly reframe these tables as KptLLM-baseline comparisons, and the abstract/conclusion should not attribute these results to KptLLM++.","section":"Sections 4.2–4.4 and Tables 1–4"},{"comment":"The 'state-of-the-art' comparisons in the joint generic detection section lack the protocol detail needed to support the central claim. Table 6 states that all top-down methods use GT bounding boxes and that no flip tests are used, but it does not report matched input resolutions, backbone sizes, training data, or whether the comparison methods were evaluated under identical single-scale conditions. The margins over strong baselines are small (e.g., 78.1 vs 77.4 for ViTPose and LocLLM on COCO), so small protocol differences could change the ranking. Table 12 compares AP for ED-Pose with PCK@0.2 for other methods on CarFusion, and the text does not specify whether ED-Pose is a top-down method using GT boxes. The authors should provide complete experimental configurations for every baseline and the proposed model, and ideally report standard deviations or significance tests where margins are narrow.","section":"Section 4.5, Tables 6–12"},{"comment":"All four ablation studies are conducted on the visual prompt-based keypoint detection task using MP-100 Split-1 with the KptLLM model, not KptLLM++. The identify-then-detect strategy, prompt feature extractor, and combined visual-textual prompts are therefore validated only for the predecessor architecture that uses special-token regression and a CLIP encoder. Because KptLLM++ changes the decoding to textual coordinates and replaces the visual encoder with DINOv2, the transferability of these ablations to the proposed model is not established. The authors should run the key ablations on KptLLM++ (or explicitly justify why the earlier model's ablations carry over), since the identify-then-detect mechanism is a central architectural claim.","section":"Section 4.6, Tables 13–16"}],"minor_comments":[{"comment":"The sentence 'We aggregate the results to derive accuracy rates ((2)' is incomplete and contains a formatting error; it should be rewritten.","section":"Section 4.1.2, paragraph (1)"},{"comment":"There is a typo in the heading: 'Indentify-then-Detect' should be 'Identify-then-Detect'.","section":"Section 3.4 heading"},{"comment":"The text says 'KptLLM accepts multiple images' but the figure caption and surrounding context refer to KptLLM++; the naming should be made consistent throughout.","section":"Section 3, first paragraph and Figure 2"},{"comment":"The 'Scaling Up' column uses symbols '×' and '✓' without a legend; the paper should clarify that '×' means no scaling and '✓' means joint training with the eight datasets.","section":"Table 5"},{"comment":"The phrase 'breaking the previously perceived performance upper bound for LLM-based approaches' is vague; the authors should cite the specific prior result or bound they are referring to.","section":"Section 4.5"},{"comment":"The model name 'LLaV A-V1.5-7B' appears with an inconsistent space; it should be written consistently as 'LLaVA-V1.5-7B'.","section":"Section 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a journal submission extending the authors' previous NeurIPS paper KptLLM, and a large portion of the experimental evidence (Tables 1–4 and all ablations) is for that earlier model. The 'unseen AnimalPose' generalization result is the most delicate point: given the explicit training pool (AnimalWeb, AP-10K, Animal Kingdom, APT-36K), the five AnimalPose categories are very likely seen during training, and the paper's own failure-case section concedes limited generalization to unseen keypoints. If a category-overlap audit confirms the overlap, the abstract's generalization claim should be substantially revised. The SOTA tables also need protocol-level detail before the comparisons can be trusted. These issues are fixable with additional experiments and careful reframing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real attempt at a meaningful problem—can one MLLM handle keypoint semantics, visual-prompt detection, and text-prompt detection in a single model—and the KptLLM++ artifact (Vicuna-7B + DINOv2 + textual coordinate decoding + 500K joint training) is new. The generic detection results are genuinely competitive: 78.1 AP on COCO, 81.2 on AP-10K, 77.3 on Human-Art, 83.9 on MacaquePose, all without test-time flip. The identify-then-detect chain-of-thought is a sensible design and the internal ablation (87.68 to 91.66 PCK on MP-100) supports it. Credit where due: the 500K scaling experiment is a real engineering effort and the in-the-wild visualizations look honest.\n\nThe soft spots are serious, though. The biggest is that Tables 1–4—semantic understanding on MP-100, visual-prompt detection, and text-prompt detection on AP-10K—report the earlier KptLLM, not KptLLM++. KptLLM++ appears only in the generic detection section (Tables 5–12). So the paper's unification story is not backed by the unified model's own numbers on those three tasks. That's a reporting gap, not a fatal flaw, but it needs fixing before anyone accepts the abstract's broad SOTA claim.\n\nThe second issue is the \"unseen\" AnimalPose result. The training pool includes AnimalWeb (334 species, including dogs and cats), AP-10K (23 families), APT-36K, and Animal Kingdom. The five AnimalPose categories—dog, cat, cow, horse, sheep—are almost certainly in there. That's not necessarily disqualifying: the keypoint definitions differ (AnimalWeb is facial, AnimalPose is whole-body), and the 79.2 PCK@0.2 still beats X-Pose. But the paper calls AnimalPose 'unseen' and presents it as zero-shot generalization, and it doesn't provide per-category breakdowns or an overlap analysis. A referee should demand a leave-family-out evaluation.\n\nMinor but worth noting: no error bars, no released code or weights, and the scaling ablation (Table 5) conflates model changes (CLIP→DINOv2, regression→text) with the dataset scaling. The stress-test note's central point holds up: the generalization claim is load-bearing and currently under-evidenced.\n\nWho is this for? People working on MLLMs for fine-grained perception, pose estimation, or open-vocabulary keypoint detection. It deserves a serious referee—the problem is important and the generic detection numbers are strong enough that the community should see this reviewed. My recommendation: send to peer review, but expect a major revision that clarifies the model naming, re-runs or explicitly scopes the task-specific tables, adds the AnimalPose overlap audit, and ideally releases code.","headline":"A worthwhile step toward a single MLLM for keypoint semantics and detection, but the evidence is uneven; send to review, expect a major revision.","tokens_in":24529,"tokens_out":3895,"would_cite":false,"duration_ms":44216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single multimodal large language model can both interpret the meaning of a keypoint and output its precise pixel position, achieving state-of-the-art accuracy on human, animal, and object benchmarks.","keywords":["keypoint detection","pose estimation","multimodal large language model","generic keypoint comprehension","category-agnostic pose estimation","open-vocabulary keypoint detection","identify-then-detect","chain-of-thought"],"falsifier":"Train the same model on the same eight datasets after deleting every training image labeled dog, cat, cow, horse, or sheep, then measure PCK@0.2 on AnimalPose; if it stays near 79.2 the generalization claim holds, and if it falls toward the 59.1 no-scaling level or below X-Pose's 73.4, the claimed unseen-category gain is an artifact of category overlap.","tokens_in":23340,"feed_emoji":"🎯","tokens_out":10154,"duration_ms":102773,"temperature":0.7,"pith_summary":"This paper claims that a multimodal large language model can become a generic keypoint engine: one model that reads the semantic meaning of a keypoint and outputs its location at pixel precision, whether the prompt is an image, a keypoint name, or a detailed text description. The proposed system, KptLLM++, uses an identify-then-detect chain-of-thought in which the model first states what the keypoint is and then writes its normalized coordinates as plain text, so no special token or regression head is needed. Trained on more than 500,000 samples spanning humans, animals, vehicles, and occluded or artistic scenes, it reports the best results among compared methods on MP-100, AP-10K, COCO, Human-Art, MacaquePose, Animal Kingdom, and CarFusion, and 79.2 PCK@0.2 on AnimalPose, which the paper treats as an unseen test set. If correct, this removes the presumed accuracy ceiling on LLM-based keypoint detection and turns keypoint understanding into a conversational, instruction-driven task.","feed_headline":"Multimodal LLM names and pinpoints keypoints at pixel precision","feed_subtitle":"A single model handles text or image prompts for any object's keypoints, beating specialist detectors.","key_machinery":"The load-bearing mechanism is the Identify-then-Detect (ItD) strategy: a single autoregressive response first names or interprets the keypoint's semantics and then emits its normalized coordinates as text, within the same chain-of-thought. Supporting components are the prompt feature extractor, which uses cross-attention layers over the support-image features to isolate keypoint-specific visual evidence and disambiguate mirror-symmetric keypoints; the DINOv2 visual encoder; LoRA fine-tuning of the LLM; and a joint training pool of eight datasets. The textual-coordinate decoder matters because it eliminates the need for a new vocabulary token and a regression head, which the paper argues makes training simpler and scales cleanly to many keypoint categories.","core_discovery":"The central claim, stated in Section 4.5, is that KptLLM++ outperforms every compared vision-only and vision-language method on the tested keypoint benchmarks, breaking the previously perceived performance upper bound for LLM-based approaches. The key design is the identify-then-detect paradigm: the LLM first interprets the semantic meaning of the prompted keypoint and then localizes it through a structured chain-of-thought, which helps resolve ambiguous left/right and symmetric-keypoint cases. Coordinates are normalized to [0,1], rounded to three decimal places, and generated directly as text tokens, replacing the special keypoint token and feed-forward regression head of the earlier KptLLM. Switching the visual encoder from CLIP to DINOv2 and scaling the training mixture to eight datasets with over half a million samples are credited with converting these choices into large gains, including a jump on AnimalPose from 55.9 to 79.2 PCK@0.2.","pith_inferences":["The AnimalPose number is probably not a clean zero-shot result: the five test categories are common animals, and the training pool includes AnimalWeb, Animal Kingdom, and AP-10K, so the 79.2 PCK likely measures familiar-category generalization, not unseen-keypoint generalization; a held-out-order experiment would settle this.","If text-based coordinate decoding is what enables scaling, then the 3-decimal rounding (one part in a thousand of image size) sets a floor on localization precision; testing finer or coarser rounding would show whether this is the current bottleneck.","The identify-then-detect recipe should transfer to other pixel-level grounding tasks, such as part segmentation or landmark-conditioned generation, wherever an LLM can verbalize the semantics before emitting coordinates; that transfer is not demonstrated in the paper.","An end-to-end multi-object variant that trains the detector together with the keypoint model could reduce sensitivity to box errors, which the paper's own failure analysis identifies as a main source of mistakes."],"forward_implications":["A single model can answer keypoint-semantic questions, visual-prompt detection, and text-prompt detection without changing architecture or adding task heads.","Textual coordinate output makes keypoint detection human-readable and lets users batch multiple keypoint queries into one conversation turn.","The same architecture without dataset scaling scores 59.1 on AnimalPose; with scaling it scores 79.2, so data diversity is presented as the main driver of out-of-domain accuracy.","Adding the identify-then-detect step raises PCK on MP-100 Split-1 from 87.68 to 91.66, so semantic interpretation is a direct contributor to localization accuracy.","Combining the model with an open-world object detector yields a multi-object pipeline, since the detector supplies crops and the model localizes keypoints within each crop."],"supporting_citations":[{"why":"It supplies the MP-100 benchmark, its category-agnostic splits, and the POMNet baseline whose average-pooling prompt feature extraction is ablated.","marker":"Xu et al., 2022a"},{"why":"It is the strongest visual-prompt baseline on MP-100 that the method must beat in the 1-shot and 5-shot comparisons.","marker":"Shi et al., 2023"},{"why":"It defines the textual-prompt evaluation on AP-10K and provides the CLAMP baseline for cross-species detection.","marker":"Zhang et al., 2023c"},{"why":"It provides the keypoint semantic labels used for MP-100 semantic-understanding accuracy and the X-Pose baseline on unseen AnimalPose.","marker":"Yang et al., 2023e"},{"why":"It supplies the textual-coordinate decoding approach that replaces the special-token regression head and is credited with simplifying scaling.","marker":"Wang et al., 2024a"},{"why":"It is the upgraded visual encoder whose replacement of CLIP is credited for improved seen and unseen accuracy.","marker":"Oquab et al., 2023"},{"why":"It is the base multimodal LLM and language-modeling training paradigm that the framework is built on.","marker":"Liu et al., 2023b"},{"why":"It is the preliminary model ablated in the comparison, isolating the gains from DINOv2, textual decoding, and dataset scaling.","marker":"Yang et al., 2024b"},{"why":"It is the supposedly unseen dataset whose 20 keypoints and five animal categories define the generalization test.","marker":"Cao et al., 2019"}],"fun_headline_variants":["LLM pinpoints keypoints with identify-then-detect reasoning","KptLLM++ reads and writes keypoints from text or images","One model for any object's keypoints, beating specialists","Chain-of-thought guides LLM to precise keypoint localization","From semantics to pixels: LLM masters keypoint detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generalization result assumes that the AnimalPose categories are genuinely unseen, but the training pool includes animal datasets that plausibly already contain dogs, cats, cows, horses, and sheep.","fun_headline_variants_meta":{"raw":{"variants":["LLM pinpoints keypoints with identify-then-detect reasoning","KptLLM++ reads and writes keypoints from text or images","One model for any object's keypoints, beating specialists","Chain-of-thought guides LLM to precise keypoint localization","From semantics to pixels: LLM masters keypoint detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000963,"raw_usage":{"total_tokens":4129,"prompt_tokens":1005,"completion_tokens":3124,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":3037}},"tokens_in":621,"tokens_out":3124,"duration_ms":23825,"temperature":1.0,"reasoning_tokens":3037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:17:00.708164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on the same eight datasets after deleting every training image labeled dog, cat, cow, horse, or sheep, then measure PCK@0.2 on AnimalPose; if it stays near 79.2 the generalization claim holds, and if it falls toward the 59.1 no-scaling level or below X-Pose's 73.4, the claimed unseen-category gain is an artifact of category overlap.","supporting_citations":[],"review_version":1}