{"id":"63388ef7-8884-4e90-83b3-ad2a1af03fc2","arxiv_id":"2502.09460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MET-POSE uses metamorphic rules to test pose-estimation systems without ground-truth labels, and on Mediapipe Holistic it detects faults at similar or higher rates than classic labeled testing.","lead":"This paper introduces MET-POSE, a method to test AI pose-estimation systems without hand-labeled data by checking whether the system stays consistent when the input image is altered, for example, blurred, mirrored, or recolored. It shows the method finds failures in Google's Mediapipe Holistic system at rates comparable to or higher than classic testing on labeled data, which matters because labeling poses is expensive and application-specific.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rule violations are counted as faults without validation; violating a metamorphic relation on a transformed image is not shown to be an actual pose error, and the classic-testing baseline does not test the same inputs.","rationale":"Reader's weakest assumption matches the core issue: metamorphic relation validity. I agree, and the threat is load-bearing because the paper's central claim is empirically about fault detection rate. The evidence in Figures 4 and 6 compares percentages of images with violations/failures, but violations are never checked against labels on transformed images. The explicit exclusion of AllRels as 'artificially high error rates' shows the authors themselves did not trust all rules, but no independent validation replaces that trust. If violations are false positives, the similarity/higher rate claim collapses into 'inconsistency under perturbation' rather than fault detection. A fair comparison with classic testing would require ground-truth labels on the same transformed inputs and identical landmark subsets. The conditional verdict is right; it should remain conditional pending validation. No additional concern about SUT diversity is needed to see this.","tokens_in":19864,"tokens_out":6350,"duration_ms":66851,"concrete_test":"On 100 randomly sampled FLIC images per SubRel configuration at threshold 0.2, have annotators label keypoints on both original and transformed images; recompute the SUT error against these transformed-image labels and compare with MET-POSE violations using the same Err_lmks and threshold. If most violations are genuine large errors and most non-violations are accurate on transformed images, the concern is resolved; if violations frequently occur where the SUT is accurate or where annotators cannot identify the subject, the reported failure rates are inflated. For RQ2, additionally recompute classic testing on the same transformed images (not just originals) and with the same keypoint subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MET-POSE's headline claim assumes a one-to-one correspondence between a metamorphic violation and a genuine fault of the SUT. That correspondence is not established. In Section V-D, Err_lmks uses the SUT's own keypoints on the original image to build the expected keypoints on the transformed image, so a system that is wrong but transformation-invariant passes, while an extreme transformation that makes the subject hard to detect is recorded as an infinite violation even though a correct system might also fail. The paper's own post-hoc selection of SubRels, after discarding AllRels configurations with 'artificially high error rates' (Section V-D), is evidence that rule validity is context-dependent and was not independently checked. Consequently, the RQ2 comparison with classic testing is not on equal terms: classic testing measures SUT error against human labels on original images only, whereas MET-POSE measures consistency between original and modified outputs. A higher failure rate under this comparison can reflect sensitivity to perturbations rather than genuine faults, so 'uncover faults at a similar or higher rate than classic testing' is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MET-POSE, a metamorphic testing framework for pose estimation systems that checks whether a system's keypoint outputs are consistent under image transformations (spatial, image-quality, and colour-space changes), thereby avoiding the need for hand-labelled ground truth. The framework is evaluated on MediaPipe Holistic using the FLIC and PHOENIX datasets, with the error metric Err_lmks and a user-configurable threshold. The experiments report violation rates as a function of threshold, compare MET-POSE with ground-truth-based testing on FLIC, and analyse subsumption relationships among metamorphic rules. The central claim is that MET-POSE can uncover faults at a similar or higher rate than classic testing using hand-labelled data.","tokens_in":20024,"tokens_out":3113,"duration_ms":33023,"significance":"If the central claim were fully supported, MET-POSE would be a valuable contribution: it addresses the oracle problem and the high cost of pose-keypoint labelling, it is presented in a system- and dataset-agnostic way, and it gives practitioners a tunable mechanism for testing pose estimators under application-relevant conditions. The paper also has concrete strengths: the framework is described formally enough to reimplement, the companion repository is referenced, the identity rule is used as a determinism sanity check, and the analysis of subsumption rates is an informative way to understand which transformations contribute distinct information. However, the headline comparison with classic testing is currently under-supported because metamorphic-rule violations are treated as faults without independent validation, and because the two testing methods measure different quantities on different inputs.","major_comments":[{"comment":"The mapping from a metamorphic-rule violation to a genuine fault of the SUT is not established. In Err_lmks, the expected keypoints on the modified image are derived from the SUT's own keypoints on the original image, so a system that is consistently wrong but transformation-invariant will pass MET-POSE, while an extreme perturbation that prevents detection is counted as an infinite violation even if a correct pose estimator would also fail to detect the subject. The paper therefore needs either to validate each relation on a trustworthy reference system or on a small set of manually labelled transformed images, or to reframe MET-POSE as measuring robustness/sensitivity rather than fault detection.","section":"Section V-D"},{"comment":"The post-hoc selection of SubRels is load-bearing and is presented as a fix for configurations that produce 'artificially high error rates'. This is an admission that some of the proposed relations are not necessary properties of correct behaviour in the tested contexts. The restriction from AllRels to SubRels is a free parameter that can inflate the apparent effectiveness of the method; the paper should report how the selection was made, justify it with external evidence, and show that the selected rules remain valid on held-out data or on a reference implementation.","section":"Section V-D and Table I"},{"comment":"The RQ2 comparison with classic testing is not on equal terms. Classic testing measures the SUT's error against human labels on the original image, whereas MET-POSE measures consistency between outputs on original and transformed images. A higher failure rate under MET-POSE can reflect sensitivity to input perturbations rather than the presence of genuine pose-estimation faults, so the abstract's claim that MET-POSE 'uncover[s] faults ... at a similar or higher rate than classic testing' is not directly supported by the plotted overlap and difference percentages. A meaningful comparison requires a fault-level definition that relates transformations to ground-truth error, or at least a manual validation of which rule violations correspond to incorrect keypoints.","section":"Section VI-B and Figure 6"},{"comment":"The empirical results are presented without confidence intervals or significance tests. Figures 4-6 compare percentages on a single SUT and two datasets, and RQ2.2 draws conclusions about overlap of failing images, yet no test such as McNemar's test is applied. Given that the central quantitative claim is about rates of fault discovery, the paper should either provide appropriate statistical support or explicitly present the results as descriptive and limit the strength of the conclusions accordingly.","section":"Sections V and VI"}],"minor_comments":[{"comment":"'Mediapipe Hollistic' is a typo for 'MediaPipe Holistic'.","section":"Section V-B"},{"comment":"The footnotes for the colour-fill values are rendered ambiguously: the superscript markers 'a' and 'b' are not clearly tied to the table entries, and the sentence 'close to skin colour on phoenix dataset' lacks a period. Please clarify the notation.","section":"Table I"},{"comment":"In the Internal Validity paragraph, the sentence beginning 'We have also added an identity metamorphic relation...' is incomplete and should be rephrased, for example: 'If the identity relation were violated, we would need to run the experiments multiple times to reach a statistically significant conclusion.'","section":"Section VIII"},{"comment":"The subsumption heatmaps are very hard to read because the axis labels are small and dense; consider using a colour scale with larger fonts and abbreviating the rule names in a legend.","section":"Figure 7"},{"comment":"The first column label '# failed rules' is potentially confusing because the rows are ranges of counts; please rename it to something like 'Number of failed rules per image' and add a note explaining that each row gives the number of images falling into that range.","section":"Table II"},{"comment":"Reference [17] has a typo in the venue name ('in MET)' instead of a proper workshop name); please correct it.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the framework is potentially useful, but the main empirical claim is currently over-stated. The authors should be given the opportunity to validate the metamorphic relations (or soften the fault-detection claim), re-frame the RQ2 comparison, and add basic statistical support. I would not recommend rejection, as the central idea is sound and the shortcomings appear addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MET-POSE is a practical extension of metamorphic testing to pose estimation. Not a big theoretical leap, but it ships a general framework, a broad rule set, and a level of transparency that many SE-for-ML papers lack. The strongest parts are the subsumption analysis across rules and the honest threats-to-validity section.\n\nWhat is actually new: prior MT work on image classifiers and hand-pose robustness exists, but this generalizes to full pose estimation with user-configurable rules and error metrics, and shows how to tailor rule sets to an application. The code is in a public repo, the experiments look reproducible, and the identity sanity check is a nice touch.\n\nThe soft spot is the one the stress-test flags: the RQ2 comparison is not on equal terms. Classic testing measures error against human labels on original images. MET-POSE measures agreement between outputs on original and transformed images. A violation means the system is not invariant under a transformation; it does not necessarily mean the keypoints are wrong on either image. For transformations like extreme brightness or motion blur, a correct system might also fail to detect the subject, so the invariant is not an independently validated property of correctness. The paper's own post-hoc selection of SubRels, dropping AllRels configs that give 'artificially high error rates', is a hint that several proposed relations are not valid invariants in all settings. This does not kill the framework - metamorphic testing can still catch real inconsistencies - but it means the abstract's wording ('uncover faults at a similar or higher rate than classic testing') outruns the evidence.\n\nMinor points: no confidence intervals or significance tests; only one SUT; the FLIC keypoint mapping is described but not deeply analyzed. These are minor relative to the main concern.\n\nWho this is for: SE-for-ML researchers and practitioners who want a no-extra-labeling way to stress-test pose estimators. A reader should treat reported failure rates as 'inconsistency rates' until the relations are validated per domain.\n\nRecommendation: send it to peer review. The framework is useful, the evaluation is transparent, and the core concern is addressable by (a) validating each metamorphic rule's invariant on a small manually labeled sample and (b) re-framing RQ2 as sensitivity analysis rather than fault detection. A good referee should ask for those changes, but desk rejection would be wrong.","headline":"A solid, transparent metamorphic-testing framework for pose estimation; the fault-rate comparison with classic testing overstates what the experiments show, but the core idea is useful and worth refereeing.","tokens_in":20589,"tokens_out":2167,"would_cite":true,"duration_ms":20760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that metamorphic testing can reveal faults in pose estimation systems as effectively as hand-labelled ground-truth testing, without the labelling cost.","keywords":["metamorphic testing","pose estimation","oracle problem","ground truth free testing","Mediapipe Holistic","metamorphic relations","computer vision testing"],"falsifier":"Run a rule on images with known ground-truth keypoints for both the original and transformed versions; if the rule reports violations on many images where the system's keypoints are accurate on both versions, the rule is not a reliable oracle, and if, conversely, every violated image has a large ground-truth error, the framework's fault signal is confirmed.","tokens_in":19625,"feed_emoji":"🧪","tokens_out":6234,"duration_ms":51124,"temperature":0.7,"pith_summary":"The paper proposes MET-POSE, a metamorphic testing framework that tests pose estimation systems without needing manually labelled ground-truth keypoints. It claims that a pose estimator can be checked by transforming input images (mirroring, stretching, blurring, changing brightness or colour) and verifying that the output keypoints change exactly as the transformation dictates; any violation counts as a fault. The authors apply MET-POSE to Mediapipe Holistic on the FLIC and PHOENIX datasets and report that it uncovers faults at a rate similar to, and for higher error thresholds greater than, classic ground-truth-based testing. If this holds, practitioners can test pose estimators on their own application-specific images at a fraction of the labelling cost, and can tune which metamorphic rules to use for the failure types they care about.","feed_headline":"Pose-estimation faults found without hand-labeled data","feed_subtitle":"Rule violations under blur, mirror, and color shifts spot pose-estimation faults without labels.","key_machinery":"The central object is the metamorphic rule: a pair (transformation, relation), where the transformation produces a modified image and the relation states what the keypoint outputs on the two images must satisfy for the system to behave correctly—for example identity for brightness or blur changes, mirroring for a mirror transformation, and rotation for a rotated image. The severity of a violation is computed by $\\mathrm{Err}_{\\mathrm{lmks}}$, the median of per-landmark normalized Euclidean distances (using Mediapipe's shoulder, iris, and wrist normalizations), with an infinite error when keypoints are detected on only one of the two images. This mechanism carries the argument because it converts an unanswerable question, 'are these output keypoints correct?', into a checkable one, 'did the outputs change the way a correct system would change them?'.","core_discovery":"On the paper's own terms, the central discovery is that the oracle problem for pose estimation can be bypassed: instead of knowing where the keypoints should be, one only needs to know how keypoint positions should change when the input image is transformed. MET-POSE formalizes each check as a metamorphic rule pairing a transformation with a relation, evaluates violations with an error metric that returns the median normalized distance between expected and detected keypoints (and an infinite error when keypoints disappear on one side of the pair), and then reports violations and their severity. The experiments on Mediapipe Holistic show that the framework finds large numbers of rule violations on both datasets, that on FLIC the violations overlap with, and at higher thresholds exceed, the failures found by comparing against the hand-labelled ground truth, and that different rules are not subsumed by one another, so different rules surface different types of faults.","pith_inferences":["A testable extension is to calibrate each metamorphic rule against a small labelled sample: compute how often rule violations agree with ground-truth failures on that sample, and drop rules with low agreement, which would filter out invariants that are not truly necessary for correctness.","The paper's own observation that some relation configurations produced artificially high error rates suggests error thresholds must be set per rule rather than globally; practitioners could derive per-rule thresholds from the distribution of violation severities on a trusted subset of images.","The pattern of which single rule an image violates could serve as a debugging signal for the model: images that violate only a colour rule point to colour over-reliance, images that violate only a blur rule point to texture dependence, and so on, turning the framework from a test oracle into a diagnostic tool.","For video-based pose estimation, temporal metamorphic relations—such as keypoint consistency under frame-rate changes or frame reordering—would extend the framework beyond the single-image rules studied here."],"forward_implications":["Users can run MET-POSE on unlabelled images from their own application domain and still obtain a fault signal, so domain shift no longer forces a new labelling effort.","At higher error thresholds the framework reports more failures than classic ground-truth testing on FLIC, so it can act as a complement that catches faults labelled testing misses.","Because rule subsumption is low and dataset-dependent, a practical test campaign should include several rule families rather than rely on one transformation.","PHOENIX has no ground-truth keypoints, so MET-POSE provides the only fault signal there; this shows testing is possible in settings where classic testing cannot be applied.","The same framework extends to any pose estimator and any landmark set by swapping the transformations, relations, and error metric, without changing the underlying testing logic."],"supporting_citations":[{"why":"Defines the oracle problem that motivates label-free testing of pose estimation systems.","marker":"[13]"},{"why":"Supplies the metamorphic-testing approach for ML image classifiers that MET-POSE adapts to pose estimation.","marker":"[15]"},{"why":"Is the closest prior metamorphic-testing study on hand pose estimation that the paper extends and contrasts with.","marker":"[17]"},{"why":"Is the pose estimation system under test, Mediapipe Holistic.","marker":"[4]"},{"why":"Describes BlazePose GHUM 3D, the underlying model of the system under test.","marker":"[25]"},{"why":"Provides the PHOENIX dataset, which has no ground-truth keypoints and demonstrates testing where classic testing cannot run.","marker":"[27]"},{"why":"Provides the FLIC dataset and its ground-truth keypoints used for the comparison with classic testing.","marker":"[28]"},{"why":"Defines the normalized distances used in the error metric.","marker":"[30]"}],"fun_headline_variants":["Metamorphic rules find pose errors without labeled data","Testing pose estimation without ground truth: metamorphic approach","No labels needed: metamorphic testing catches pose faults","From blurred to mirrored: label-free pose testing via metamorphic rules","Metamorphic testing exposes pose errors without manual labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For each metamorphic rule to be a valid fault detector, the relation it encodes must be a necessary property of any correct pose estimator; if a correct system could legitimately lose track of a subject under an extreme transformation, the rule's violation is a property of the transformation, not a fault of the system.","fun_headline_variants_meta":{"raw":{"variants":["Metamorphic rules find pose errors without labeled data","Testing pose estimation without ground truth: metamorphic approach","No labels needed: metamorphic testing catches pose faults","From blurred to mirrored: label-free pose testing via metamorphic rules","Metamorphic testing exposes pose errors without manual labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2581,"prompt_tokens":1022,"completion_tokens":1559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":1480}},"tokens_in":638,"tokens_out":1559,"duration_ms":10509,"temperature":1.0,"reasoning_tokens":1480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:22:31.311406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a rule on images with known ground-truth keypoints for both the original and transformed versions; if the rule reports violations on many images where the system's keypoints are accurate on both versions, the rule is not a reliable oracle, and if, conversely, every violated image has a large ground-truth error, the framework's fault signal is confirmed.","supporting_citations":[{"cited_title":"The oracle problem in software testing: A survey,","cited_arxiv_id":null,"evidence_quote":"Defines the oracle problem that motivates label-free testing of pose estimation systems."},{"cited_title":"Identifying implementation bugs in machine learning based image classifiers using metamorphic testing,","cited_arxiv_id":null,"evidence_quote":"Supplies the metamorphic-testing approach for ML image classifiers that MET-POSE adapts to pose estimation."},{"cited_title":"Robustness evaluation in hand pose estimation models using metamorphic testing,","cited_arxiv_id":null,"evidence_quote":"Is the closest prior metamorphic-testing study on hand pose estimation that the paper extends and contrasts with."},{"cited_title":"MediaPipe Holistic — Simultaneous Face, Hand and Pose Prediction, on Device,","cited_arxiv_id":null,"evidence_quote":"Is the pose estimation system under test, Mediapipe Holistic."},{"cited_title":"Neural sign language translation,","cited_arxiv_id":null,"evidence_quote":"Provides the PHOENIX dataset, which has no ground-truth keypoints and demonstrates testing where classic testing cannot run."},{"cited_title":"Modec: Multimodal decomposable models for human pose estimation,","cited_arxiv_id":null,"evidence_quote":"Provides the FLIC dataset and its ground-truth keypoints used for the comparison with classic testing."},{"cited_title":"Model Card: MediaPipe BlazePose GHUM 3D,","cited_arxiv_id":null,"evidence_quote":"Defines the normalized distances used in the error metric."}],"review_version":1}