{"id":"360c13ac-1da3-425e-8fef-3eb1d0f22c84","arxiv_id":"2505.16931","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PIIvot anonymizes tutoring dialogues by labeling potential PII and generating context-consistent surrogate replacements, enabling release of the QATD2k dataset without materially changing basic text statistics.","lead":"This paper introduces PIIvot, an anonymization pipeline that labels possible personal information in tutoring chats and then replaces it with realistic fake names and places using an LLM. It also releases QATD2k, a large real-world math tutoring dialogue dataset, and reports that anonymization barely changes the text statistics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth in §2.3 is built by resolving machine–annotator discrepancies, so Table 2's F1 may be circular; a blind re-annotation is needed.","rationale":"The reader's conditional verdict is appropriate, and my concern does not move it. The most load-bearing issue is not the HIPS assumption alone, although that is real and honestly acknowledged; it is that the paper's central quantitative evidence—Table 2's detection metrics—may be inflated by a circular ground-truth construction. The paper does a number of things well: it releases the dataset, documents consent and filtering, and reports inter-rater reliability. But Section 2.3's ambiguity about how machine–annotator discrepancies were resolved means the reported F1 cannot be taken as an independent estimate of detection quality. A blind re-annotation study would settle whether the metrics are trustworthy; the reader's HIPS concern remains important for downstream utility but is secondary to the privacy-relevant detection claim. I therefore recommend no change to the conditional verdict, with the condition strengthened to require the blind re-annotation check.","tokens_in":11395,"tokens_out":10126,"duration_ms":78941,"concrete_test":"Draw a random sample of 500 dialogues (~10% of QATD2k), have two annotators independently label potential PII from the raw pre-anonymization text with no access to PIIvot's predictions or to the existing gold labels, and reconcile disagreements with a third blind adjudicator. Recompute PIIvot's precision/recall against this blind gold set. If the dialogue F1 falls more than ~0.02 below the 0.984 reported in Table 2, the discrepancy-resolution protocol has materially inflated the headline metrics and the residual-privacy risk should be re-estimated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 says the evaluation ground truth was produced by 'resolving' discrepancies between the machine's labels and annotator labels. If the PIIvot model's predictions influence adjudication, model false positives can be converted into true positives in the final gold set, inflating precisely the precision/recall numbers in Table 2 (dialogue F1=0.984, question F1=0.820) that serve as the main quantitative evidence for the framework's detection step. The paper does not specify who resolved the discrepancies, whether they were blind to the model's outputs, or how many disagreements were settled in the model's favor. An inflated detection estimate weakens the privacy guarantee of the released QATD2k, which is the paper's headline artifact. A secondary gap is that Table 1 only reports surface text statistics (word counts, n-gram entropy, uptake), which do not test the Hidden-In-Plain-Sight assumption behind the data-integrity claim; but the circular ground-truth issue is the more load-bearing problem because it undermines the core quantitative result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents PIIvot, a two-stage anonymization framework. The first stage uses a fine-tuned DeBERTa model to label potential-PII spans with a recall-first objective; the second uses GPT-4o to replace those spans with context-consistent surrogates under a 'Hidden-In-Plain-Sight' assumption. As a case study, the authors release QATD2k, about 1,971 real tutoring dialogues with metadata and talk-move labels, claiming it is the largest open-source real-world tutoring dataset of its kind. They report micro-averaged dialogue F1=0.984 and question F1=0.820 for the detection stage, a held-out model F1=0.94, and surface text statistics suggesting minimal impact of anonymization. The central claim is that reframing PII detection as potential-PII labeling, followed by context-aware surrogate replacement, enables practical anonymization at scale for educational dialogue data.","tokens_in":11599,"tokens_out":8528,"duration_ms":67764,"significance":"Strengths: the dataset and code are publicly released, the NER model is evaluated on an independent held-out test set (Table 3), the annotation codebook shows high inter-rater reliability (minimum weighted F1 of 0.98), and the ethics and limitations sections are candid about residual PII risk and the domain dependence of the Hidden-In-Plain-Sight assumption. If the detection and replacement claims are validated, PIIvot would be a practical recipe for opening sensitive educational dialogue data, and QATD2k would fill a real gap in real-world tutoring benchmarks. The main concerns below concern the validity of the headline detection numbers and the evidence for the data-integrity claim; both are addressable without changing the scope of the paper.","major_comments":[{"comment":"The evaluation ground truth for Table 2 is established by 'resolving discrepancies between the machine and annotator labels' (§2.3). If the PIIvot model's predictions are visible to or influence the adjudicators, then the model's false positives can be converted into gold labels, mechanically inflating the reported precision/recall (dialogue F1=0.984, question F1=0.820) and making the annotator row in the same table a resubstitution estimate rather than an independent benchmark. The manuscript does not say who resolved the discrepancies, whether they were blind to the model outputs, or how many disagreements were settled in the model's favor. Please add a blind re-annotation of a random sample in which gold labels are defined by annotator agreement alone (or by an independent adjudicator blind to machine labels), report inter-annotator agreement on the new sample, and quantify how often the machine label disagreed with the final gold label. Without this, the core detection result and the privacy guarantee for QATD2k are not established.","section":"§2.3, Table 2"},{"comment":"The claim that anonymization preserves data integrity and has little to no effect on text-based metrics is supported only by surface statistics in Table 1, namely words per turn, n-gram entropy, and turn uptake. These metrics do not test whether surrogate replacements preserve semantic coherence, consistency of named entities across a dialogue, or the mathematical content of word problems. The paper itself states that the Hidden-In-Plain-Sight assumption 'is not true across many domains' (§5.1), which is an honest limitation; however, for QATD2k the assumption is asserted rather than validated. Please add at least one content-level validation, for example a small downstream reading or math comprehension probe before and after anonymization, human judgments of coherence and replacement consistency, or a check that all references to a replaced entity remain aligned in the released dataset. This is load-bearing because data utility is a central part of the contribution.","section":"§2.1.2, §5.1, Table 1"}],"minor_comments":[{"comment":"The filtering thresholds (at least 20 total messages and 7 from either participant, at most 8 dialogues per DQ, and a 1000 unique DQ cap) are presented without rationale; a sentence justifying each choice would improve replicability.","section":"§2.2.1"},{"comment":"The 'No PIIvot' row is undefined; clarify whether it is the same dialogues before replacement, a separate baseline, or a re-computation on pre-anonymized text, and report the number of dialogues in that row.","section":"Table 1"},{"comment":"The <Getting Students to Relate> F1 is 0.0000 in the validation set; while the paper excludes this label from downsampling and reports metrics without it, the talk-move labels are part of the released dataset, so the dataset documentation should flag this label as unreliable for 1:1 tutoring.","section":"Appendix B, Table 5"},{"comment":"In the final paragraph of Section 5.2, 'we acknowledge the positionally of the authors' appears to be a typo for 'positionality'.","section":"§5.2"},{"comment":"The reference entry 'Commission and others' is not a proper author name; replace it with 'Federal Trade Commission' and the full report title and year.","section":"References"},{"comment":"The abstract uses 'QATD-2k' while the rest of the paper uses 'QATD2k'; please unify the notation throughout.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The circular ground-truth issue in §2.3 is the main reason for major revision; it is fixable with a blind re-annotation study, which is a standard and proportionate request. The dataset release is valuable and the paper is otherwise within scope for an applied NLP/EDM venue. I would not reject; the revision should focus on de-circularizing the detection evaluation and adding a content-level validation of the data-integrity claim. I would also gently encourage the authors to add at least one baseline PII detector comparison, since the 'lighter-weight' framing would be much stronger with a direct quantitative contrast."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, QATD2k is a genuinely useful new resource: nearly two thousand real human-human math tutoring dialogues with question metadata, tutor consent, and talk-move labels. That alone justifies a look. Second, the paper's quantitative case for the PIIvot detection step is weaker than it appears. Section 2.3 says the evaluation ground truth was produced by resolving discrepancies between the machine's labels and the annotators' labels. The paper never says who resolved those disagreements, how many there were, whether the resolver was blind to the model's output, or how often the model's judgment won. Without that, Table 2's dialogue F1 of 0.984 is not trustworthy as reported. The separate held-out model test in Table 3 is on a different 40k-utterance corpus, so it does not rescue the dialogue numbers.\n\nWhat the paper does well: the framework is a sensible domain-specific combination of recall-first NER and Hidden-In-Plain-Sight surrogate generation, and the authors are clear that the HIPS assumption does not hold everywhere. The codebook work and the contextual error analysis for the talk-move classifier show real care. They also ship the code and the dataset, which is exactly the kind of reproducible output that deserves credit.\n\nThe soft spots beyond the circularity: there is no head-to-head comparison with existing PII detection tools, and no downstream-task check of whether the anonymized data still supports tutoring-related modeling. The HIPS assumption is acknowledged as domain-specific, so that is a stated limitation rather than a hidden one. The talk-move label issue with 'Getting Students to Relate' is disclosed with an error analysis, which is honest.\n\nNet: the dataset is worth having, and the framework is a reasonable applied contribution. But the privacy claim depends on detection quality, and the current evaluation protocol is too inside-out to take the detection numbers at face value. A blind re-annotation of a sample, or at least a detailed description of the adjudication process, is needed before recommending the framework for high-stakes anonymization.\n\nFor whom: educational dialogue researchers, privacy-preserving NLP people, and anyone thinking about releasing real-world tutoring data. It deserves serious peer review, but with a request for a cleaner evaluation. I would bring it to our reading group and would cite the dataset in future work, while being careful not to repeat the circular evaluation claim.","headline":"Useful new tutoring dataset, but the detection F1 rests on a ground truth the model helped shape.","tokens_in":12123,"tokens_out":1923,"would_cite":true,"duration_ms":17578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PIIvot reframes PII detection as potential-PII labeling and uses LLM-generated, conversation-consistent surrogate replacements to anonymize a large real-world tutoring dataset without meaningfully changing its text statistics.","keywords":["anonymization","personally identifiable information","named entity recognition","LLM surrogate generation","tutoring dialogue","educational data sharing","math word problems","open dataset"],"falsifier":"Run a downstream tutoring-quality evaluation (for example, word-problem solving accuracy or human ratings of tutor helpfulness) on the original versus PIIvot-anonymized dialogues: if the anonymized version performs materially worse, the Hidden-In-Plain-Sight assumption fails for this domain. A complementary check is a linkage attack: if surrogate names combined with external records recover a student or tutor identity, the privacy guarantee is broken.","tokens_in":11198,"feed_emoji":"🛡️","tokens_out":11814,"duration_ms":90247,"temperature":0.7,"pith_summary":"This paper proposes that the hard part of anonymizing educational dialogue—deciding which spans truly identify a person—can be dropped when the data's purpose is known. PIIvot instead labels every entity that might be PII (names, locations, schools, emails, phones, URLs, birth dates) and uses an LLM to replace those spans with realistic fake entities that stay consistent across the conversation. On 1,971 real math-tutoring dialogues from a UK-based platform, the pipeline produced QATD2k, which the authors describe as the largest open-source real-world tutoring dataset of its kind, while leaving words-per-turn, entropy, and conversational uptake essentially unchanged. The authors' aim is to show that a recall-first 'potential-PII' label set plus Hidden-In-Plain-Sight surrogate replacement is a practical path for open-science data sharing in domains where entity content carries no task meaning.","feed_headline":"Potential-PII labeling opens a 1,971-dialogue tutoring dataset.","feed_subtitle":"Realistic fake names and places keep 1,971 real tutoring dialogues open for research.","key_machinery":"The load-bearing mechanism is two-step: (1) recall-first potential-PII labeling—an NER system that deliberately over-labels entities rather than judging whether a span is truly identifying, using an inside/outside (IO) token scheme with a centered context window and punctuation cleaning; and (2) Hidden-In-Plain-Sight (HIPS) surrogate generation—an LLM creates a mapping from original spans to fake but contextually appropriate replacements, conditioned on the entire chat so the same entity gets the same surrogate throughout, with feedback-based reprompting enforcing that replacements differ enough from originals and preserve requested qualities. This mapping, kept consistent across turns, is what preserves referential coherence while removing identifying content.","core_discovery":"The central claim is that anonymization for question-anchored tutoring can be decomposed into two simpler steps without sacrificing utility. First, a fine-tuned DeBERTa NER model labels spans with word-level IO tags for seven entity types, treating over-detection as acceptable—hence 'potential-PII' rather than PII. Second, GPT-4o builds a per-dialogue mapping from each original span to a surrogate, conditioned on the full chat history and on qualities the prompt asks it to preserve (e.g., gender and ethnic background), with feedback-based reprompting to ensure the replacement is materially different; emails and URLs are obfuscated directly. The empirical demonstration is QATD2k: 1,971 dialogues and 46,249 turns collected from real tutor-student interactions, moderated, tutor-consented, and downsampled by talk-move diversity. On this dataset the anonymized text matches the original on the paper's text-based metrics, and potential-PII detection reaches 0.984 micro-F1 on dialogues, though only 0.820 on LaTeX-formatted question text, where the authors fall back on human annotation.","pith_inferences":["The paper measures utility only with surface text statistics; a stronger test it does not run is downstream tutoring quality (e.g., whether a tutor model's responses or student outcomes change on anonymized data), and that is the natural next experiment.","The authors concede that question text (LaTeX) defeated their NER model and required human annotation; a format-aware variant or a policy of not anonymizing question metadata (which carries no student PII) could remove that bottleneck at scale.","The paper promises within-dialogue surrogate consistency but does not analyze cross-dialogue entity frequency; a practical extension would enforce globally unique surrogates or measure whether repeated names across the corpus enable linkage attacks.","The surrogate generator is a proprietary API, and the authors themselves warn about prompt logging; a self-hosted open-weights LLM with the same prompt-and-reprompt loop would test whether the method is reproducible without commercial dependence."],"forward_implications":["Other text domains where entity content is decorative—fictional assessment items, templated exercises, simulated scenarios—can adopt the same two-step pipeline and expect similar utility preservation.","Dataset publishers can treat PII detection as a recall-first labeling task rather than a precision-sensitive classification task, lowering the cost and raising the recall of anonymization pipelines.","QATD2k provides real human-to-human tutoring dialogues with talk-move labels, question metadata, and train/test splits, giving modelers an alternative to synthetic or crowd-sourced tutoring data for training and benchmarking.","Because PIIvot keeps surrogates consistent within a conversation, downstream models retain cross-turn reference coherence, which is the main reason text-based metrics stay flat."],"supporting_citations":[{"why":"is the source of the HIPS idea: realistic surrogates reduce exposure of protected information in clinical text, which PIIvot adapts to tutoring data.","marker":"Carrell et al., 2013"},{"why":"documents the precision/recall limits of PII detection on student essays and is the comparison for the potential-PII labeling claim.","marker":"Buchh, 2024"},{"why":"represents the redaction-based anonymization approach in education that PIIvot moves beyond.","marker":"Bosch et al., 2020"},{"why":"shows LLM-based de-identification of student PII and argues that separating PII from non-PII is hard, motivating potential-PII labels.","marker":"Singhal et al., 2024"},{"why":"demonstrates rules-and-transformers de-identification for student writing, grounding PIIvot's span-labeling approach.","marker":"Holmes et al., 2023"},{"why":"provided the talk-move classifier whose labels drive the diversity-based downsampling to QATD2k.","marker":"Moreau-Pernet et al., 2024"},{"why":"motivates the filtering thresholds (20 total messages, 7 from one participant) used to select candidate sessions.","marker":"Chen et al., 2019"},{"why":"contributes MathDial and frames entropy as a dataset-quality metric, the preservation of which supports the utility claim.","marker":"Macina et al., 2023a"}],"fun_headline_variants":["PIIvot: Lightweight anonymization for tutoring dialogues","Potential-PII labeling unlocks 1,971-tutoring-dialogue dataset","DeBERTa + GPT-4o anonymize tutoring chats in two-step PIIvot","QATD-2k: Open dataset with 1,971 real tutoring dialogues anonymized","Lightweight PIIvot framework anonymizes tutoring dialogues for open science"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the names, locations, schools, and similar labeled entities in these dialogues carry no meaning for the task the data will serve, so replacing them with consistent fakes cannot degrade dataset quality; the authors state this is true for their math word problems but not for many other domains.","fun_headline_variants_meta":{"raw":{"variants":["PIIvot: Lightweight anonymization for tutoring dialogues","Potential-PII labeling unlocks 1,971-tutoring-dialogue dataset","DeBERTa + GPT-4o anonymize tutoring chats in two-step PIIvot","QATD-2k: Open dataset with 1,971 real tutoring dialogues anonymized","Lightweight PIIvot framework anonymizes tutoring dialogues for open science"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1363,"prompt_tokens":888,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":504,"tokens_out":475,"duration_ms":3776,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:52:17.856620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a downstream tutoring-quality evaluation (for example, word-problem solving accuracy or human ratings of tutor helpfulness) on the original versus PIIvot-anonymized dialogues: if the anonymized version performs materially worse, the Hidden-In-Plain-Sight assumption fails for this domain. A complementary check is a linkage attack: if surrogate names combined with external records recover a student or tutor identity, the privacy guarantee is broken.","supporting_citations":[{"cited_title":"Hello , [ REDACTED ]","cited_arxiv_id":null,"evidence_quote":"represents the redaction-based anonymization approach in education that PIIvot moves beyond."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"demonstrates rules-and-transformers de-identification for student writing, grounding PIIvot's span-labeling approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"motivates the filtering thresholds (20 total messages, 7 from one participant) used to select candidate sessions."}],"review_version":1}