{"id":"812f247e-4bab-4f32-8902-b9e7e018d738","arxiv_id":"2608.09790","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CARD combines planning, controlled generation, and metric-guided revision to produce credit card Reddit threads whose aggregate statistics match real threads better than OASIS and SynthPAI.","lead":"This paper introduces CARD, a system that generates realistic Reddit-style credit card discussions by planning each comment's role, writing the comments, and then revising them until the collection's statistics match real threads. It reports that CARD is closer to real credit card discussions than two existing social simulation baselines across four language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 91.7% non-rejection rate is not probative: CARD calibrates on the same metrics used for evaluation and copies reply structure from the matched real thread, so the headline comparison partly measures target access and stopping-rule behavior rather than generalizable realism.","rationale":"The reader's weakest assumption overlaps with my concern: the calibration loop optimizes toward the same metric distributions used to declare success. I partially agree and sharpen the point by adding two concrete mechanisms not emphasized by the reader: (1) the planner copies reply depth, comment count, and branch structure from the matched real thread, which directly inflates structural metrics such as average reply depth and structural virality; and (2) a non-rejection rate near 91.7% is close to the 95% rate expected under the null, so the headline number is not by itself evidence of realism. This does not change the overall verdict: the framework is a plausible systems contribution, and the baseline comparison is informative, but the central claim should be read as conditional. The paper ships no code or data, and key calibration tolerances (epsilon_t, protected-metric tolerances) are unspecified, which are secondary but reinforce the need for an independent check. The proposed held-out test with pre-registered metrics, including content-level metrics the paper explicitly lists as future work, would settle whether the concern lands. The Future Work section's concession that content-level objectives such as topic coverage, claim alignment, and factual consistency are absent is an explicit limitation that should be weighed in the verdict.","tokens_in":12761,"tokens_out":8176,"duration_ms":85391,"concrete_test":"Design a held-out generalization test: split the 150 seed posts into 100 calibration posts and 50 held-out posts. For held-out posts, give CARD only the seed post (no matched real thread for the planner's structural controls and no calibration on held-out threads or metrics), and compare the resulting held-out distributions against real threads on a pre-registered metric set that includes at least two content metrics not used in calibration (e.g., topic coverage and factual consistency) in addition to the MiroBench metrics. If the non-rejection rate drops substantially below 91.7% or CARD no longer beats SynthPAI on the held-out metric set, the headline result is best explained by target access and calibration overfit rather than by a generalizable simulation capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CARD's central claim (abstract and Table 2) is that generated thread distributions match real ones better than OASIS and SynthPAI. The evaluation is not an independent test of this claim. In §3.3.3, the self-looped calibrator selects the target metric from the same MiroBench metric set used in §4.2, revises comments until the Mann–Whitney U and Kolmogorov–Smirnov tests no longer reject (p > 0.05) or a maximum of seven iterations is reached, and then Table 2 reports non-rejection rates on exactly those metrics. Under the null, p > 0.05 is expected 95% of the time, so the reported 91.7% is close to the trivial chance level and far from an equivalence claim; the paper itself notes that non-rejection does not establish equivalence. More concretely, §3.3.1 says the planner derives comment count, reply depth, and branch structure 'from the matched real discussion,' and §4.2 evaluates average reply depth and structural virality, so the structural component of Table 2 is partly constructed by copying the target. Baselines do not receive the matched real thread, making the comparison unequal. Future Work (§6) concedes content-level realism (topic coverage, factual consistency, claim alignment) is absent. Thus the evidence supports that CARD can be optimized toward a known target distribution, not that it generates realistic credit card discussions in settings where no matched real thread exists.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CARD, a framework for generating simulated Reddit discussion threads about credit cards. Given a seed post and its matched real thread, a planner specifies discussion structure and comment-level roles (function, stance, tone, reasoning move, issue type), a writer generates comments with an LLM, and a self-looped calibrator revises comments to reduce distributional mismatches with the real thread collection across lexical, semantic, behavioral, and structural metrics. The authors evaluate CARD on 150 real Reddit credit-card seed posts against OASIS and SynthPAI baselines, reporting Mann–Whitney U and Kolmogorov–Smirnov p-values, Cliff's delta and Wasserstein distances, and a blind Turing test. The headline result is that CARD achieves non-rejection rates up to 91.7% of metric–test pairs (p>0.05) and smaller distributional distances than baselines.","tokens_in":13057,"tokens_out":3204,"duration_ms":30990,"significance":"If the claims are well supported, CARD would be a useful step toward controllable, realistic simulation of consumer-finance discussions, and the planner–writer–calibrator architecture is a sensible way to inject domain structure into LLM generation. The paper is strong in transparency: the revision loop, protection policy, and rollback mechanism are described in detail, and the evaluation covers multiple backbone models and reports effect sizes in addition to p-values. However, the central evidence is weakened by a calibration–evaluation circularity and by unequal access to the matched real thread across methods, so the significance depends on whether the authors can provide a more independent test of realism.","major_comments":[{"comment":"The self-looped calibrator selects target metrics from the same MiroBench metric set used in the evaluation, revises comments until the Mann–Whitney U and Kolmogorov–Smirnov tests no longer reject (p>0.05) or a maximum of seven iterations is reached, and Table 2 reports non-rejection rates on exactly those metrics. This makes the headline non-rejection rates posterior to fitting the evaluation distribution, not independent evidence of realism. Under the null, p>0.05 is expected with probability 0.95 per test, so the reported 91.7% is close to the trivial chance level. The paper itself correctly notes (§4.2) that non-rejection does not establish equivalence. To substantiate the central claim, the authors should report results on held-out metrics that the calibration loop does not optimize, or otherwise demonstrate that the improvement transfers beyond the calibrated metric set.","section":"§3.3.3, §4.2, Table 2"},{"comment":"The planner derives comment count, reply depth, and branch structure 'from the matched real discussion,' and the evaluation measures average reply depth and structural virality. Consequently, part of the reported structural match is constructed by copying the target thread's structure, while the baselines (OASIS and SynthPAI) do not receive the matched real thread at all. This makes the comparison unequal and weakens the claim that CARD generates realistic discussion structures in a target-free setting. The authors should either provide the same matched-thread information to the baselines or evaluate CARD in a setting where no matched real thread is used for planning.","section":"§3.3.1, §4.2, Table 5"},{"comment":"The Future Work section explicitly concedes that content-level objectives—topic coverage, claim alignment, and factual consistency—are absent from the current framework. The abstract's phrase 'matches the distributions of real credit card discussions' thus overstates the scope of the realism claim; the evidence supports structural, lexical, and stylistic matching, not content-level realism. The conclusions and title should be restricted accordingly, and the evaluation should avoid implying equivalence from p>0.05.","section":"§6, Abstract"}],"minor_comments":[{"comment":"There is a typo in the description of the matching procedure: 'CARD retrieves one matched real with the same about the same financial scenario' should read 'one matched real thread about the same financial scenario.'","section":"§3.3"},{"comment":"The human evaluation description reports '20 matched discussion pairs for each method' and '72 unique discussion pairs across all methods,' but with two comparison methods (OASIS and SynthPAI) one would expect 40 unique pairs; the counting should be clarified.","section":"§4.3, Figure 3"},{"comment":"The matching procedure for selecting the 'matched real thread' for each seed post is not described; because the planner and the evaluator both rely on this matched thread, the authors should specify how matching was performed and report matching quality statistics.","section":"§3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation is undermined by a circularity that the authors themselves partially acknowledge: the calibration loop optimizes the very metrics used for evaluation. In addition, the matched-real-thread privilege given to CARD but not to baselines makes the direct comparison unfair. The central idea is salvageable, but the authors need to either redesign the evaluation with held-out metrics or substantially reframe the claims. I would encourage the editor to ask for a revised version that addresses these points before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CARD is a genuinely integrated system — planner, role-controlled writer, and a self-loop that revises comments to shrink distributional gaps — and it is applied to a domain nobody else has simulated. That part is real and worth a look.\n\nWhat it does well: the control taxonomy (financial function, stance, reasoning move, issue type) is carefully chosen for credit card talk, the writing is clear, and the comparison against OASIS and SynthPAI is informative. The Table 4 breakdown of revision behavior is nice: you can see which backbones actually benefit from the loop and which just burn candidates. The paper is also honest in places: it states that p>0.05 is not equivalence, and it concedes in Future Work that content-level realism (topic coverage, factual consistency, claim alignment) is not modeled.\n\nNow the soft spots, in order of size.\n\nFirst, the main evaluation is partly circular. The self-loop in §3.3.3 targets the same MiroBench metrics used in §4.2, and the acceptance rule is reduction of the target gap. So Table 2's high non-rejection rates are posterior to optimization, not independent evidence. The 91.7% number is not probative: under the null you expect 95% of p-values above 0.05, and the loop is explicitly trying to get there. The effect sizes and Wasserstein distances are computed on the same metrics the loop squeezed, so they inherit the same problem.\n\nSecond, the structural metrics are partly constructed. The planner derives comment count, reply depth, and branch structure from the matched real thread, and the evaluation rewards matching average depth and virality. That means part of the 'match' is copying the target's skeleton. Baselines do not get the matched thread, so the comparison is unfair in that dimension. The paper frames the task as 'given a seed post and its matched real thread,' which is fine as a task definition, but the abstract's 'matches the distributions of real credit card discussions' should be read as 'when given the real thread as scaffolding.'\n\nThird, the independent evidence is thin. The human Turing test uses 20 matched pairs per method, five annotators, and no reported agreement statistics. The matching procedure for choosing the real thread per seed post is unreported. No code or data are released, and calibration tolerances (epsilon_t, protected tolerance) are not specified.\n\nAre these fatal? No. The architecture is sensible and the domain is underserved. The fix is straightforward: evaluate on a held-out metric set not used in the loop, release artifacts, get more human judgments, and include a condition where CARD has to generate structure without the matched thread. I would send this to review, but with the expectation of major revision.\n\nWho is it for? People building LLM social simulations, specifically financial discussion simulation; also anyone comparing agentic generation frameworks. It deserves a serious referee.","headline":"A useful system paper whose headline realism numbers are inflated by optimizing the same metrics it evaluates on; the framework itself is sound and worth building on.","tokens_in":13628,"tokens_out":2950,"would_cite":true,"duration_ms":27861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CARD generates credit card discussion threads whose aggregate statistics are close enough to real Reddit threads that statistical tests find no difference on 91.7% of metric-test pairs, outperforming prior simulators.","keywords":["credit card discussion simulation","LLM agents","distributional realism","discussion thread generation","self-loop calibration","consumer finance communication","Reddit threads"],"falsifier":"Reproduce the evaluation after randomly permuting which real thread is paired with each seed post; if the non-rejection rate stays near 91.7% under this permutation, the claimed match is not actually driven by the matched-pair design. A content-level audit would then be needed, since the benchmark's metrics do not check factual accuracy.","tokens_in":12536,"feed_emoji":"💳","tokens_out":8616,"duration_ms":74889,"temperature":0.7,"pith_summary":"CARD is a framework for simulating online credit card discussions, and the paper claims it produces whole threads whose collective statistics match real Reddit threads far better than existing social simulators. The central evidence is distributional: across lexical, semantic, behavioral, and structural metrics, the generated collection is close enough to the real collection that standard tests fail to detect a difference on 91.7% of metric-test pairs, while the strongest comparison baseline reaches 37.5%. The realism comes from planning each comment's communicative role before any text is written, then revising the entire collection toward real distributions while protecting each comment's planned financial function. If the claim holds, researchers gain a controlled way to study how consumers talk about fees, rewards, eligibility, and trade-offs without running large real-world interventions.","feed_headline":"Simulated credit card threads match real ones on 91.7% of tests","feed_subtitle":"A planner-writer-calibration loop reproduces how consumers argue, recommend, and explain credit cards.","key_machinery":"The load-bearing mechanism is the self-looped population calibrator, a revision loop that works metric by metric over the whole generated collection. In each round it diagnoses which metric shows a significant generated-versus-real difference, designates every other metric as protected, selects the threads and comments with the largest contribution to the gap, samples best-of-N candidate rewrites conditioned on the comment's planned financial role, locally accepts rewrites that reduce the target gap while passing a role-preservation check, and retains a whole round only if protected metrics stay within tolerance. The planner's control vector—one tuple per comment encoding parent, depth, length, financial discussion function, claim stance, reasoning move, issue type, tone, and opening style—is what anchors each rewrite so that calibration improves statistics without turning a recommendation into an explanation or dropping a cost–benefit trade-off. Context dropout during writing, inspired by dropout in neural networks, gives later comments varied and partial views of the discussion, which reduces repeated semantic patterns.","core_discovery":"The paper's central claim is that thread-level realism can be engineered by separating what a comment does within a discussion from what it literally says. CARD first plans each thread as a sequence of abstract communicative acts—reply structure, financial discussion function, claim stance, reasoning move, issue type, tone, length, and opening style—then a writer turns each plan into text conditioned on the seed post and a partial view of the conversation. A self-looped calibration pass then selects the threads and comments contributing most to a distributional gap, generates candidate rewrites, and accepts only those that shrink the target gap while preserving the planned financial role. Across the benchmark's twelve metrics, CARD with the two strongest backbones reaches a non-rejection rate of 91.7% on both Mann–Whitney U and Kolmogorov–Smirnov tests, and it shows smaller Cliff's $\\delta$ and Wasserstein distances than the baselines. A blind Turing test also identifies CARD threads as human-written or indistinguishable more often than the baselines.","pith_inferences":["Editorial inference: the reported 91.7% is a claim about distributional similarity, not about content accuracy; the benchmark's metrics do not check whether fee amounts, reward rules, or eligibility claims are factually correct, so two statistically matching threads could still disagree on substance.","Editorial inference: a direct stress test would vary the matching procedure that pairs each seed post with a real thread; if non-rejection rates swing with the choice of matched thread, much of the result depends on the calibration target rather than on the framework's planning and revision machinery.","Editorial inference: applied to high-stakes domains such as medical or legal advice, the same recipe would need content-level guards; distributional realism alone would not certify that generated advice is safe, since a thread can match human style while asserting false rules."],"forward_implications":["Credit card discussion threads generated this way can stand in for real threads in studies of how consumers discuss rewards, fees, eligibility, and trade-offs, at least on the distributional dimensions the benchmark measures.","The same planner-writer-calibrator structure should transfer to other financial or non-financial discussion domains once domain-specific discussion functions and metrics are defined, as the paper's Future Work states.","Because calibration preserves each comment's planned role, statistical matching can be improved without distorting the substance of financial advice, disagreement, or product comparison.","The result strengthens the case that realism in social simulation is a collection-level property, not just a property of individual comments."],"supporting_citations":[{"why":"Supplies the thread-level evaluation metrics and the matched real Reddit threads used as the calibration target and comparison set.","marker":"[29]"},{"why":"Serves as the general social-media simulation baseline whose distributional alignment is compared against CARD.","marker":"[28]"},{"why":"Serves as the profile-conditioned Reddit-style thread generation baseline compared against CARD.","marker":"[30]"},{"why":"Motivates the context dropout mechanism the writer uses to vary each generated comment's view of the discussion.","marker":"[25]"}],"fun_headline_variants":["Simulated Reddit threads: 91.7% statistical match to real ones","Plan, write, calibrate: recipe for realistic fake Reddit threads","AI credit-card chats beat baselines in blind human test","Non-verbatim guidance makes AI Reddit threads feel human","CARD: separating planning from writing yields human-like discussions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the benchmark's metric set, together with one matched real thread per seed post, adequately captures what makes a credit card discussion realistic; if the matching is unrepresentative or the metrics miss content-level accuracy, calibration can fit statistics while missing substance.","fun_headline_variants_meta":{"raw":{"variants":["Simulated Reddit threads: 91.7% statistical match to real ones","Plan, write, calibrate: recipe for realistic fake Reddit threads","AI credit-card chats beat baselines in blind human test","Non-verbatim guidance makes AI Reddit threads feel human","CARD: separating planning from writing yields human-like discussions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3107,"prompt_tokens":922,"completion_tokens":2185,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":2095}},"tokens_in":538,"tokens_out":2185,"duration_ms":16554,"temperature":1.0,"reasoning_tokens":2095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:14:53.255856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the evaluation after randomly permuting which real thread is paired with each seed post; if the non-rejection rate stays near 91.7% under this permutation, the claimed match is not actually driven by the matched-pair design. A content-level audit would then be needed, since the benchmark's metrics do not check factual accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the profile-conditioned Reddit-style thread generation baseline compared against CARD."}],"review_version":2}