An LLM's Apology: Outsourcing Awkwardness in the Age of AI

Anna Soligo; Twm Stone

REVIEW 4 major objections 5 minor 9 references

An LLM's Apology: Outsourcing Awkwardness in the Age of AI

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Modern LLMs can already produce kind, believable, hard-to-verify excuses for cancelling plans.

desk verdict FLAKE-Bench is a genuinely new dataset and a fun paper, but its Anthropic-talent ranking is just GPT-4o grading itself — treat the numbers as a caution, not a result. read the letter →

arxiv 2506.13685 v1 pith:WLQP2P4T submitted 2025-06-16 cs.CY cs.HC

Twm Stone , Anna Soligo This is my paper · ORCID

classification cs.CYcs.HC

keywords LLMevaluationexcusegenerationsocialflakingghostingFLAKE-BenchbenchmarkdatasetAIethicsdeception

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that modern large language models are already surprisingly good at the socially delicate task of cancelling plans at short notice: given a description of an event, a desired outcome, and personal context, frontier chatbots can produce excuses that read as effective, kind, and human. To test this, it introduces FLAKE-Bench, a set of 250 manually written scenarios spanning romantic, social, and professional obligations, and scores ten models on three axes—efficacy, kindness, and humanity—using a GPT-4o judge. The headline result is that Anthropic's models outperform the others, with Claude Sonnet 3.7 scoring highest overall, and that category difficulty is strikingly stable across providers. The paper frames this as evidence that flaking can be delegated, relieving users of the anxiety and creative burden that leads to ghosting.

What carries the argument

The load-bearing object is FLAKE-Bench, a benchmark built from 250 manually written cancellation scenarios split equally among romantic, individual-social, group-social, internal-professional, and external-professional situations. Each scenario supplies an event, an outcome, and personal context that the model must weave into its excuse; the model is told to produce something specific, believable, difficult to verify as false, and kind, with no placeholders. Responses are scored by GPT-4o on three 0–100 axes—efficacy, kindness, and humanity—and the three scores are combined with a geometric mean so that a single weak axis drags the total down.

What would settle it

Take the 250 FLAKE-Bench scenarios, have a group of human raters score the same model outputs without knowing which AI produced them, and compare their composite ratings to GPT-4o's: if the human ordering does not put Anthropic's models at the top, or if humans flag as obviously artificial messages GPT-4o called human, the central claim about flaking effectiveness fails.

Watch

Extended reading notes

Core claim

The central discovery is that frontier LLMs do not merely produce grammatically polite cancellations; they can generate context-specific, hard-to-verify excuses that a judge rates as sincere and human-sounding across most of a broad scenario space. Sonnet 3.7 scored 0.710 overall against Sonnet 3.5's 0.705 and Haiku 3.5's 0.676, with the best model scoring above 0.7 in every one of the five scenario categories; at the bottom, o1-mini scored 0.454. Almost all models did best on social-group events and worst on external professional commitments, and each provider's newest model beat its older ones. The paper takes this as evidence that excusing oneself from commitments can be outsourced to silicon without the creative burden or moral qualms falling on the user.

Load-bearing premise

The ranking stands on the assumption that GPT-4o, when asked to grade efficacy, kindness, and humanity, rewards what human recipients would reward, and the paper never checks this against human raters.

Editorial extensions

If this is right

Outsourcing cancellation messages is already practical: the best models can do it in a few hundred tokens, so the task costs fractions of a cent per excuse.
Newer, more expensive models consistently beat their predecessors for every provider, so excuse-generation capability is improving with the general frontier.
Because category ordering is stable across models, social-group flaking is inherently easier and external-professional flaking harder; benchmark designers can use that gap as a calibration signal.
The geometric-mean scoring heavily penalises a model that is effective but cold in one category, matching the social intuition that one unkind line can spoil an otherwise fine apology.
If providers recognise this capability, emotionally skilled models will carry commercial value in personal-assistant contexts, not just reasoning benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An editorial inference: because GPT-4o grades itself, the ranking may partly measure stylistic affinity between judge and writer; a human-rating replication would test whether the claimed effectiveness transfers to real recipients.
The benchmark samples one-shot written excuses only; real cancellations face follow-up pressure, so a multi-turn version—resisting counter-requests or offers to reschedule—would be a stricter test of the paper's positive claim.
If the category ordering is culturally robust, then the social-group versus professional-external difficulty gap reflects how scripted each kind of cancellation is in ordinary life, which is a testable social-science claim beyond the paper.
The paper's framing treats an effective excuse as a good outcome, but the recipient's experience is not measured; a natural next study would ask whether such messages actually reduce resentment compared with honest cancellations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a funny, well-executed SIGBOVIK-style paper that ships a real dataset and code, but its headline ranking — Anthropic models are particularly talented at making excuses — rests on a single self-grading GPT-4o judge with no human validation. If you take the ranking at face value, you'll be misled. If you read it as a satirical benchmark that doubles as a cautionary tale about LLM-as-judge circularity, it's genuinely useful.

What's new: FLAKE-Bench, 250 manually written cancellation scenarios across five categories, with a clear rubric and a reproducible Inspect-based harness. The dataset and code are on GitHub. The writing is clear, and the limitations section is unusually honest — they explicitly note that GPT-4o fails to flag obvious AI text, rates placeholders as sincere, and may have idiosyncratic views on social acceptability.

The soft spots are real and load-bearing. The judge is GPT-4o, which is also one of the ten models being ranked. The humanity score, which feeds the geometric mean, is GPT-4o's opinion of humanness, and the paper's own evidence shows that opinion is unreliable. There's no human rater panel, no error bars, and a single run (though the authors say some re-evaluation was consistent). Also, four scenarios had to be dropped for o1-mini due to content filtering, changing the sample for that model. So the provider ordering — Anthropic > DeepSeek > OpenAI — is not established as a fact about the models; it's a fact about GPT-4o's grading preferences. The stress-test note is right: this is the central flaw, and it's not a minor one.

That said, the paper doesn't deceive. The satirical framing and the candid limitations make the authors' epistemic humility clear. They even joke that they're sure 'someone else is looking at' the deception concerns. As a serious empirical claim, it fails. As a piece of work that models a new evaluation task and openly demonstrates the pitfalls of self-judging, it has value.

Who's it for? Readers working on LLM evaluation, especially anyone using LLM-as-judge, and people who want a light but genuine dataset for testing cancellations. I wouldn't block publication anywhere that welcomes playful but reproducible evaluations, but I'd want the central claim reframed as 'GPT-4o's judgments' rather than a statement about human social acceptability.

Recommendation: send it to peer review at a venue that tolerates satire, with a request for human validation and a caveat on the Anthropic conclusion. It's not a desk reject for me.

Referee Report

4 major / 5 minor

Summary. The paper introduces FLAKE-Bench, a benchmark for evaluating how well LLMs generate socially acceptable excuses for canceling commitments. Ten models from OpenAI, Anthropic, and DeepSeek are prompted with 250 manually created scenarios spanning professional, social, and romantic categories. Each response is scored by GPT-4o on three dimensions—Efficacy, Kindness, and Humanity—and combined via a geometric mean. The reported results rank Anthropic models at the top and OpenAI models at the bottom, and the authors conclude that LLMs are 'surprisingly effective' at this task and that Anthropic's models show 'particular talent'. The dataset and evaluation code are open-sourced.

Significance. If the benchmark were validated, it would be a useful and original resource for studying LLM-mediated social interactions, deception, and 'flaking'. The authors are explicit about several limitations, which is commendable, and the open-source dataset and Inspect-based implementation are concrete contributions. However, the central measurement is not validated: the judge is GPT-4o, which is also one of the models being ranked, and no human-rater agreement is reported. Since the construct being measured—social acceptability—is inherently a human judgment, the reported ordering can support the paper's conclusions only after the judge is shown to agree with human judgments. Until then, the paper's main claim is conditional at best.

major comments (4)

[§2.4, §4.2, §5] The central conclusion that LLMs are 'surprisingly effective at generating socially acceptable excuses' and that Anthropic's models show 'particular talent' rests entirely on scores assigned by GPT-4o, which is itself one of the ten models being ranked. No human-rater validation is reported for any of the three rubric dimensions. This is not merely a methodological nicety: the Humanity subscore, which contributes to the geometric-mean total, is explicitly a judgment about whether text appears AI-generated, and §4.2 documents that GPT-4o misses responses that begin with 'As a large language model'. Because 'social acceptability' is a human construct, the provider ordering in Table 1 could be an artifact of GPT-4o's grading preferences rather than a property of the models. A human-rater study, or at minimum a report of agreement statistics with human judges, is required before the ranking can support the stated conclusions.
[§3, Table 1] All reported scores are from a single run with no confidence intervals or statistical significance tests. The differences between adjacent models are small (e.g., Sonnet 3.7 = 0.710 versus Sonnet 3.5 = 0.705; o1 = 0.581 versus V3 = 0.574), and the claim in §4.1 that 'later reevaluation of some of the models returned reasonably consistent results' is not backed by any reported numbers. Without multiple independent runs or an error model, the conclusions about a 'general increase in capabilities over time' and the provider-level ordering are not robust.
[§2.3, Appendix B] The o1-mini results are based on a different dataset than all other models: four items were removed to avoid content filtering, including one social item and two romantic items. Direct comparisons between o1-mini and other models therefore confound model capability with a difference in evaluation samples. The paper should report whether the ranking is stable when all models are evaluated only on the common subset of items, especially because o1-mini is the lowest-ranked model.
[§4.1, §5] The phrase 'surprisingly effective' implies a comparison to an expectation or to human performance, but no such comparison is provided. The paper reports raw GPT-4o scores only, with no human-generated excuses scored by the same judge and no pre-registered threshold for 'effectiveness'. As written, the claim is not falsifiable, because any positive score could be interpreted as 'effective' and any ordering as 'surprising'. Adding a human baseline, or at least a clear criterion for what would count as effective, is needed to support the conclusion.

minor comments (5)

[Abstract] The abstract says 'user's social life'; the plural possessive 'users' social life' is intended.
[Appendix D] Appendix D states that '40 summary results' were calculated, but the table lists 60 rows (10 models × 6 rows per model, including Overall). Please reconcile the counts.
[Introduction, Figure 1] The reference to the 'adjacent figure (Munroe, 2025)' is not self-contained; if the xkcd comic is not reproduced, the description of it should be clarified.
[§4.3] The observation that GPT-4o 'rated itself 8th of 10 models' does not address the evaluator-bias concern, since a biased judge can be harsh on its own outputs. This sentence should be clarified or removed.
[§4.1] The 'weak trend towards longer excuses being rated as better' is not quantified; reporting a correlation coefficient or a fitted relationship would make the observation verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the ranking is an empirical observation from an unvalidated judge, not a definitional reduction or a fitted parameter renamed as a prediction.

full rationale

The claimed derivation chain is: rubric and dataset -> GPT-4o judge -> category scores -> geometric-mean totals -> model ranking -> conclusion that Anthropic models show particular talent. None of these steps is definitionally identical to another, and no equation in the paper reduces a predicted quantity to an input by construction. The fact that GPT-4o is both the judge and one of the ten evaluated models creates a genuine measurement-validity / evaluator-bias risk, and the absence of human raters means the 'social acceptability' construct is not externally validated. However, that is a correctness and validity limitation, not a circular reduction: the paper does not define social acceptability as 'whatever GPT-4o scores', and it explicitly reports evidence against self-preference (GPT-4o ranked 8th of 10). The rubric articulates human-facing criteria (efficacy, kindness, humanity) and then uses GPT-4o as a fallible instrument, not as the definition of the construct. The paper's own limitation sections flag GPT-4o's failure to detect AI text and its possibly idiosyncratic social-acceptability judgments, which strengthens the case that this is an acknowledged validity threat rather than a hidden circular step. The only self-citation, Stone and Stoddart (2024), concerns dataset inspiration and is not load-bearing for the ranking. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is repackaged under new coordinates. Therefore the paper's central claim is not circular; it is empirically derived from a questionable but non-circular measurement instrument.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central results rest on GPT-4o as judge, the author-built scenario set, and the content-filtering procedure. None are externally benchmarked or validated against human raters, so the benchmark is an internally consistent measurement of GPT-4o's grading behavior rather than a validated measure of social acceptability.

free parameters (1)

Grading rubric anchors and thresholds = Default score 40%; efficacy >70% rare; AI-suspicion scores <50%; placeholder scores <20
These hand-set anchors in Appendix A directly determine every model's numeric score and ranking. They are not data-derived.

assumptions (4)

domain assumption GPT-4o can reliably judge whether a message is kind, effective, and human-like.
Section 2.4 selects GPT-4o as the LLM-judge on the basis of prior emotional-intelligence benchmarks, but no human ground truth or validation is provided.
domain assumption The 250 manually created scenarios are representative of real-world flaking situations.
Section 2.3 states the dataset was created by the first author and reflects his hobbies and life experiences; section 4.3 acknowledges intrinsic bias.
domain assumption Content-filtering exclusions do not materially change comparative conclusions.
Section 4.3 and Appendix B remove up to four scenarios for o1-mini and omit 'immoral' or illegal scenarios entirely, which changes the set of evaluated contexts.
domain assumption Humanity is a real, independently rateable property of the text.
The Humanity criterion assumes an LLM judge can distinguish human from AI text, but the paper's own limitations note GPT-4o fails to detect placeholders and AI markers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An LLM's Apology: Outsourcing Awkwardness in the Age of AI." pith.science (2026). https://pith.science/paper/WLQP2P4T

@misc{pith2026250613685,
  author       = {Pith},
  title        = {Pith review of: An LLM's Apology: Outsourcing Awkwardness in the Age of AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WLQP2P4T}},
  note         = {Machine review of arXiv:2506.13685}
}

read the original abstract

A key part of modern social dynamics is flaking at short notice. However, anxiety in coming up with believable and socially acceptable reasons to do so can instead lead to 'ghosting', awkwardness, or implausible excuses, risking emotional harm and resentment in the other party. The ability to delegate this task to a Large Language Model (LLM) could substantially reduce friction and enhance the flexibility of user's social life while greatly minimising the aforementioned creative burden and moral qualms. We introduce FLAKE-Bench, an evaluation of models' capacity to effectively, kindly, and humanely extract themselves from a diverse set of social, professional and romantic scenarios. We report the efficacy of 10 frontier or recently-frontier LLMs in bailing on prior commitments, because nothing says "I value our friendship" like having AI generate your cancellation texts. We open-source FLAKE-Bench at github.com/Cloakless/flake-bench to support future research.

Figures

Figures reproduced from arXiv: 2506.13685 by the authors.

**Figure 1.** Pre-LLM options To address this serious gap in the current literature, we introduce a new benchmark for assessing the capabilities of models in Fabricating Legitimate Alibis Kindly and Effectively, or FLAKE-Bench for short. FLAKE-Bench measures a model’s ability to generate socially acceptable excuses that balance believability, kindness, and effectivearXiv:2506.13685v1 [cs.CY] 16 Jun 2025 [PITH_FULL_IMAGE:figures… view at source ↗

**Figure 2.** Model results for each grading criterion [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Anthropic results by category The price of tokens for each model, at the time of writing, is listed in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: OpenAI results by category [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: DeepSeek results by category [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Other suggestions included fairly explicit sexual or violent content and were not suitable for publication... D Full results 50 evals were run, each with approximately (see Appendix B) 50 samples, and then 40 ‘summary’ results were calculated. The full results are pre…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 8 canonical work pages

[1]

User-focused telehealth powered by llms: Bridging the gap between technology and human-centric care delivery

Ho Yan Kwan. User-focused telehealth powered by llms: Bridging the gap between technology and human-centric care delivery. In 2024 4th International Conference on Computer Communication and Artificial Intelligence (CCAI), pages 187--191, 2024. doi:10.1109/CCAI61966.2024.10603150

work page arXiv 2024
[2]

Google Duplex : An AI system for accomplishing real-world tasks over the phone

Google . Google Duplex : An AI system for accomplishing real-world tasks over the phone. Google Research Blog, May 2018. URL https://research.google/blog/google-duplex-an-ai-system-for-accomplishing-real-world-tasks-over-the-phone/

work page 2018
[3]

Excusing Yourself

Randall Munroe. Excusing Yourself . xkcd, 2025. URL https://xkcd.com/3057/

work page 2025
[4]

Powell, Benjamin Le, and Kipling D

Gili Freedman, Darcey N. Powell, Benjamin Le, and Kipling D. Williams. Emotional experiences of ghosting. The Journal of Social Psychology, 164 0 (3): 0 367--386, 2024. doi:10.1080/00224545.2022.2081528. URL https://doi.org/10.1080/00224545.2022.2081528. PMID: 35621208

work page arXiv 2024
[5]

Mittelst \"a dt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes

Justin M. Mittelst \"a dt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes. Large language models can outperform humans in social situational judgments. Scientific Reports, 14 0 (27449), 2024. doi:10.1038/s41598-024-79048-0. URL https://doi.org/10.1038/s41598-024-79048-0

work page doi:10.1038/s41598-024-79048-0 2024
[6]

The Ballmer Peak: An Empirical Search

Twm Stone and Jaz Stoddart. The B allmer P eak: An E mpirical S earch. In Proceedings of SIGBOVIK, Pittsburgh, PA USA, 2024. URL https://arxiv.org/abs/2404.10002

work page Pith review arXiv 2024
[7]

Liu, Jinfeng Zhou, Alvionna S

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M.C. Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional intelligence of large language models. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2402.12071

arXiv 2024
[8]

Inspect AI: Framework for Large Language Model Evaluations , 2024

UK AI Security Institute. Inspect AI: Framework for Large Language Model Evaluations , 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai

work page 2024

Show all 9 references

[9]

The dead grandmother/exam syndrome

Mike Adams. The dead grandmother/exam syndrome. Annals of Improbable Research, 5 0 (6): 0 3--6, November/December 1999. URL https://improbable.com/airchives/paperair/volume5/v5i6/GrandmotherEffect

1999

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

User-focused telehealth powered by llms: Bridging the gap between technology and human-centric care delivery

Ho Yan Kwan. User-focused telehealth powered by llms: Bridging the gap between technology and human-centric care delivery. In 2024 4th International Conference on Computer Communication and Artificial Intelligence (CCAI), pages 187--191, 2024. doi:10.1109/CCAI61966.2024.10603150

work page arXiv 2024

[2] [2]

Google Duplex : An AI system for accomplishing real-world tasks over the phone

Google . Google Duplex : An AI system for accomplishing real-world tasks over the phone. Google Research Blog, May 2018. URL https://research.google/blog/google-duplex-an-ai-system-for-accomplishing-real-world-tasks-over-the-phone/

work page 2018

[3] [3]

Excusing Yourself

Randall Munroe. Excusing Yourself . xkcd, 2025. URL https://xkcd.com/3057/

work page 2025

[4] [4]

Powell, Benjamin Le, and Kipling D

Gili Freedman, Darcey N. Powell, Benjamin Le, and Kipling D. Williams. Emotional experiences of ghosting. The Journal of Social Psychology, 164 0 (3): 0 367--386, 2024. doi:10.1080/00224545.2022.2081528. URL https://doi.org/10.1080/00224545.2022.2081528. PMID: 35621208

work page arXiv 2024

[5] [5]

Mittelst \"a dt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes

Justin M. Mittelst \"a dt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes. Large language models can outperform humans in social situational judgments. Scientific Reports, 14 0 (27449), 2024. doi:10.1038/s41598-024-79048-0. URL https://doi.org/10.1038/s41598-024-79048-0

work page doi:10.1038/s41598-024-79048-0 2024

[6] [6]

The Ballmer Peak: An Empirical Search

Twm Stone and Jaz Stoddart. The B allmer P eak: An E mpirical S earch. In Proceedings of SIGBOVIK, Pittsburgh, PA USA, 2024. URL https://arxiv.org/abs/2404.10002

work page Pith review arXiv 2024

[7] [7]

Liu, Jinfeng Zhou, Alvionna S

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M.C. Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional intelligence of large language models. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2402.12071

arXiv 2024

[8] [8]

Inspect AI: Framework for Large Language Model Evaluations , 2024

UK AI Security Institute. Inspect AI: Framework for Large Language Model Evaluations , 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai

work page 2024