Pith. sign in

REVIEW 3 major objections 5 minor 9 references

A dialogue filter kept 87.4% of chatbot sentences on the learner's CEFR level, versus 54.1% without filtering.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:24 UTC pith:5YLOBGCV

load-bearing objection Useful pilot system, but the headline 87% vs 54% 'level-appropriateness' result is not evidence: the BERT classifier that filters the dialogue is the same one that scores the outcome. the 3 major comments →

arxiv 2607.21887 v1 pith:5YLOBGCV submitted 2026-07-24 cs.HC cs.CL

Towards Reducing Foreign Language Anxiety Using Level-Appropriate Embodied Conversational Agents

Krishan Rajaratnam , Wenbin Gan , Yuan Sun This is my paper
classification cs.HC cs.CL
keywords foreign language anxietyCEFRconversational agentslarge language modelsembodied agentslevel-appropriate dialoguegenerate-evaluate-regeneratesecond language acquisition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether an embodied conversational agent can reduce foreign language anxiety by matching its dialogue to the learner's proficiency. It proposes a generate-evaluate-regenerate loop in which a classifier trained on a CEFR-labeled dataset of conversational texts filters a large language model's utterances, with a feedback agent guiding simplifications until a threshold is met. In a three-person pilot with Japanese university students, the filtered agent produced 87.4% of sentences within one predicted CEFR level of each learner's self-assessed proficiency, versus 54.1% for the unfiltered agent. The pilot found no statistically significant drop in anxiety scores, but two of three participants showed lower anxiety in the filtered condition, and all preferred it.

Core claim

The central claim is that an iterative level-filtering loop can make an LLM-based embodied conversation partner reliably produce language at or near a learner's CEFR level, and that this alignment is a plausible route to lower foreign language anxiety. The paper's quantitative evidence is the large gap in on-level output between filtered and unfiltered conditions (87.4% vs 54.1%). It treats this as evidence that the system works as designed, while being explicit that the anxiety-reduction hypothesis remains unproven due to the tiny sample.

What carries the argument

The load-bearing component is the generate-evaluate-regenerate loop: a large language model outputs dialogue; the response is split into sentences; a fine-tuned BERT classifier assigns each sentence a CEFR level; a scoring rule (sentences at or below the target count fully, sentences one level above count half, compared against a 0.66 threshold) decides acceptance; if rejected, a separate feedback agent interprets the CEFR predictions and suggests simplifications, and the conversational agent rewrites the response. Only accepted responses are kept in the agent's memory, preserving context for the ongoing conversation.

Load-bearing premise

The approach depends on the classifier's difficulty labels being a valid stand-in for what a particular learner can comfortably understand; the pilot never checks this against human judgment or an objective proficiency measure.

What would settle it

Have independent human raters assign CEFR levels to transcripts from both conditions; if the filtered transcripts are not rated substantially closer to learners' self-assessed levels than the unfiltered ones, the 87.4%-vs-54.1% gap is an artifact of the classifier. A larger randomized trial with objective proficiency measurement would settle whether the filtered condition actually lowers anxiety.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Automatic CEFR filtering is a practical way to keep LLM-based tutors from speaking over learners' heads, if the level-appropriateness result holds.
  • Because the filter targets self-assessed proficiency, it shifts the design problem from content generation to accurate, low-cost proficiency estimation.
  • The observed negative relationship between on-level sentence share and reported anxiety, if causal, implies that even imperfect filtering might improve learner comfort.
  • The pilot's preference for the filtered condition, despite participants not consciously noticing the complexity difference, suggests perceived difficulty may operate partly below awareness.
  • Conversational context, such as a familiar self-introduction topic, can override system-level filtering as a difficulty control, so future studies need to constrain or systematically vary topics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: this filtering pattern could generalize to any domain with difficulty-labeled text, not just CEFR, so long as a reliable labeler exists.
  • Beyond the paper: the 87.4% figure should be re-checked with independent human raters, because the same classifier that filters the dialogue also computes the outcome metric.
  • Beyond the paper: participants' anxiety about response pauses (ma) points to timing as a separate design lever; future agents could log hesitation and vary pause length to test it.
  • Beyond the paper: since filtering is calibrated to self-assessed proficiency, inaccurate self-assessments would lead to mistargeted simplification; objective measures or adaptive estimation could strengthen the approach.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a multi-agent embodied conversational system that filters LLM-generated English dialogue to match a learner's self-assessed CEFR level. The method uses a fine-tuned BERT classifier in a generate-evaluate-regenerate loop. A pilot study with three Japanese university students compares a filtered agent with an unfiltered agent, reporting that 87.4% of filtered-condition sentences versus 54.1% of unfiltered-condition sentences fell within one predicted CEFR level of the learner's self-assessed proficiency. The paper additionally reports non-significant FLA differences, qualitative usability findings such as the culturally unfamiliar café scenario, and the participant-perceived pressure of response pauses (ma).

Significance. The design idea—using a CEFR classifier to adapt conversational output to learner proficiency and doing so in an embodied agent—is potentially useful for SLA applications, and the qualitative findings about cultural fit and interaction pacing are informative for future system design. The pilot is honestly reported as small-scale and does not overclaim statistical significance. However, the central quantitative claim about level-appropriateness is not supported because the same BERT classifier is used both as the filter's acceptance criterion and as the evaluation metric. As a result, the headline 87.4%-versus-54.1% result is an artifact of selection on the evaluation statistic and cannot be read as evidence that the system produces genuinely more level-appropriate output. Without an external validation criterion, the paper's main contribution is reduced to a usability/design case study.

major comments (3)
  1. [§2.1 and §3, Figs. 5–6] The central level-appropriateness comparison is methodologically circular. In §2.1, the same fine-tuned BERT CEFR classifier computes each response's acceptance score—the proportion of sentences at or below target, with half weight for one level above—and accepts a response only when that score exceeds ε=0.66. In §3, 'level-appropriate' is measured as the percentage of sentences classified by the same BERT model as within one CEFR level of the learner's self-assessed proficiency. The filter is therefore optimizing essentially the same statistic used as the outcome. Even with a perfectly calibrated classifier, the filtered condition must score higher on this metric simply because high-scoring responses were selected; a biased classifier is worse, because the filter actively selects sentences that exploit the classifier's errors. The citation to Kogan et al.'s validation of the base model
  2. [§3, Fig. 4] The 'negative relationship' between FLA and level-appropriateness is based on six points from three participants, two per person, with no independence and no statistical analysis. Participant 3 exhibits the opposite ordering between conditions. The paper is appropriately cautious in noting the absence of statistical significance, but the conclusions still lean on this scatterplot as supporting evidence (e.g., 'the observed negative relationship... indicates that both content complexity and topic familiarity may influence learner comfort'). At most this can be described as a descriptive pattern. The combination of circularity and non-independence means this figure should not be used to infer a relationship between complexity and anxiety.
  3. [§2.2 and Table 2] The filter is calibrated to learners' self-assessed CEFR levels rather than to objective proficiency. Self-assessment is explicitly acknowledged as a limitation, but it remains load-bearing for the main quantitative claim: 'within one predicted CEFR level of the learner's self-assessed proficiency' is not the same as 'within one CEFR level of the learner's actual proficiency.' If self-assessment is inaccurate, the system may be filtering to the wrong target. This weakens construct validity and should be either addressed with an objective proficiency measure or removed from the headline claim.
minor comments (5)
  1. [§2.1] The scoring rule is described verbally but not formalized. Please provide an explicit equation for the acceptance score, including how 'one level above' and 'at or below' are counted, and clarify how the 87.4% 'within one level' metric is computed relative to the filter's half-weight rule. The two definitions are not obviously the same: the filter counts one-level-above sentences at half weight, while the outcome metric appears to count them fully as 'within one level.'
  2. [§3.1 and §4] The conclusion states the study 'offers preliminary evidence that proficiency-aligned conversational agents may help reduce FLA,' but the FLA comparisons are non-significant, one of three participants showed the opposite pattern, and the level-appropriateness metric is circular. The conclusion should be reworded to reflect the evidence actually presented: a usability and design-feasibility pilot with suggestive but inconclusive FLA outcomes.
  3. [§2.2 and Fig. 2] The first session is always filtered and the second/third are randomized; this is acknowledged, but the paper should also note that the 'intro session' FLA scores in Table 2 are not a true baseline because the intro conversation itself is filtered. The phrase 'Pre-intervention FLA' and 'After Intro Conversation' should be interpreted accordingly.
  4. [References] Reference [7] points to a MEXT-hosted PDF rather than a formal publication; please provide the official CEFR-J source or a stable DOI. Reference [5] is cited in the acknowledgements as 'Zimu Cheng' while the reference lists 'Z. Cheng, X. Pan, and M. Gillies'; please align the name conventions.
  5. [Numerous places] There are minor typographical errors, e.g., 'hinder ing' in §1, and inconsistent hyphenation of 'self-assessment'/'self assessment.' A careful proofread is needed.

Circularity Check

1 steps flagged

The 87.4%-vs-54.1% level-appropriateness result is not independent: the same fine-tuned BERT classifier filters dialogue (§2.1) and scores the outcome (§3), so the filtered condition is selected on the evaluation statistic.

specific steps
  1. fitted input called prediction [§2.1 (Iterative CEFR Leveling Loop) and §3 (Results): acceptance-score threshold ε=0.66 vs. 'within one predicted CEFR level' metric.]
    "Instead, we score each response by calculating the proportion of sentences at or below the target level, assigning half weight to sentences that are one level above the target to reward proximity. This score is then compared against a threshold ε (set to 0.66 in our experiments). If the score exceeds ε, the response is returned to the user and stored in the agent’s memory. / Overall, 87.4% of sentences generated in the filtered condition fell within one predicted CEFR level of the learner’s self-assessed proficiency, compared to 54.1% in the unfiltered condition."

    The filter's acceptance score and the reported outcome are both computed from the same fine-tuned BERT classifier's per-sentence CEFR predictions. The acceptance rule keeps a response only when the weighted fraction of sentences at/below target (with half weight for one level above) exceeds 0.66, which mathematically implies that the fraction of sentences within one predicted level is also above 0.66. The §3 metric is exactly that within-one-predicted-level fraction, so the filtered condition is guaranteed to score high on the evaluation by construction. The unfiltered condition has no such constraint, making the 87.4% vs 54.1% gap substantially an artifact of optimizing the evaluation statistic rather than independent evidence of level appropriateness. The paper's FLA results are non-sign

full rationale

The central quantitative claim—that the multi-agent system produces more level-appropriate output—is supported by a metric that is not independent of the system's filtering mechanism. The same fine-tuned BERT classifier is used twice: inside the generate-evaluate-regenerate loop to accept or reject responses based on a weighted fraction of sentences at or below target, and in §3 to compute the reported outcome as the percentage of sentences the classifier places within one CEFR level. Because acceptance requires that weighted fraction to exceed ε=0.66, every accepted response is guaranteed to have a high within-one-level fraction by the same classifier's judgment; thus the observed gap is partly predetermined by the selection rule, not by genuine pedagogical appropriateness. This is a clear case of a fitted model's predictions being used both as the filter and as the ground-truth-like outcome. No self-citation chain, imported uniqueness theorem, or ansatz-by-citation circularity is present; the external Kogan et al. validation of the base classifier is real evidence, but it does not validate the in-pipeline selection metric or the generated dialogues. The paper honestly reports non-significant FLA results and notes that self-assessment 'does not replace objective proficiency measurement,' but those caveats do not repair the circularity of the level-appropriateness statistic. Overall, one central 'prediction' reduces substantially by construction, yielding a partial circularity score of 6.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's central evidence rests on a fitted classifier whose predictions are used both as the filter criterion and as the evaluation metric, plus self-reported proficiency as the target level. These are the main unverified inputs; no new physical or conceptual entities are introduced.

free parameters (4)
  • Filter acceptance threshold ε = 0.66
    Hand-set threshold in §2.1; the leveling loop accepts a response only if the classifier-based score exceeds ε. No sensitivity analysis reported.
  • Half-weight for one-level-above sentences = 0.5 (weight)
    Arbitrary design choice in the response-level score function; affects which responses pass the filter.
  • Fine-tuned BERT CEFR classifier weights = not reported
    Model fitted to the Ace-CEFR dataset; its predictions are used both for filtering and for the outcome metric, so its parameters are load-bearing.
  • Self-assessment can-do agreement threshold = 79%
    Criterion used to map self-reported listening/speaking to a CEFR level; chosen without stated justification.
axioms (4)
  • domain assumption The BERT CEFR classifier's predicted levels are a valid measure of text difficulty and learner appropriateness.
    Used to score, accept, and regenerate dialogue (§2.1) and to compute the success metric (§3). No human-level validation is provided in this paper.
  • domain assumption Self-assessed CEFR level is a reliable proxy for the learner's actual proficiency and the appropriate target level.
    The target level is set from two self-evaluation scales; the paper notes this 'does not replace objective proficiency measurement' (§2.2).
  • domain assumption LLM agents can rewrite dialogue to a lower CEFR level while preserving conversational coherence.
    The generate-evaluate-regenerate loop assumes the conversational agent can comply with feedback and simplify text without breaking the interaction.
  • domain assumption The Japanese-translated 15-item FLCAS measures foreign language anxiety in this non-classroom setting.
    The paper itself cites Motoda and cultural concerns about FLCAS validity in Japanese learners (§4).

pith-pipeline@v1.3.0-alltime-deepseek · 5767 in / 11861 out tokens · 108361 ms · 2026-08-01T06:24:17.058689+00:00 · methodology

0 comments
read the original abstract

Foreign language anxiety (FLA) can be a major barrier to second language acquisition (SLA), especially in conversational contexts. With the proliferation of large language models (LLMs) throughout all areas of life, recent work suggests that interacting with LLM agents can be instrumental within the field of SLA and foreign language education, especially for reducing FLA. Related work also suggests that linguistic demands and task complexity can be predictors of FLA, implying that the use of demanding, complex language could lead to learners experiencing higher FLA. In this paper, we propose a novel multi-agent embodied conversational system that generates level-appropriate dialogue for English language learners. These levels are based on those defined by the Common European Framework of Reference for Languages (CEFR) to describe non-native listener and speaker proficiency. Using a "generate-evaluate-regenerate" loop with multiple LLM agents and a level classifier, it achieves a desired simplicity that is adaptive to the user's proficiency level. We also share the results of a preliminary small-sample pilot study that tested this system with Japanese university students, to see whether it would yield lower FLA levels than an unsimplified embodied conversational agent. Analysis of conversational output showed that 87.4% of dialogue sentences generated by the proposed multi-agent system fell within one predicted CEFR level of the learner's self-assessed proficiency, compared to 54.1% for the unsimplified agent. This suggests that the novel system is better able to produce output at an appropriate level for the learner. Though this study did not yield statistically significant evidence that the system reduces FLA levels in Japanese learners of English, likely due to a small sample size, it provides usability findings and culturally-informed design insights that will inform future study.

Figures

Figures reproduced from arXiv: 2607.21887 by Krishan Rajaratnam, Wenbin Gan, Yuan Sun.

Figure 1
Figure 1. Figure 1: Leveled Conversational Agent Framework 2.1 Iterative CEFR Leveling Loop Due to Kogan et al.’s dataset consisting of short conversational excerpts, the LLM’s initial output is first segmented into individual sentences, which are then classified into projected CEFR levels using a fine￾tuned BERT model. This process can yield multiple CEFR predictions for a single response. While it might seem natural to comp… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Experimental Study Procedure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Immersive User Speaking Interface with Embodied Conversational Agent [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows a scatter plot of reported FLA against the percentage of sentences classified within one level of the target proficiency level or below. The plot suggests a negative relationship between the proportion of level-appropriate sentences and reported FLA. Additionally, Participant 3, who reported slightly higher FLA after the filtered conversation than after the unfiltered conversation, did not exhibit as… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of CEFR Classifications for Sentences in Filtered Conversations [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 2 canonical work pages

  1. [1]

    Foreign language classroom anxiety,

    E. K. Horwitz, M. B. Horwitz, and J. Cope, “Foreign language classroom anxiety,” The Modern Language Journal, vol. 70, no. 2, pp. 125–132, 1986

    1986

  2. [2]

    Exploring student perceptions of language learning affordances of Large Language Models: A Q methodology study,

    K. Li, L. Lun, and P. Hu, “Exploring student perceptions of language learning affordances of Large Language Models: A Q methodology study,” Education and Information Technologies vol. 30, pp. 13691–13711, 2025. doi:10.1007/s10639-025-13362-6. 7

    work page doi:10.1007/s10639-025-13362-6 2025

  3. [3]

    The influence of large language models as collaborative dialogue partners on EFL English oral proficiency and foreign language anxiety,

    Y. Zheng, Y. Zhou, X. Chen, and X. Ye, “The influence of large language models as collaborative dialogue partners on EFL English oral proficiency and foreign language anxiety,” Computer Assisted Language Learning, pp. 1–27, 2025. doi:10.1080/09588221.2025.2453191

    arXiv 2025

  4. [4]

    Prakash, ELLA: An Episodic Language Learning Assistant for Language-Anxious Learners

    K. Prakash, ELLA: An Episodic Language Learning Assistant for Language-Anxious Learners. Delft University of Technology, 2024. Retrieved from https://repository.tudelft.nl/file/File_846601fa-bc14-485d-a082-5adc5d428022

    2024

  5. [5]

    Reducing Foreign Language Anxiety in Icebreaker Sessions Using LLM-Controlled Multilingual Virtual Agents in VR,

    Z. Cheng, X. Pan, and M. Gillies, “Reducing Foreign Language Anxiety in Icebreaker Sessions Using LLM-Controlled Multilingual Virtual Agents in VR,” IVA’25: Proceedings of the 25 th ACM International Conference on Intelligent Virtual Agents, No. 47, pp. 1–4, 2025. doi:10.1145/3717511.3749308

    arXiv 2025

  6. [6]

    Ace-CEFR -- A Dataset for Automated Evaluation on the Linguistic Difficulty of Conversational Texts for LLM Applications,

    D. Kogan, M. Schumacher, S. Nguyen, M. Suzuki, M. Smith, C. S. Bellows, and J. Bernstein, “Ace-CEFR -- A Dataset for Automated Evaluation on the Linguistic Difficulty of Conversational Texts for LLM Applications,” arXiv preprint. vol. 2506, no. 1404, 2025. doi:10.48550/arXiv.2506.14046

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.14046 2025

  7. [7]

    Tono, CEFR-J を活用するための ‘Can Do’ Descriptor リスト

    Y. Tono, CEFR-J を活用するための ‘Can Do’ Descriptor リスト. Tokyo University of Foreign Studies, 2012. Retrieved from https://www.mext.go.jp/b_menu/shingi/chousa/shotou/092/shiryo/__icsFiles/afieldfile/2012/09/24 /1325972_2_3.pdf

    2012

  8. [8]

    日本語不安尺度の作成とその検討: 目標言語使用環境における第二言語不安の測定,

    S. Motoda, “日本語不安尺度の作成とその検討: 目標言語使用環境における第二言語不安の測定,” 教育心理学研究, vol. 48, no. 4, pp. 422–432,

  9. [2000]

    doi:10.5926/jjep1953.48.4_422 8

    work page doi:10.5926/jjep1953.48.4_422