AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures

Donghoon Chang; Kyeongeon Lee; Seungryeol Baek; Taehong Kim; Wonjun Yang

REVIEW 2 major objections 5 minor 1 references

AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A readability-oriented post-ASR workflow for Korean-English medical lectures substantially increased English-script rendering of medical terms but left or introduced semantic drift in about 12% of reference sentences and polarity failures i

desk verdict Small, honest pilot: the headline drift numbers hinge on a single annotator, but the core claim — score script rendering and meaning separately — holds. read the letter →

arxiv 2607.17237 v1 pith:P74ZTUP4 submitted 2026-07-19 cs.CL

Kyeongeon Lee , Donghoon Chang , Seungryeol Baek , Taehong Kim , Wonjun Yang This is my paper · ORCID

classification cs.CL

keywords English-scriptrenderingsemanticdriftpolaritypreservationKorean-Englishmedicallecturespost-ASRrewritingreadabilityversusfaithfulnessASRevaluationLLMpost-processing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a post-ASR rewriting stage can improve how medical terms are displayed in English script while simultaneously failing to preserve the medical meaning: a readability gain is not a faithfulness gain. The pilot evaluates one historical workflow on four lectures and shows that post-processing raised the English-script rendering rate from 0.39 to 0.71 on one input path and from 0.26 to 0.65 on another, yet 34 or 36 of 282 reference sentences drifted semantically and 11 or 13 of 101 polarity rows failed. The authors argue that such surface and meaning-level measures must be reported separately, because a transcript that looks cleaner can still teach a wrong medical claim. The paper's scope is deliberately modest: a four-lecture, single-annotator pilot that documents failure modes rather than population rates.

What carries the argument

The key mechanism is the post-ASR rewriting stage of the AI_LectureNote workflow: after raw speech-to-text, a generative LLM pass 'Englishs' Korean-phonetic medical terms into Latin script and regularizes wording. The central analytical machinery is the separation between (a) the English-script rendering rate - a recall-like measure of whether curated domain terms appear in English/Latin script, regardless of correctness - and (b) human-annotated semantic drift and critical polarity labels, which capture whether the medical meaning is preserved. A cross-input control applies the same rewriting stage to a different raw ASR front-end to distinguish errors introduced by post-processing from err

What would settle it

An independent annotator, blinded to the paper's labels, would re-annotate the 282 reference sentences and 101 polarity rows for the two post-processed conditions; if the re-annotation found no semantic drift or no polarity failures, or found that the Jaccard overlap pattern reversed (drift overlapping more than polarity), the central trade-off claim would fail.

Watch

Extended reading notes

Core claim

The central claim is that readability-oriented post-ASR rewriting improves English-script rendering of medical terms but does not imply semantic faithfulness. In the two post-processed conditions, the macro English-script rendering rate rose to 0.71 and 0.65 (from 0.39 and 0.26, respectively), while semantic drift appeared in 34 and 36 of 282 reference sentences and polarity failures in 11 and 13 of 101 polarity-cue rows. A descriptive cross-input comparison found that polarity-failure sets overlapped more across front-ends (Jaccard 0.60) than general semantic-drift sets (Jaccard 0.23), suggesting that some polarity errors are post-processing-associated while broader drift is more input-sens

Load-bearing premise

The load-bearing premise is that the author-created references and the single author-annotator's semantic and polarity labels are a valid, unbiased measure of 'medical meaning preservation'; if a second independent annotator re-labeled them differently, the headline drift and polarity numbers could change in magnitude or distribution.

Editorial extensions

If this is right

Transcript utility for students cannot be read off CER/WER; the same workflow can lower error rates in some lectures and raise them in others, so surface accuracy and medical meaning must be reported as separate axes.
A readability-oriented rewriting stage can introduce or carry forward errors that a term-recall score would miss, so evaluations of post-ASR correction should measure semantic faithfulness and polarity, not just rendering.
Polarity failures (e.g., hypo-/hyper- flips) can persist or be created even when English-script rendering improves, making them a distinct clinical-safety concern for study materials.
Chunk-level script consistency varies sharply within a lecture, indicating that aggregate rendering rates can hide unstable term-script selection.
Because post-processing can both correct and introduce errors (the paper notes restorations like 'Inhalation' for 'Annihilation' alongside new substitutions), the net benefit of such rewriting is conditional on input quality and lecture content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the single-annotator bias is real, the specific drift and polarity rates could shift with independent annotation, but the qualitative readability-faithfulness trade-off is likely robust, since a 0.3-0.4 point rendering gain coexisting with 10-13% drift is a large effect.
The Jaccard asymmetry suggests a testable hypothesis: whole-sentence rewriting stages systematically create polarity flips independent of raw ASR errors, while general semantic drift depends on the upstream transcript's specific mistakes.
A constrained post-processor that replaces transliterated terms via a medical glossary rather than free-form rewriting might retain most rendering gains while reducing introduced polarity errors - this is a natural next experiment.
The paper's reference policy (lightly cleaned study transcripts rather than verbatim ASR gold) means the observed drift partly reflects a deliberate target definition; re-testing against verbatim references would separate rewriting-induced changes from reference-matching differences.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a small, honest pilot with one load-bearing caveat. The headline drift and polarity numbers rest on a single author-annotator and author-created references, but the paper's real claim — that script-rendering gains and medical meaning preservation have to be evaluated separately — survives that caveat.

What's actually new: a five-condition protocol with a cross-input control (same post-processor, different raw front-ends), plus the specific failure-overlap finding (polarity Jaccard 0.60 vs drift Jaccard 0.23). The rendering-rate computations are deterministic, the raw baselines show the metric isn't trivially satisfied (whisper-1 at 0.39, gpt-4o at 0.26), and the paper ships code, references, and annotation files. That is real evidence, and it earns more credit than the typical LLM-correction paper.

The paper also does something too rare: it states its limits in plain sight. The Limitations section names the single-annotator problem, the confounds, the 2026 whisper-1 rerun, the author-created reference policy, and the fact that the phonetic variant list is not exhaustive. That doesn't fix the problems, but it lets you read the results at face value as a bounded pilot.

The soft spots are exactly where the reader's stress-test points. The 34/36 drift counts, the 11/13 polarity failures, and the Jaccard overlap all come from one pass by one person who built the system and wrote the references. No inter-annotator agreement, no adjudication, no blinding. The introduced-vs-propagated attribution in §6.4 is the most fragile part because the annotator knows which output is raw and which is rewritten. An independent annotator could shift the numbers in magnitude, maybe not in direction. The rendering metric also depends on author-curated term lists, so the 0.39→0.71 is a measured computation but the target is a choice.

That said, the central argument holds up. The paper never claims population rates; it says descriptive pilot, and it behaves like one. The claim that surface metrics and medical meaning need to be scored separately is supported by the results, not just asserted. I would not treat the self-referential evaluation as fatal — in a retrospective study of your own tool, full independence is often impossible, and the authors say so. That is a limitation, not a disqualifier.

The conditional verdict is right. This deserves a serious referee: it's reproducible, honestly scoped, and methodologically useful for the code-switched/medical post-ASR niche. A good referee should push for a second annotator or a blinded re-labeling, but that's revision work, not a rejection.

Recommendation: send to peer review. I'd bring it to a reading group on evaluation of LLM post-correction, and I'd cite it if I were writing about that area.

Referee Report

2 major / 5 minor

Summary. This paper retrospectively evaluates AI_LectureNote, a post-ASR rewriting workflow for Korean–English medical lectures, on four author-recorded lectures across five conditions: raw whisper-1, AI_LectureNote post-processed whisper-1, raw gpt-4o-transcribe, prompted gpt-4o-transcribe, and gpt-4o-transcribe with AI_LectureNote post-processing. The headline finding is that the post-processing raises English/Latin-script rendering of curated medical terms (macro rate 0.39→0.71 for whisper-1; 0.26→0.65 for gpt-4o) while semantic drift is observed in 34/282 and 36/282 reference sentences and polarity failures in 11/101 and 13/101 polarity-cue rows for the two post-processed conditions. A cross-input Jaccard comparison suggests polarity failures overlap more across front-ends (0.60) than general drift sets (0.23). The paper argues from this pilot that surface accuracy, term-script rendering, chunk-level consistency, and medical-meaning preservation should be evaluated separately. The authors explicitly frame all results as descriptive, single-annotator, four-lecture pilot findings, with code and data released.

Significance. If the findings hold, the paper makes a useful methodological point: in code-switched medical lecture transcription, script-level readability metrics can improve substantially under free-form post-processing while meaning-level errors persist or are introduced. The strengths are the reproducible public repository, metrics frozen before analysis (§5.1), the deterministic rendering computation, and the cross-input control that applies the same post-processing to a different raw front-end. The main weakness is that all semantic/polarity measurements rest on a single author-annotator and author-created references. The qualitative trade-off is likely robust, but the exact headline counts are not independently verified. The contribution is diagnostic and pilot-scale rather than a benchmark.

major comments (2)

[§5.2, §4.2, Tables 3–4] The headline semantic-drift and polarity counts (34/36 drifted rows; 11/13 polarity failures; Jaccard 0.60 vs 0.23 in §6.4) are produced by a single author-annotator in a single pass, against references created by the same team. The paper discloses this in §8, but the Abstract and §6 present the numbers as results. Because the central trade-off claim uses these counts, the measurement instrument needs independent replication: a second annotator with agreement statistics, or a blinded protocol that re-binds references. Without this, the exact magnitudes in the Abstract are not independently supported.
[§6.4, Appendix B] The 'introduced' vs 'propagated' attribution is made on selected examples rather than a systematic coding of all drifted rows, and the annotator knows which output is raw and which is post-processed. The observation that 'several rows that all raw front-ends transcribed correctly flipped to the same wrong value after post-processing' is not accompanied by an exhaustive count or a coding protocol. This weakens the causal reading that post-processing created specific polarity failures. The authors should either quantify the introduced/propagated split with a formal, ideally blinded, coding or explicitly restrict the claim to the documented examples.

minor comments (5)

[§2, References] Typos: 'diﬀicult' should be 'difficult'; 'T able 1' should be 'Table 1'; 'i2b2/V A challenge' should be 'i2b2/VA challenge'.
[§5.1] The 'capped' in 'capped recall' is not formally defined. Please state the cap (e.g., each occurrence counted at most once) and the aggregation formula over lectures.
[§6.4] The Jaccard values are correct (9/15=0.60; 13/57≈0.23), but the denominators should be identified explicitly: the 15 unioned failures over 101 polarity rows and the 57 unioned drift rows over 282 reference sentences.
[Table 3] The 'pooled total' row omits the faithful/minor breakdown, making it inconsistent with the per-lecture rows. Please show the pooled faithful/minor/drift counts or restructure the table.
[§5.1, Table 2] Report the number of curated domain-term occurrences per lecture and per condition. Without this, the pooled Korean phonetic counts and the rendering rates are difficult to compare across lectures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pilot's headline numbers are empirical observations against author-created references; the disclosed single-annotator design is a validity limitation, not a constructional reduction.

full rationale

No load-bearing circular step was found. The paper is a retrospective evaluation of a historical, unchanged post-ASR pipeline (§3, §4.1), not a derivation of predictions from the data it evaluates. The headline English-script rendering rate (§5.1) is a capped recall of author-curated reference terms, so it measures exactly the behavior the post-processor was designed to produce; however, no parameter was fitted to this metric, and the gain is not forced by construction—the paper reports lectures where post-processing *lowered* rendering (anthrax_01 0.56→0.54; anticancerdrugs_02 0.81→0.75, §6.2), showing the result is contingent. The semantic-drift and polarity counts come from a single author-annotator (§5.2), and the paper explicitly discloses this in §8: "References and labels were created via single-pass annotation by the system developers (single author-annotator) with no second pass, independent replication, adjudication, or inter-annotator agreement." That is a real threat to the reliability and generalizability of the counts, but it is not a circular reduction: the claims are reports of labels, not equations whose output equals their input by definition. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation to prior author work. The paper also bounds its own claims (Abstract: "This single-annotator pilot documents concrete failure modes rather than population rates"). Accordingly, the correct finding is no significant circularity; the evaluator-overlap concern belongs under validity risk, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No numerical parameters are fitted and no entities are invented; the hand-chosen inputs are the curated term lists, the chunking policy, the prompt, and the annotation scheme, all of which directly shape the reported rates. The paper's own limitation statements (§8) acknowledge these choices bound every claim.

free parameters (4)

Curated per-lecture domain-term lists (canonical English forms, accepted variants, korean_phonetic variants) = hand-curated sets from references and observed outputs
The English-script rendering rate is computed over these author-chosen term lists (§5.1); the headline 0.39→0.71 improvement is directly sensitive to which terms and accepted variants are included.
3-minute non-overlapping chunking policy (gpt-4o conditions) = 3 minutes
Hand-chosen policy that defines the chunk-level script-consistency metric and the chunked raw/prompted gpt-4o outputs (§4.1, §5.1).
Condition-4 minimal Korean prompt = “의학 강의 녹음입니다. 영어 의학 용어는 영어로 표기합니다.”
Prompt design choice for the prompt-sensitivity check; effects varied in sign across lectures (§6.2), and the authors treat it as preliminary only.
Semantic-drift taxonomy and 101 polarity-cue row set = 9-label taxonomy; 101 rows identified by annotator
The 34/36 drift counts and 11/13 polarity failures derive entirely from this single-annotator scheme (§5.2); a different annotator could re-bound both row sets and labels.

assumptions (4)

domain assumption The author-created reference transcripts are a valid content-preserving target for semantic-faithfulness evaluation
§4.2 states references 'preserve medical content... polarity/direction cues' but they are not independent gold standards; all drift and polarity results are measured against them.
domain assumption A single author-annotator's labels are reliable enough to support descriptive claims
§5.2/§8: one pass, no adjudication or inter-annotator agreement; the headline numbers rest on this premise and the paper bounds its claims accordingly.
domain assumption English-script rendering is a meaningful readability proxy for Korean-English medical study transcripts
§5.1 defines the headline metric; the claim that script rendering aids scan/search/alignment is an assumption about student utility not directly measured in the study.
domain assumption 2026 re-execution of whisper-1 adequately represents the historical path
§3/§10: whisper-1 is re-run in 2026 and 'should not be read as a 2023 snapshot'; internal API changes could shift the baseline that the post-processing gain is measured from.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures." pith.science (2026). https://pith.science/paper/P74ZTUP4

@misc{pith2026260717237,
  author       = {Pith},
  title        = {Pith review of: AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P74ZTUP4}},
  note         = {Machine review of arXiv:2607.17237}
}

read the original abstract

AI_LectureNote is a historical, readability-oriented post-ASR workflow for Korean-English medical lectures. It rewrites speech-to-text output into study transcripts while restoring Latin-script medical terms rather than Korean phonetic transliterations. We retrospectively evaluate the workflow on four author-recorded lectures across five conditions. In this pilot, post-processing raised the macro English-script rendering rate from 0.39 to 0.71 on the whisper-1 path and from 0.26 to 0.65 when applied to 3-minute chunked gpt-4o-transcribe output. However, English-script rendering did not imply semantic faithfulness: the two post-processed conditions showed semantic drift in 34 and 36 of 282 reference sentences and polarity failures in 11 and 13 of 101 polarity-cue rows. A descriptive cross-input comparison suggested different candidate failure patterns: polarity-failure sets overlapped more strongly across front-ends (Jaccard 0.60; 9 shared of 15 unioned failures) than general semantic-drift sets (Jaccard 0.23; 13 shared of 57 unioned drifts). This single-annotator pilot documents concrete failure modes rather than population rates and supports evaluating surface accuracy, term-script rendering, chunk-level script consistency, and medical-meaning preservation separately.

Figures

Figures reproduced from arXiv: 2607.17237 by the authors.

**Figure 1.** Evaluation design. Four author-recorded lectures were processed into five outputs and scored against author-created study-transcript references. The two post-processed conditions use the same historical AI_LectureNote rewriting stage. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Readability gain versus observed faithfulness risk, N=4. Left: macro English-script rendering by condition. Right: per-lecture semantic-drift rate for the two post-processed conditions. Chunking and prompt sensitivity. Under the 3-minute gpt-4o chunking policy, per-chunk rendering varied sharply within lectures (acuteinflammation_02 0.94 then 0.07; diuretics_01 0.00–0.29; anticancerdrugs_02 0.38 then 0.96), consiste… view at source ↗

**Figure 3.** Semantic-drift taxonomy for the two post-processed conditions. Bars show drift-label incidences by lecture and error type; labels are not mutually exclusive, so category counts can exceed unique drifted-row totals [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Critical medical polarity, N=4. Panel (a) shows wrong and omitted polarity-cue rows by condition; panel (b) shows the whisper-1 → AI_LectureNote polarity transition for diuretics_01. Polarity failures were rarer than general semantic drift but more clinically salient, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 linked inside Pith

[1]

• Adedeji, A., Joshi, S., & Doohan, B. (2024). The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models. arXiv:2402.07658. • Adedeji, A., Sanni, M., Ayodele, E., Joshi, S., & Olatunji, T. (2025). The Multi- cultural Medical Assistant: Can LLMs Improve Medical ASR Errors Across Borders? arXiv:2501.15310. • Chapman, W...

arXiv 2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

• Adedeji, A., Joshi, S., & Doohan, B. (2024). The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models. arXiv:2402.07658. • Adedeji, A., Sanni, M., Ayodele, E., Joshi, S., & Olatunji, T. (2025). The Multi- cultural Medical Assistant: Can LLMs Improve Medical ASR Errors Across Borders? arXiv:2501.15310. • Chapman, W...

arXiv 2024