Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

Laura Diosan; Mihai Nadas

REVIEW 4 major objections 5 minor 22 references

GPT-4o with a three-shot prompt restores Romanian diacritics at 96% average accuracy, outperforming nine other large language models and beating an echo baseline by 19%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:51 UTC pith:SVKSCKYY

load-bearing objection Useful benchmark with reproducible code, but the headline numbers measure text copying as much as diacritic restoration. the 4 major comments →

arxiv 2511.13182 v3 pith:SVKSCKYY submitted 2025-11-17 cs.CL

Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

Mihai Nadas , Laura Diosan This is my paper

classification cs.CL

keywords diacritic restorationRomanianlarge language modelsfew-shot promptingGPT-4obenchmark evaluationprompt engineeringerror analysis

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether off-the-shelf large language models can restore diacritics in Romanian text—a common problem when diacritics are lost in digital input—and how they compare to a trivial baseline that simply echoes the stripped input. Testing ten models with prompt templates ranging from zero-shot to three-shot instructions, it finds that GPT-4o with a three-shot prompt achieves the highest average score, about 19% above the echo baseline, while models like Llama 2 fall far below it. The paper also shows prompt design matters: adding a few examples ('shots') consistently improves restoration, and error analysis pinpoints the â/î distinction as the hardest case. A sympathetic reader would care because the results suggest that for some languages, a general LLM with carefully chosen examples can restore diacritics without task-specific fine-tuning.

Core claim

The paper's central discovery is that GPT-4o with a three-shot prompt template achieves a Total Average Score of 0.9639 across eight evaluators on two Romanian datasets, 19% higher than the Summa Echo baseline (0.8100), and that this result holds across all evaluated OpenAI models (RPR >= 1.11). It also finds that the three-shot template is the best overall prompt for most models, while Meta's Llama 2 models score near zero because they fail to follow the instruction to output only restored text. Error analysis shows that most remaining errors come from confusion between â and î, sentence-initial capitals, and over-generation by models like Mixtral. The paper frames these findings as evidenc

What carries the argument

The key machinery is the evaluation protocol centered on the Total Average Score (TAS), the arithmetic mean of eight evaluators—restoration accuracy and Levenshtein-based error rates at character and word levels, each in case-sensitive and case-insensitive variants—computed over two 1,000-sentence subsets (DLRLC and CRAWLER). Against this, the 'Echo' baseline (merely copying the diacritic-stripped input) supplies a reference point, and the Relative Performance Ratio (RPR) divides a model's maximum TAS by the baseline's TAS to express how much better (or worse) a model is at restoring diacritics. The three-shot prompt template, which gives explicit input-output examples, is the prompt design

Load-bearing premise

The evaluation's headline numbers assume that averaging accuracy across every character and word is a faithful measure of diacritic restoration; since most characters and words have no diacritics, the baseline's 0.81 score is mostly non-diacritic similarity.

What would settle it

Compute the models' diacritic-only accuracy—say, F1 on the set of characters that carry diacritics in the reference—on the same 2,000 sentences; if GPT-4o's score on those tokens is not clearly above the echo baseline (which would score 0 on diacritics), the central claim that LLMs can restore diacritics would collapse. Equivalently, report the TAS after removing all non-diacritic characters from both output and reference.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

GPT-4o in three-shot mode can be used directly as a Romanian diacritic restorer at near-96% average accuracy, no fine-tuning required.
Few-shot examples in prompts are a reliable lever: adding up to three examples improved every model except Meta's, so prompt engineering, not just model choice, drives success.
Many open-source models (Llama 2, Llama 3 8B, Mixtral, RoLlama) deliver below-baseline results out of the box, meaning they need targeted fine-tuning before use for this task.
The dominant error pattern—â/î confusion—means a simple rule-based post-correction step could eliminate roughly a fifth of remaining errors.
For task-specific NLP in diacritic-rich languages, a small number of model choices currently separate acceptable performance from failure; model size alone does not guarantee competence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the baseline score is inflated by the many non-diacritic characters and words that any accurate copy will match, the real gap between GPT-4o and the echo on diacritics alone is likely much larger than the reported 19%—and the failures of the below-baseline models are even more severe than TAS suggests.
The three-shot template's third example uses pre-1993 orthography (î vs â); on post-1993 test data this might actually confuse some models, so separating orthographic regimes could change rankings.
The paper's reported correlation between model size and performance may be confounded by prompt compliance; a smaller instruction-tuned model (GPT-4o) outperformed larger ones, suggesting architectural instruction-following capacity is the real driver.
A natural extension would be to evaluate on diacritic-level precision/recall metrics instead of overall character accuracy, which would give a clearer signal for practical use and for fine-tuning targets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful benchmark with reproducible code, but the headline numbers measure text copying as much as diacritic restoration.

read the letter

Quick take: the paper does real work and ships code, but the main quantitative claims don't measure what they claim. The Echo baseline—a model that restores nothing—scores 0.81 on the Total Average Score. That alone tells you most of the score comes from matching characters that don't need diacritics. So the '19% above baseline' and the RPR values are not a clean measure of diacritic restoration.

What's genuinely new: a broad comparison of nine models, five prompt templates, two datasets, and eight evaluators for Romanian diacritic restoration. Nobody else has published this specific matrix. The three-shot result—GPT-4o improving with more shots—is a useful empirical observation. The public repository with code, prompts, and eval scripts deserves credit. Section 7's error analysis is the most credible part: it tracks confusion counts for â/î, context effects, and over/under-generation, and it points to concrete post-processing gains.

Where it falls down: the main evaluation doesn't isolate diacritics. The eight evaluators are accuracy and Levenshtein-based scores over whole text, so a model that simply echoes the stripped input gets high marks for all the characters it didn't need to change. The baseline's 0.81 is not a sign that the baseline is 'good at diacritics'; it's a sign that the metric is mostly about copying. As a result, the RPR values below 1 for Llama 3 8B, RoLlama, Mixtral, and Llama 2 don't tell you how many diacritics they restored. In fact, the Llama 2 examples in Appendix E show TAS = 0.0 because the model wrapped the answer in extra prose—a format failure, not a diacritic failure. The paper even says this, then treats it as underperformance on the task.

The authors have no confidence intervals, and there's no diacritic-only accuracy in the main results. If they recompute the headline scores on diacritic positions only, the ranking and baseline comparisons may change materially. The qualitative direction—GPT-4o/OpenAI strong, Llama 2 poorly compliant—is plausible, but the evidence as presented is not reliable.

Verdict: this is a solid starting point, not a finished evaluation. The error analysis and open resources make it worth engaging with, and the flaw is fixable. I'd send it to peer review and ask for a diacritic-only metric and a format-compliance analysis. A reader interested in Romanian diacritic restoration will get value from the prompts and the error taxonomy, but should not trust the TAS table as it stands.

Referee Report

4 major / 5 minor

Summary. This paper evaluates a set of proprietary and open-weight LLMs on the task of restoring Romanian diacritics in stripped text. The authors build two 1,000-statement subsets from dexonline data, design five prompt templates (zero-shot to three-shot), and score outputs with eight character- and word-level accuracy/error-rate evaluators aggregated into a Total Average Score (TAS). The main result is that GPT-4o with the three-shot prompt achieves TAS 0.9639, 19% higher than an 'Echo' baseline that simply regurgitates the diacritic-stripped input, while several open models (e.g., Llama 2 7B, Llama 2 70B, Mixtral) fall below that baseline. The paper also includes a Section 7 error analysis on DLRLC-1000 with diacritic confusion counts and over/under-generation statistics, and the authors provide code and prompts in a public repository.

Significance. The paper is practically relevant: Romanian diacritic restoration is a real preprocessing problem, and a systematic comparison of current LLMs with prompt variants is useful to practitioners. The release of code, prompt templates, and evaluation scripts is a strength that supports reproducibility. If the headline TAS/RPR results were specifically about diacritic restoration, the finding that GPT-4o outperforms an echo baseline by 19% and that some open models fail to beat the baseline would be a useful contribution. However, the central quantitative claim is currently undermined by the aggregate evaluator design, as I detail below. The paper's own Section 7 error analysis contains diacritic-level signal, but it is not integrated into the headline metrics, so the main comparison remains ambiguous.

major comments (4)

[Sections 4.3, 5, Appendix C; Table 4] The TAS/RPR framework does not isolate diacritic restoration. The Echo baseline outputs the stripped input and hence restores zero diacritics, yet it receives TAS 0.8100 (Table 4). This is only possible if the eight evaluators (RA_*, RER_*) credit characters and words that never required a diacritic. Appendix C defines RA as 'the percentage of characters correctly restored with diacritics,' but the 0.8100 baseline demonstrates that the operative implementations are not restricted to diacritic-bearing positions. Consequently, the Section 6 claim that GPT-4o's TAS is '19% higher than the baseline' is not specifically about diacritics; it conflates copying fidelity with restoration accuracy. For a model such as Llama 2 7B (TAS 0.002), 'underperforming the echo' may reflect format failures rather than inability to restore diacritics. I recommend recomputing all headline scores using diacriti
[Section 6, Table 4] The RPR comparisons inherit the same artifact. RPR is defined as MTAS(LLM)/MTAS(Echo). Since the Echo's denominator is inflated by non-diacritic text matches, the RPR values (e.g., 1.190, 1.154, 0.946, 0.002) are not interpretable as ratios of diacritic restoration ability. A model could score above 1.0 by reproducing most of the stripped text and adding a fraction of diacritics, or a model could score below 1.0 by adding extra commentary even while restoring all diacritics. The paper should either replace these ratios with diacritic-specific metrics or clearly present them as 'general output fidelity' rather than restoration performance. As written, the claim that OpenAI models 'outperform the baseline by at least 11%' is not supported for the actual task.
[Section 6, first paragraph; Appendix C] The paper reports no confidence intervals or significance tests. The headline TAS values (0.9639 vs 0.8100 vs 0.7663, etc.) are point estimates over 2,000 statements. Given the small sample and model stochasticity, the ranking among mid-tier models (e.g., Llama 3 8B at 0.7663 vs Echo at 0.8100) could reasonably be within noise. I request per-metric standard errors or a bootstrap/permutation test, at least for the top models and the baseline. This is especially important because the main claims are comparative.
[Section 7] The error analysis is the most task-relevant part of the paper (confusion counts for ă/î/â, position statistics, over-generation). However, it is not connected to the TAS/RPR results. Section 7.5 states that rule-aware post-processing could eliminate 19% of remaining mistakes, but the paper does not report a diacritic-only aggregate score for each model. If the authors added a column to Table 4 for diacritic-level accuracy (or F1), the reader could verify whether the ranking survives the metric fix. Without that, the headline story and the error analysis remain disjoint.

minor comments (5)

[Table 1 and Table 2] The 'Average Words per Statement' and 'Average Diacritics per Statement' values appear to be off by a factor of 1000: total words 11,975 over 1,000 statements is 11.975, not 11,975; total diacritics 4,924 over 1,000 statements is 4.924, not 4,924. The same applies to the crawler subset (26.163 vs 26,163; 8.736 vs 8,736). Please correct the decimal separators.
[Section 6, paragraph 3] The phrase 'outperforming GPT-3.5 Turbo in RPR by 0,72' is unclear; presumably a value like 0.072 is intended, but the sentence should be rewritten for precision.
[References] Reference [11] (OpenAI GPT-4o announcement) has a typo: 'Gello gpt-4o' should be 'Hello GPT-4o'. Reference [3] uses a placeholder URL ('https://osf.io/your-preprint-url') and should be completed.
[Appendix B, Stage Five] The third shot example intentionally uses pre-1993 orthography ('mîine', 'Cîinele') to test historical forms, but the paper's stated scope (Section 3.2) is post-reform orthography. Please clarify whether the three-shot template with a pre-reform example helps or harms models, and whether that example was chosen deliberately.
[Section 6, Table 4 ordering] Table 4 lists the models in descending MTAS; the 'Llama 2 70B' row shows 0.146, while Section 6 text mentions 0.6008 for Llama 2 70B in a list of underperformers. There is a discrepancy: the text says 'Llama 2 7B, and Llama 2 70B models recorded ... of 0.6463, 0.6079, 0.7663, 0.2501, and 0.6008, respectively' but Table 4 lists 0.002 and 0.146. Please reconcile the numbers and verify the reporting.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical benchmark; TAS and RPR are descriptive aggregations of measured outputs, and the Echo baseline is a deliberately trivial comparator, not a fitted parameter.

full rationale

The paper makes no derivation claim that reduces to its own inputs. It measures black-box LLM outputs on a fixed corpus, aggregates eight evaluator scores into TAS, and divides by the Echo model's TAS to form RPR. RPR is a definitional normalization, not a hidden reuse of the target result, and no fitted parameter is later renamed as a prediction. The Echo baseline is an explicit no-op baseline (Section 4.3) rather than a learned model, so its score does not create a fitted-input/prediction circularity. The strongest concern is that the Echo baseline achieves TAS 0.8100 despite restoring zero diacritics, which suggests the aggregate metrics credit non-diacritic characters and words; this is a measurement-validity limitation, noted in Section 10.4, not a circular-reasoning defect. There is also no load-bearing self-citation chain: the central comparison rests on the authors' own logged evaluations and on externally documented model outputs. Therefore no enumerated circular step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is an empirical benchmark, not a theoretical derivation, so there are no fitted free parameters or invented entities. The load-bearing assumptions are the representativeness of the two 1,000-sentence subsets, the validity of the aggregate evaluators, and the suitability of the Echo baseline.

axioms (4)

domain assumption The 1,000-statement subsets preserve the complexity and diversity of the full dexonline datasets.
Section 3.2 asserts representativeness based on average diacritics per word but does not describe the sampling method; all model scores come from these two subsets.
domain assumption TAS over eight accuracy and error evaluators is a valid measure of diacritic restoration.
Appendix C defines evaluators over all characters and words; because non-diacritic characters dominate, the Echo baseline scores 0.81 while restoring no diacritics, so the assumption is questionable.
domain assumption The Echo model that outputs the diacritics-stripped input is a meaningful neutral baseline.
Section 4.3 introduces this baseline; under the aggregate metric it rewards copying, so it is not neutral with respect to diacritic restoration.
domain assumption The evaluation targets post-1993 Romanian orthography, treating â-in-words as correct and pre-1993 î forms as errors.
Sections 3.1 and 10.1 select data to match the 1993 reform; conclusions about 'correct' restoration are restricted to this orthographic norm.

pith-pipeline@v1.3.0-alltime-deepseek · 13804 in / 11676 out tokens · 110077 ms · 2026-08-03T21:51:41.872129+00:00 · methodology

0 comments

read the original abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.

Figures

Figures reproduced from arXiv: 2511.13182 by Laura Diosan, Mihai Nadas.

**Figure 1.** Figure 1: Performance of Large Language Models (LLMs) in Diacritics Restoration Tasks: Results based on Total Average Score (TAS). The highest score for each model represents the Maximum Total Average Score (MTAS) Model Performance Comparison Against a Baseline In assessing the capabilities of Large Language Models (LLMs) in the task of diacritic restoration, it becomes essential to compare their performance against… view at source ↗

**Figure 2.** Figure 2: Reference diacritic distribution (pie chart) and per-model restoration counts (heat-map); darker cells indicate over-generation. of training data plays a crucial role in the models’ ability to accurately process and restore diacritics. 7 Error Analysis To address the reviewers’ request for a systematic error analysis we performed an additional pass on the DLRLC-1000 evaluation split using the extended err… view at source ↗

**Figure 3.** Figure 3: Performance of Large Language Models (LLMs) in Diacritics Restoration Tasks: Fully Detailed Results. This table showcases the comparative effectiveness of different LLMs, including OpenAI GPT-4 and its variants, against the Echo model baseline. Highlighted scores demonstrate the LLMs’ superior capability in accurately restoring diacritics in Romanian texts across various prompt templates, with detailed ins… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

22 extracted references · 6 linked inside Pith

[1]

arXiv preprint arXiv:2303.08774 (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

Pith/arXiv arXiv 2023
[2]

https://dexonline.ro/static/download/dex-database.sql.gz (2014), accessed: 2024-06-03

dexonline.ro, P.: dexonline.ro sql database. https://dexonline.ro/static/download/dex-database.sql.gz (2014), accessed: 2024-06-03

2014
[3]

OSF Preprint, https://osf.io/your-preprint-url (2023), accessed: 2024-06-03

Dillion, D., Mondal, D., Tandon, N., Gray, K.: Large language models as moral experts? gpt-4o outperforms expert ethicist in providing moral guidance. OSF Preprint, https://osf.io/your-preprint-url (2023), accessed: 2024-06-03

2023
[4]

https://github.com/jondurbin/airoboros (2023), accessed: 2024-06-03

Dorbin, J.: airoboros github repository. https://github.com/jondurbin/airoboros (2023), accessed: 2024-06-03

2023
[5]

https://wiki.dexonline.ro/wiki (2014), accessed: 2024-06-03

Frâncu, C.: Information about project dexonline.ro (ro). https://wiki.dexonline.ro/wiki (2014), accessed: 2024-06-03

2014
[6]

arXiv preprint arXiv:2401.04088 (2024)

Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.d.l., Hanna, E.B., Bressand, F., et al.: Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

Pith/arXiv arXiv 2024
[7]

In: ICAI

Laki, L.J., Yang, Z.G.: Automatic diacritic restoration with transformer model based neural machine translation for east-central european languages. In: ICAI. pp. 190–202 (2020)

2020
[8]

arXiv preprint arXiv:2405.07703 (2024)

Masala, M., Ilie-Ablachim, D.C., Corlatescu, D., Zavelca, M., Leordeanu, M., Velicu, H., Popescu, M., Dascalu, M., Rebedea, T.: Openllm-ro–technical re- port on open-source romanian llms trained starting from llama 2. arXiv preprint arXiv:2405.07703 (2024)

Pith/arXiv arXiv 2024
[9]

In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018) (2018)

Náplava, J., Straka, M., Straˇnák, P., Hajic, J.: Diacritics restoration using neural networks. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018) (2018)

2018
[10]

In: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP)

Nu¸ tu, M., L˝orincz, B., Stan, A.: Deep learning for automatic diacritics restoration in romanian. In: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP). pp. 235–240. IEEE (2019)

2019
[11]

https://openai.com/index/hello-gpt-4o/ (2024), accessed: 2024-06-03

OpenAI: Gello gpt-4o. https://openai.com/index/hello-gpt-4o/ (2024), accessed: 2024-06-03

2024
[12]

arXiv preprint arXiv:2312.11805 (2023)

Team, G., Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.B., Yu, J., Soricut, R., Schalk- wyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

Pith/arXiv arXiv 2023
[13]

arXiv preprint arXiv:2307.09288 (2023)

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

Pith/arXiv arXiv 2023
[14]

Dict,ionarul limbii române literare contemporane

Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y ., Zhou, Z., Gong, C., Shen, Y ., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420 (2023) 18 Mihai Dan NAD ˘AS, and Laura DIOS, AN A Data Corpus Source A: Dict,ionarul limbii române literare contemporane (DEX-DLRLC)Our research utiliz...

Pith/arXiv arXiv 2023
[15]

It evaluates precision in diacritic restoration while maintaining case fidelity

Restoration Accuracy - Case Sensitive, Character Level (RA_CS_CL): Computes the percentage of characters correctly restored with diacritics, considering case sensitivity. It evaluates precision in diacritic restoration while maintaining case fidelity
[16]

Restoration Accuracy - Case Insensitive, Character Level (RA_CI_CL): Focuses on character-level accuracy without case sensitivity, aiming to assess the model’s effectiveness in diacritic restoration irrespective of letter casing
[17]

It assesses each word as a whole unit, comparing the model’s output with the reference

Restoration Accuracy - Case Sensitive, Word Level (RA_CS_WL): Evaluates the accuracy of restoring entire words with diacritics accurately, with case sensitivity. It assesses each word as a whole unit, comparing the model’s output with the reference
[18]

C.2 Restoration Error Rate Evaluators The error rate is quantified by assessing the frequency and severity of inaccuracies in diacritic restoration against a reference text

Restoration Accuracy - Case Insensitive, Word Level (RA_CI_WL): Measures word-level accuracy without considering case sensitivity, providing insights into the model’s capacity to restore words with correct diacritics, overlooking case distinctions. C.2 Restoration Error Rate Evaluators The error rate is quantified by assessing the frequency and severity o...
[19]

It leverages the Levenshtein distance to quantify the minimum number of edits required

Restoration Error Rate - Case Sensitive, Character Level (RER_CS_CL): Mea- sures the frequency and severity of errors in character-level diacritic restoration, accounting for case sensitivity. It leverages the Levenshtein distance to quantify the minimum number of edits required
[20]

Restoration Error Rate - Case Insensitive, Character Level (RER_CI_CL): Calculates the character-level error rate without case sensitivity, focusing on errors purely related to incorrect diacritic placement
[21]

Restoration Error Rate - Case Sensitive, Word Level (RER_CS_WL): Determines the error rate at the word level with case sensitivity, using the Levenshtein distance to assess the extent and nature of errors
[22]

0.0 Llama 2 70B, Basic - Zero-Shot Sure! Here’s the sentence with the diacritics restored:

Restoration Error Rate - Case Insensitive, Word Level (RER_CI_WL): Evaluates the word-level error rate without case sensitivity, focusing on diacritic placement over case correctness. These evaluators provide nuanced insights into the models’ capabilities and limi- tations in restoring diacritics, contributing valuable knowledge to the field of natural la...

[1] [1]

arXiv preprint arXiv:2303.08774 (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

Pith/arXiv arXiv 2023

[2] [2]

https://dexonline.ro/static/download/dex-database.sql.gz (2014), accessed: 2024-06-03

dexonline.ro, P.: dexonline.ro sql database. https://dexonline.ro/static/download/dex-database.sql.gz (2014), accessed: 2024-06-03

2014

[3] [3]

OSF Preprint, https://osf.io/your-preprint-url (2023), accessed: 2024-06-03

Dillion, D., Mondal, D., Tandon, N., Gray, K.: Large language models as moral experts? gpt-4o outperforms expert ethicist in providing moral guidance. OSF Preprint, https://osf.io/your-preprint-url (2023), accessed: 2024-06-03

2023

[4] [4]

https://github.com/jondurbin/airoboros (2023), accessed: 2024-06-03

Dorbin, J.: airoboros github repository. https://github.com/jondurbin/airoboros (2023), accessed: 2024-06-03

2023

[5] [5]

https://wiki.dexonline.ro/wiki (2014), accessed: 2024-06-03

Frâncu, C.: Information about project dexonline.ro (ro). https://wiki.dexonline.ro/wiki (2014), accessed: 2024-06-03

2014

[6] [6]

arXiv preprint arXiv:2401.04088 (2024)

Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.d.l., Hanna, E.B., Bressand, F., et al.: Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

Pith/arXiv arXiv 2024

[7] [7]

In: ICAI

Laki, L.J., Yang, Z.G.: Automatic diacritic restoration with transformer model based neural machine translation for east-central european languages. In: ICAI. pp. 190–202 (2020)

2020

[8] [8]

arXiv preprint arXiv:2405.07703 (2024)

Masala, M., Ilie-Ablachim, D.C., Corlatescu, D., Zavelca, M., Leordeanu, M., Velicu, H., Popescu, M., Dascalu, M., Rebedea, T.: Openllm-ro–technical re- port on open-source romanian llms trained starting from llama 2. arXiv preprint arXiv:2405.07703 (2024)

Pith/arXiv arXiv 2024

[9] [9]

In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018) (2018)

Náplava, J., Straka, M., Straˇnák, P., Hajic, J.: Diacritics restoration using neural networks. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018) (2018)

2018

[10] [10]

In: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP)

Nu¸ tu, M., L˝orincz, B., Stan, A.: Deep learning for automatic diacritics restoration in romanian. In: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP). pp. 235–240. IEEE (2019)

2019

[11] [11]

https://openai.com/index/hello-gpt-4o/ (2024), accessed: 2024-06-03

OpenAI: Gello gpt-4o. https://openai.com/index/hello-gpt-4o/ (2024), accessed: 2024-06-03

2024

[12] [12]

arXiv preprint arXiv:2312.11805 (2023)

Team, G., Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.B., Yu, J., Soricut, R., Schalk- wyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

Pith/arXiv arXiv 2023

[13] [13]

arXiv preprint arXiv:2307.09288 (2023)

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

Pith/arXiv arXiv 2023

[14] [14]

Dict,ionarul limbii române literare contemporane

Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y ., Zhou, Z., Gong, C., Shen, Y ., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420 (2023) 18 Mihai Dan NAD ˘AS, and Laura DIOS, AN A Data Corpus Source A: Dict,ionarul limbii române literare contemporane (DEX-DLRLC)Our research utiliz...

Pith/arXiv arXiv 2023

[15] [15]

It evaluates precision in diacritic restoration while maintaining case fidelity

Restoration Accuracy - Case Sensitive, Character Level (RA_CS_CL): Computes the percentage of characters correctly restored with diacritics, considering case sensitivity. It evaluates precision in diacritic restoration while maintaining case fidelity

[16] [16]

Restoration Accuracy - Case Insensitive, Character Level (RA_CI_CL): Focuses on character-level accuracy without case sensitivity, aiming to assess the model’s effectiveness in diacritic restoration irrespective of letter casing

[17] [17]

It assesses each word as a whole unit, comparing the model’s output with the reference

Restoration Accuracy - Case Sensitive, Word Level (RA_CS_WL): Evaluates the accuracy of restoring entire words with diacritics accurately, with case sensitivity. It assesses each word as a whole unit, comparing the model’s output with the reference

[18] [18]

C.2 Restoration Error Rate Evaluators The error rate is quantified by assessing the frequency and severity of inaccuracies in diacritic restoration against a reference text

Restoration Accuracy - Case Insensitive, Word Level (RA_CI_WL): Measures word-level accuracy without considering case sensitivity, providing insights into the model’s capacity to restore words with correct diacritics, overlooking case distinctions. C.2 Restoration Error Rate Evaluators The error rate is quantified by assessing the frequency and severity o...

[19] [19]

It leverages the Levenshtein distance to quantify the minimum number of edits required

Restoration Error Rate - Case Sensitive, Character Level (RER_CS_CL): Mea- sures the frequency and severity of errors in character-level diacritic restoration, accounting for case sensitivity. It leverages the Levenshtein distance to quantify the minimum number of edits required

[20] [20]

Restoration Error Rate - Case Insensitive, Character Level (RER_CI_CL): Calculates the character-level error rate without case sensitivity, focusing on errors purely related to incorrect diacritic placement

[21] [21]

Restoration Error Rate - Case Sensitive, Word Level (RER_CS_WL): Determines the error rate at the word level with case sensitivity, using the Levenshtein distance to assess the extent and nature of errors

[22] [22]

0.0 Llama 2 70B, Basic - Zero-Shot Sure! Here’s the sentence with the diacritics restored:

Restoration Error Rate - Case Insensitive, Word Level (RER_CI_WL): Evaluates the word-level error rate without case sensitivity, focusing on diacritic placement over case correctness. These evaluators provide nuanced insights into the models’ capabilities and limi- tations in restoring diacritics, contributing valuable knowledge to the field of natural la...