{"id":"63722d8f-9548-446f-a5f0-53428fb6ab2a","arxiv_id":"2412.11203","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A Rasa-based Wolof task-oriented dialog system, trained on French MASSIVE data projected through an in-house French-Wolof machine translation system, achieves near-French intent classification but weaker slot filling.","lead":"The authors built a task-oriented chatbot for Wolof, the main language of Senegal, by translating French training examples and carrying their labels over with an automated projection method. The Wolof intent classifier reaches macro F1 0.995 versus 0.999 for French, while slot filling drops from 0.97 to 0.89.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation set is generated by the same projection pipeline as training, so the 0.995 Wolof intent F1 may reward fitting internally consistent but incorrect labels; human-annotated Wolof test data is needed to support the parity claim.","rationale":"The reader's weakest assumption is exactly the one I find load-bearing: the only evaluation of the Wolof classifier uses labels produced by the same automatic pipeline that created the training data. The paper is otherwise straightforward: it uses standard Rasa DIET with LaBSE and reports F1 on a French-to-Wolof projection of MASSIVE. The method section is explicit that identifiers replace labeled spans and that labels are copied from source, so the test-set labels are not independent. The paper's own limitation statements in Section 6 and the conclusion confirm the translation system's influence. There is no internal contradiction in the reported numbers; the problem is external validity. I credit the authors for a clear system description and for reporting the slot-filling gap, which is the most honest evidence of projection error in the paper. However, the headline claim of parity with French is not yet established for actual Wolof use. The proposed human-labeling check is the minimal experiment that would settle the concern: if native speakers agree with the projected labels and the classifier's F1 on their labels is similar, the claim holds; if not, the paper should be revised to describe the result as a synthetic benchmark only. Since the reader's conditional verdict already reflects this uncertainty, I recommend no change to the verdict.","tokens_in":10939,"tokens_out":5713,"duration_ms":55324,"concrete_test":"Select 200 held-out utterances from the projected Wolof test split, masking the projected intent and slot labels. Have two native Wolof speakers, blind to the French source and to each other, independently label each utterance for intent and slot spans, then measure inter-annotator agreement and compare their labels against the projected labels. If per-intent F1 against the human labels drops materially below the reported 0.995, or if a 95% confidence interval excludes parity with the synthetic result, the parity claim is an artifact of label projection rather than a property of real Wolof intent classification. As a complement, a second sample of 200 French utterances should be translated by human translators into natural Wolof and labeled by the same annotators, to test out-of-distribution robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central parity claim rests on the assumption that the projected Wolof labels are the labels a native speaker would assign. The Wolof test set is produced by the same in-house MT plus identifier back-filling pipeline that generated the training set, and intent labels are simply copied from the French MASSIVE source. Consequently, the 0.995 macro F1 in Table 2 measures the model's ability to fit the projection pipeline's label assignments on translationese, not its ability to classify natural Wolof. If the MT system makes a systematic meaning-changing error, the example still carries the original French intent label; a model trained on many such examples will learn the wrong mapping and still score highly on an identically projected test set. The paper's own Section 6 cautions that 'annotation projection is heavily influenced by the machine translation system' and that translating annotations outside context degrades slot quality, with slot micro F1 falling from 0.97 to 0.89 in Table 3. No human-annotated Wolof set, no error bars, and no release of the generated data or code are provided, so the parity claim is currently only evidence about the internal consistency of the synthetic pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for building a task-oriented dialog system in Wolof by projecting annotations from the French portion of the MASSIVE dataset using an in-house French–Wolof machine translation system. The projection replaces annotated spans with numeric identifiers wrapped in dollar signs before translation, then back-fills the translated spans. The authors train a Rasa/DIET intent classifier and slot filler with LaBSE embeddings on the resulting synthetic Wolof data. They report an intent-classification macro F1 of 0.995 on Wolof versus 0.999 on French, and a slot-filling micro F1 of 0.89 on Wolof versus 0.97 on French, concluding that the Wolof classifier performs similarly to the French one and that the approach is extensible to other low-resource languages.","tokens_in":11113,"tokens_out":4589,"duration_ms":43334,"significance":"If the claims are validated, the contribution is a practical, low-cost recipe for bootstrapping task-oriented dialog data in low-resource languages without explicit word-alignment models: a marker-based projection method coupled with a language-agnostic classifier pipeline. The use of the public MASSIVE dataset and the explicit discussion of translation-quality limitations are strengths. However, the central evaluation is conducted on a Wolof test set generated by the same MT and projection pipeline as the training set, so the reported F1 values currently measure internal consistency of the pipeline rather than performance on authentic Wolof. The slot-filling gap in Table 3 suggests that the projection is noisy, which further underlines the need for human-annotated evaluation.","major_comments":[{"comment":"The Wolof test set is produced by the same in-house French–Wolof MT system and annotation-projection pipeline that generated the training data, and intent labels are copied from the French MASSIVE source. The macro F1 of 0.995 therefore measures how well the model fits the projection pipeline's label assignments on translationese, not how well it classifies natural Wolof. The paper's own statement that 'annotation projection is heavily influenced by the machine translation system' and the slot-filling drop in Table 3 (micro F1 0.89 vs 0.97) indicate that the projected labels are noisy. To support the abstract's parity claim, the authors should evaluate on a human-annotated Wolof test set (e.g., a professionally translated and annotated sample) or, at minimum, report a manual inspection of a random sample of the projected test set, together with confidence intervals for the F1 scores.","section":"Section 6, Table 2"},{"comment":"The marker/identifier choice (dollar-sign-prefixed numeric identifiers) is justified only by an informal 'empirically tested' procedure on sample sentences, with no quantitative assessment of projection correctness. The paper calls the methodology 'robust' but provides no direct evidence of projection quality. Please add a projection-quality evaluation: for example, manually annotate a random sample of projected Wolof sentences and report precision/recall of slot projection, and compare against an existing projection baseline such as word alignment (López de Lacalle et al., 2020) or mark-then-translate (Chen et al., 2023). This is necessary to substantiate the robustness claim.","section":"Section 4"},{"comment":"The experimental setup is underspecified: no random seed, number of runs, or hyperparameter details for DIET/LaBSE are given, and the 80/20 split is described in a single sentence. Given that the reported intent F1 differences are small (0.999 vs 0.995), the reader cannot determine whether these differences are significant. Please report the variance over multiple runs or bootstrap confidence intervals, and state the exact train/test split procedure.","section":"Sections 5–6"},{"comment":"No code, configuration, or synthetic Wolof dataset is released, and the in-house MT system is described only by a BLEU score (26.38) without architectural or training-data details. Since the main contribution is a data-generation methodology and a chatbot engine, the results cannot be verified or built upon without these artifacts. Please provide a public release or a detailed availability statement, including the generated Wolof dataset and the Rasa configuration files.","section":"Reproducibility"}],"minor_comments":[{"comment":"The identifier format is written as '$0N$', which is ambiguous; please clarify whether the placeholder is '$' + '0' + N + '$' or something else, and align the notation between the text and Figure 1.","section":"Section 4"},{"comment":"The caption says 'Intent confidence matrix' but the figure appears to be a confusion matrix; please clarify the caption and fix the typo 'reccomandation_events' in the text.","section":"Section 6, Figure 6"},{"comment":"The phrase \"We've used it\" should be formalized to \"We used it\".","section":"Section 5"},{"comment":"Please specify whether slot filling is evaluated at the span level or with BIO tagging, and define what 'accuracy' means for slot filling.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the self-generated evaluation set: the parity claim in the abstract is not supported without human-annotated Wolof data. The slot-filling gap in Table 3 and the paper's own caveats suggest the pipeline may be producing internally consistent but incorrect labels. I would not accept the paper in its current form, but the issue is fixable within the manuscript's scope by adding a human evaluation component, quantitative projection-quality assessment, and a data/code release. The paper is borderline for a journal; if the authors address the above, it could become a useful contribution to low-resource ToDS."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take.\n\nThe paper is a clean, readable systems paper. The new bit is the $0N$ identifier trick for annotation projection: replacing labeled spans with dollar-signed numbers before MT, which preserves them through translation better than plain markers. That is a small but genuine contribution, and the authors did the right thing by empirically testing marker variants instead of assuming one would work. The engineering around Rasa/DIET/LaBSE is also sensible and described clearly. I also credit the authors for acknowledging in Section 6 that annotation projection is heavily influenced by MT quality and that translating annotations out of context degrades slot filling.\n\nThe elephant in the room is the evaluation. The Wolof train and test sets both come from the same in-house French-to-Wolof MT and projection pipeline, with intent labels copied from French. So the 0.995 macro F1 on Wolof measures how well the model fits the pipeline's label assignments, not how well it handles human Wolof. The parity claim with French is therefore only about internal consistency. The stress-test note is exactly right. The slot-filling gap (0.89 vs 0.97 micro F1) is the honest signal here, and even that is on synthetic data. Without a human-annotated Wolof test set, error bars, or a release of the generated data, the robustness and extensibility claims are not yet supported.\n\nI also note the absence of baseline comparisons. The paper discusses marker-based projection (Chen et al., 2023) and alignment-based methods, but does not empirically compare against them on the same Wolof data. That weakens the claim that the $0N$ approach is better than alternatives.\n\nWho is this for? Practitioners building task-oriented bots for Wolof or similar low-resource languages. For them, the recipe alone is valuable even before evaluation is tightened. The paper is not a conceptual breakthrough, but it is a legitimate extension of translate-train with a practical twist.\n\nMy recommendation: send it to peer review, but with the clear expectation of major revision. The authors need to (1) evaluate on human-annotated Wolof, ideally with a small native-speaker study; (2) report variance across seeds; (3) compare against at least one baseline projection method; and (4) release the generated Wolof data and code. The core idea is plausible and worth refereeing; the current evidence just does not justify the headline parity claim.\n\nI'd bring it to a reading group as a case study in evaluation circularity, but I probably won't cite it in my own work until the data is out.","headline":"A practical Wolof chatbot pipeline with a neat identifier trick, but the headline parity claim rests on a self-generated test set and needs human-validated evaluation before it can be believed.","tokens_in":11681,"tokens_out":1309,"would_cite":false,"duration_ms":14896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a Wolof task-oriented chatbot can be trained on machine-translated French examples, achieving 0.995 macro F1 for intent classification, close to the 0.999 French score.","keywords":["Wolof","task-oriented dialog systems","annotation projection","cross-lingual transfer","low-resource languages","intent classification","slot filling","machine translation"],"falsifier":"Hand-translate and hand-annotate a few hundred Wolof utterances from the same 27 MASSIVE intents, run the trained Wolof intent classifier on them, and compare the macro F1 with the reported 0.995; a substantially lower score on human-annotated data would show that the near-parity is an artifact of evaluating on self-consistent machine-generated labels.","tokens_in":10693,"feed_emoji":"🤖","tokens_out":11257,"duration_ms":92346,"temperature":0.7,"pith_summary":"The paper tries to show that a usable task-oriented dialog system for Wolof can be built without manually annotated Wolof data, by projecting French training annotations into Wolof through an in-house machine-translation system. The core idea is to replace each annotated French span with a dollar-sign-and-number identifier before translation, then back-fill the translated sentence with the stored translation of the span, avoiding the label-loss problems of tags and brackets. On 27 intents and 9,638 utterances from the MASSIVE French subset, the generated Wolof intent classifier reaches 0.995 macro F1 versus 0.999 for French, which the authors read as evidence that the synthetic Wolof data preserves intent structure. Slot filling is weaker, at 0.86 macro F1 versus 0.96 for French, because multi-word annotations are translated outside their sentence context. The authors also claim the language-agnostic pipeline makes the approach extendable to other low-resource languages, which matters because such languages are typically left out of LLM-based assistants.","feed_headline":"Wolof chatbot intent accuracy matches French via translated data","feed_subtitle":"A modular chatbot from machine-translated French examples reaches 0.995 macro F1 on Wolof intents.","key_machinery":"The load-bearing mechanism is the identifier-based annotation projection procedure: the system replaces every labeled French span with a token of the form `$0N$`, translates the modified sentence with a fine-tuned French-to-Wolof machine-translation model, then replaces each surviving identifier with the stored translation of the original span and reattaches its label. This avoids separate word-alignment models and, the authors report, preserves labels through translation more consistently than XML tags, brackets, parentheses, or bare numbers. The second component is the chatbot generation engine, built on the Rasa open-source dialogue framework, that compiles Excel domain files into a working chatbot with a fixed language-agnostic pipeline; the pipeline uses LaBSE multilingual sentence embeddings and the DIET model (Dual Intent and Entity Transformer) for intent detection and slot filling. The seed data is the French subset of MASSIVE, 9,638 examples across 27 intents and 10 domains.","core_discovery":"The central claim, stated on the paper's own terms, is that annotation projection can be made reliable enough for intent classification by substituting labeled spans with `$0N$` identifiers before machine translation. The authors tested XML tags, brackets, parentheses, and plain numeric identifiers, and found that only the dollar-sign form consistently survives translation, so the translated sentence retains placeholders that can be replaced by the stored translations of the original spans. Feeding the resulting Wolof synthetic data through a fixed language-agnostic intent-and-slot pipeline gives a Wolof intent classifier with 0.995 macro F1 on the 27-intent benchmark, effectively matching the French classifier trained on the source data (0.999 macro F1). The authors attribute the lower slot-filling score (0.86 macro F1) to annotation fragments being translated independently of their sentence context, and they read the intent result as evidence that the methodology is sound and portable.","pith_inferences":["Editorial: Because the Wolof test set is synthetic, the 0.995 figure is best read as internal consistency with the projection pipeline; a human-annotated Wolof benchmark is the test the paper does not provide.","Editorial: The `$0N$` finding suggests a cheap design heuristic for other low-resource projections: before building alignment machinery, search for identifier token forms that survive the target machine-translation model, since a surviving placeholder removes the need for word alignment.","Editorial: Improving translation of multi-word slot annotations with context, or retranslating each annotation along with its surrounding sentence, would likely close the slot-filling gap faster than improving intent classification.","Editorial: Field deployment will additionally face the orthography split between official and conventional Wolof writing; the paper names spelling correction as future work, and without it a benchmark trained on official forms may underperform on real user text."],"forward_implications":["A Wolof intent classifier with 0.995 macro F1 can be produced from translated French MASSIVE data without any human Wolof annotation.","The same chatbot-generation engine can be pointed at any language with a source dataset and a translation system, because the intent pipeline does not change with the language.","Modular chatbots built this way offer output control that the authors contrast with LLM-based assistants, whose hallucination risk motivates the work.","Slot filling, at 0.86 macro F1, is the weaker link, and the authors tie it to decontextualized translation of span annotations.","Improving the underlying French-to-Wolof translation quality is the lever that would most directly raise the quality of the synthetic Wolof dataset."],"supporting_citations":[{"why":"Supplies the MASSIVE multilingual corpus, whose French subset is the source of all training and evaluation data.","marker":"FitzGerald et al., 2023"},{"why":"Provides the earlier French-to-Wolof machine-translation work and parallel data that the in-house translation system builds on.","marker":"Mbaye et al., 2024"},{"why":"Small-100 is the multilingual translation model fine-tuned for French-to-Wolof, and its quality directly bounds the annotation projection.","marker":"Mohammadshahi et al., 2022"},{"why":"Marker-based label projection approach that the authors test and then move away from, motivating the `$0N$` identifier strategy.","marker":"Chen et al., 2023"},{"why":"Rasa is the open-source dialogue framework that the chatbot generation engine compiles Excel files into.","marker":"Bocklisch et al., 2017"},{"why":"LaBSE is the language-agnostic sentence embedder that lets the intent pipeline treat French and Wolof uniformly.","marker":"Feng et al., 2022"},{"why":"DIET is the model inside the Rasa pipeline that performs the intent classification and slot filling being evaluated.","marker":"Bunk et al., 2020"},{"why":"Establishes the translate-train cross-lingual transfer paradigm for task-oriented dialog that this approach extends to Wolof.","marker":"Schuster et al., 2019"}],"fun_headline_variants":["Wolof intent classifier matches French via translated data","Machine-translated data lifts Wolof chatbot to French accuracy","Low-resource Wolof chatbot achieves French-like intent scores","Modular approach gives Wolof chatbot 0.995 intent F1","Translated placeholders yield Wolof chatbot on par with French"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the French-to-Wolof translation preserves both the intent of each utterance and the meaning of each labeled span, so the projected Wolof labels are correct; the paper cannot verify this because its Wolof test set is produced by the same translation and projection pipeline.","fun_headline_variants_meta":{"raw":{"variants":["Wolof intent classifier matches French via translated data","Machine-translated data lifts Wolof chatbot to French accuracy","Low-resource Wolof chatbot achieves French-like intent scores","Modular approach gives Wolof chatbot 0.995 intent F1","Translated placeholders yield Wolof chatbot on par with French"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1468,"prompt_tokens":906,"completion_tokens":562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":522,"tokens_out":562,"duration_ms":4933,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:10:44.981284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-translate and hand-annotate a few hundred Wolof utterances from the same 27 MASSIVE intents, run the trained Wolof intent classifier on them, and compare the macro F1 with the reported 0.995; a substantially lower score on human-annotated data would show that the near-parity is an artifact of evaluating on self-consistent machine-generated labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the earlier French-to-Wolof machine-translation work and parallel data that the in-house translation system builds on."},{"cited_title":"SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages","cited_arxiv_id":"2210.11621","evidence_quote":"Small-100 is the multilingual translation model fine-tuned for French-to-Wolof, and its quality directly bounds the annotation projection."}],"review_version":1}