SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?

Referee: [Section 3] Section 3 (Benchmark Construction): The description of SEA-BED dataset curation, task selection, and data sourcing is insufficiently detailed. Without explicit information on how examples were collected, balanced, or quality-controlled across the 10 languages, it is impossible to rule out confounds that could produce the observed non-uniformity.

Authors: We agree that greater methodological transparency is needed. In the revised manuscript, we will substantially expand Section 3 to include: (1) detailed sourcing information for each task and language (including original data providers and any translation or adaptation steps); (2) explicit balancing criteria used to ensure comparable example counts and difficulty distributions across the 10 languages; and (3) quality-control procedures, such as automated filtering rules, human verification protocols, and inter-annotator agreement metrics where applicable. These additions will allow readers to better assess potential confounds. revision: yes

Referee: [Section 4] Section 4 (Results): Performance figures and language-task matrices are presented without statistical significance tests, confidence intervals, or error bars. This omission makes it difficult to assess whether the reported variations across languages and tasks exceed what would be expected from sampling noise alone.

Authors: We acknowledge this limitation in statistical reporting. We will update all performance tables and figures in Section 4 to include 95% confidence intervals (computed via bootstrapping) and error bars. In addition, we will report results of statistical tests (e.g., Friedman tests for overall language and task effects followed by post-hoc Wilcoxon signed-rank tests with Bonferroni correction) to evaluate whether observed differences across languages and tasks are statistically significant beyond sampling variability. revision: yes

Referee: [Section 5] Section 5 (Discussion): The interpretation that non-uniform accuracy demonstrates representational inconsistency does not address the plausible alternative that gaps track differences in pretraining data volume or token coverage for each SEA language. A post-hoc correlation between model performance and estimated pretraining exposure per language would be required to support the stronger claim.

Authors: Our primary claim concerns the empirical observation of non-uniform performance across languages and tasks, which already challenges the assumption of a perspective-independent semantic space. We agree, however, that discussing alternative explanations strengthens the paper. In the revision we will add a dedicated paragraph in Section 5 that (a) acknowledges pretraining data imbalance as a plausible contributing factor and (b) presents a post-hoc analysis correlating model performance with available proxies for language-specific pretraining exposure (e.g., cited web-crawl statistics and model documentation). Because exact token counts for proprietary models are not publicly disclosed, the analysis will necessarily rely on these proxies and will be presented as suggestive rather than definitive. revision: partial