Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

Sophia Maria

REVIEW 4 major objections 5 minor 3 cited by

The paper claims Compass-v3, a 245B-parameter mixture-of-experts model with 71B active parameters, is the current best e-commerce LLM for Southeast Asia, beating general frontier models and preserving general ability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:41 UTC pith:MSLSR27S

load-bearing objection Real engineering, overstated claims: the MoE systems work is plausible and useful, but the 'surpassing GPT-4 and DeepSeek' result rests on an in-house benchmark with no error bars, no release, and a competing model as judge. the 4 major comments →

arxiv 2509.09121 v1 pith:MSLSR27S submitted 2025-09-11 cs.CL

Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

Sophia Maria This is my paper

classification cs.CL

keywords domain-specific LLMmixture-of-expertsmultilingual e-commerceSoutheast Asian languageslow-resource multilingualoptimal transport preference optimizationinstruction tuningquantization

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Compass-v3 is a vertical-domain language model for Southeast Asian e-commerce: 245B total parameters with 71B active, a sparse mixture-of-experts (MoE) architecture with 16 large experts (4 active per token), and a 12T-token training mixture that combines curated e-commerce text, multilingual SEA corpora, and synthetic commerce instructions. The paper's central claim is that this domain-centered recipe—large experts, staged training, and an optimal-transport-based preference optimization called OTPO—makes the model state of the art on e-commerce tasks, surpassing the general-purpose frontier models it is compared with, including GPT-4 series, DeepSeek-V3.1, and Qwen3-235B. The claim matters because it would show that a comparatively compact vertical model can beat far larger general models in its niche while staying competitive on general benchmarks, and that such a model is practically deployable: the paper reports 86.74% accuracy on an in-house seven-language e-commerce benchmark, a 2x inference speedup from expert-aware FP8 quantization, and over 70% of LLM traffic on the platform now served by Compass-v3. If right, it shifts the scaling question from 'more parameters' toward 'the right data and alignment for a domain.'

Core claim

On its own terms, the paper claims that Compass-v3, a mixture-of-experts model with 245B total and 71B active parameters, is currently the best-performing LLM on Southeast Asian e-commerce tasks, surpassing the frontier general models it is compared with (including GPT-4 series, DeepSeek-V3.1, and Qwen3-235B). It reports 86.74% overall accuracy on its in-house e-commerce benchmark spanning seven languages and real business scenarios such as product guidance, after-sales service, and product understanding, and top-three results on open-source e-commerce suites. The paper further claims strong multilingual capability in low-resource Southeast Asian languages and competitive general English per

What carries the argument

The argument rests on three coupled mechanisms. (1) A sparse MoE architecture with 16 large experts, only 4 active per token: fewer-but-larger experts keep grouped GEMMs efficient and give each activated expert high representational capacity under a 71B-active budget. (2) A multi-stage training pipeline—general pretraining, e-commerce/SEA multilingual enhancement, reasoning, long-context extension to 128K tokens—with multi-token prediction (MTP) layers that share the embedding and LM head but update only their own parameters, speeding convergence and later inference. (3) Optimal-Transport Direct Preference Optimization (OTPO), the named alignment mechanism: optimal transport computes a token

Load-bearing premise

The load-bearing premise is that the private, seven-language in-house benchmark—with open-ended outputs scored by GPT-4.1, one of the models being compared, and no reported error bars or human-judge agreement—faithfully measures 'state-of-the-art e-commerce performance.'

What would settle it

Run a blind evaluation of the same e-commerce task families (product guidance, after-sales, attribute extraction, title generation) on a public dataset with human annotators: if GPT-4.1 or DeepSeek-V3.1 ties or beats Compass-v3 beyond inter-annotator variation, the core 'surpassing' claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

If Compass-v3's results are correct, a vertical MoE trained on 12T tokens can outperform general-purpose frontier models in its niche, so domain-specific data and alignment can compensate for fewer active parameters.
The multilingual results imply that low-resource Southeast Asian languages can be served well by a single model without giving up English-language general performance.
The expert-aware FP8 quantization results imply practical deployment: roughly 2x decode speedup with near-parity accuracy, making the 245B model affordable enough for production.
The production statistic—over 70% of LLM traffic on the platform now served by Compass-v3—is a direct consequence the paper uses to argue industrial viability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial: The headline 'surpassing' depends on private benchmarks and GPT-4.1 as judge, so the ranking may not survive independent replication; releasing the benchmark or a human-evaluated subset would settle it.
Editorial: The recipe (fewer-but-larger experts, mixed training stages, OTPO) is portable; applying it to finance, law, or biomedicine with public benchmarks would show whether domain-specific data or the recipe itself does the work.
Editorial: OTPO's token-level weighting should help any preference learning task with long, noisy contexts, such as multilingual dialogue or summarization alignment; the paper does not test those settings.
Editorial: The traffic-replacement claim implies an economic advantage over API-based general models, but the paper does not report cost, latency, or service metrics that would quantify that advantage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Real engineering, overstated claims: the MoE systems work is plausible and useful, but the 'surpassing GPT-4 and DeepSeek' result rests on an in-house benchmark with no error bars, no release, and a competing model as judge.

read the letter

First the bottom line: Compass-v3 is a real systems engineering effort — a 245B/71B-active MoE trained on 12T tokens and deployed at Shopee — and the paper contains some genuinely useful engineering details, especially the expert-aware FP8 quantization that oversamples rare experts during calibration. But the headline claim, that it surpasses DeepSeek-V3.1, GPT-4 series, and Qwen3-235B, is not supported by the evidence provided. The evaluation carrying that claim is an in-house benchmark built from Shopee business data, scored without error bars, without released items, and with GPT-4.1 as the judge for open-ended outputs — GPT-4.1 is also one of the baselines. On the public benchmarks (ECInstruct, Shopping MMLU) the paper itself claims only top-three, not first. So the abstract overstates what the data show.

What is actually new: the system optimizations for training and inference (intra-node expert parallelism, memcpy kernel fix, virtual and uneven pipeline parallelism, and the FP8 scheme) are concrete and plausible. The mixed-loss SFT (full-sequence loss for e-commerce prompts, answer-only for general tasks) is a reasonable design choice. The paper also cites prior work for OTPO, RegMix, and WizardLM — honest in the body, which makes it more puzzling that the abstract says 'we propose OTPO' when Section 4.2 attributes it to Li et al. 2025. That mismatch needs fixing.

The soft spots are in the evaluation methodology. No error bars or sample sizes means the fractional-point margins in Table 3 could easily be noise. GPT-4.1 as an evaluator in a comparison that includes GPT-4.1 creates a clear conflict. And the in-house benchmark is derived from the same business data used for training — the circularity burden is real. The paper would be much stronger if it released the benchmark, reported agreement with human judges, and ran significance tests. As it stands, the 'surpassing frontier models' claim cannot be independently checked.

Who should read this: practitioners working on domain-specific MoE and inference quantization will find the engineering section useful. Researchers looking for a rigorous comparison of model capabilities will not. It deserves a serious referee because the engineering contributions are real and the evaluation can be improved, but the current form overclaims.

Recommendation: send it to peer review with the expectation of major revision — require benchmark release or at minimum detailed uncertainty quantification and human agreement, and align the abstract with the body's attribution of OTPO.

Referee Report

4 major / 5 minor

Summary. The paper describes Compass-v3, a 245B-parameter (71B active) Mixture-of-Experts LLM for Southeast Asian e-commerce. It combines: (i) large-expert MoE with multi-token prediction; (ii) a multi-stage pretraining pipeline over 12T tokens with data-mixture search; (iii) single-stage SFT with mixed full-sequence/answer-only losses and a closed-loop 'data flywheel'; (iv) mixed-policy RL with Optimal-Transport Direct Preference Optimization (OTPO) and a domain reward model; and (v) expert-aware FP8 quantization for inference. The central claim is that Compass-v3 achieves state-of-the-art e-commerce performance, surpassing DeepSeek-V3.1, GPT-4 series, and Qwen3-235B, while remaining competitive on general benchmarks and strong in low-resource SEA languages. The main supporting evidence is an in-house Shopee benchmark (Sections 6.1.2, 6.2, 6.4), with open-ended outputs judged by GPT-4.1.

Significance. If the SOTA claim could be supported, this would be a notable industrial-scale result: a domain-specific 245B MoE outperforming general-purpose frontier models in e-commerce and several low-resource SEA languages, deployed at Shopee scale. The system-level contributions—intra-node expert parallelism, memcpy kernel optimization, expert-aware FP8, OTPO—are concrete and likely useful. The paper also provides partial external validation on ECInstruct and Shopping MMLU (Section 6.3). However, the manuscript does not release the model, benchmark items, or evaluation code, and the flagship evaluation is private and not statistically grounded; as written, the central 'surpassing' claim is not reproducibly supported.

major comments (4)

[§6.2, Table 3 (also Tables 5–6)] The central claim that Compass-v3 'surpasses' GPT-4.1, DeepSeek-V3.1, and Qwen3-235B rests on scores from a private in-house benchmark. No sample sizes, per-task confidence intervals, bootstrap errors, or significance tests are reported. Several margins are only a few percentage points (e.g., the reported overall margin over GPT-4.1), which may be within item-sampling or rubric noise. Without uncertainty quantification, the claimed ordering is not established. This is load-bearing for the abstract's SOTA claim.
[§6.1.2 vs. §2.4 and §3.2] The in-house benchmark is constructed from Shopee business data, the same source as the training corpus and the data-flywheel loop; no contamination/overlap analysis is provided. The independent open-source benchmarks in Section 6.3, however, are only claimed to place Compass-v3 'among the top three' and 'comparable to top-performing systems,' not uniformly ahead of all competitors. Thus the external evidence does not independently substantiate the abstract's 'surpassing DeepSeek-V3.1, GPT-4 series, and Qwen3-235B' claim.
[§6.1] The manuscript states that for domain-specific open-ended tasks it employs GPT-4.1 as an evaluator, and GPT-4.1 is also a baseline in Table 3. Evaluator self-preference is a known risk; no human-judge agreement (e.g., Cohen's kappa) or comparison with non-competing judges is reported. The open-ended scores in Tables 3 and 5 are therefore not attributable to model quality rather than evaluator bias.
[§4.3, Table 2] The reward model and OTPO are validated on 'shopee-internal' evaluations with pairwise accuracy only, again without sample sizes or error bars. Since the preference data include deployment feedback from the model itself (Section 3.2.1), the alignment improvement is partly circular. This weakens the OTPO contribution claim, though the main model-level claim is not solely dependent on it.

minor comments (5)

[Abstract, Section 1] Typos and OCR artifacts: 'Taglog' should be 'Tagalog'; equations in Section 2.3.4 are garbled and should be re-set.
[§6.1.1] Baselines use 'official api defaults' while Compass-v3 uses custom decoding parameters (max_tokens, top_k, top_p, temperature). Report exact decoding settings for all baselines to ensure a fair comparison.
[§2.4.2] The claim that proxy validation loss is 'strongly correlated' with benchmark scores is made without reporting correlation coefficients, scatter plots, or the specific validation sets used.
[§5.3] The speedup numbers ('1.2×', 'up to 1,? tokens/s') are not accompanied by sufficient hardware details (GPU model, batch size, sequence length, number of nodes) for reproduction.
[Tables 4–6] No uncertainty estimates (standard deviations across seeds or bootstrap) are given, even for the open-source benchmarks, making it difficult to assess the stability of the reported rankings.

Circularity Check

2 steps flagged

SOTA claim rests on in-house Shopee benchmark built from the same business data used for training, scored by a competing baseline.

specific steps

fitted input called prediction [Section 6.1.2 (In-House Dataset Construction) and Section 3.2.1 (Large-scale E-commerce Instructions)]
"we construct an in-house e-commerce evaluation dataset, primarily sourced from real business scenarios ... real-world business scenarios: developed in close collaboration with shopee business units to guarantee alignment with practical application needs"

The benchmark that supports the abstract's SOTA claim (Table 3: Compass-v3 86.74% vs GPT-4.1 83.71%) is built from the same Shopee business scenarios used to construct the SFT/flywheel training data. The model was fitted to this distribution through pretraining, SFT, and RL; reporting superiority on a test set drawn from that same distribution is a measure of fit to the training input, not an independent prediction. The baselines (DeepSeek, GPT, Qwen) were not trained on this distribution, so the comparison is structurally forced by the choice of test data.
other [Section 6.1 (Evaluation Setting)]
"for domain-specific, open-ended tasks, we employ gpt-4.1 as an evaluator to ensure fairness and consistency"

GPT-4.1 is itself a baseline in the same benchmark (Table 3: 83.71%). Using a competing model as the judge for open-ended outputs means the relative ranking of Compass-v3 and GPT-4.1 is produced by one of the parties being compared. No human-judge agreement or released items are provided, so the 'surpassing GPT-4.1' claim is not an independent measurement; it is GPT-4.1's own scoring of outputs from both systems.

full rationale

The central 'state-of-the-art' claim is not derived from first principles; it is an empirical evaluation. The evaluation's two key components are both closed loops: (1) the in-house benchmark is sourced from the same Shopee business scenarios as the training data (Sections 6.1.2 and 3.2.1), and (2) the open-ended outputs are scored by GPT-4.1, which is itself a compared baseline (Section 6.1). Together these make the flagship claim partially circular: the model is fitted to Shopee business data and then evaluated on Shopee business data using a competitor as judge. The external open-source benchmarks (ECInstruct, Shopping MMLU) provide some independent evidence, but the paper only claims 'top three' on them, not the uniform superiority stated in the abstract. Other self-citations (Compass-v2 taxonomy, OTPO from Li et al.) are not the load-bearing support for the central claim. Therefore the partial circularity score is 6.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claims rest on proxy-model extrapolation, judge fairness, private-benchmark validity, translation validity, and standard math. The paper pulls its evaluation instruments from its own pipeline and its own prior report (Compass-v2), while the headline alignment method is explicitly borrowed from Li et al. 2025. No free-parameter estimates are given for the loss weights or calibration criteria, yet the results depend on them. No new unobserved entities are introduced; the model, benchmarks, and flywheel procedures are artifacts of the reported system, not postulates requiring independent evidence.

free parameters (5)

load-balancing auxiliary loss coefficient beta (Eq. 4) = not reported
Coefficient weighting the auxiliary router-balancing loss in the total training objective; reported to decay over training, value never given. It determines routing behavior and therefore model quality.
z-loss coefficient alpha (Eq. 5) = not reported
Stabilizing loss for router logits; value and decay schedule not reported.
Compass-v3 evaluation decoding parameters = max_tokens=2048, seed=42, repetition_penalty=1.05, top_k=6, top_p=0.95, temperature=0.4
Chosen in Section 6.1.1 while Qwen uses temperature 0.6/top_p 0.9/top_k 20 and GPT/DeepSeek use API defaults; decoding settings materially affect open-ended benchmark scores and are not shared across models.
FP8 expert calibration oversampling criterion = not reported
Experts are oversampled 'until their activation counts satisfy a pre-defined criterion' (Section 5.2); the criterion is undefined, and it determines quantization accuracy.
mixed-loss task split (e-commerce vs general tasks) = not reported
Which samples are treated as e-commerce (full-sequence loss) vs general (answer-only loss) is determined by a task taxonomy that is not released, so the switch in Eq. 3 cannot be reproduced.

axioms (6)

domain assumption 8M-parameter proxies trained on 1B tokens (and a 574M/1.5B-token check) predict the outcome of training a 245B model on 12T tokens; proxy validation loss 'strongly correlates' with benchmark scores.
Invoked in Section 2.4.2 to justify selecting the data mixture; correlation numbers and plots are not shown, and no scaling-law gap analysis is provided.
domain assumption GPT-4.1-as-judge produces fair and consistent accuracy labels for open-ended e-commerce outputs, while being one of the compared baselines.
Stated in Section 6 as the evaluation choice for open-ended tasks; no inter-annotator agreement with humans or judge-vs-human validation is reported.
domain assumption Accuracy on the authors' private in-house benchmarks (built from Shopee business scenarios) is a faithful measure of real-world e-commerce quality, with sub-point differences meaningful.
Used throughout Sections 6.2-6.4; benchmark items, labels, and human-validation protocol are not released.
domain assumption The output-space-size heuristic in Table 1 determines when off-policy vs on-policy preference data is sufficient.
Presented as motivation for mixed-policy RL in Section 4.1; no ablation isolates this rule from model and domain confounds.
domain assumption Translated multilingual benchmarks (random subsets of OpenBookQA, XCOPA, MMLU, HellaSwag translated into six languages) are valid instruments across languages.
Section 6.5 states subsets are translated but the translation method, quality checks, and prompt localization are not described.
standard math Standard ML machinery: cross-entropy SFT loss, auxiliary router-balancing loss, z-loss, and Bradley-Terry objectives are taken as given.
Standard and correctly instantiated in Eqs. 1-5 and Section 4.3; no novelty or risk here.

pith-pipeline@v1.3.0-alltime-deepseek · 22071 in / 25620 out tokens · 272846 ms · 2026-08-04T19:41:43.179618+00:00 · methodology

0 comments

Cite this review

Pith. "Pith review of Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia." pith.science (2026). https://pith.science/paper/MSLSR27S

@misc{pith2026250909121,
  author       = {Pith},
  title        = {Pith review of: Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MSLSR27S}},
  note         = {Machine review of arXiv:2509.09121}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Large language models (LLMs) excel in general-domain applications, yet their performance often degrades in specialized tasks requiring domain-specific knowledge. E-commerce is particularly challenging, as its data are noisy, heterogeneous, multilingual, and highly dynamic. We present Compass-v3, a vertical-domain Mixture-of-Experts (MoE) model with 245B total parameters and 71B active per token, designed for Southeast Asian e-commerce. Compass-v3 adopts fewer but larger experts, combined with hardware-efficient optimizations-such as intra-node expert parallelism and a customized memcpy operator-to maximize GPU utilization. The model is trained on 12T tokens of curated multilingual corpora and large-scale synthetic e-commerce instructions using a mixed-training strategy. To enhance alignment, we propose Optimal-Transport Direct Preference Optimization (OTPO), which captures token-level distinctions and improves instruction adherence in commerce-specific scenarios. Extensive evaluations demonstrate that Compass-v3 delivers state-of-the-art e-commerce performance, surpassing DeepSeek-V3.1, GPT-4 series, and Qwen3-235B. Moreover, Compass-v3 demonstrates strong multilingual capability across low-resource Southeast Asian languages (Indonesian, Thai, Filipino, Vietnamese, Malay, Taglog) and Portuguese while sustaining competitive performance on general benchmarks. It has already been widely applied in Shopee's industrial-scale e-commerce platform and is gradually replacing OpenAI's traffic, now accounting for over 70\% of total LLM usage, highlighting its dual strengths in specialized commerce expertise and broad linguistic competence.

discussion (0)

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Valley3: Scaling Omni Foundation Models for E-commerce
cs.AI 2026-05 unverdicted novelty 4.0

Valley3 is an omni MLLM for e-commerce that uses a four-stage pre-training pipeline plus post-training for controllable reasoning and agentic search, outperforming baselines on e-commerce benchmarks while staying comp...
JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications
cs.AI 2026-06 unverdicted novelty 3.0

Oxygen AIIC is an industrial platform using LLMs and VLMs for scalable item knowledge production and service at JD.com, reporting 94.2% precision and 82.8% recall along with business metric improvements.
JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications
cs.AI 2026-06 unverdicted novelty 2.0

JD.com's Oxygen AIIC applies LLMs/VLMs with a Semantic Search then Discrimination architecture and self-evolving models to produce item knowledge at industrial scale, reporting 94.2% precision and 82.8% recall.

Reference graph

Works this paper leans on

1 extracted references · 1 linked inside Pith · cited by 2 Pith papers

[1]

jN achiamL sN adler L sN a©arwalL lN ahmadL iN akkayaL fN lN alemanL dN almeidaL jN altenschmidtL sN altmanL sN anadkatL et alN gptMT technical reportNarxiv preprint arxiv:RSPSNPX77TL RPRSN rN aN bradley and mN eN terry N rank analysis of incomplete block desi©nsZ iN the method of paired comparisonsNbiometrikaL SYHSOTIZSRT–STUL QYURN mN chenL jN tworekL h...

Pith/arXiv arXiv

[1] [1]

jN achiamL sN adler L sN a©arwalL lN ahmadL iN akkayaL fN lN alemanL dN almeidaL jN altenschmidtL sN altmanL sN anadkatL et alN gptMT technical reportNarxiv preprint arxiv:RSPSNPX77TL RPRSN rN aN bradley and mN eN terry N rank analysis of incomplete block desi©nsZ iN the method of paired comparisonsNbiometrikaL SYHSOTIZSRT–STUL QYURN mN chenL jN tworekL h...

Pith/arXiv arXiv

Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it