{"id":"6463295c-0dd0-433e-a102-fdddc1138c13","arxiv_id":"2607.05968","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"A 4B-model cascade for Thai KOL matching reaches 94.1% P@5 on 11 queries, matching a frontier model, with pairwise SimPO training transferring end-to-end while pointwise SFT+GRPO does not.","lead":"The paper builds a three-stage pipeline (retrieve→rerank→reason) using 4B open-weight models to match Thai influencers to marketing briefs, matching frontier-model accuracy at ~35× fewer tokens. A smart generalist might read it for the empirical finding that pairwise preference training transfers to deployment while pointwise fine-tuning doesn't — a practical lesson in supervision design.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Headline 'matches frontier quality' rests on a 2.3-point P@5 margin over n=11 queries; the n=31 replication reverses the ordering, making the central comparison statistically unstable.","rationale":"The reader correctly identified the most load-bearing concern: the headline comparison rests on n=11 queries with a 2.3-point margin that is within the noise floor, and the n=31 replication reverses the ordering. This is the soft spot of the paper. The reader's verdict of CONDITIONAL with MODERATE confidence is appropriate. The paper has genuine merits — the offline-online inversion analysis is thoughtful, the cascade design is well-motivated, and the cost-accuracy tradeoff is clearly characterized. But the central claim as stated ('matches frontier quality') overstates what the evidence supports. The paper's own Limitations section acknowledges the small sample issue, which is commendable, but the abstract and conclusion still present the comparison as a finding rather than a directional observation. No code or data is released, which further limits independent verification. The concern does not reveal an internal inconsistency or methodological error — the experiments are cleanly designed and honestly reported. The issue is simply that the evidence base is too thin to support the strength of the central claim. A CONDITIONAL verdict with the recommendation to either soften the headline claim or expand the evaluation set is the right call. I agree with the reader's assessment and see no reason to adjust the verdict.","tokens_in":12750,"tokens_out":1948,"duration_ms":157014,"concrete_test":"Run a paired bootstrap resampling test (≥10,000 resamples) over the per-query P@5 scores on both Set 1 and Set 2, computing the distribution of the Base-minus-Kimi difference. If the 95% bootstrap interval for the difference on Set 1 includes zero (which is likely given n=11 and a 2.3-point gap), and Set 2's interval also includes zero or is negative, the 'matches frontier quality' claim should be revised to 'approaches frontier quality within sampling uncertainty.' Additionally, collecting a third set of ≥50 queries with all candidates labeled would provide the statistical power needed to settle the comparison definitively.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that the 4B cascade 'matches frontier quality' — is supported by one data point: 94.1% vs 91.8% P@5 on Set 1 (n=11). The authors themselves note that a single flipped triplet moves EM by ~2.4pp; for P@5 at k=5 over 11 queries, a single query's P@5 can shift the macro-average by roughly 9 points. The 2.3-point margin is well within this noise envelope. Critically, Set 2 (n=31) reverses the ranking: Kimi-K2.6 (91.9) beats Base (89.3), a 2.6-point deficit. The paper frames this as 'gaps compress,' but it is a direct contradiction of the headline on a larger sample. The paper does not report confidence intervals, bootstrap distributions, or any significance test. Without these, the claim that the cascade 'matches' the frontier model is not distinguishable from noise. The more defensible claims — that the cascade architecture (rerank-then-reason) is cheaper and better than reason-on-all-50, and that pairwise supervision transfers while pointwise does not — are directionally supported across both sets and do not depend on the fragile 2.3-point margin. The headline claim does.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"InfluMatch presents a three-stage cascade (retrieval → rerank → reason) for matching Thai marketing briefs to influencers (KOLs), using only 4B open-weight models. The reranker is a SimPO-tuned 4B model scoring candidates by logp(Yes); the reasoner is an untuned 4B model grading shortlisted candidates per criterion. The paper reports 94.1% P@5 on an 11-query set (Set 1), matching a frontier model (Kimi-K2.6, 91.8%) at ~35× fewer output tokens. A key secondary finding is that pairwise SimPO fine-tuning transfers end-to-end while pointwise SFT+GRPO does not, despite the latter winning offline — an inversion traced via an evidence audit to label quality issues.","tokens_in":13644,"tokens_out":1472,"duration_ms":264109,"significance":"The paper makes several valuable contributions: (1) a deployable cascade architecture with explicit per-stage cost budgets, (2) a SimPO formulation for single-token reranking (Eq. 1) that achieves training-serving consistency by design, (3) a well-motivated evidence audit (Table 2) that grounds the offline–end-to-end inversion in measurable label quality problems, and (4) a practical finding that pairwise supervision transfers while pointwise does not. The cost–accuracy Pareto analysis (Table 10) is informative and the system-level engineering is sound. The work is positioned within the relevant neural IR and LLM-as-judge literature.","major_comments":[{"comment":"§6.3, Tables 8–9: The headline claim that the cascade 'matches frontier quality' rests on a 2.3-point P@5 margin (94.1 vs. 91.8) over n=11 queries. The authors acknowledge in §Limitations that a single flipped triplet moves EM by ~2.4pp; for P@5 at k=5 over 11 queries, a single query reordering can shift the macro-average by roughly 9 points. Critically, Set 2 (n=31) reverses the ordering: Kimi-K2.6 (91.9) beats Base (89.3). The paper frames this as 'gaps compress,' but it is a direct reversal of the headline comparison on a larger sample. No confidence intervals, bootstrap distributions, or significance tests are reported. The claim should be softened to reflect that the cascade is 'competitive with' rather than 'matching' frontier quality, or the authors should provide bootstrap CIs to support the stronger claim.","section":null},{"comment":"§6.3, Table 8: The cascade result (94.1% P@5) is notably higher than reason-on-all-50 (80.0% P@5) — a 14-point gain from filtering. This is a counterintuitive and load-bearing finding (it underpins the cascade design). The explanation in §7 (filtering removes diluting candidates, reducing ties) is plausible but not tested directly. A controlled experiment — e.g., injecting random noise into the top-10 vs. top-50 pools, or measuring tie rates at each pool size — would strengthen this claim. As stated, the 14-point gap could also reflect that the top-10 by SimPO rerank happens to contain more relevant KOLs by construction, which would be a retrieval-quality effect rather than a reasoning-quality effect.","section":null},{"comment":"§4.3, Table 2: The evidence audit is produced by Kimi-K2.6, the same model used as the frontier baseline throughout the paper. While the authors note that quote verification is model-independent (exact substring check), the selection of which quotes to verify and the grounding judgment itself inherit the auditor's biases, as acknowledged in §Limitations. Since the audit underpins the central claim that pointwise labels are noisy (and thus that the offline–end-to-end inversion is label-imposed rather than capacity-imposed), using the same model as both auditor and baseline competitor creates a potential conflict. The authors should discuss whether an alternative auditor (or human spot-check of the audit) was considered.","section":null}],"minor_comments":[{"comment":"§3.4, Eq. (1): The values of β and γ are not stated in the text. These are free parameters that affect the margin and calibration of the SimPO objective. Please report the chosen values and briefly justify them.","section":null},{"comment":"§5.3: The relevance criterion uses binary majority voting with a tie-breaking rule (f=0.5 dropped). The number of annotators per KOL and inter-annotator agreement statistics are not reported. This information would help calibrate the reliability of the end-to-end labels.","section":null},{"comment":"Table 6: The SimPO row reports 78.0 EM (32/41) on the indirect logp(Yes) path, matching Kimi-K2.6's 78.0 EM (32/41) on the direct judge path. The raw counts (32/41) are the same, but the items being judged differ (indirect scoring of individual KOLs vs. direct A-vs-B comparison). The text should clarify that these are not the same 32 correct items.","section":null},{"comment":"§4.2: The T2 annotation interface did not provide a scoring rubric defining 0/1/2 boundaries. This is discussed later in §7 as a contributing factor to the inversion, but it would be more transparent to flag this design choice when the annotation protocol is first described, so readers understand the label quality before reaching §7.","section":null},{"comment":"§6.4, Table 10: Wall-clock timings are noted as not like-for-like across models (different concurrency levels). The text correctly directs attention to the token axis, but the abstract's claim of '~20s on one A100' could be misread as a fair latency comparison. Consider adding a parenthetical in the abstract noting that wall-clock is indicative and cost comparisons rest on tokens.","section":null},{"comment":"The abstract states 'a 50-KOL query in ~20s on one A100.' Table 10 reports 20.2s for the deployed configuration. It would be useful to clarify whether this includes retrieval time or only the rerank+reason stages.","section":null},{"comment":"References [20] and [21] cite model release blogs rather than technical reports. If technical reports or model cards with architecture details are available, citing those would help reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper has solid engineering and an interesting finding about pairwise vs. pointwise supervision transfer. The main issue is statistical: the headline claim is not distinguishable from noise at n=11, and the n=31 replication reverses the ordering. If the authors soften the headline to 'competitive with' and add bootstrap CIs, the paper is publishable. The evidence audit using Kimi-K2.6 as both auditor and baseline is a minor concern but worth flagging to the authors. I would not reject over this — the cascade design and supervision-signal analysis are the more durable contributions, and those survive the statistical fragility of the headline."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive reading. The three major comments are well-taken; we address each below and indicate revisions where the manuscript will change.","responses":[{"response":"The referee is correct on all counts. The 2.3-point margin on n=11 is not statistically distinguishable from zero, and Set 2 does reverse the ordering. We acknowledge that 'matches frontier quality' overstates what the data support. We will make two changes in the revision. First, we will soften the headline claim throughout the paper (abstract, §1.1, §6.3, §8) to 'competitive with frontier quality' or 'within frontier range,' and we will explicitly note the Set 2 reversal in §6.3 rather than framing it as 'gaps compress.' Second, we will add bootstrap 95% confidence intervals (10,000 resamples, stratified by query) for all P@5 comparisons in Tables 8 and 9. We expect these intervals to be wide and overlapping, which is itself the honest message: the 4B cascade and the frontier model are in the same ballpark, but we cannot claim superiority at these sample sizes. We agree this is the correct framing.","revision_made":"yes","referee_comment":"§6.3, Tables 8–9: The headline claim that the cascade 'matches frontier quality' rests on a 2.3-point P@5 margin (94.1 vs. 91.8) over n=11 queries. Set 2 (n=31) reverses the ordering. No confidence intervals, bootstrap distributions, or significance tests are reported. The claim should be softened or bootstrap CIs provided."},{"response":"This is a fair point and we agree the current explanation is plausible but untested. The referee's alternative hypothesis — that the top-10 by SimPO rerank simply contains more relevant KOLs by construction, so the reasoner starts from a better pool — is partially correct but does not fully explain the pattern. Note that 'reason on all 50' already has access to the same relevant KOLs; the question is why seeing more candidates hurts. That said, we cannot rule out that the larger pool introduces tie-breaking noise that depresses the reasoner's ranking, which is a retrieval-pool-size effect rather than a 'filtering removes dilution' effect. We will add a controlled experiment in the revision: we will measure tie rates (fraction of candidate pairs with identical summed scores) at pool sizes 10, 20, and 50, holding the reasoner fixed. If tie rates increase substantially at larger pool sizes, this directly supports the dilution mechanism. We will also report the number of relevant KOLs in the top-10 vs. top-50 pools per query, to quantify how much of the gap is a retrieval-quality effect. We agree that without this experiment the 14-point claim is under-supported as currently stated, and we will temper the language in §7 accordingly.","revision_made":"partial","referee_comment":"§6.3, Table 8: The 14-point gain from filtering (94.1 vs. 80.0) is counterintuitive and load-bearing but not directly tested. The gap could reflect retrieval-quality effects rather than reasoning-quality effects. A controlled experiment would strengthen this claim."},{"response":"We agree this is a legitimate concern. To be precise about what does and does not inherit the auditor's bias: the quote verification step (exact substring match against the profile text) is model-independent and mechanical — a quote either appears verbatim or it does not. What does inherit potential bias is (a) which quotes the auditor selects to extract and (b) the binary grounding judgment (is this quote sufficient evidence for the score?). These are genuine sources of auditor bias. We did not run an alternative auditor or human spot-check in the current study, and we cannot retroactively claim independence. In the revision we will: (1) add an explicit discussion of this conflict in §4.3 and §Limitations, acknowledging that the audit's central finding (score-2 labels are weakly grounded) could be partially an artifact of the auditor's quote-selection tendencies; (2) report inter-rater agreement on a small human spot-check of the audit if we can complete it before the revision deadline — we will attempt this but cannot guarantee it at this time. If the spot-check is not feasible within the revision window, we will at minimum reframe the audit's role: rather than presenting it as definitive proof that pointwise labels are noisy, we will present it as consistent with the label-noise hypothesis, noting that the offline–end-to-end inversion itself (SFT+GRPO wins offline, loses end-to-end) is a model-independent observation that holds regardless of the audit. The audit is corroborating evidence, not the sole basis for the claim.","revision_made":"partial","referee_comment":"§4.3, Table 2: The evidence audit is produced by Kimi-K2.6, the same model used as the frontier baseline. The audit underpins the central claim that pointwise labels are noisy. Using the same model as auditor and baseline competitor creates a potential conflict. Should an alternative auditor or human spot-check be considered?"}],"tokens_in":12595,"tokens_out":1243,"duration_ms":79486,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper you should know about here is an applied systems paper for Thai influencer matching. The headline — a 4B cascade matching a frontier model — is oversold. The genuinely interesting result is the offline-to-end-to-end inversion: pointwise supervision wins offline but degrades deployed ranking, while pairwise SimPO transfers. That finding is real and worth reading for, even if the framing around it is too strong for the evidence base. The cascade architecture itself (retrieval → rerank → reason) is standard neural IR, and the paper is honest about that. What it does well is the analysis of *why* pointwise supervision fails. The evidence audit (Table 2) showing that only 15.9% of score-2 labels are backed by verbatim profile quotes is a concrete, model-independent grounding check that supports the inversion story. The clear-cut subset analysis (SFT rising from 57.7 to 73.4 wf1 on grounded labels) is good evidence that the plateau is label-imposed, not capacity-imposed. The cost analysis (Table 10) is also useful — the Pareto frontier is laid out clearly, and the token-axis comparison is the right metric given the concurrency differences. The SimPO reranker matching Kimi-K2.6 at 78.0 EM on 41 triplets is directionally solid, though again small-n. Now the soft spots. The central claim — 94.1% vs 91.8% P@5 over Kimi-K2.6 — is on n=11 queries. The authors themselves note a single flipped triplet moves EM by ~2.4pp; for P@5 at k=5 over 11 queries, a single query reordering can shift the macro-average by roughly 9 points. The 2.3-point margin is noise. Set 2 (n=31) reverses the ordering: Kimi beats Base (91.9 vs 89.3). The paper frames this as 'gaps compress,' but it is a direct reversal on a larger sample. No confidence intervals, no bootstrap, no significance test. The headline claim does not survive the larger evaluation. The more defensible claims — that filtering before reasoning is both cheaper and better, and that pairwise supervision transfers while pointwise does not — hold directionally across both sets and do not depend on the fragile margin. The evidence audit uses Kimi-K2.6 as auditor, which is also the baseline being compared against. The audit's quote-verification step is model-independent (exact substring match), so this is a minor concern, but worth noting. No code or data is released. The free parameters (β, γ, cutoffs, aggregation) are not extensively ablated. This paper is for practitioners building deployed IR/reranking pipelines in low-resource languages, and for researchers thinking about supervision-design choices in annotation pipelines. The supervision inversion finding has broader relevance. It deserves a serious referee. The referee should push hard on the evaluation: larger query sets, confidence intervals, and released artifacts. If the authors can show the inversion holds on a bigger sample with proper statistics, this is a solid contribution. If the headline claim is the main selling point, it does not stand.","headline":"Applied KOL-matching cascade with a real supervision-design finding, but the headline 'matches frontier' claim rests on n=11 and reverses on n=31.","tokens_in":13771,"tokens_out":742,"would_cite":false,"duration_ms":176898,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"4B models match frontier LLM on influencer search, 35× fewer tokens","keywords":[],"falsifier":"Run the same cascade and frontier baseline on a larger query set (e.g., 100+ queries with full candidate labeling). If the 4B cascade's P@5 drops below the frontier baseline by a statistically significant margin, or if a cleaner pointwise labeling protocol eliminates the offline–end-to-end inversion, the paper's two central claims would be weakened.","tokens_in":13040,"feed_emoji":"🔍","tokens_out":1236,"duration_ms":286602,"temperature":0.7,"pith_summary":"The paper claims that a three-stage cascade of small (4B-parameter) open-weight models can match a frontier LLM on a real influencer-matching task — ranking Thai KOLs against free-form marketing briefs — at roughly 1/35th the output-token cost and ~4× lower latency. The cascade works by splitting labor: dense retrieval casts a wide net (50 candidates), a SimPO-tuned 4B reranker filters to a top-10 shortlist using a single Yes-token log-probability score, and an untuned 4B reasoner grades each shortlisted KOL per criterion with a Thai rationale. The key design insight is that filtering before reasoning is both cheaper and more accurate than reasoning over the full pool, because the reasoner's coarse per-criterion scoring becomes decisive when weak candidates are removed. The paper's second claim is about supervision: fine-tuning on pairwise (relative) preferences transfers to end-to-end deployment, while fine-tuning on pointwise (absolute) scores wins offline but degrades the deployed system — an inversion traced to noisy, under-specified absolute labels where only 16% of 'match' judgments are backed by verbatim profile evidence.","feed_headline":"4B models match frontier LLM on KOL search at 35× lower token cost","feed_subtitle":"A three-stage cascade of small open-weight models matches a frontier model on Thai influencer matching — but the headline gap rests on 11","key_machinery":"Three-stage cascade (retrieval → rerank → reason); SimPO pairwise preference training on a single Yes-token log-probability; pointwise {0,1,2} rubric scoring with even-sum aggregation; evidence audit verifying label grounding via exact-substring profile quotes","core_discovery":"The central discovery is a division-of-labor effect in retrieval cascades: a pairwise-preference-trained reranker that is mediocre at absolute judgment (topping out at ~67% P@5 alone) becomes a strong filter that enables an untuned reasoner to reach 94.1% P@5 — beating the same reasoner applied to all 50 candidates (80.0%) by 14 points while using half the tokens. The paper traces this to the reasoner's coarse resolution (five criteria scored 0/1/2 produces ties over large pools) being eliminated when the candidate set is small. The secondary discovery is the offline–end-to-end inversion: pointwise SFT+GRPO fine-tuning of the reasoner tops the offline per-criterion leaderboard (59.0 weighted","pith_inferences":["The 2.3-point P@5 gap between the 4B cascade (94.1%) and the frontier baseline (91.8%) on Set 1 (n=11) is well within the noise floor — a single query reordering shifts macro-averaged P@5 by roughly 9 points at this sample size. The Set 2 replication (n=31) reverses the ranking (frontier 91.9% > 4B 89.3%), suggesting the 'matching frontier quality' claim is directional rather than statistically es","The inversion finding (offline-best fine-tuned model is worst end-to-end) may be partly an artifact of the specific label-collection design rather than a general property of pointwise supervision. The paper itself attributes the gap to three compounding design choices (no rubric, grouped annotation, criterion-vs-holistic mismatch); a cleaner pointwise labeling protocol might close the inversion, w","The '35× fewer output tokens' comparison is between the 4B cascade (2,974 output tokens/query) and Kimi-K2.6 in think mode (102,862 output tokens/query). A fairer cost comparison might use the frontier model in no-think mode (19,457 tokens, ~6.5× more), which narrows the efficiency gap substantially while still favoring the small-model cascade."],"forward_implications":["If relative-preference supervision consistently transfers better than absolute scoring, annotation pipelines for LLM-based retrieval should default to best/worst comparisons over rating scales, reducing label noise and improving downstream ranking.","The finding that filtering before reasoning improves both cost and accuracy suggests that multi-stage cascades with cheap filters and expensive reasoners may generalize beyond KOL matching to any domain where per-item scoring has coarse resolution over large candidate sets.","The evidence-audit methodology — checking whether human labels are backed by verbatim evidence — offers a practical diagnostic for detecting when a model's disagreements with human labels reflect label noise rather than model error, which could reframe how fine-tuning success is evaluated.","The deployment of 4B open-weight models at frontier-adjacent quality on a non-English (Thai) task suggests that the cost gap between frontier and small models may be narrowest in domain-specific, well-structured retrieval tasks rather than general-purpose reasoning."],"fun_headline_variants":["Pairwise reranker lifts untuned 4B reasoner to 94% P@5 on KOL search","Divide-and-filter: mediocre rerankers make strong 4B reasoners","Pairwise-trained reranker halves tokens, lifts P@5 by 14 points","Untuned reasoner beats fine-tuned on filtered top-10","Cascade of 4B models matches frontier KOL search at fraction of cost"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The headline accuracy comparison rests on 11 queries where a single reordering shifts the metric by roughly 9 percentage points, and the larger replication set (31 queries) reverses which model wins — so the claim that the cascade 'matches frontier quality' is not statistically distinguishable from a tie at the reported sample sizes.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise reranker lifts untuned 4B reasoner to 94% P@5 on KOL search","Divide-and-filter: mediocre rerankers make strong 4B reasoners","Pairwise-trained reranker halves tokens, lifts P@5 by 14 points","Untuned reasoner beats fine-tuned on filtered top-10","Cascade of 4B models matches frontier KOL search at fraction of cost"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":848,"prompt_tokens":740,"completion_tokens":108,"prompt_tokens_details":null},"tokens_in":740,"tokens_out":108,"duration_ms":36639,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T19:39:03.834933+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run the same cascade and frontier baseline on a larger query set (e.g., 100+ queries with full candidate labeling). If the 4B cascade's P@5 drops below the frontier baseline by a statistically significant margin, or if a cleaner pointwise labeling protocol eliminates the offline–end-to-end inversion, the paper's two central claims would be weakened.","supporting_citations":[],"review_version":1}