Pith. sign in

REVIEW 3 major objections 6 minor 71 references

Under a fixed multimodal training recipe, goal-driven 1× data subsets reach a 512k uniform baseline with far fewer samples and higher accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 22:17 UTC pith:35254I5X

load-bearing objection Clean fixed-recipe result: goal-driven 1× subsets hit a 512k Uni-10x bar much earlier with small accuracy gains, but the title claim is only half-secured without same-budget 1× selectors. the 3 major comments →

arxiv 2603.12478 v2 pith:35254I5X submitted 2026-03-12 cs.CV cs.LG

Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

Rujie Wu , Haozhe Zhao , Hai Ci , Yizhou Wang This is my paper
classification cs.CV cs.LG
keywords Data OptimizationInstruction TuningVision Language Modelsmultimodal learningvideo understandingsample selectionefficient fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal instruction tuning often wastes compute by spreading a fixed budget evenly over large mixed image–video pools whose samples have very uneven value. This paper introduces Goal-Driven Data Optimization (GDO): every candidate is scored with six probe-derived descriptors, then small optimized training subsets are built for explicit goals such as minimum loss, diversity, or stronger temporal video emphasis. Holding the model, one-epoch recipe, checkpoints, and evaluation fixed, these 1× subsets reach the performance of a fixed 512k-sample uniform baseline after roughly 27k–35k samples on four video-understanding benchmarks, while also finishing higher in accuracy. Gains are largest on subtask-focused temporal benchmarks and grow as the allocation goal stresses temporally informative video; ultra-long-video benchmarks improve more modestly because the training pool is short-video and image dominant. The result matters because it treats data allocation itself as a first-class lever for efficiency and capability once the backbone and recipe are locked.

Core claim

Under one fixed one-epoch Qwen3-VL-8B-Instruct train/eval contract, goal-driven optimized 1× subsets reach the fixed 512k-sample Uni-10x reference after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench, while improving Accuracy by +1.38, +1.67, +3.08, and +0.84 percentage points respectively. Across MinLoss, Diverse, Temp, and Temp+, stronger temporal emphasis yields steadily better long-video understanding behavior.

What carries the argument

Goal-Driven Data Optimization (GDO): six sample descriptors (flow magnitude, video-dependence score, temporal necessity, self-consistency, PPL-like difficulty, coverage) feed one shared fixed scorer; goal-specific feasibility presets then control budget, video ratio, temporal-positive coverage, source floors, and oversampling to build 1× subsets. Only subset construction changes; model, optimizer, and evaluation stay fixed.

Load-bearing premise

The six descriptors from one frozen probe model, mixed by a single hand-fixed shared scoring formula, rank true training utility well enough that only budget and composition presets need to change across goals.

What would settle it

Train the identical fixed recipe on random or purely difficulty-based 1× subsets of the same sizes as the GDO profiles; if those controls match or exceed GDO’s peak-match sample counts and final accuracy deltas on MVBench, VideoMME, MLVU, and LVBench, the claim that the descriptors and goal presets drive the gains fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal SFT budgets can be cut by more than 10× relative to uniform 10× sampling while matching or exceeding the same reference accuracy.
  • Different explicit allocation goals produce distinct efficiency–capability trade-offs under an identical train/eval contract.
  • Stronger temporal data pressure systematically improves motion, order, and temporal-perception subtasks.
  • Gains stay smaller when the target regime (ultra-long video) is poorly matched by a short-video/image training pool.
  • Once backbone and recipe are fixed, data allocation becomes a controllable design lever rather than a fixed background choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same descriptor-plus-feasibility pattern could be reused as a pre-filter for other uneven multimodal corpora, including long-context or audio-visual instruction pools.
  • Adding native ultra-long-video coverage to the pool would likely let Temp-style presets close more of the LVBench gap without changing the scorer.
  • That one fixed scorer works across goals suggests admissibility constraints, not preference ranking, are the main dial for goal specialization.
  • If the frozen probe were a weaker or different model family, the utility ranking might degrade, so probe–backbone alignment itself is a testable design variable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Goal-Driven Data Optimization (GDO) for multimodal instruction tuning. It extracts six probe-derived sample descriptors (flow, VDS, temporal necessity, self-consistency, PPL-like difficulty, coverage), ranks candidates with a shared hand-weighted scorer (Eqs. 9–11), and builds 1× subsets under goal-specific feasibility presets (MinLoss, Diverse, Temp, Temp+). Under a fixed one-epoch Qwen3-VL-8B-Instruct SFT recipe, the optimized subsets are compared to a fixed 512k Uni-10x uniform baseline on MVBench, VideoMME, MLVU, and LVBench. The headline result is that GDO reaches the Uni-10x reference after 26.6k–35.4k samples while improving Accuracy by +0.84 to +3.08 pp, with stronger temporal presets improving long-video behavior. Subtask heatmaps, trajectory plots, and Temp+ ablations support a structured, capability-dependent effect rather than a uniform score lift.

Significance. If the result holds under stronger same-budget controls, the paper makes a useful systems contribution: it treats multimodal SFT data allocation as a first-class design lever under a deliberately strict train/eval contract (fixed model, optimizer, checkpoints, benchmarks; benchmark-blind construction). The released code, four-goal spectrum, peak-match framing, and honest LVBench mismatch discussion are concrete strengths. The work is timely for video-centric post-training, where mixed image–video pools are large and unevenly useful. Significance is currently limited by the missing same-budget selection baselines; with those, the paper would more cleanly separate “less data helps” from “this goal-driven scorer/feasibility design helps.”

major comments (3)
  1. [Table 1, Fig. 1, §3.2] Table 1 / Fig. 1 / §3.2: The central less-data and peak-match claims compare optimized 1× subsets only to a fixed 512k Uni-10x uniform baseline. This confounds (i) discarding most of the pool with (ii) the six-descriptor scorer and goal presets. Without same-budget 1× controls—at minimum random 1× at each Ng, and preferably quality-only, VDS-only, or prior instruction-selection baselines (e.g., LESS/AlpaGasus-style ranking)—the reported 14–19× reductions and +0.84–+3.08 pp gains may largely reflect any non-uniform curation beating 10× uniform rather than GDO’s specific design. This is load-bearing for the title claim and should be added under the same fixed train/eval contract.
  2. [§2.3, Eqs. (9)–(11)] §2.3, Eqs. (9)–(11) and Supp. Table 6: The shared scorer uses fixed hand-chosen mixture weights (e.g., 0.35/0.95/0.35 for video; 0.90/0.15 for image; bvid coefficients 0.85/0.9/0.55/0.15), and the four goals mainly change budgets and feasibility floors rather than the scorer. Table 5 ablates score components for Temp+ but does not test sensitivity of the mixture weights or compare against a pure feasibility-only / pure score-only builder at matched Ng. A short sensitivity or alternative-weight experiment is needed to support the claim that the shared scorer plus goal presets—not an accidental coefficient choice—drive the frontier shifts.
  3. [§2.2, Eqs. (4)–(7)] §2.1–2.2 and §3.1: Descriptors (VDS, temporal necessity, self-consistency, PPL) are obtained from a frozen Qwen3-VL-8B-Instruct probe that is the same model family as the SFT target. While construction is benchmark-blind and not circular by construction, the paper should quantify how much the result depends on this same-family oracle—e.g., by recomputing a subset of descriptors with a different frozen probe, or by reporting correlation of VDS/PPL ranks across probes. Without that, generalization of GDO beyond “self-probe the model you will fine-tune” remains an open assumption behind the utility oracle.
minor comments (6)
  1. [§2.1, Eq. (1)] §2.1 Eq. (1) defines a per-goal 10× uniform control U_g = Uniform(D; 10|S_g|), but all reported tables use a single fixed 512k Uni-10x. Clarify whether per-goal 10× controls were run and, if not, why the fixed 512k anchor is preferred.
  2. [Table 2] Table 2: MinLoss and Diverse show negative LVBench deltas while Temp/Temp+ are positive. A one-sentence interpretation in the table caption would help readers see the intended goal spectrum without flipping to the discussion.
  3. [Fig. 4, Table 4] Fig. 4 heatmap and Table 4: adverse subtasks (e.g., MLVU Ego under MinLoss −15.1 pp) are important; consider marking statistically or practically large regressions more explicitly so redistribution is as visible as gains.
  4. [§2.2, Eq. (5)] Notation: m_tnc is defined as T(q) from a probe judgment, but the prompt template and mapping to [0,1] are only sketched. A short appendix box with the exact probe prompt would improve reproducibility beyond the code link.
  5. [§4] Related Work §4 is comprehensive but dense; a short paragraph explicitly contrasting GDO’s fixed-contract comparison with curriculum/reweighting/active-selection pipelines would sharpen the positioning.
  6. [Abstract / Tables] Minor typos/consistency: “ML VU” / “MLVU” and “L VBench” / “LVBench” spacing varies across abstract, tables, and body; unify.

Circularity Check

0 steps flagged

No significant circularity: subset construction is benchmark-blind; claimed gains are measured after fixed SFT on external benchmarks, not forced by the descriptors or scorer.

full rationale

GDO’s chain is: (1) compute six probe-derived descriptors on a shared pool, (2) rank with a fixed shared scorer (Eqs. 9–11) and goal feasibility presets Cg, (3) train Qwen3-VL-8B-Instruct under one fixed one-epoch recipe, (4) evaluate Accuracy on MVBench/VideoMME/MLVU/LVBench. The paper states subset construction is benchmark-blind—test identities and answers never enter descriptors, scoring, or filtering—so peak-match sample counts and pp gains (Table 1, Fig. 1/3) are not true by construction. Score coefficients are fixed mixture weights shared across goals, not refit to the reported benchmarks; ablations (Table 5) remove terms rather than re-optimize to the targets. Using a frozen same-family probe for VDS/PPL/SC/TNC is standard data-selection practice and does not make post-SFT external scores tautological. Self-citations (LongViTU, Bongard-OpenWorld, etc.) appear only in related work and are not load-bearing for the less-data/faster-convergence claim. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or ansatz-via-citation reduces the central empirical claim to its inputs. Residual design choices (hand-set weights, temporal presets) affect capability allocation but do not make the reported frontier crossings definitional.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central claim rests on empirical training under a fixed recipe, not a theorem. Load-bearing free choices are the hand-set scorer mixture weights, goal feasibility numbers (subset size, video ratio, VDS+ targets), and the assumption that probe-derived descriptors predict SFT utility for the same model. Invented entities are methodological constructs (GDO, descriptor vector, goal presets), not physical objects; independent evidence is the released training trajectories and ablations, not external theory.

free parameters (4)
  • video scorer mixture weights (0.35 tanh(bvid/3) + 0.95 z_vds3 + 0.35 z_qual; bvid coeffs 0.85/0.9/0.55/0.15)
    Fixed by authors once as relative mixture weights; not learned; ablations remove terms but do not re-fit weights.
  • image scorer mixture weights (0.90 tanh(bimg/3) + 0.15 z_qual; q_text weight 1.10)
    Hand-set image-side coefficients shared across goals.
  • goal profile budgets and feasibility targets (Ng, rv, VDS+tgt, rmin_t|v etc. in Table 6)
    Preset sizes 12.9k–53.3k and video/temporal floors define the four goals; central less-data claims depend on these choices.
  • Uni-10x reference budget of 512k samples
    Fixed control size against which Peak Match and Reduction are defined; reduction factors scale with this choice.
axioms (4)
  • domain assumption Holding model, optimizer, checkpoints, and evaluation fixed isolates data-allocation effects as the cause of observed accuracy/trajectory differences.
    Stated as the comparison contract in §2.1; standard causal assumption in controlled SFT ablations.
  • domain assumption Probe-derived descriptors (flow, VDS, temporal necessity, self-consistency, PPL-like difficulty, coverage) are informative of sample training utility for the same backbone.
    Core of §2.2–2.3; without this, ranking and feasibility controls would not improve downstream benchmarks.
  • domain assumption Subset construction is benchmark-blind (test identities/answers never enter descriptors, scoring, or filtering).
    Claimed in §2.1; needed to interpret gains as genuine transfer rather than leakage.
  • ad hoc to paper A single frozen Qwen3-VL-8B-Instruct probe is an adequate oracle for VDS, temporal necessity, self-consistency, and loss-based difficulty.
    Method instantiates all probe scores with this model; no multi-probe validation is reported.
invented entities (3)
  • Goal-Driven Data Optimization (GDO) framework no independent evidence
    purpose: Package shared scoring plus goal-specific feasibility presets to build optimized 1× multimodal SFT subsets under a fixed train/eval contract.
    Primary methodological object of the paper; evidence is empirical subset performance, not independent theory.
  • Six-dimensional sample descriptor vector m(x) no independent evidence
    purpose: Provide comparable cues for ranking and composition (motion, video dependence, temporal demand, stability, difficulty, coverage).
    Defined in §2.2; components reuse known signals but the joint vector is paper-specific scaffolding.
  • Four goal profiles (MinLoss, Diverse, Temp, Temp+) no independent evidence
    purpose: Instantiate different allocation goals via budget/mixture/coverage controls under one scorer.
    §2.4 and Table 6; they are design presets, not discovered natural kinds.

pith-pipeline@v1.1.0-grok45 · 21793 in / 3596 out tokens · 30659 ms · 2026-07-14T22:17:43.715071+00:00 · methodology

0 comments
read the original abstract

Multimodal instruction tuning is often compute-inefficient because training budgets are spread across large mixed image-video pools whose utility is highly uneven. We present Goal-Driven Data Optimization (GDO), a framework that computes six sample descriptors for each candidate and constructs optimized 1$\times$ training subsets for different goals. Under a fixed one-epoch Qwen3-VL-8B-Instruct training and evaluation recipe on 8 H20 GPUs, GDO uses far fewer training samples than the Uni-10x baseline while converging faster and achieving higher accuracy. Relative to the fixed 512k-sample Uni-10x baseline, GDO reaches the Uni-10x reference after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench, while improving Accuracy by +1.38, +1.67, +3.08, and +0.84 percentage points, respectively. The gains are largest on MVBench and MLVU, while LVBench improves more modestly, consistent with its ultra-long-video setting and the mismatch between that benchmark and the short-video/image-dominant training pool. Across MinLoss, Diverse, Temp, and Temp+, stronger temporal emphasis yields steadily better long-video understanding behavior. Overall, GDO provides a goal-driven data optimization framework that enables faster convergence with fewer training samples under a fixed training protocol. Code is available at https://github.com/rujiewu/GDO.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 45 linked inside Pith

  1. [1]

    In: International Conference on Learning Representations (2023)

    Abbas,A.,Tirumala,K.,Simig,D.,etal.:Semdedup:Data-efficientlearningatweb scale through semantic deduplication. In: International Conference on Learning Representations (2023)

    2023

  2. [2]

    arXiv preprint arXiv:2204.14198 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., et al.: Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 (2022)

    Pith/arXiv arXiv 2022

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., et al.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2409.12191 (2024)

    Bai, J., Huang, S., Yang, Z., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  5. [5]

    In: In- ternational Conference on Machine Learning (2009)

    Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: In- ternational Conference on Machine Learning (2009)

    2009

  6. [6]

    arXiv preprint arXiv:2209.06794 (2022)

    Chen, X., Wang, X., Changpinyo, S., et al.: Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 (2022)

    Pith/arXiv arXiv 2022

  7. [7]

    arXiv preprint arXiv:2305.18565 (2023)

    Chen, X., Wang, X., Zhai, X., et al.: Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565 (2023)

    Pith/arXiv arXiv 2023

  8. [8]

    arXiv preprint arXiv:2312.14238 (2024)

    Chen, Z., Wang, K., Huang, J., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    arXiv preprint arXiv:2305.06500 (2023)

    Dai, W., Li, J., Li, D., et al.: Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 (2023)

    Pith/arXiv arXiv 2023

  10. [10]

    arXiv preprint arXiv:2307.08701 (2023)

    Dubois, Y., Li, X., Taori, R., et al.: Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701 (2023)

    Pith/arXiv arXiv 2023

  11. [11]

    In: International Conference on Learning Representations (2018)

    Fan, Y., Tian, F., Qin, T., et al.: Learning to teach. In: International Conference on Learning Representations (2018)

    2018

  12. [12]

    arXiv preprint arXiv:2403.11481 (2024)

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., Li, Q.: Videoagent: A memory-augmented multimodal agent for video understanding. arXiv preprint arXiv:2403.11481 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    In: Advances in Neural Information Processing Systems (Datasets and Benchmarks Track) (2024)

    Fu, C., Dai, Y., Luo, Y., et al.: Video-mme: The first-ever comprehensive eval- uation benchmark of multi-modal llms in video analysis. In: Advances in Neural Information Processing Systems (Datasets and Benchmarks Track) (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2406.11794 (2024)

    Gadre, S.Y., Pappu, A., Bansal, R., et al.: Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    In: International Conference on Machine Learning (2019)

    Ghorbani, A., Zou, J.: Data shapley: Equitable valuation of data for machine learn- ing. In: International Conference on Machine Learning (2019)

    2019

  16. [16]

    arXiv preprint arXiv:1704.03003 (2017)

    Graves, A., Bellemare, M.G., Menick, J., Munos, R., Kavukcuoglu, K.: Automated curriculum learning for neural networks. arXiv preprint arXiv:1704.03003 (2017)

    Pith/arXiv arXiv 2017

  17. [17]

    arXiv preprint arXiv:2402.19479 (2024)

    He, K., Fan, H., Li, Y., et al.: Panda-70m: Captioning and retrieving 70m high-quality video-text pairs for video-language modeling. arXiv preprint arXiv:2402.19479 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    arXiv preprint arXiv:2203.15556 (2022)

    Hoffmann, J., Borgeaud, S., Mensch, A., et al.: Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)

    Pith/arXiv arXiv 2022

  19. [19]

    arXiv preprint arXiv:2302.14045 (2023)

    Huang, S., Dong, L., Wang, W., et al.: Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023)

    Pith/arXiv arXiv 2023

  20. [20]

    In: International Conference on Artificial Intelligence and Statistics (2019) 16 R

    Jia, R., Dao, D., Wang, B., et al.: Towards efficient data valuation based on the shapley value. In: International Conference on Artificial Intelligence and Statistics (2019) 16 R. Wu et al

    2019

  21. [21]

    In: International Conference on Machine Learning (2018)

    Jiang,L.,Zhou,Z.,Leung,T.,Li,L.J.,Fei-Fei,L.:Mentornet:Learningdata-driven curriculum for very deep neural networks on corrupted labels. In: International Conference on Machine Learning (2018)

    2018

  22. [22]

    arXiv preprint arXiv:2311.08046 (2024)

    Jin, P., Takanobu, R., Zhang, W., et al.: Chat-univi: Unified visual representa- tion empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    arXiv preprint arXiv:2001.08361 (2020)

    Kaplan, J., McCandlish, S., Henighan, T., et al.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    Pith/arXiv arXiv 2001

  24. [24]

    In: Advances in Neural Information Processing Systems (2019)

    Kirsch, A., van Amersfoort, J., Gal, Y.: Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In: Advances in Neural Information Processing Systems (2019)

    2019

  25. [25]

    In: Advances in Neural Information Processing Systems (2010)

    Kumar, M.P., Packer, B., Koller, D.: Self-paced learning for latent variable models. In: Advances in Neural Information Processing Systems (2010)

    2010

  26. [26]

    arXiv preprint arXiv:2304.00043 (2023)

    Kwon, Y., Zou, J.: Data-oob: Out-of-bag estimate as a simple and efficient data value. arXiv preprint arXiv:2304.00043 (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Chen, D., Zhang, Y., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    arXiv preprint arXiv:2411.14411 (2024)

    Li, B., Zhang, Y., Chen, D., et al.: Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2411.14411 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Li, B., Zhang, Y., Chen, L., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  30. [30]

    arXiv preprint arXiv:2301.12597 (2023)

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Zhu, B., Ye, Y., et al.: Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    Pith/arXiv arXiv 2023

  32. [32]

    arXiv preprint arXiv:2407.12345 (2024)

    Liu, H., Li, Z., Xu, J., et al.: Finevideo: A fine-grained vision-centric video instruc- tion dataset. arXiv preprint arXiv:2407.12345 (2024)

    Pith/arXiv arXiv 2024

  33. [33]

    arXiv preprint arXiv:2310.03744 (2023)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 (2023)

    Pith/arXiv arXiv 2023

  34. [34]

    arXiv preprint arXiv:2304.08485 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    arXiv preprint arXiv:2403.12945 (2024)

    Liu, X., Zhao, R., Wang, Z., et al.: Videofactory: Constructing large-scale high- quality video corpora with automatic filtering. arXiv preprint arXiv:2403.12945 (2024)

    Pith/arXiv arXiv 2024

  36. [36]

    arXiv preprint arXiv:2412.05271 (2024)

    Liu, Z., Zhao, X., Chen, Z., et al.: Internvl2: Better multimodal foundation models with scalable vision-language alignment. arXiv preprint arXiv:2412.05271 (2024)

    Pith/arXiv arXiv 2024

  37. [37]

    arXiv preprint arXiv:2402.10452 (2024)

    Liu, Z., Zhang, T., Zhou, C., et al.: Instruction-following difficulty as a data selec- tion signal for efficient alignment. arXiv preprint arXiv:2402.10452 (2024)

    arXiv 2024

  38. [38]

    arXiv preprint arXiv:2306.05424 (2023)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards de- tailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    arXiv preprint arXiv:2312.15685 (2023)

    Marion, P., West, P., Dziri, N., et al.: What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685 (2023)

    Pith/arXiv arXiv 2023

  40. [40]

    In: ACL Short Papers (2010)

    Moore, R.C., Lewis, W.: Intelligent selection of language model training data. In: ACL Short Papers (2010)

    2010

  41. [41]

    arXiv preprint arXiv:2303.08774 (2023)

    OpenAI: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Technical report (2023) Goal-Driven Data Optimization for Multimodal Instruction Tuning 17

    OpenAI: Gpt-4v system card. Technical report (2023) Goal-Driven Data Optimization for Multimodal Instruction Tuning 17

    2023

  43. [43]

    arXiv preprint arXiv:2107.07075 (2021)

    Paul, M., Ganguli, S., Dziugaite, G.K.: Deep learning on a data diet: Finding important examples early in training. arXiv preprint arXiv:2107.07075 (2021)

    Pith/arXiv arXiv 2021

  44. [44]

    In: North American Chapter of the Association for Computational Linguistics (2024)

    Qin, Y., Wei, F., Zhou, M., et al.: Cherry llm: Data curation for effective instruc- tion tuning. In: North American Chapter of the Association for Computational Linguistics (2024)

    2024

  45. [45]

    In: International Conference on Machine Learn- ing (2021)

    Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learn- ing (2021)

    2021

  46. [46]

    In: International Conference on Machine Learning (2018)

    Ren, M., Zeng, W., Yang, B., Urtasun, R.: Learning to reweight examples for robust deep learning. In: International Conference on Machine Learning (2018)

    2018

  47. [47]

    In: International Conference on Learning Representations (2018)

    Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core- set approach. In: International Conference on Learning Representations (2018)

    2018

  48. [48]

    Computer Sciences Technical Report 1648, University of Wisconsin–Madison (2009)

    Settles, B.: Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison (2009)

    2009

  49. [49]

    arXiv preprint arXiv:2307.16449 (2024)

    Song, Y., Wang, S., He, K., et al.: Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 (2024)

    Pith/arXiv arXiv 2024

  50. [50]

    Advances in Neural Information Processing Systems (2023)

    Sorscher, B., Geirhos, R., Shekhar, S., et al.: Beyond neural scaling laws: Beating power law scaling via data pruning. Advances in Neural Information Processing Systems (2023)

    2023

  51. [51]

    Technical report (2023)

    Taori, R., Gulrajani, I., Zhang, T., et al.: Stanford alpaca: An instruction-following llama model. Technical report (2023)

    2023

  52. [52]

    arXiv preprint arXiv:2501.01234 (2025)

    Team, Q.: Qwen3-vl technical report. arXiv preprint arXiv:2501.01234 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    In: International Conference on Learning Representations (2018)

    Toneva, M., Sordoni, A., Combes, R., et al.: An empirical study of example forget- ting during deep neural network learning. In: International Conference on Learning Representations (2018)

    2018

  54. [54]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang,W.,He,Z.,Hong,W.,Cheng,Y.,Zhang,X.,Qi,J.,Ding,M.,Gu,X.,Huang, S., Xu, B., et al.: Lvbench: An extreme long video understanding benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22958–22967 (2025)

    2025

  55. [55]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., et al.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    Pith/arXiv arXiv 2022

  56. [56]

    arXiv preprint arXiv:2306.04387 (2023)

    Wang, Y., Mishra, S., Alipoormolabashi, P., et al.: M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387 (2023)

    Pith/arXiv arXiv 2023

  57. [57]

    In: International Conference on Lan- guage Resources and Evaluation (2020)

    Wenzek, G., Lachaux, M.A., Conneau, A., et al.: Ccnet: Extracting high quality monolingual datasets from web crawl data. In: International Conference on Lan- guage Resources and Evaluation (2020)

    2020

  58. [58]

    arXiv preprint arXiv:2501.05037 (2025)

    Wu, R., Ma, X., Ci, H., Fan, Y., Wang, Y., Zhao, H., Li, Q., Wang, Y.: Longvitu: Instruction tuning for long-form video understanding. arXiv preprint arXiv:2501.05037 (2025)

    Pith/arXiv arXiv 2025

  59. [59]

    arXiv preprint arXiv:2310.10207 (2023)

    Wu, R., Ma, X., Zhang, Z., Wang, W., Li, Q., Zhu, S.C., Wang, Y.: Bongard- openworld: Few-shot reasoning for free-form visual concepts in the real world. arXiv preprint arXiv:2310.10207 (2023)

    Pith/arXiv arXiv 2023

  60. [60]

    In: International Conference on Machine Learning (2024)

    Xia, M., Wu, T., Ge, T., et al.: Less: Selecting influential data for targeted instruc- tion tuning. In: International Conference on Machine Learning (2024)

    2024

  61. [61]

    arXiv preprint arXiv:2402.11999 (2024)

    Xie, S.M., Gururangan, S., Xin, J., et al.: Dclm: Data curriculum for language models. arXiv preprint arXiv:2402.11999 (2024)

    Pith/arXiv arXiv 2024

  62. [62]

    arXiv preprint arXiv:2304.14178 (2023)

    Ye, Q., Xu, H., Xu, M., et al.: mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 (2023)

    Pith/arXiv arXiv 2023

  63. [63]

    arXiv preprint arXiv:2205.01917 (2022) 18 R

    Yu, J., Wang, Z., Vasudevan, V., et al.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022) 18 R. Wu et al

    Pith/arXiv arXiv 2022

  64. [64]

    arXiv preprint arXiv:2406.16846 (2024)

    Zhang, P., Xu, X., Li, Y., et al.: Longva: Scaling long-context visual language models for long videos and beyond. arXiv preprint arXiv:2406.16846 (2024)

    Pith/arXiv arXiv 2024

  65. [65]

    arXiv preprint arXiv:2312.06653 (2023)

    Zhang, R., Feng, Y., Li, B., et al.: Active instruction tuning: Improving cross-task generalization by actively selecting instruction data. arXiv preprint arXiv:2312.06653 (2023)

    Pith/arXiv arXiv 2023

  66. [66]

    arXiv preprint arXiv:2410.02713 (2024)

    Zhang, Y., Wang, Y., Liu, Z., et al.: Llava-video: 178k high-quality video instruction-tuning data for multi-modal models. arXiv preprint arXiv:2410.02713 (2024)

    Pith/arXiv arXiv 2024

  67. [67]

    arXiv preprint arXiv:2406.04264 (2024)

    Zhang, Y., Zhou, J., Sun, H., et al.: Vista-400k: A high-quality video instruction- tuning dataset for general-purpose video-language models. arXiv preprint arXiv:2406.04264 (2024)

    Pith/arXiv arXiv 2024

  68. [68]

    arXiv preprint arXiv:2407.05282 (2024)

    Zhao, H., Ma, X., Chen, L., Si, S., Wu, R., An, K., Yu, P., Zhang, M., Li, Q., Chang, B.: Ultraedit: Instruction-based fine-grained image editing at scale. arXiv preprint arXiv:2407.05282 (2024)

    Pith/arXiv arXiv 2024

  69. [69]

    Advances in Neural Information Processing Systems (2023)

    Zhou, C., Lu, Y., Mishra, S., et al.: Lima: Less is more for alignment. Advances in Neural Information Processing Systems (2023)

    2023

  70. [70]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al.: Mlvu: Benchmarking multi-task long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13691–13701 (2025)

    2025

  71. [71]

    Zhu, D., Chen, J., Shen, X., et al.: Minigpt-4: Enhancing vision-language under- standing with advanced large language models. arXiv preprint arXiv:2304.10592 (2023) Supplementary Material 19 Supplementary Material for Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning A Additional Analysis A.1 Trajectories Figu...

    Pith/arXiv arXiv 2023