{"id":"52b21db8-8389-4bfd-8970-74725e53b21d","arxiv_id":"2606.05445","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Brick-Composer trains MLLMs on brick assembly via three signals, raising step-level success from under 1% to around 15% on the new BC-Bench benchmark.","lead":"The paper introduces Brick-Composer, a training framework that uses human design examples, environmental feedback, and synthetic data to improve multimodal LLMs on sequential brick selection and placement tasks. A smart generalist might read it to understand current limits of AI spatial reasoning and how targeted training can advance embodied construction capabilities.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Simulation fidelity for World Feedback and Synthetic Experience unvalidated against real physics","rationale":"The reader's weakest_assumption already isolates the exact unstated assumption about simulation fidelity. The full text supplies implementation details but does not add empirical sim-to-real validation, leaving the same load-bearing risk.","tokens_in":1804,"tokens_out":261,"duration_ms":22498,"concrete_test":"Take the 20 highest-success objects from the trained Qwen-3-8B evaluation; re-execute the same predicted sequences on a physical robot with identical bricks and compare per-step pose error and strict success rate to the simulator numbers. A drop >50% in real success would confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes large gains (3x selection accuracy, step success from <1% to ~15%) to the three training signals. World Feedback and Synthetic Experience both depend on a simulator to supply visual/physical consequences and scaled experience. No section in the manuscript validates that the simulator reproduces real brick contact, friction, stability, or gravity effects at the precision needed for pose estimation. If simulator-reality mismatch exists, the reported BC-Bench numbers reflect simulator-specific learning rather than transferable assembly skill.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces BC-Bench, the first benchmark for MLLM evaluation on sequential brick assembly (brick selection and pose estimation subtasks), and proposes Brick-Composer, a training framework that combines Human Design Sparks, World Feedback, and Synthetic Experience to fine-tune models such as Qwen-3-8B. It reports concrete gains: brick selection accuracy improved by over 3x, reduced pose estimation errors, strict step-level success raised from <1% to ~15%, and up to 42% of steps correctly composed for complete objects.","tokens_in":1912,"tokens_out":485,"duration_ms":11634,"significance":"If the empirical results hold under real-world conditions, the work provides the first systematic demonstration that MLLMs can acquire grounded assembly skills via the three proposed signals, establishing a reproducible benchmark and training recipe that could extend to other sequential physical construction tasks.","major_comments":[{"comment":"Abstract and §4 (World Feedback, Synthetic Experience): the central performance claims (3x selection accuracy, step success from <1% to ~15%) rest on simulator-generated signals, yet no section validates that the simulator reproduces real brick contact forces, friction coefficients, or stability under gravity at the precision required for pose estimation transfer; without this, the BC-Bench numbers may reflect simulator-specific overfitting rather than transferable skill.","section":"Abstract, §4"},{"comment":"§5 (Experiments): the reported improvements lack details on data splits, statistical significance tests, number of runs, and whether post-hoc hyperparameter choices were made after seeing test results; these omissions make it impossible to assess whether the gains are robust or could be artifacts of the experimental protocol.","section":"§5"}],"minor_comments":[{"comment":"Notation for the two subtasks (selection vs. pose) is introduced in the abstract but not consistently referenced with equation numbers in the methods; adding explicit definitions would improve clarity.","section":"Abstract, §3"},{"comment":"Figure captions for BC-Bench examples should explicitly state the number of candidate bricks and the exact success criteria used for the 15% and 42% figures.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and note the planned revisions.","responses":[{"response":"We agree that all experiments, including BC-Bench and the three training signals, are performed in simulation and that the manuscript provides no direct validation of simulator physics (contact forces, friction, gravity stability) against real bricks. The work positions itself as an initial study of MLLM assembly capabilities within a reproducible simulated environment rather than a claim of immediate real-world transfer. We will revise the abstract and §4 to state this scope explicitly and add a limitations paragraph discussing simulator assumptions and the sim-to-real gap.","revision_made":"partial","referee_comment":"[Abstract, §4] Abstract and §4 (World Feedback, Synthetic Experience): the central performance claims (3x selection accuracy, step success from <1% to ~15%) rest on simulator-generated signals, yet no section validates that the simulator reproduces real brick contact forces, friction coefficients, or stability under gravity at the precision required for pose estimation transfer; without this, the BC-Bench numbers may reflect simulator-specific overfitting rather than transferable skill."},{"response":"We acknowledge that the current manuscript omits these experimental details. The revised version will add a dedicated experimental protocol subsection specifying the train/validation/test splits, the number of independent runs, the statistical significance tests performed, and confirmation that hyperparameter selection preceded test-set evaluation.","revision_made":"yes","referee_comment":"[§5] §5 (Experiments): the reported improvements lack details on data splits, statistical significance tests, number of runs, and whether post-hoc hyperparameter choices were made after seeing test results; these omissions make it impossible to assess whether the gains are robust or could be artifacts of the experimental protocol."}],"tokens_in":1432,"tokens_out":394,"duration_ms":26462,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a new benchmark called BC-Bench plus a three-signal training setup that lifts MLLM performance on brick selection and placement in simulation. A Qwen-3-8B model reaches roughly 15% strict step success and 42% steps on full objects after training, which is a measurable step up from the near-zero baselines they report.\n\nWhat stands out is the concrete framing of the task into selection and pose estimation subtasks, plus the three signals: human design examples, simulator feedback on outcomes, and extra synthetic trajectories. The numbers on accuracy and error reduction are stated plainly, and the work tests an off-the-shelf model rather than claiming a new architecture.\n\nThe soft spot is the lack of any check on whether the simulator reproduces real brick contact, friction, or stability. World Feedback and Synthetic Experience both depend on that simulator, yet the paper gives no comparison to physical measurements or real-robot trials. If the sim diverges from reality on those properties, the reported gains stay inside the simulator and do not yet show transferable assembly skill.\n\nThis paper is aimed at researchers working on embodied MLLMs and sequential decision tasks in robotics. Readers who want a starting benchmark for brick-style assembly will get something usable to build on. The experimental claims are specific enough that a serious referee could evaluate them and ask for the missing sim validation runs.\n\nI would send it to peer review with a note to add at least basic real-world or high-fidelity physics checks before final acceptance.","headline":"Brick-Composer shows clear gains on a new assembly benchmark but leaves the simulator-to-reality gap unaddressed.","tokens_in":2424,"tokens_out":377,"would_cite":false,"duration_ms":20302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MLLMs acquire brick assembly skills through three training signals, tripling selection accuracy and raising step success to 15%.","keywords":["brick assembly","multimodal large language models","assembly planning","spatial reasoning","BC-Bench","pose estimation","sequential decision making","construction tasks"],"falsifier":"Retraining the same base MLLM with only two of the three signals and measuring whether step-level assembly success remains below 5% on BC-Bench would test whether the full combination is required for the reported gains.","tokens_in":2723,"feed_emoji":"🧱","tokens_out":728,"duration_ms":34206,"temperature":0.7,"pith_summary":"The paper tries to establish that multimodal large language models lack fine-grained visual grounding and spatial reasoning for brick assembly but can acquire usable skills when trained with a specific framework. It formulates each assembly step as selecting the correct brick from candidates then estimating its placement pose, introduces a benchmark showing current models fail at both, and demonstrates large gains from the new method. A sympathetic reader would care because the work sketches a route for general AI models to handle sequential physical construction with reusable parts rather than staying limited to language or image tasks.","feed_headline":"MLLMs triple brick selection accuracy after targeted training","feed_subtitle":"Human design sparks, world feedback and synthetic experience lift step success from under 1% to 15% for object construction.","key_machinery":"Brick-Composer learning framework that integrates Human Design Sparks for demonstrations, World Feedback for physical grounding, and Synthetic Experience for scaling to train MLLMs on sequential brick selection and pose estimation.","core_discovery":"Brick assembly is formulated as a sequential decision-making problem where each step requires brick selection from candidates and pose estimation for placement. Current state-of-the-art MLLMs struggle with both subtasks. Brick-Composer equips MLLMs with assembly capabilities by combining Human Design Sparks that supply affordance-rich construction demonstrations, World Feedback that grounds predictions in visual and physical outcomes, and Synthetic Experience that scales training beyond existing designs. The result is brick selection accuracy improved by over three times, substantially lower pose estimation errors, and strict step-level assembly success increased from less than 1% to around","pith_inferences":["The same three-signal approach might transfer to assembly tasks with different building blocks such as furniture or modular robots.","BC-Bench could serve as a reusable testbed for measuring progress in MLLM spatial reasoning over time.","Closing the remaining gap to reliable full-object assembly would likely require tighter integration between the learned policy and real-world robot execution."],"forward_implications":["Brick selection accuracy improves by over three times compared with baseline MLLMs.","Pose estimation errors are substantially reduced.","Strict step-level assembly success rises from less than 1% to around 15%.","A fine-tuned Qwen-3-8B model can correctly compose up to 42% of the steps for a complete object."],"fun_headline_variants":["Brick-Composer triples MLLM brick selection accuracy","MLLM assembly steps reach 15% success after training","Design sparks and feedback lift brick pose estimation","Synthetic experience scales MLLM brick assembly skills"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The three proposed training signals can be combined and applied to MLLMs to produce the stated performance gains without requiring additional unstated assumptions about simulation fidelity or model fine-tuning details.","fun_headline_variants_meta":{"raw":{"variants":["Brick-Composer triples MLLM brick selection accuracy","MLLM assembly steps reach 15% success after training","Design sparks and feedback lift brick pose estimation","Synthetic experience scales MLLM brick assembly skills"]},"model":"grok-4.3","cost_usd":0.003944,"raw_usage":{"total_tokens":2070,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":39437000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1241,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":59,"duration_ms":12103,"temperature":1.0,"reasoning_tokens":1241,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T05:57:13.372824+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Retraining the same base MLLM with only two of the three signals and measuring whether step-level assembly success remains below 5% on BC-Bench would test whether the full combination is required for the reported gains.","supporting_citations":[],"review_version":1}