{"id":"861e7278-f3b1-4268-b9a7-aa81e4353fce","arxiv_id":"2505.11108","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PARSEC provides a large crowdsourced benchmark for personalized object rearrangement, and ContextSortLM, an LLM-based model using structured preference representations, achieves top computational performance and competitive human-rated alignment.","lead":"This paper introduces PARSEC, a benchmark with 110,000 crowdsourced examples of how 72 people organize household objects, plus an LLM-based model, ContextSortLM, that places objects by combining prior and current scene context. Household robots need to infer a user's organization style without explicit instructions, and this work provides data and a method to study that.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ContextSortLM's computational superiority is a single point estimate with no error bars or significance tests; the 0.06 PA gap over APRICOT-NonInteractive may be within noise.","rationale":"The reader's weakest_assumption focused on the external validity of crowdsourced 2D arrangements as a proxy for real-home preferences, which is a legitimate concern about benchmark transfer. However, the more immediately load-bearing issue for the paper's central claim is that the computational comparison supporting 'ContextSortLM outperforms other models' lacks any measure of uncertainty. The reader's rationale did mention that Table II lacks error bars and statistical tests, so there is partial agreement. I recommend keeping the CONDITIONAL verdict, because the concern does not invalidate the benchmark contribution or the method, but it does require the authors to add statistical support or soften the wording before the claim can be accepted as stated.","tokens_in":10842,"tokens_out":4575,"duration_ms":45813,"concrete_test":"Re-run the KnownEnv evaluation of ContextSortLM, APRICOT-NonInteractive, TidyBot-Random, and ConSOR at least 10 times with different random seeds for fold generation and, for the LLM methods, with temperatures {0, 0.3, 0.7}. For each model, report mean PA with 95% bootstrap confidence intervals over folds and runs, and run paired significance tests (e.g., Wilcoxon signed-rank) between ContextSortLM and APRICOT-NonInteractive for each environment category and overall. If the PA differences are not significant at p<0.05, the claim that ContextSortLM outperforms other models should be reworded to 'numerically higher' or 'trending higher'.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that ContextSortLM outperforms other models in computational evaluation rests entirely on Table II, which reports Placement Accuracy as a single value per model and environment category, averaged over cross-validation folds. No confidence intervals, standard deviations, number of runs, or significance tests are provided for any PA value. This is especially problematic because the top models are LLM-based (gpt-4-0613) and their outputs are stochastic; the paper never states the sampling temperature, the number of repeated generations, or whether the reported numbers come from a single run. The margin between ContextSortLM (0.59) and APRICOT-NonInteractive (0.53) on average, and the per-category differences (e.g., 0.54 vs 0.50 in Uniform-1D), are small enough that they could be explained by LLM sampling variability or by fold selection. Figure 6 even shows high variance in ContextSortLM's SED scores in densely occupied environments, yet the corresponding PA advantage is presented as conclusive. Without statistical support, the abstract's claim that ContextSortLM 'outperforms other models' is not established; only a numerical trend is shown. The human evaluation does include statistical tests, but those are a separate, small-scale study (36 examples, 3 raters each) and cannot validate the computational ranking. This undermines the paper's primary comparative claim, even if the benchmark itself and the proposed method remain valuable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PARSEC, a benchmark and dataset for personalized robotic object rearrangement in partially arranged scenes, together with ContextSortLM, a zero-shot LLM-based method that builds a JSON-style preference representation from prior arrangements and then places objects via code completion conditioned on the current partial arrangement. The dataset contains 110K rearrangement examples generated from 432 crowdsourced arrangements by 72 users across 15 environments and five organizational tasks. The authors compare ContextSortLM with six graph-based baselines using Placement Accuracy, Scene Edit Distance, and Incorrectly Grouped Objects, and complement this with a crowdsourced human study in which 108 raters rank predictions from four models. The main reported findings are that models using both prior and current scene context outperform single-context models, and that ContextSortLM achieves the highest Placement Accuracy and ranks among the top two in human rater preference across environment categories.","tokens_in":11086,"tokens_out":8260,"duration_ms":81606,"significance":"The paper provides a concrete, released benchmark and dataset for an under-served problem, and its zero-shot evaluation is an appropriate way to test generalization to unseen users and environments. The structured JSON preference representation for multiple valid placements is a sensible and potentially reusable design choice, and the human evaluation includes significance testing, which is a strength. The main limitation is that the computational comparison, which underpins the abstract's 'outperforms' claim, is reported without uncertainty quantification; the benchmark's external validity also rests on an unvalidated crowdsourcing proxy. If the authors add repeated runs and statistical intervals, and qualify the claims about real-world relevance, the paper would be a solid benchmark contribution.","major_comments":[{"comment":"Placement Accuracy is reported as a single number per model and environment category with no error bars, standard deviation, confidence intervals, or statistical tests. Because the three leading models are stochastic LLMs using gpt-4-0613 and the paper does not state sampling temperature, number of repeated generations, or seed handling, the margins between ContextSortLM and APRICOT-NonInteractive (0.59 vs 0.53 average; 0.54 vs 0.50 in Uniform-1D) could plausibly be explained by sampling noise or fold selection. Since this table is the primary support for the abstract's statement that ContextSortLM 'outperforms other models,' that claim is currently not established. Please report repeated-run statistics and pairwise significance tests, or explicitly state deterministic decoding and show fold-level stability.","section":"§VI.A, Table II"},{"comment":"The paper describes PARSEC as capturing 'real-world organizational habits,' but the data come from 72 Prolific workers arranging objects in simulated 2D environments, and no evidence is provided that these crowdsourced arrangements correspond to how people organize physical homes. This assumption underlies every conclusion drawn from the PARSEC evaluation and should be explicitly validated, or the claims about real-world relevance should be qualified in the abstract and discussion.","section":"§IV and §VIII"},{"comment":"The human evaluation, while statistically analyzed, is small (36 examples, 3 raters each) and its results are more mixed than the abstract suggests: APRICOT-NonInteractive has a higher alignment score than ContextSortLM in Uniform-1D (45.2% vs 40.5%) and Uniform-2D (43.2% vs 37.8%). The paper explains this through over-clustering, and the rank scores do support ContextSortLM being among the top two, but the text should more prominently acknowledge that the human evidence does not fully corroborate the computational superiority claim.","section":"§VII.A, Table III"}],"minor_comments":[{"comment":"The sentence 'designated the other five as observed user arrangements ... generating C(5,2) pairs' is internally inconsistent: if all five other arrangements are used as the observed set, there is one observed set per target, not ten. Please clarify the generation protocol and report the exact number of distinct (AO, A*G) pairs before random omission of objects.","section":"§IV, Dataset Generation"},{"comment":"The text says real user arrangements exhibit lower within-user similarity scores (Si, Si) compared to rule-based arrangements (Uj, Uj), but earlier S1, S2 denote rule-based personas and U1–U5 denote real users; the labels appear to be swapped.","section":"§IV, Figure 3"},{"comment":"The procedure is described as k-fold cross-validation with 'one of five users per environment category' excluded per fold, but the number of folds k is never stated; please specify the exact fold construction.","section":"§VI, Experimental Setup"},{"comment":"The phrase 'Bonferroni correction of α = 6' is unclear; normally one corrects α by dividing by the number of comparisons, so please state the number of pairwise tests and the resulting corrected significance threshold.","section":"§VII.A, Statistical Analysis"},{"comment":"The columns for KnownEnv and NovelEnvCategory are identical for ContextSortLM, APRICOT-NonInteractive, and TidyBot-Random because these models are evaluated zero-shot; consider reporting these results once and stating explicitly that they are constant across the two conditions by construction.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The paper is a reasonable benchmark contribution, and the code/data release is a strength. The main risk is overstatement in the abstract relative to the lack of uncertainty quantification in Table II. The external-validity concern is common to crowdsourced benchmarks and can be addressed with qualifying language rather than a large new study. I do not see citation or novelty issues; the self-citation of ConSOR is relevant prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a benchmark paper first, a methods paper second. The new dataset is the real contribution; ContextSortLM is a reasonable LLM extension, but the comparative claims outrun the statistics.\n\nWhat's actually new: PARSEC is a substantial crowdsourced dataset—110K rearrangement examples from 72 users, 93 object categories, 15 environments—that fills a real gap. Existing datasets are rule-based or small, and the benchmark formulation around partially arranged scenes plus prior observations is cleanly specified. ContextSortLM's idea, summarizing prior observed arrangements into a JSON-style preference representation that explicitly allows multiple valid placements, is sensible and a modest step beyond TidyBot and APRICOT. The authors also deserve credit for Section VIII, where they state ContextSortLM's limitations (dense environments, sensitivity to noisy observations) rather than burying them.\n\nSoft spots: The main issue is Table II. Placement Accuracy is reported as a single point per model and category, with no confidence intervals, no number of runs, and no sampling temperature or generation count for the stochastic LLM-based models. The gap between ContextSortLM (0.59) and APRICOT-NonInteractive (0.53) could plausibly be LLM sampling noise. That doesn't sink the benchmark, but it does mean the abstract's \"outperforms other models\" is not established by the computational evaluation. The human evaluation partly rescues this: the rank-score analysis with Friedman/Wilcoxon shows ContextSortLM significantly better than APRICOT-NonInteractive in Non-Uniform and better than TidyBot-Random in Uniform-1D, though not everywhere. That's an honest result, and the paper presents it as such. The human study is small (36 examples, 14 users), so those significance tests should be read cautiously. Also, crowdsourced arrangements are a proxy for real-home preferences; the authors acknowledge some of this in the limitations.\n\nOverall: the benchmark deserves a serious referee. The dataset is reproducible (code and data released), the problem formulation is clean, and the evaluation is mostly careful. The paper needs revisions: add error bars and sampling details to Table II, soften the abstract, and clarify the scope of the human evaluation. But as a resource paper it's useful, and the flaws are fixable.\n\nRecommendation: engage with it, send it to peer review. The benchmark will likely be used regardless of what happens to ContextSortLM.","headline":"A valuable new crowdsourced benchmark whose main comparative claim is underpowered by the statistics; the dataset itself justifies peer review.","tokens_in":11595,"tokens_out":1948,"would_cite":true,"duration_ms":19809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a household robot can infer a user's organizational preferences from scene context alone, and that preserving multiple valid placements for each object produces better rearrangements than single-rule or single-context…","keywords":["object rearrangement","personalized robotics","LLM preference learning","scene context","PARSEC benchmark","crowdsourced dataset","partial arrangement","semantic environment"],"falsifier":"An ablation where ContextSortLM collapses its JSON-style meta-preference into a single textual rule list and still matches its placement accuracy would show the representation is not load-bearing; conversely, a real-home study where ContextSortLM's placements are rated no closer to user preference than a single-context baseline would falsify the transfer claim.","tokens_in":10633,"feed_emoji":"🤖","tokens_out":5689,"duration_ms":56633,"temperature":0.7,"pith_summary":"The paper argues that a household robot can learn a specific person's organizational preferences from scene context alone, without explicit instructions, and place new objects in a partially arranged room. It introduces PARSEC, a benchmark built from 110K rearrangement examples crowdsourced from 72 users across 93 object categories and 15 environments. It proposes ContextSortLM, an LLM-based model that summarizes observed arrangements into a structured JSON-style preference representation that allows multiple valid placements for the same object. On PARSEC, models using both prior and current scene context outperform models using only one, and ContextSortLM achieves the best placement accuracy as well as top-two human rater rankings in all three environment categories. The paper's central claim is that combining multiple scene context sources through a flexible preference representation best captures real organizational habits.","feed_headline":"Robots can learn your tidying style just from past arrangements","feed_subtitle":"PARSEC data set and ContextSortLM show that blending what you did before with the room's current state places objects best.","key_machinery":"The load-bearing mechanism is ContextSortLM's meta-preference representation $M_O$, a JSON-style consolidation of rules extracted from the user's previously observed arrangements. Unlike a single textual rule list, $M_O$ preserves multiple valid placement locations for an object category, so flexible preferences are not lost. Placement itself is done by appending $M_O$ to a prompt that rewrites the current partial arrangement $A_P$ as Python-style pick-place commands and asking an LLM to complete the code. This structure is what lets the model reason jointly over prior and current scene context while avoiding the overgeneralization that occurs when an LLM compresses observations into one description.","core_discovery":"The central discovery is that explicitly representing the fact that a user may find several locations acceptable for the same object, rather than collapsing observations into one rule or one textual summary, lets an LLM-based rearrangement system place objects more faithfully in partially arranged environments. ContextSortLM first extracts placement rules from each observed arrangement, then consolidates them into a single JSON-style preference representation that keeps multiple valid surfaces per object. At test time it rewrites the current partial arrangement as pick-place code and asks the LLM to complete the program against this representation. In the PARSEC benchmark its placements match the target user's arrangements better than any compared model, and in a crowdsourced ranking of predictions by 108 raters it ranks among the top two in every environment category. The paper also establishes a benchmark-level finding: personalized rearrangement methods that integrate prior observations with the environment's current arrangement outperform methods relying on either source alone.","pith_inferences":["Editorial inference: Because PARSEC environments are simulated 2D layouts, the benchmark likely understates the role of 3D spatial cues such as reachability and visibility; a model that exploits those cues could outperform ContextSortLM's rankings in real homes.","Editorial inference: The JSON-style meta-preference representation could generalize to other LLM tasks where multiple outputs are acceptable, such as mobile manipulation goal selection or human-in-the-loop planning.","Editorial inference: The observation that human raters tolerate reasonable placement variations suggests future evaluations should measure 'any acceptable placement' as well as exact matches.","Editorial inference: An active-learning extension that asks the user one targeted question only when ContextSortLM's valid placements conflict could close the gap in densely occupied environments without burdening the user."],"forward_implications":["A robot using ContextSortLM can begin personalizing after a few passive observations, with no explicit user instructions.","Household robots operating in already-occupied spaces benefit more from combining prior and current context than from either source alone.","Designers of LLM-based rearrangement prompts should preserve multiple valid placements explicitly; flattening them into a single rule loses user alignment.","PARSEC provides a standard comparison mode for future rearrangement models across different environment types.","ContextSortLM's weakness in densely occupied scenes points toward hybrid designs that let learned policies resolve conflicts with the current environment."],"supporting_citations":[{"why":"TidyBot provides the LLM rule-generation template that ContextSortLM's rule extraction is inspired by, and serves as the single-context baseline.","marker":"[9]"},{"why":"APRICOT is the dual-context LLM baseline whose NonInteractive variant ContextSortLM is compared against; its textual preference summary is the contrast that motivates the JSON-style representation.","marker":"[8]"},{"why":"ConSOR supplies the Scene Edit Distance metric and is the current-scene-context baseline evaluated across occupancy levels.","marker":"[6]"},{"why":"NeatNet is the graph-based prior-context baseline that needs training per environment, used to test generalization limits.","marker":"[12]"},{"why":"CF+ is the earlier dual-context collaborative filtering method that works only in seen environments, motivating the PARSEC benchmark.","marker":"[7]"},{"why":"CF is the collaborative filtering baseline whose pairwise object similarities miss broader scene context.","marker":"[4]"}],"fun_headline_variants":["Robots learn tidying preferences from scene context alone","ContextSortLM beats others in personalized object placement","PARSEC: 110K examples teach robots your placement style","LLM places objects by respecting your multiple valid spots","Personalized rearrangement from past arrangements, no instructions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The crowdsourced arrangements made by 72 online workers in simulated 2D scenes are a valid proxy for how real people organize their physical homes.","fun_headline_variants_meta":{"raw":{"variants":["Robots learn tidying preferences from scene context alone","ContextSortLM beats others in personalized object placement","PARSEC: 110K examples teach robots your placement style","LLM places objects by respecting your multiple valid spots","Personalized rearrangement from past arrangements, no instructions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":1954,"prompt_tokens":965,"completion_tokens":989,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":912}},"tokens_in":581,"tokens_out":989,"duration_ms":9804,"temperature":1.0,"reasoning_tokens":912,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:25.999344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation where ContextSortLM collapses its JSON-style meta-preference into a single textual rule list and still matches its placement accuracy would show the representation is not load-bearing; conversely, a real-home study where ContextSortLM's placements are rated no closer to user preference than a single-context baseline would falsify the transfer claim.","supporting_citations":[{"cited_title":"APRICOT: Active preference learning and constraint-aware task planning with LLMs,","cited_arxiv_id":null,"evidence_quote":"APRICOT is the dual-context LLM baseline whose NonInteractive variant ContextSortLM is compared against; its textual preference summary is the contrast that motivates the JSON-style representation."},{"cited_title":"ConSOR: A context- aware semantic object rearrangement framework for partially arranged scenes,","cited_arxiv_id":null,"evidence_quote":"ConSOR supplies the Scene Edit Distance metric and is the current-scene-context baseline evaluated across occupancy levels."},{"cited_title":"My house, my rules: Learning tidying preferences with graph neural networks,","cited_arxiv_id":null,"evidence_quote":"NeatNet is the graph-based prior-context baseline that needs training per environment, used to test generalization limits."},{"cited_title":"Learning user’s preferred household organization via collaborative filtering methods.,","cited_arxiv_id":null,"evidence_quote":"CF+ is the earlier dual-context collaborative filtering method that works only in seen environments, motivating the PARSEC benchmark."},{"cited_title":"Organizing objects by predicting user preferences through collaborative filtering,","cited_arxiv_id":null,"evidence_quote":"CF is the collaborative filtering baseline whose pairwise object similarities miss broader scene context."}],"review_version":1}