Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

BiFold: Bimanual Cloth Folding with Language Guidance

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BiFold repurposes a pre-trained vision-language model to convert text commands into bimanual pick-and-place actions for cloth folding, reporting state-of-the-art results on an existing language-conditioned folding benchmark and the best…

desk verdict Solid empirical contribution with a useful auto-annotated bimanual dataset, but the language-guidance claim is untested because no ablation removes the language input. read the letter →

arxiv 2501.16458 v2 pith:UBMQQEK6 submitted 2025-01-27 cs.RO cs.CV

Oriol Barbany , Adrià Colomé , Carme Torras This is my paper · ORCID
classification cs.ROcs.CV
keywords bimanualmanipulationclothfoldinglanguage-conditionedpolicyvision-languagemodelautomaticannotationpick-and-placegarmentdeformationrobotlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single model can translate high-level text instructions, such as "fold the left sleeve inward to the halfway point", directly into two-arm pick-and-place actions for folding garments. The proposed model, BiFold, uses a pre-trained vision-language transformer with most weights frozen and lightly adapted, then decodes separate probability heatmaps for the left and right grasp and release positions. On an existing unimanual language-conditioned folding benchmark, BiFold reports higher success rates than earlier text-conditioned policies, and on a new bimanual dataset that the authors annotate automatically, it reports the best image-space keypoint metrics and the lowest simulation mesh errors. The paper also contributes a scalable annotation pipeline that turns roughly four thousand human virtual-reality folding demonstrations into over a thousand distinct language-aligned prompts, enabling training without manual labels.

What carries the argument

The load-bearing component is a contrastive vision-language transformer whose image and text branches are kept largely frozen and adapted with low-rank updates, followed by a transformer encoder that fuses token sequences and convolutional decoders that emit per-arm pick-and-place heatmaps. The other essential mechanism is the automatic dataset-annotation pipeline: it maps garment vertices to a per-category canonical coordinate space, thresholds those coordinates into semantic regions such as sleeves and waistbands, merges the left and right hand labels with a hand-designed rule table, and instantiates template sentences into hundreds of varied instructions. Together these allow the model to be trained on bimanual human demonstrations with no manual annotation.

What would settle it

Take a random sample of the new bimanual dataset, have a human annotator verify each automatically generated instruction and pick-and-place pair, and measure the label agreement rate; if a substantial fraction of the "place" labels or instructions are wrong, the bimanual evaluation is not a reliable measure of language-conditioned folding.

Watch

Extended reading notes

Core claim

The central discovery is that a frozen, low-rank-adapted vision-language transformer provides a sufficiently rich shared representation of garment images and natural-language folding instructions that the model can predict bimanual actions it was never explicitly taught. The policy produces pixel-space distributions over pick and place locations for each arm, constrained so that picks fall on the cloth mask, and it conditions on up to three previous keyframes to resolve ambiguities such as which side of a symmetric cloth is "top". The authors show that this design outperforms a prior transformer-based language-conditioned folding policy on an existing benchmark, and that on their own bimanual dataset it achieves the best average precision, lowest keypoint error, and lowest simulation mesh error while generalizing to new garments, new paraphrased instructions, and real images.

Load-bearing premise

The bimanual results rest on the automatic annotation pipeline producing correct language instructions and pick-and-place labels; if those labels are systematically noisy, the reported bimanual improvements would be overstated.

Editorial extensions

If this is right

  • If the architecture is right, a frozen vision-language backbone plus small adaptation is enough for language-conditioned deformable-object manipulation, so the main barrier becomes labelled data rather than representation learning.
  • The automatic annotation pipeline can be reapplied to other tracked demonstration datasets, lowering the cost of producing language-aligned manipulation benchmarks.
  • Conditioning on a short history of keyframes improves pick-and-place precision in the bimanual setting, so memory of past states should be part of future folding policies.
  • Predicting actions in pixel space, instead of on a downsampled point cloud, allows place positions to lie outside the current cloth silhouette, which the paper shows is needed for most bimanual folds.
  • The reported gains on unseen tasks in the unimanual benchmark indicate that the text-image alignment transfers beyond the exact instruction templates used in training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, which the paper leaves open, would be to couple BiFold's action heatmaps with an instruction-breaking planner so a single high-level goal yields a whole folding sequence; nothing in the paper rules this out.
  • The annotation pipeline's reliance on canonical-coordinate thresholds assumes consistent garment topology within a category; testing it on highly varied designer garments would reveal how far the approach scales.
  • Because the real-world evaluation is offline and qualitative, the strongest testable extension would be a full closed-loop dual-arm deployment on the same garments, measuring physical fold success rather than heatmap accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces BiFold, a vision-language model for bimanual cloth folding from RGB images and natural language instructions. The model uses a frozen SigLIP encoder adapted with LoRA, fuses image and text tokens in a transformer, optionally conditions on H previous observations, and outputs pick and place heatmaps for left and right arms. To train it, the authors augment the VR-Folding dataset with automatically generated language instructions via NOCS-based semantic labeling and template prompts. They evaluate on the unimanual Deng et al. benchmark (Table I), on their new bimanual dataset in simulation and image space (Tables II-III), and on real images offline (Fig. 4c). The paper claims state-of-the-art unimanual performance, best bimanual performance, and strong generalization to new instructions, garments, and environments.

Significance. If the results hold, the paper makes useful contributions: a practical recipe for adapting a pretrained vision-language model to bimanual cloth-folding action prediction, a fully automatic pipeline for generating language-aligned action labels from existing human demonstrations, and a new bimanual benchmark. The context mechanism and the move from point-cloud-based to pixel-space prediction are sensible design choices, and the external unimanual benchmark provides a useful sanity check. However, the central attribution of performance to language guidance is not yet supported by the experiments, and the evaluation lacks statistical grounding and quantitative real-world validation; these gaps currently limit the strength of the stated contributions.

major comments (4)
  1. [IV-D, Table IV] The central claim that language guidance is what enables BiFold's performance is not supported because no experiment removes or corrupts the language input. Section III-A defines the policy as πθ(at | ℓt, ot, ...) and all reported models retain the SigLIP text branch; Table IV only swaps the text encoder (T5) or changes the fusion/decoder architecture, never training a vision-only variant or one with scrambled text. Since the instruction templates are generated from the same semantic labels used to define the task, text is highly redundant with the visual observation and action distribution, and a vision-only policy with the same context could plausibly match many of the seen-instruction and unseen-instruction scores. The 'unseen task' rows in Table I are the only place where language seems necessary, and there BiFold is strong (e.g., Corner 100.0 with 1000 demonstrations), but without a no-language control the 'language guidance' attribution in the abstract remains unestablished.
  2. [IV-B, IV-C, Tables I-III] No error bars, confidence intervals, or training seeds are reported for any of the quantitative results. Tables I-III present single numbers for success rates, AP, KP-MSE, mIoU, and success; differences of a few percentage points (e.g., Table I, Half with 1000 demonstrations: BiFold 69.3% vs. Deng et al. 74.0%; Table III, Skirt SuccessIoU≥80: BiFold 31.7% vs. BiFold w/o context 34.9%) are within the range one would expect from stochastic training on small datasets, so the claimed state-of-the-art and consistent-outperformance conclusions are not statistically grounded.
  3. [IV-C, Fig. 4c] The real-world evaluation is offline and purely qualitative: the paper states 'we perform an offline qualitative evaluation on test images' and shows predicted actions in Fig. 4c, with no physical folding, no success metric, and no comparison to baselines. The abstract's claim of 'strong generalization to new instructions, garments, and environments' is therefore only partially supported; the 'environments' part is not demonstrated quantitatively. At minimum, the claim should be softened or the evaluation supplemented with a quantitative real-world study.
  4. [III-B, Algorithm 1, Appendix IV-B] The bimanual benchmark is self-created with an automatic annotation pipeline whose outputs are not validated. Algorithm 1 contains an explicit comment that 'Place vertices may be wrong,' and Appendix IV-B acknowledges that NOCS thresholding may fail for garment categories with high shape diversity. If the automatically parsed semantic labels and templates are systematically noisy, the bimanual results in Tables II and III and the associated generalization claims could be inflated. The paper should report annotation quality (e.g., human agreement on a sample, or a manual audit of parsed actions) and show that the reported bimanual results are not an artifact of label noise.
minor comments (4)
  1. [Appendix I-C] There is a typo in the sentence 'Wwe can observe that when the simulator becomes unstable...' — 'Wwe' should be 'We'.
  2. [Throughout] The reference to the prior work appears as 'Denget al.' without a space; it should be 'Deng et al.' in all occurrences.
  3. [IV-B, Table I] The baseline numbers for the unimanual benchmark are taken from Deng et al. without retraining; the paper should clarify whether the same data splits, augmentations, and evaluation protocol were used, since the comparison could be sensitive to such details.
  4. [III-A] The fixed context size H=3 is motivated by dataset statistics, but the paper does not discuss how the model behaves when a test sequence has more than three actions, which would require either truncating the context or using a longer horizon; a brief comment would clarify the expected failure mode.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the unimanual benchmark is external and the bimanual targets come from held-out human demonstrations, while the self-citations are not load-bearing.

full rationale

BiFold's derivation chain is self-contained and non-circular. The unimanual state-of-the-art claim (Table I) is evaluated on the external Deng et al. benchmark, with baseline numbers taken from that prior work and the model trained on that same fixed dataset; no parameter fitted by BiFold enters the definition of the target success metric. The bimanual claim is weaker because the dataset is self-created, but the ground-truth pick-and-place targets are human VR demonstration vertices re-rendered in simulation, not outputs of the model, and the test partition is held out. The language templates are generated from NOCS semantic labels, which may make the text branch partially redundant with the visual state, but this is a missing vision-only ablation rather than a circular derivation. The model is trained with a standard BCE loss against Gaussian-smoothed ground-truth positions, independent of the evaluation metrics. The paper explicitly flags its own limitations in Appendix IV-C (oracle ambiguity with human demonstrations, simulator physics, NOCS failure for diverse topologies) and in Section V, and these admissions do not conceal a circular step. Self-citations such as [46], [15], and [2] support only side claims about simulator inaccuracy, VR data collection, and bimanual benchmarking; they are not the load-bearing premise for the reported predictions. The strongest non-circular concern is the absence of a no-language or text-scrambled control, which affects the attribution of gains to language guidance, but that is an experimental-control issue, not a definitional or self-citation circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claims rest on standard supervised learning plus several domain assumptions: feature transfer from SigLIP, semantic consistency of NOCS labels, sufficiency of pick-and-place primitives, accuracy of SoftGym simulation, and availability of segmentation masks. These are not proven but are reasonable for the field; they lower confidence in the headline claims. No new physical entities, forces, dimensions, or latent constructs are introduced.

free parameters (6)
  • Context window size H = 3
    Set so that over 95% of bimanual sequences fit; controls how much history conditions each pick-and-place prediction.
  • Heatmap label variance Sigma = 5.0 I
    Gaussian spread around ground-truth pick and place pixels when constructing training targets; taken from prior work rather than learned.
  • LoRA configuration = rank 8, alpha 32, dropout 0.01
    Adapter hyperparameters chosen to fine-tune SigLIP; not derived from theory or tuned on the test set.
  • Action filtering thresholds = span > 5 frames, distance >= 0.1 m
    Actions shorter or closer than these thresholds are discarded as spurious, altering the training label set.
  • Simulation divergence filter = z-score ratio > 3.5 filtered
    Meshes with unstable simulation are removed using a normalized maximum-edge-length criterion; this changes the dataset distribution.
  • Success thresholds = 0.0125 m vertex distance; IoU >= 80%
    Task success is defined by these cutoffs; different thresholds would change reported success rates.
assumptions (6)
  • domain assumption SigLIP's pretrained vision-language representations transfer to cloth folding after LoRA adaptation
    The whole method relies on this transfer; no independent evidence in the paper demonstrates that SigLIP features are suitable for deformable-object manipulation beyond the reported results.
  • domain assumption Pick-and-place primitives are sufficient to express all folding strategies
    The policy deliberately excludes dynamic actions; if valid folds require flings or drags, the action space is incomplete.
  • domain assumption NOCS coordinates remain semantically consistent throughout a manipulation sequence
    Language labels are generated by thresholding NOCS values; the paper assumes category-level canonical correspondence, which may fail for diverse garment topologies.
  • domain assumption SoftGym cloth physics is accurate enough for evaluating folding success
    All quantitative success metrics come from SoftGym; the supplementary cites a benchmark showing SoftGym cloth dynamics are inaccurate, weakening this premise.
  • domain assumption A segmentation mask of the cloth is available at inference
    Pick positions are forced inside the mask; in real experiments SAM is used, so mask quality directly affects predictions.
  • domain assumption Human VR demonstrations plus predefined folding protocols are valid ground-truth folding behavior
    The dataset treats volunteer VR actions as correct demonstrations even though the paper notes some demonstrations are suboptimal and some volunteers ignore the protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BiFold: Bimanual Cloth Folding with Language Guidance." pith.science (2026). https://pith.science/paper/UBMQQEK6

@misc{pith2026250116458,
  author       = {Pith},
  title        = {Pith review of: BiFold: Bimanual Cloth Folding with Language Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBMQQEK6}},
  note         = {Machine review of arXiv:2501.16458}
}
read the original abstract

Cloth folding is a complex task due to the inevitable self-occlusions of clothes, their complicated dynamics, and the disparate materials, geometries, and textures that garments can have. In this work, we learn folding actions conditioned on text commands. Translating high-level, abstract instructions into precise robotic actions requires sophisticated language understanding and manipulation capabilities. To do that, we leverage a pre-trained vision-language model and repurpose it to predict manipulation actions. Our model, BiFold, can take context into account and achieves state-of-the-art performance on an existing language-conditioned folding benchmark. To address the lack of annotated bimanual folding data, we introduce a novel dataset with automatically parsed actions and language-aligned instructions, enabling better learning of text-conditioned manipulation. BiFold attains the best performance on our dataset and demonstrates strong generalization to new instructions, garments, and environments.

Figures

Figures reproduced from arXiv: 2501.16458 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. BiFold model architecture: We use a frozen SigLIP [16] model and adapt it using LoRA [17] to obtain tokens from an RGB image and an input text. The same encoders are used to incorporate past observations that provide context to the model. The domain of each token is indicated using a modality encoding and the sequence order using positional encodings. The concatenated sequence is processed using a transformer encode… view at source ↗
Figure 3
Figure 3. Dataset samples: Examples of language-aligned bimanual cloth folding instructions obtained using our pro￾posed annotation pipeline. Pick and place positions for right and left actions are represented as the origin and endpoints of an arrow. Each action uses eight vertices, which might be distinct and fall into different pixels. the Normalized Object Canonical Space (NOCS) mesh [33]. Parsing actions: We remove spurio… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Qualitative examples: Action predictions obtained with our model. Pick and place actions for right and left are represented as the origin and endpoints of arrows: red and green for ground truth, and blue and light blue for our model. V. CONCLUSIONS In this paper, we pr…
Figure 5
Figure 5. Figure 5: VR-Folding sequence: Sample sequence from the VR-Folding dataset in which hands are included using the sensed positions from the VR gloves. In this image we can see the tiling green pattern and constant orange interior. This design choice can lead to inputs that can be…
Figure 6
Figure 6. Figure 6: Re-rendering step: The only visual input in the original dataset is given as a colored point cloud with uniform colors and patterns (left). We take the simulation mesh, and randomly choose camera position (center). Finally, we apply a texture to the mesh and render RGB…
Figure 7
Figure 7. Figure 7: Continuous NOCS tracking across pick–and–place subactions. Each panel shows a single pick-and-place step rendered in two columns: the left image corresponds to the pre-action state, and the right to the post-action state. We overlay the user’s VR-tracked hand positions…
Figure 8
Figure 8. Figure 8: Semantic pick and place positions: We obtain the semantic location of the grip by mapping the picked vertices on the NOCS and thresholding its coordinates. In this figure, we show an example of each category colored by thresholding the left-right and top-bottom directi…
Figure 9
Figure 9. Figure 9: Divergent sequence: In this sequence, we can see that the manipulated top has become unstable in the cloth simulator, yielding unrealistic shapes. The top uses the CLOTH3D [32] mesh with identifier 00156 and corresponds to the sequence 00156 Top 000013 t in the VR-Fold…
Figure 10
Figure 10. Figure 10: Sequence length per category: Here we show a histogram of the number of actions of each sequence grouped by clothing category in linear scale and stacked (left) and logarithmic scale and separated (right) for the BiFold dataset. Note that some sequences can have only …
Figure 11
Figure 11. Figure 11: Additional BiFold dataset statistics When referring to folding sleeves, Fig. 11c shows a similar distribution of equal preference for the left and right arm with the latter being slightly more used for the first fold. Finally, in Fig. 11b we can see that there is a no…
Figure 12
Figure 12. Figure 12: Patch artifact: When using transformer decoders, we observe patch artifacts in the heatmap predictions. 3) The picker moves to the pick position and closes the gripper. 4) The picker moves to the position in 2). 5) The picker is moved to the placing position but at a …
Figure 13
Figure 13. Figure 13: Real dataset: The real dataset is composed of eight garments with different materials, topologies, and textures. In this figure we show the initial configuration, where the clothes are flattened on a table [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Qualitative unimanual SoftGym: For the unimanual version, training data comes from SoftGym and is generated through scripted movements, leading to high success rates. In each example, we indicate the type of instruction: Seen Instruction (SI), UnSeen Instruction (USI)…
Figure 15
Figure 15. Figure 15: Additional qualitative examples on our dataset: BiFold can learn different folding patterns conditioned on language and a starting folding position. The obtained actions successfully replicate the human demonstrations with different sizes, cloth textures, and camera p…
Figure 16
Figure 16. Figure 16: Qualitative bimanual SoftGym: Despite the apparent distribution shift between the training dataset and the SoftGym environment, BiFold is able to predict coherent actions [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Additional qualitative examples on real data: We present the results obtained with the different configurations of the real dataset and trying a diverse set of prompts taken from the templates of our dataset [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: End-to-end unimanual folding: Example of a whole folding rollout using the BiFold with context trained with 1000 demonstrations on the unimanual folding dataset [5]. B. Out of distribution prompts BiFold predicts a single pick-and-place action at a time. Hence, we do …
Figure 19
Figure 19. Figure 19: "Fold a T-shirt into a square". When using the instruction "Fold the trousers in L shape" we obtain outputs like the ones in [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: "Fold the trousers in L shape". not replicate a way to achieve an L-fold that would grasp the end of one of the legs with one or two arms and move it diagonally up and to the other side, which we can expect due to the absence of similar folds in the dataset [PITH_FUL…
Figure 21
Figure 21. Figure 21: Failures: Collection of the worse predictions obtained for each dataset and cloth category [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Sub sampled real-world point cloud: In this figure we show the point cloud with 200 points obtained by following the approach by Deng et al. [5]. On the right, we show the sampled points in green on top of the real-world point cloud. For the sake of visualization, we …
Figure 23
Figure 23. Figure 23: Number of pixels in the segmentation mask: We show a histogram of the number of pixels in the cloth region on the BiFold dataset, with the count indicated in the left axis. We also provide the cumulative distribution of these values, where the cumulative probability i…
Figure 24
Figure 24. Figure 24: Distance in pixels to the segmentation mask: This plot shows the histogram of distances from place positions to the cloth segmentation mask across the train and test partition of the BiFold dataset, with the count in the left axis. We also provide the cumulative distr…
Figure 25
Figure 25. Figure 25: Extreme examples of place position out of segmentation mask: Pick and place positions for right and left actions are represented as the origin and endpoints of an arrow. Each action uses eight vertices, which might be distinct and fall into different pixels. (a) Side …
Figure 26
Figure 26. Figure 26: Input difference between real garments and those in the unimanual dataset: We present a comparison between two T-shirts from the real-world dataset (in red) and the unimanual dataset (in pink and yellow). As we can see, the captured real-world point clouds are denser …
Figure 27
Figure 27. Figure 27: Grasping failure: Example of an instance in which the root of the failure is on the motion primitive. In this case, the language instruction is "Fold the trousers, orientating from the bottom right towards the top left.". BiFold produces satisfactory pick and place lo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language-Guided Long Horizon Manipulation with LLM-based Planning and Visual Perception

    cs.RO 2025-09 conditional novelty 5.0 of 10

    A robot folds cloth from spoken language by decomposing instructions with GPT-4o and grounding each step with a SigLIP2-based pick-and-place perception module.

  2. Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding

    cs.RO 2025-05 conditional novelty 4.0 of 10

    Keyframe-based temporal context and LoRA fine-tuning improve language-guided pick-and-place predictions in the BiFold cloth-folding model.

Reference graph

Works this paper leans on

62 extracted references · 48 canonical work pages · cited by 2 Pith papers

  1. [1]

    Household cloth object set: Fostering benchmarking in deformable object manipulation,

    I. Garcia-Camacho, J. Borr `as, B. Calli, A. Norton, and G. Aleny `a, “Household cloth object set: Fostering benchmarking in deformable object manipulation,”IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 5866–5873, 2022

    work page 2022

  2. [2]

    Benchmarking bimanual cloth manipulation,

    I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Aleny `a, and D. Kragic, “Benchmarking bimanual cloth manipulation,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1111–1118, 2020

    work page 2020

  3. [3]

    Folds- former: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention,

    K. Mo, C. Xia, X. Wang, Y . Deng, X. Gao, and B. Liang, “Folds- former: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention,”IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 760–767, 2023

    work page 2023

  4. [4]

    CLIPort: What and Where Pathways for Robotic Manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “CLIPort: What and Where Pathways for Robotic Manipulation,” inCoRL, 2021

    work page 2021

  5. [5]

    Learning Language- Conditioned Deformable Object Manipulation with Graph Dynamics,

    Y . Deng, K. Mo, C. Xia, and X. Wang, “Learning Language- Conditioned Deformable Object Manipulation with Graph Dynamics,” inICRA, 2024

    work page 2024

  6. [6]

    SpeedFolding: Learning Efficient Bimanual Folding of Garments,

    Y . Avigal, L. Berscheid, T. Asfour, T. Kr ¨oger, and K. Goldberg, “SpeedFolding: Learning Efficient Bimanual Folding of Garments,” inIROS, 2022, pp. 1–8

    work page 2022

  7. [7]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inMedical Image Com- puting and Computer-Assisted Intervention, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds., 2015, pp. 234–241

    work page 2015

  8. [8]

    Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation,

    A. Canberk, C. Chi, H. Ha, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, “Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation,” inICRA, 2022

    work page 2022

Show all 62 references
  1. [9]

    Unifolding: Towards sample-efficient, scalable, and generalizable robotic garment folding,

    H. Xue, Y . Li, W. Xu, H. Li, D. Zheng, and C. Lu, “Unifolding: Towards sample-efficient, scalable, and generalizable robotic garment folding,” inCoRL, 2023

    2023

  2. [10]

    Learning Transferable Visual Models From Natural Language Super- vision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Super- vision,” inICML, 2021

    2021

  3. [11]

    Transporter networks: Rearranging the visual world for robotic ma- nipulation,

    A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V . Sindhwani, and J. Lee, “Transporter networks: Rearranging the visual world for robotic ma- nipulation,” inCoRL, 2020

    2020

  4. [12]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” inNeurIPS, vol. 30, 2017

    2017

  5. [13]

    Learning Visible Connec- tivity Dynamics for Cloth Smoothing,

    X. Lin, Y . Wang, Z. Huang, and D. Held, “Learning Visible Connec- tivity Dynamics for Cloth Smoothing,” inCoRL, 2021

    2021

  6. [14]

    GarmentTracking: Category-Level Garment Pose Tracking,

    H. Xue, W. Xu, J. Zhang, T. Tang, Y . Li, W. Du, R. Ye, and C. Lu, “GarmentTracking: Category-Level Garment Pose Tracking,” inCVPR, June 2023, pp. 21 233–21 242

    2023

  7. [15]

    A virtual reality framework for fast dataset creation applied to cloth manipulation with automatic semantic labelling,

    J. Borr `as, A. Boix-Granell, S. Foix, and C. Torras, “A virtual reality framework for fast dataset creation applied to cloth manipulation with automatic semantic labelling,” inICRA, 2023, pp. 11 605–11 611

    2023

  8. [16]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inICCV, 2023, pp. 11 975–11 986

    2023

  9. [17]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  10. [18]

    Reproducible scaling laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” inCVPR, 2023, pp. 2818–2829

    2023

  11. [19]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,” arXiv:2303.15389, 2023

    2023 arXiv

  12. [20]

    DINOv2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Je- gou, J. Mairal, P. Labatu...

    2024

  13. [21]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” inCVPR, 2022, pp. 15 979– 15 988

    2022

  14. [22]

    Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs,

    S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y . LeCun, and S. Xie, “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs,” arXiv:2406.16860, 2024

    2024 arXiv

  15. [23]

    Probing the 3d awareness of visual foundation models,

    M. El Banani, A. Raj, K.-K. Maninis, A. Kar, Y . Li, M. Rubinstein, D. Sun, L. Guibas, J. Johnson, and V . Jampani, “Probing the 3d awareness of visual foundation models,” inCVPR, 2024, pp. 21 795– 21 806

    2024

  16. [24]

    PaliGemma: A versatile 3B VLM for transfer,

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Un- terthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bo ...

    2024 arXiv

  17. [25]

    Pali-3 vision language models: Smaller, faster, stronger,

    X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. V oigtlaender, B. Mustafa, S. Goodman, I. Alabdulmohsin, P. Padlewski, D. Salz, X. Xiong, D. Vlasic, F. Pavetic, K. Rong, T. Yu, D. Keysers, X. Zhai, and R. Soricut, “Pali-3 vision language models: Smaller, faster, stronger,...

    arXiv

  18. [26]

    Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world,

    K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y . Kim, W. Han, A. Herrasti, R. Krishna, D. Schwenk, E. VanderBilt, and A. Kembhavi, “Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world,...

    2024

  19. [27]

    4M: Massively multimodal masked modeling,

    D. Mizrahi, R. Bachmann, O. F. Kar, T. Yeo, M. Gao, A. Dehghan, and A. Zamir, “4M: Massively multimodal masked modeling,” inAdvances in Neural Information Processing Systems, 2023

    2023

  20. [28]

    4M-21: An any-to-any vision model for tens of tasks and modalities,

    R. Bachmann, O. F. Kar, D. Mizrahi, A. Garjani, M. Gao, D. Griffiths, J. Hu, A. Dehghan, and A. Zamir, “4M-21: An any-to-any vision model for tens of tasks and modalities,” arXiv:2406.09406, 2024

    2024 arXiv

  21. [29]

    Octo: An open-source generalist robot policy,

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine, “Octo: An open-source generalist robot policy,” inProceedings of Robotics...

    2024

  22. [30]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choro- manski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Her- zog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalash- nikov, Y . Kuang...

    2023 arXiv

  23. [31]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. R. Selvaraju, A. D. Gotmare, S. Joty, C. Xiong, and S. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” inNeurIPS, 2021

    2021

  24. [32]

    CLOTH3D: Clothed 3D Humans,

    H. Bertiche, M. Madadi, and S. Escalera, “CLOTH3D: Clothed 3D Humans,” inECCV, 2020, pp. 344–359

    2020

  25. [33]

    Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation,

    H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation,” inCVPR, June 2019

    2019

  26. [34]

    Posescript: 3d human poses from natural language,

    G. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer, and G. Ro- gez, “Posescript: 3d human poses from natural language,” inECCV, 2022, p. 346–362

    2022

  27. [35]

    Adam: A Method for Stochastic Optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimiza- tion,” inICLR, 2015, pp. 1–15

    2015

  28. [36]

    SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation,

    X. Lin, Y . Wang, J. Olkin, and D. Held, “SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation,” inCoRL, 2021

    2021

  29. [37]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollar, and R. Gir- shick, “Segment anything,” inICCV, 2023, pp. 4015–4026

    2023

  30. [38]

    Gpt-fabric: Folding and smoothing fabric by leveraging pre-trained foundation models,

    V . Raval, E. Zhao, H. Zhang, S. Nikolaidis, and D. Seita, “Gpt-fabric: Folding and smoothing fabric by leveraging pre-trained foundation models,”arXiv preprint arXiv:2406.09640, 2024

    2024 arXiv

  31. [39]

    Fab- ricflownet: Bimanual cloth manipulation with a flow-based policy,

    T. Weng, S. Bajracharya, Y . Wang, K. Agrawal, and D. Held, “Fab- ricflownet: Bimanual cloth manipulation with a flow-based policy,” in CoRL, 2021

    2021

  32. [40]

    Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor,

    D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, K. Yamane, S. Iba, J. Canny, and K. Goldberg, “Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor,” in IEEE/RSJ Intern...

    2020

  33. [41]

    Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding,

    H. Ha and S. Song, “Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding,” inConference on Robotic Learning (CoRL), 2021

    2021

  34. [42]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. C. Courville, “Film: Visual reasoning with a general conditioning layer,” inAAAI, 2018

    2018

  35. [43]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  36. [44]

    RT-1: Robotics Transformer for Real-World Control at Scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalash- nikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Ma...

    2022 arXiv

  37. [45]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,” inCoRL, 2023. [Online]. Available: https://openreview.net/forum?id=9 8LF30mOC

    2023

  38. [46]

    Benchmarking the Sim-to-Real Gap in Cloth Manipulation,

    D. Blanco-Mulero, O. Barbany, G. Alcan, A. Colom ´e, C. Torras, and V . Kyrki, “Benchmarking the Sim-to-Real Gap in Cloth Manipulation,” IEEE Robotics and Automation Letters, 2024

    2024

  39. [47]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023

    2023

  40. [48]

    Hierarchical diffusion policy for kinematics-aware multi-task robotic manipulation,

    X. Ma, S. Patidar, I. Haughton, and S. James, “Hierarchical diffusion policy for kinematics-aware multi-task robotic manipulation,”CVPR, 2024

    2024

  41. [49]

    Distilled Feature Fields Enable Few-Shot Language-Guided Manip- ulation,

    W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola, “Distilled Feature Fields Enable Few-Shot Language-Guided Manip- ulation,” inCoRL, 2023

    2023

  42. [50]

    ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environment,

    B. Zhou, H. Zhou, T. Liang, Q. Yu, S. Zhao, Y . Zeng, J. Lv, S. Luo, Q. Wang, X. Yu, H. Chen, C. Lu, and L. Shao, “ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environment,” inICCV, 2023

    2023

  43. [51]

    BlenderProc2: A Procedural Pipeline for Photorealistic Rendering,

    M. Denninger, D. Winkelbauer, M. Sundermeyer, W. Boerdijk, M. Knauer, K. H. Strobl, M. Humt, and R. Triebel, “BlenderProc2: A Procedural Pipeline for Photorealistic Rendering,”Journal of Open Source Software, vol. 8, no. 82, p. 4901, 2023

    2023

  44. [52]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” inICLR, 2021. [Online]. Available: http...

    2021

  45. [53]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV, 2020, p. 213–229

    2020

  46. [54]

    PyBullet, a Python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “PyBullet, a Python module for physics simulation for games, robotics and machine learning,” http://pybullet. org, 2016–2021

    2016

  47. [55]

    MuJoCo: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “MuJoCo: A physics engine for model-based control,” inIEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. BiFold: Bimanual Cloth Folding with Language Guidance Supplementary Material CONTENTS I INTRODUCTION 1 II RELATE...

    2012

  48. [56]

    Set pick and place heights using the radius of the picker, regardless of the world coordinate of the vertex

  49. [57]

    2https://huggingface.co/docs/transformers/en/model_doc/siglip Fig

    The picker is moved to the picking position but at a predefined height. 2https://huggingface.co/docs/transformers/en/model_doc/siglip Fig. 12:Patch artifact:When using transformer decoders, we observe patch artifacts in the heatmap predictions

  50. [58]

    The picker moves to the pick position and closes the gripper

  51. [59]

    The picker moves to the position in 2)

  52. [60]

    The picker is moved to the placing position but at a predefined height

  53. [61]

    The picker goes to the placing position and opens the gripper

  54. [62]

    Fold a T-shirt into a square

    The picker moves to the place position at the same predefined height as in 2). All the movements are performed at a speed of 5 mm/action except steps (2) and (7), which we perform 100 times faster as they are supposed to not interact with the cloth. The bimanual primitive uses...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.