Pith. sign in

REVIEW 2 major objections 6 minor 36 references

Assessment of protein assembly prediction in CASP13

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The CASP13 assembly assessment reports clear uptake in participation, more oligomeric targets, and a consistent, albeit modest, 5–15 percent improvement over CASP12 across all four scoring measures.

desk verdict A transparent, valuable CASP13 assembly assessment whose headline 5–15% improvement claim rests on a comparability assumption the paper itself does not substantiate; the homomeric-contact finding is the strongest new result. read the letter →

arxiv 1908.07662 v1 pith:L3CON3PL submitted 2019-08-21 q-bio.BM

Dmytro Guzenko , Aleix Lafita , Bohdan Monastyrskyy , Andriy Kryshtafovych , Jose M. Duarte This is my paper · ORCID
classification q-bio.BM
keywords CASP13proteinassemblypredictionquaternarystructureoligomericstateinterfacehomologymodellingcontactstructuralbiologyassessment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper assesses the second dedicated protein assembly category in CASP13, comparing predictions from 45 groups on 42 oligomeric targets against the CASP12 assembly round. It claims consistent, albeit modest, progress: 5–15 percent improvement across all four evaluation scores, with 9 of 42 targets solved on all metrics. It also finds that ignoring the oligomeric state harms even tertiary-structure predictions, that homomeric interfacial contacts are already predictable but were not used for assembly modelling, and that human-assisted homology modelling dominates the field. For a general reader, the takeaway is that quaternary structure is intrinsic to protein modelling and must be built into prediction pipelines from the start.

What carries the argument

The machinery is the scoring and ranking protocol. Four scores—ICS (F1) and IPS (Jaccard) for interfaces, plus lDDT_O and GDT_O for the whole assembly—require mapping chains between model and target, done with the 13-score algorithm (13-align for the largest target). Per-target z-scores with outlier removal and leave-one-out ranking guard against inflated scores on hard targets. For the CASP12-vs-CASP13 comparison, scores are matched by percentiles under the assumption of similar target difficulty, with a naive assembly method as the per-target baseline.

What would settle it

Recompute the percentile-matched comparison with targets stratified by the paper's own easy/medium/difficult classes, or on a subset of targets matched for template availability and interface complexity. If the 5–15 percent improvements shrink to zero or reverse within matched difficulty strata, the claimed progress is explained by target selection rather than method improvement.

Watch

Extended reading notes

Core claim

The central discovery is that protein assembly prediction improved measurably but not dramatically between CASP12 and CASP13. Using four scores—interface contact similarity (F1), interface patch similarity (Jaccard), and oligomeric versions of lDDT and GDT—the paper finds a consistent 5–15 percent improvement across score percentiles, and identifies 9 of 42 assembly targets as solved by all four scores above 0.5. The gains track human-assisted homology modelling; fully automated servers rank near the naive baseline. A second finding is that homomeric interfacial contacts are already predicted well by some contact prediction groups, yet these predictions are currently counted as false positives and were not fed into assembly modelling. The paper also shows that treating chains as independent folding units degrades even tertiary predictions for targets with intertwined interfaces.

Load-bearing premise

The progress claim stands on the assumption that CASP12 and CASP13 assembly targets have roughly the same difficulty distribution; if the 2019 targets are easier, the 5–15 percent gains are an artifact of target selection.

Editorial extensions

If this is right

  • Predictors that ignore the oligomeric state will fail on intertwined interfaces: targets whose evaluation unit was the monomer received poor tertiary predictions despite good subunit templates.
  • Contact prediction groups already produce accurate homomeric interface contacts, so the next step is to fold multiple chains simultaneously from contact matrices rather than discarding interfacial contacts as false positives.
  • The 5–15 percent improvement is real but modest and largely attributable to human-assisted homology modelling; automated servers rank close to the naive baseline.
  • Data-assisted predictions using SAXS, crosslinking, or NMR showed no systematic improvement over regular predictions in CASP13, apart from one target with favorable crosslinks.
  • Only two servers participate in the fully automated multimeric CAMEO experiment, indicating that automation in assembly modelling lags behind tertiary modelling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If homomeric interface contacts were added to contact-prediction evaluation, some group rankings would change (the paper notes H0968S2 as an example); a future CASP round that scores them explicitly would likely reward groups that already predict them.
  • The dominance of human-assisted homology modelling suggests the modest gains may saturate as structural templates become the limiting factor; a direct test is whether deep-learning pipelines trained on oligomeric targets can beat the homology baseline on difficult targets with no assembly templates.
  • The data-assisted comparison is currently confounded because the best non-assisted groups did not join the assisted category; a cleaner test would compare the same groups' assisted and non-assisted predictions on identical targets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This manuscript presents the CASP13 assembly category assessment. The authors evaluate predictions for 42 oligomeric targets using four scores (ICS, IPS, oligomeric lDDT and GDT), define a 'solved' criterion requiring all four scores to exceed 0.5, compare CASP13 with CASP12 by percentile matching of scores, and rank groups using leave-one-out Z-scores. They report increased participation, a modest 5–15% improvement over CASP12, an unchanged solved-target proportion (9/42 vs 6/30), a dominant role for human-assisted homology modeling, successful prediction of homomeric interface contacts that was nevertheless not used in assembly modeling, and no systematic benefit from data-assisted targets except target 80957.

Significance. If the cross-edition comparison is valid, this assessment provides a useful benchmark for protein assembly prediction and a clear statement of the field's state. The paper's strengths include a transparent evaluation protocol, a baseline comparison, the use of four complementary scores, and the novel observation that homomeric interface contacts are predictable but currently unused. However, the headline quantitative claim of 5–15% improvement rests on an untested comparability assumption and lacks uncertainty estimates; the paper's own solved-target comparison is essentially flat and does not independently support the improvement claim. The manuscript would be substantially strengthened by reporting the difficulty-class distributions for CASP12 and CASP13 and by adding confidence intervals or bootstrap estimates for the percentile-matched differences.

major comments (2)
  1. [§3.1, Figure 3] The '5–15% improvement for all scores across the board' claim is load-bearing and currently rests on the unsupported assumption that the difficulty of CASP12 and CASP13 assembly targets has roughly the same distribution. The only support offered is the unresolved placeholder citation '(EVIDENCE IN; REFTHIS YEAR'S DOMAIN PREDICTION ASSESSMENT=)', which is not a completed bibliographic entry and, even if completed, refers to domain prediction rather than assembly-specific difficulty. The manuscript itself defines three difficulty classes in §2.2 but never reports their distributions in the two editions, so the assumption cannot be checked from the presented data. Moreover, no confidence intervals or bootstrap estimates accompany the percentile-matched values, and the paper's own solved-target comparison (9/42 vs 6/30) is essentially flat and therefore does not independently corroborate improvement. Please either provide difficulty-class distributions and uncertainty estimates, or weaken the progress claim to something like 'observed score gains are consistent with modest improvement, assuming comparable target difficulty.'
  2. [§3.1 vs §3.2] The statement in §3.1 that 'absence of detectable assembly templates with near-complete coverage guarantees absence of good models' is too strong and is internally contradicted by the paper's own example of target 40976 in §3.2, where successful dimeric models were produced from a monomeric template whose interdomain interfaces resembled the dimeric interface. As written, the 'guarantee' is false; if the intended meaning is that this is the dominant pattern for most targets, the sentence should be revised to state that pattern and to specify how templates were defined and detected.
minor comments (6)
  1. [§3.1] The placeholder citation '(EVIDENCE IN; REFTHIS YEAR'S DOMAIN PREDICTION ASSESSMENT=)' must be resolved to a proper bibliographic entry before publication; a citation to the domain prediction assessment, even if completed, would not by itself establish equivalence of assembly-target difficulty.
  2. [§2.3, Figure 4] The statement that the maximum and minimum leave-one-out total scores 'can be used to assess the significance of the differences between closely ranked groups' is not a valid significance test; leave-one-out variation measures sensitivity to individual targets, not sampling uncertainty in group ability. The ranking itself is fine as a descriptive result, but the significance language should be removed or accompanied by an appropriate test.
  3. [§3.5] The conclusion that data-assisted scores are 'not significantly different' from regular predictions is made without any statistical test or confidence interval; given the small number of targets (7), the wording should be descriptive rather than inferential, or the relevant test should be reported.
  4. [§2.3] The notation '13-score algorithm' is unclear; the text appears to intend a standard structural alignment score such as TM-score, and the acronym should be written out consistently with the cited reference.
  5. [§3.1] The phrase '6(EASY) TARGETS OUT OF 30' is ambiguous: it is not clear whether all six solved CASP12 targets were in the EASY difficulty class or whether '(EASY)' is a typographical artifact. Please clarify.
  6. [Figures 2 and 3] The captions describe rich information, but the figures as provided in this manuscript version do not show the target identifiers or score distributions in a way that a reader can use to verify the claims about individual targets; please ensure the published figures include clearly legible labels.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: CASP13 assembly assessment is a data report with one unresolved CASP12/13 comparability caveat.

full rationale

This is a community-wide assessment paper, not a derivation: the central quantitative claims are summary statistics of submitted models computed with pre-defined, externally published scores (ICS/IPS from the CASP12 assessment, LDDT, GDT). No parameter is fitted to a subset of the data and then renamed as a prediction; the solved-target counts (9/42 vs 6/30) and percentile-matched score shifts are observed from the submissions themselves. The only questionable load-bearing step is the Section 3.1 assumption that CASP12 and CASP13 assembly-target difficulties have roughly the same distribution, supported by the unresolved placeholder '(EVIDENCE IN; REFTHIS YEAR'S DOMAIN PREDICTION ASSESSMENT=)'. That is a missing-evidence/citation-gap concern about comparability, not circularity: the assumption is stated openly, is not derived from the CASP13 assembly scores, and does not make the improvement numbers true by construction. Reuse of the authors' own CASP12 metrics and difficulty classes (ref. 21) is methodological continuity, and the CASP12 scores were recalculated on the same scale; no equation in the paper reduces a predicted quantity to an input. I therefore find no significant circularity, only a minor self-referenced citation gap that should be resolved.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are the solved-target threshold and the Z-score outlier cutoff, both chosen by convention or hand. The axioms are operational assumptions about target difficulty comparability, ground-truth assignment, chain mapping, baseline choice, and score validity, all of which the central claims depend on.

free parameters (3)
  • solved_threshold = 0.5 on each of ICS, IPS, lDDT_O, GDT_O
    Defines the 9 solved targets in Section 3.1; changing the threshold changes the success count and related claims.
  • z_outlier_cutoff = -2
    CASP convention to remove outliers before ranking in Section 2.3; affects group totals and the ranking figure.
  • target_difficulty_classes = Easy, Medium, Difficult assignments per target (Table S1)
    Used throughout the ranking analysis; assignments are manual and depend on predictor-available information, not on outcome.
assumptions (6)
  • domain assumption Ground-truth oligomeric state assignment for each target is correct.
    Section 2.1 relies on experimentalist indication, EPPIC, PISA, and HHpred homology consensus; five cases remained ambiguous and were assigned with low confidence, so evaluation against a possibly wrong ground truth affects all scores.
  • domain assumption CASP12 and CASP13 assembly target difficulty distributions are roughly the same.
    Section 3.1 invokes a cited companion assessment; the 5 to 15 percent improvement estimate is only as good as this comparability assumption.
  • domain assumption Chain mapping between target and prediction preserves biological equivalence.
    Section 2.3 uses TM-score and TM-align to map chains; ambiguous chain order or symmetry could misassign chains and distort GDT_O and lDDT_O scores.
  • domain assumption The first submitted model is the best model from each group.
    Section 2.3 uses only the first submitted model for each target, a CASP convention; if groups submitted varied models, this may understate their best performance.
  • domain assumption Seok-naive_assembly is an adequate baseline.
    Section 3.1 uses it as an indication of baseline per target; if the naive method is too strong or too weak, statements about groups beating baseline change.
  • domain assumption The four scores used (ICS, IPS, lDDT_O, GDT_O) are valid proxies for assembly prediction quality.
    The whole assessment computes these scores and treats them as the ground truth for quality; any metric bias affects all conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessment of protein assembly prediction in CASP13." pith.science (2026). https://pith.science/paper/L3CON3PL

@misc{pith2026190807662,
  author       = {Pith},
  title        = {Pith review of: Assessment of protein assembly prediction in CASP13},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3CON3PL}},
  note         = {Machine review of arXiv:1908.07662}
}
read the original abstract

We present the assembly category assessment in the 13th edition of the CASP community-wide experiment. For the second time, protein assemblies constitute an independent assessment category. Compared to the last edition we see a clear uptake in participation, more oligomeric targets released, and consistent, albeit modest, improvement of the predictions quality. Looking at the tertiary structure predictions we observe that ignoring the oligomeric state of the targets hinders modelling success. We also note that some contact prediction groups successfully predicted homomeric interfacial contacts, though it appears that these predictions were not used for assembly modelling. Homology modelling with sizeable human intervention appears to form the basis of the assembly prediction techniques in this round of CASP. Future developments should see more integrated approaches to modelling where multiple subunits are a natural part of the modelling process, which would benefit the structure prediction field as a whole.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages

  1. [1]

    The application of the oil turbine type of ultracentrifuge to the study of the stability region of carbon monoxide-hemoglobin

    Svedberg T, Nichols J. The application of the oil turbine type of ultracentrifuge to the study of the stability region of carbon monoxide-hemoglobin. Journal of the American Chemical Society 1927;49(11):2920--2934

    work page 1927

  2. [2]

    Structural symmetry and protein function

    Goodsell DS, Olson AJ. Structural symmetry and protein function. Annual review of biophysics and biomolecular structure 2000;29(1):105--153

    work page 2000

  3. [3]

    Dynamic dissociating homo-oligomers and the control of protein function

    Selwood T, Jaffe EK. Dynamic dissociating homo-oligomers and the control of protein function. Archives of Biochemistry and Biophysics 2012;519(2):131--143

    work page 2012

  4. [4]

    Mechanisms of protein oligomerization, the critical role of insertions and deletions in maintaining different oligomeric states

    Hashimoto K, Panchenko AR. Mechanisms of protein oligomerization, the critical role of insertions and deletions in maintaining different oligomeric states. Proceedings of the National Academy of Sciences 2010;107(47):20352--20357

    work page 2010

  5. [5]

    RCSB Protein Data Bank: biological macromolecular structures enabling research and education in fundamental biology, biomedicine, biotechnology and energy

    Burley SK, Berman HM, Bhikadiya C, Bi C, Chen L, Di Costanzo L, et al. RCSB Protein Data Bank: biological macromolecular structures enabling research and education in fundamental biology, biomedicine, biotechnology and energy. Nucleic acids research 2018;47(D1):D464--D474

    work page 2018

  6. [6]

    Inside a living cell

    Goodsell DS. Inside a living cell. Trends in biochemical sciences 1991;16:203--206

    work page 1991

  7. [7]

    Protein folding via binding and vice versa

    Tsai CJ, Xu D, Nussinov R. Protein folding via binding and vice versa. Folding and Design 1998;3(4):R71--R80

    work page 1998

  8. [8]

    Diversity of protein--protein interactions

    Nooren IM, Thornton JM. Diversity of protein--protein interactions. The EMBO journal 2003;22(14):3486--3492

    work page 2003

Show all 36 references
  1. [9]

    A survey of protein--protein complex crystallizations

    Radaev S, Li S, Sun PD. A survey of protein--protein complex crystallizations. Acta Crystallographica Section D: Biological Crystallography 2006;62(6):605--612

    2006

  2. [10]

    The resolution revolution

    K \"u hlbrandt W. The resolution revolution. Science 2014;343(6178):1443--1444

    2014

  3. [11]

    Frontiers in Cryo Electron Microscopy of Complex Macromolecular Assemblies

    Ognjenovi \'c J, Grisshammer R, Subramaniam S. Frontiers in Cryo Electron Microscopy of Complex Macromolecular Assemblies. Annual review of biomedical engineering 2019;21

    2019

  4. [12]

    CASP: A driving force in protein structure modeling

    Kryshtafovych A, Fidelis K, Moult J. CASP: A driving force in protein structure modeling. Introduction to Protein Structure Prediction: Methods and Algorithms 2010;p. 15--32

    2010

  5. [13]

    Assessment of hard target modeling in CASP12 reveals an emerging role of alignment-based contact prediction methods

    Abriata LA, Tam \`o GE, Monastyrskyy B, Kryshtafovych A, Dal Peraro M. Assessment of hard target modeling in CASP12 reveals an emerging role of alignment-based contact prediction methods. Proteins: Structure, Function, and Bioinformatics 2018;86:97--112

    2018

  6. [14]

    Assessment of contact predictions in CASP12: Co-evolution and deep learning coming of age

    Schaarschmidt J, Monastyrskyy B, Kryshtafovych A, Bonvin AM. Assessment of contact predictions in CASP12: Co-evolution and deep learning coming of age. Proteins: Structure, Function, and Bioinformatics 2018;86:51--66

    2018

  7. [15]

    Critical assessment of methods of protein structure prediction (CASP)—Round XII

    Moult J, Fidelis K, Kryshtafovych A, Schwede T, Tramontano A. Critical assessment of methods of protein structure prediction (CASP)—Round XII. Proteins: Structure, Function, and Bioinformatics 2018;86:7--15

    2018

  8. [16]

    Understanding the fabric of protein crystals: computational classification of biological interfaces and crystal contacts

    Capitani G, Duarte JM, Baskaran K, Bliven S, Somody JC. Understanding the fabric of protein crystals: computational classification of biological interfaces and crystal contacts. Bioinformatics 2015;32(4):481--489

    2015

  9. [17]

    Automated evaluation of quaternary structures from protein crystals

    Bliven S, Lafita A, Parker A, Capitani G, Duarte JM. Automated evaluation of quaternary structures from protein crystals. PLoS computational biology 2018;14(4):e1006104

    2018

  10. [18]

    Stock-based detection of protein oligomeric states in jsPISA

    Krissinel E. Stock-based detection of protein oligomeric states in jsPISA. Nucleic acids research 2015;43(W1):W314--W319

    2015

  11. [19]

    A completely reimplemented MPI bioinformatics toolkit with a new HHpred server at its core

    Zimmermann L, Stephens A, Nam SZ, Rau D, K \"u bler J, Lozajic M, et al. A completely reimplemented MPI bioinformatics toolkit with a new HHpred server at its core. Journal of molecular biology 2018;430(15):2237--2243

    2018

  12. [20]

    Principles and characteristics of biological assemblies in experimentally determined protein structures

    Xu Q, Dunbrack Jr RL. Principles and characteristics of biological assemblies in experimentally determined protein structures. Current opinion in structural biology 2019;55:34--49

    2019

  13. [21]

    Assessment of protein assembly prediction in CASP12

    Lafita A, Bliven S, Kryshtafovych A, Bertoni M, Monastyrskyy B, Duarte JM, et al. Assessment of protein assembly prediction in CASP12. Proteins: Structure, Function, and Bioinformatics 2018;86:247--256

    2018

  14. [22]

    lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests

    Mariani V, Biasini M, Barbato A, Schwede T. lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests. Bioinformatics 2013;29(21):2722--2728

    2013

  15. [23]

    Processing and analysis of CASP3 protein structure predictions

    Zemla A, Venclovas C , Moult J, Fidelis K. Processing and analysis of CASP3 protein structure predictions. Proteins: Structure, Function, and Bioinformatics 1999;37(S3):22--29

    1999

  16. [24]

    Modeling protein quaternary structure of homo-and hetero-oligomers beyond binary interactions by homology

    Bertoni M, Kiefer F, Biasini M, Bordoli L, Schwede T. Modeling protein quaternary structure of homo-and hetero-oligomers beyond binary interactions by homology. Scientific reports 2017;7(1):10480

    2017

  17. [25]

    BioJava 5: A community driven open-source bioinformatics library

    Lafita A, Bliven S, Prli \'c A, Guzenko D, Rose PW, Bradley A, et al. BioJava 5: A community driven open-source bioinformatics library. PLoS computational biology 2019;15(2):e1006791

    2019

  18. [26]

    Evaluation of template-based models in CASP8 with standard measures

    Cozzetto D, Kryshtafovych A, Fidelis K, Moult J, Rost B, Tramontano A. Evaluation of template-based models in CASP8 with standard measures. Proteins: Structure, Function, and Bioinformatics 2009;77(S9):18--28

    2009

  19. [27]

    The challenge of modeling protein assemblies: the CASP12-CAPRI experiment

    Lensink MF, Velankar S, Baek M, Heo L, Seok C, Wodak SJ. The challenge of modeling protein assemblies: the CASP12-CAPRI experiment. Proteins: Structure, Function, and Bioinformatics 2018;86:257--273

    2018

  20. [28]

    CATH: an expanded resource to predict protein function through structure and sequence

    Dawson NL, Lewis TE, Das S, Lees JG, Lee D, Ashford P, et al. CATH: an expanded resource to predict protein function through structure and sequence. Nucleic acids research 2016;45(D1):D289--D295

    2016

  21. [29]

    Optimal contact definition for reconstruction of contact maps

    Duarte JM, Sathyapriya R, Stehr H, Filippis I, Lappe M. Optimal contact definition for reconstruction of contact maps. BMC bioinformatics 2010;11(1):283

    2010

  22. [30]

    Defining an essence of structure determining residue contacts in proteins

    Sathyapriya R, Duarte JM, Stehr H, Filippis I, Lappe M. Defining an essence of structure determining residue contacts in proteins. PLoS computational biology 2009;5(12):e1000584

    2009

  23. [31]

    Sequence co-evolution gives 3D contacts and structures of protein complexes

    Hopf TA, Sch \"a rfe CP, Rodrigues JP, Green AG, Kohlbacher O, Sander C, et al. Sequence co-evolution gives 3D contacts and structures of protein complexes. Elife 2014;3:e03430

    2014

  24. [32]

    Robust and accurate prediction of residue--residue interactions across protein interfaces using evolutionary information

    Ovchinnikov S, Kamisetty H, Baker D. Robust and accurate prediction of residue--residue interactions across protein interfaces using evolutionary information. Elife 2014;3:e02030

    2014

  25. [33]

    Small angle X-ray scattering and cross-linking for data assisted protein structure prediction in CASP 12 with prospects for improved accuracy

    Ogorzalek TL, Hura GL, Belsom A, Burnett KH, Kryshtafovych A, Tainer JA, et al. Small angle X-ray scattering and cross-linking for data assisted protein structure prediction in CASP 12 with prospects for improved accuracy. Proteins: Structure, Function, and Bioinformatics 2018...

    2018

  26. [34]

    SWISS-MODEL: homology modelling of protein structures and complexes

    Waterhouse A, Bertoni M, Bienert S, Studer G, Tauriello G, Gumienny R, et al. SWISS-MODEL: homology modelling of protein structures and complexes. Nucleic acids research 2018;46(W1):W296--W303

    2018

  27. [35]

    Protein structure prediction and analysis using the Robetta server

    Kim DE, Chivian D, Baker D. Protein structure prediction and analysis using the Robetta server. Nucleic acids research 2004;32(suppl\_2):W526--W531

    2004

  28. [36]

    Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12

    Haas J, Barbato A, Behringer D, Studer G, Roth S, Bertoni M, et al. Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12. Proteins: Structure, Function, and Bioinformatics 2018;86:387--398

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.