Sixth Community Wide Experiment on the Critical Assessment of Techniques for Protein Structure Prediction

Description of the CASP6 experiment

Goals Scope Timetable Participation Targets Predictions Assessment Results Meeting Organizers Assessors

Goals

CASP6 main goal was to obtain an in-depth and objective assessment of our current abilities and inabilities in the area of protein structure prediction. Participants tried to predict as much as possible about a set of soon to be known structures. These were true predictions, not ‘post-dictions’ made on already known structures.

The broad goals of the CASP6 experiment were to address the following questions about the current state of the art in protein structure prediction:

  1. Are the models produced similar to the corresponding experimental structure?
  2. Is the mapping of the target sequence onto the proposed structure (i.e. the alignment) correct?
  3. Have similar structures that a model can be based on been identified?
  4. Are the details of the models correct?
  5. Has there been progress from the earlier CASPs?
  6. What methods are most effective?
  7. Where can future effort be most productively focused?

CASP6 Scope and Related Experiments

All types of methods for predicting protein structure were considered, ranging from comparative modeling through fold recognition and 'new fold' prediction. Contact prediction category remained with CASP6 while secondary structure prediction category was dropped. The new category, prediction of disordered regions in proteins, introduced in CASP5, was also evaluated. Two new categories, prediction of domain boundaries and protein function prediction were introduced in CASP6. In addition, there were additional activities closely related to CASP6 experiment:

CAFASP4      EVA      LIVEBENCH      SAC-CASP6       Post-CASP workshop

Timetable

Registration for the experiment started in March. The prediction season ran June through August. Automatic evaluation of the predictions and their estimation by independent assessors took place in September - October. During this time all predictors could submit abstracts describing their methods. In November the best groups were identified and invited to prepare a talk for the December CASP meeting, where the results of prediction assessment were discussed.

Participation

Participation was open to all. Intending participants, both human experts and fully automated servers, might register. Those interested in receiving mailings concerning progress of the experiment might also register as 'observers'.

Targets

We invited protein crystallographers and NMR spectroscopists to provide details of structures they expected to have made public before October 1, 2004. Prediction targets were made available through our web site. All targets were assigned an expiry date, and predictions had to be received and accepted before that expiration date.

Predictions

Predictions should have been submitted in CASP format (for 3D coordinate predictions it is simple PDB format with consecutive numbering of residues 1 -> N and added required headers).

Assessment of Predictions

Independent assessors evaluated the predictions. There were three assessors, representing expertise in the comparative modeling, fold recognition and new fold prediction areas. Assessors were provided with the results of numerical evaluation of the predictions, and judged the results primarily on that basis. They were asked to focus particularly on the effectiveness of different methods. Numerical evaluation criteria were very similar to those used in CASP5, although some assessors used additional ones.

Results and Publication

All CASP predictions and evaluations were made available through the web site shortly before the meeting. Results of server predictions were made available immediately after the target was closed for the server-type predictions (72 hours after the target release). The proceedings of the meeting were published. All participants were also encouraged to fully report their results on the FORCASP web site. The site also provided a forum for more extensive discussion of the results.

Meeting

The meeting to evaluate the results of the prediction experiment was held at the Hotel Serapo in Gaeta (Italy) on December 4-8, 2004.
250 participants attended the meeting with precedence given to active predictors. Some financial assistance was available for the most successful predictors.

Organizing Committee

       John Moult, CARB, University of Maryland, USA
       Krzysztof Fidelis, Lawrence Livermore National Laboratory, USA
       Tim Hubbard, Wellcome Trust Sanger Institute, Hinxton, UK
       Burkhard Rost, Columbia University, USA
       Anna Tramontano, University of Rome, Italy

Assessors

According to CASP policy, assessors are not allowed to be directly involved in the organization of the experiment, nor can they take part in the experiment in the role of predictors. The following people kindly agreed to be CASP6 assessors:

       Alfonso Valencia (CNB, Madrid) - for comparative modelling
       Roland Dunbrack (Fox Chase Cancer Center, Philadelphia) - for fold recognition
       BK Lee (NCI/NIH, Bethesda) - for new folds

Click here for the list of all CASPs assessors.

[Home] Protein Structure Prediction Center