Detailed description of the experiment
The main goal of the CASP experiments is to obtain an in-depth and
objective assessment of our current
abilities and inabilities in the area of protein structure prediction.
To this end, participants will produce models of soon to be released
experimental structures. These models will be true predictions,
not ‘post-dictions’ made on already known structures.
In addition to the established components of CASP, there are two
major new initiatives in CASP11:
-
Assessment of models in terms of how well they address relevant biological questions.
-
Collaboration with CAPRI (Critical Assessment of Protein Interactions)
to assess modeling of oligomeric relationships and of inter-domain relationships.
There will also be a strong focus on refinement methods, new methods
for predicting three-dimensional contacts, and the extent to which
sparse experimental data and predicted contacts can improve the accuracy
of models. Also new in this CASP will be evaluation of models based
on a perfect alignment to a template.
Specifically, CASP11 will address the following questions:
-
How well do the models help answering relevant biological questions?
-
How similar are the models to the corresponding experimental structure?
Are domain orientation and quaternary structure predicted correctly?
- Is the mapping of the target sequence onto the proposed
structure (i.e. the alignment) correct? How much can the models
be improved provided knowledge of the correct alignment?
- Have similar structures that a model can be based on (templates)
been identified?
- How much better are template-based models than those that
can be obtained by simply copying the best template?
-
How reliable are the model error estimates ('B-factors') provided
with the models?
- How much can current refinement methods improve the accuracy
of models?
-
How effective are newly emerging methods at predicting protein three dimensional contacts?
- How helpful is additional information (e.g., contacts or
sparse crystallography or spectroscopy data) in structure
modeling?
- Has there been progress from earlier CASPs?
- What methods are most effective?
- Where can future effort be most productively focused?
Tertiary structure predictions (TS):
-
The Template Based Modeling (TBM) category will include domains where a
suitable template can be identified that covers all or nearly all of
the target. For CASP11, the assessor will assess models on the basis
of how well they provide answers to relevant biological questions,
in addition to assessing aspects of accuracy that include alignment,
backbone accuracy, and side chain placement.
-
The Template free modeling (FM) category will include models
of proteins for which no suitable template can be identified.
-
The Refinement category will include selected targets from among
those released in the main modeling experiment to analyze success in refining
models beyond the accuracy obtained in the initial submissions.
For suitable CASP11 targets, we will select one of the best models received during
the prediction season, and reissue it as a starting structure for refinement.
-
The contact-based and contact-assisted structure modeling categories will
show how the knowledge of some long-range contacts (mimicking sparse experimental
data or predicted) influences the ability of predictors to
model the complete structure. This experiment will be carried out only for the more
challenging CASP11 targets where we can get coordinates in advance and have at least
two weeks for re-prediction.
-
The alignment-assisted structure modeling category will test the extent
to which other factors besides alignment correctness affect model accuracy.
For selected targets, where initial models have signicant alignment errors,
a second set of predictions will be invited, given knowledge of the correct alignment.
-
For all TS predictions, domain orientation and quaternary structure will be
assessed using the CAPRI criteria.
Other prediction categories:
- Detecting residue-residue contacts in proteins (RR).
- Identifying disordered regions in target proteins (DR).
- Quality assessment of models in general (without
knowledge of the experimental structure) and the reliability of predicting certain
residues in particular (QA).
There will be additional activities included in or related to CASP11,
extending the scope of the experiment.
Rolling CASP
will continue to run in parallel with CASP11 in May-July.
We will discuss the results of the CASP ROLL experiment
at the CASP11 predictors' meeting.
CAPRI.
In CASP11, there will be a close collaboration between CASP and CAPRI,
with the aim of stimulating increased interest in modeling of protein interactions.
Members of both the CASP and CAPRI communities will be invited to model the
interfaces of protein complexes, homo-multimers and domain interactions
in appropriate CASP11 targets. The results will be evaluated using established
CAPRI criteria. We plan to discuss the results of both experiments at the
CASP11 conference.
FORCASP:
There will be discussion of predictions and methods
on our
FORCASP forum.
Registration for CASP11 will start in the last week of March 2014.
Testing of server connectivity ("dry run" for server predictors) will
be conducted starting April 17, 2014. The first prediction targets will be
released not earlier than May 1; the last prediction targets will be
released not later than July 18; the prediction season will end not later
than July 31. Refinement, contact-assisted and alignment-assisted experiments
will end not later than August 18.
Registration for the meeting will open in July.
Abstracts describing the methods tested in CASP11 will be collected in
September.
The program of the meeting will be available in November.
The meeting will take place on December 7-10, and approximately one
month before that groups with the most accurate predictions and interesting
methods will receive invitations to give talks.
Participation is open to all. If you already have an account with the
Prediction Center, you will be able to go directly to the
CASP11 registration page.
Please check, though, that your basic registration information is
current. If it has changed - please update it through the My Personal
Data link from the main Menu. If you are new to CASP and don't have
an account with us, you will have to register with the Prediction
Center first, and after that - for CASP11. Separate
registration forms
for different types of registration will be available through this
website. Predictors with servers are requested to register as soon as
possible as we are planning on starting a
"dry run" for servers in the second decade of April.
If you are currently participating in CASP ROLL, you will have to
register separately for CASP11 (your group numbers for CASP11 and
CASP ROLL will be different).
For the experiment to succeed, it is essential that we obtain the help
of the experimental
community. As in previous CASPs, we invite protein crystallographers and NMR
spectroscopists to provide details of structures they expect to have
made public before September 15, 2014. The last day for suggesting
proteins as CASP targets is July 16, 2014.
A target submission form is available at this website.
During the prediction season, targets are posted daily on the
Target List page
and, additionally, automatically pushed to the registered prediction servers.
Targets in CASP11 will be split in two prediction tracks: those for 1) all groups
(long deadline) and 2) servers only (short deadline). Assignment of a target
to a particular track is made by the organizers and communicated to the predictors
through the Target List page. Priority for inclusion in the all groups modeling track
will be given to targets with low sequence identity to known targets.
Target release is planned for business days only, around 9am PDT.
Tarballs for QA predictions will be released at noon, PDT.
Sequence and other relevant information about the targets will be
posted at the Target List web page.
Requests to the participating servers will be sent shortly after the
target release. We plan to release not more than 3
targets per day for servers and, usually, one target per day for regular
groups. All targets are assigned two expiration dates (one - for server predictors
and another - for regular groups). All predictions must be received
and accepted before noon, 12pm PDT on the corresponding expiration date.
We are planning to release around 50 targets for evaluation in the all-group track
(long deadline) and as many targets as we can get in the server-only track.
Server groups are expected to submit their predictions for all targets.
Manual groups are expected to submit their predictions for long deadline
targets (all-group) in TS and RR categories, for all targets
(all-group + server-only) in DR and QA categories, and for all targets
designated as 'CAPRI' if submitting oligomeric predictions.
Those manual groups that wish to submit RR and monomeric TS predictions
for server-only targets are welcome to do so, but these predictions
will not be officially evaluated in CASP (multimeric
predictions for some server-only targets will be evaluated in CAPRI -
see next paragraph). The DR predictions on server-only targets are due
on the server expiration date.
In CASP11, some targets will be selected for additional assessment
of inter-chain (inter-domain) interfaces. These targets will be marked as
CAPRI-eligible in the Target List and their multimeric (or regular multidomain)
preditions will be passed to CAPRI for evaluation. Please note that CAPRI-eligible
targets will be selected from both all-group and server-only targets
(actually, we expect more server-only targets to qualify). Therefore,
if you participate in CASP with a long-deadline (human) group and are
planning on submitting multimeric predictions - please make sure that
you send your predictions on CAPRI-eligible targets before their
human expiration date (which may be shorter than 21 days on
server-only targets).
Predictions can be submitted through the Prediction Submission form available from
this web site or by the email provided on the format page. Please comply with the
instructions on CASP11 submission procedures and format.
Server predictions will be made publicly available shortly after the closing of the prediction
window for a specific target. To enable the newly established testing procedure for QA methods,
we will be releasing server predictions in 3 stages: (1) up to 20 selected predictions spanning
the whole range of model accuracy will be released 2 days after the server TS deadline;
(2) best 150 server predictions (according to the ranking from the naive consensus QA method) -
4 days after the TS deadline; (3) all server predictions - 6 days after the server TS deadline
(see the QA format description at the format page).
If you are currently participating in CASP ROLL, you will have to submit
your predictions on CASP11 targets only to CASP11. Your group numbers for
CASP11 and CASP ROLL will be kept different.
As in previous CASPs, independent assessors will evaluate the predictions.
Assessors will be provided with the results of numerical prediction evaluations performed
at the Prediction Center, and will judge the results primarily on that basis.
They will be asked to focus particularly on the effectiveness of different methods.
Evaluation criteria will as far as possible be similar to those used in previous CASPs,
although the assessors are welcome to introduce additional measures.
Assessment procedures for relating models to biological questions will be developed
by the assessors. Evaluation of interfaces will be performed by CAPRI organizers,
using established metrics.
There will be two assessors, focusing on the following areas of prediction:
-
Template based modeling -
Roland Dunbrack (Fox Chase Cancer Center, Philadelphia, USA)
-
Template free modeling /contact-assisted -
Nick Grishin (University of Texas, Dallas, USA)
Other prediction categories (contacts, quality assessment, contact-assisted and refinement)
will be evaluated by the selected assessors or the organizers.
Click here
for the list of assessors in all CASPs held so far.
In accordance with CASP policy, assessors are not directly
involved in the organization of the experiment, nor can they take part
in the relevant parts of the experiment as predictors. Predictors must not
contact assessors directly with queries, but rather these should be
sent to the
email
address.
All CASP predictions and results of numerical evaluation will be made available through
this web site shortly before the meeting.
The proceedings of the meeting will be published in a scientific journal
(see
publications of previous experiments).
All participants will also be required to describe their methods
in the abstracts (published locally at our web site) and encouraged to
discuss them on the
FORCASP forum.
These contributions will be discussed and scored
by other predictors, and this material will be taken into account in
choosing
some presentations at the meeting. Also, predictors
presenting posters at the meeting should be prepared to give a short
presentation at one of the main sessions, as some talks will be
invited during the meeting based on the
discussion of poster sessions.
The meeting to discuss results of the experiment will be held at the
Paraiso Maya all-inclusive resort
on the Riviera Maya, Mexico on December 7-10, 2014
(starting at 6pm on the 7th and ending in the afternoon of the 10th).
The meeting program will be available in mid-November, 2014.
The total cost of the meeting, including the early registration fee
and an all-inclusive lodging fee (room, all meals and coffee breaks
for 3 nights) is expected to be around $1000 (detailed pricing will
be published as soon as the prices are finalized with the resort).
Some financial assistance will be available for the most successful
predictors and students.
John Moult, CASP president; IBBR, University of Maryland, USA
Krzysztof Fidelis, University of California, Davis, USA
Andriy Kryshtafovych, University of California, Davis, USA
Torsten Schwede, University of Basel, Switzerland
Anna Tramontano, University of Rome, Italy
David Baker, University of Washington
Nick Grishin, University of Texas
David Jones, University College, London
Justin MacCallum, University of Calgary
Michael Sternberg, Imperial College, London
|
