Building from the ground up
Like a tower, validation is made of a stack of bricks. The first brick is analytical confirmation. This is the type of validation for which researchers have to ask themselves whether they get the same result from the same experiment done all over again or if a different method on the same sample set gives them the same answer.

Next are the bricks of independent repeatability and replication. Researchers not connected with the original group must see if they can carry out the same experiments and get the same answers. If the analysis has clinical implications, it should also be carried out in larger cohorts to see if the same results emerge.

The next brick of validation is interpretation, and this one “is the toughest of all,” says Ioannidis. “Even when everything has been repeatable, reproducible and replicable, there is some room for differences in opinion.” Ioannidis says that, while he believes in the freedom of researchers to interpret data as they see fit, some standards need to be set in how to interpret data for different fields.

The final brick is asking whether the newly discovered information helps us. “Even if you know what a variant means, and even if it is one you can act on, does acting on it actually improve public health?” asks Adam Felsenfeld at the National Human Genome Research Institute. “It is a huge issue that has to be tackled not just by the clinical community but by health-care economists” and others. He gives the example of the prostate cancer screening test, whose true clinical utility in reducing the burden of disease has been debated. He says that kind of consideration for clinical utility should be built into -omics research as early as possible.

No one-size-fits-all solution
In discussing validation, it’s important to appreciate that the different -omics fields can’t be lumped together. The information gleaned from these fields “encompasses so many different kinds of data. Each one of them has its own technical challenges with respect to validation,” says Ralph Bradshaw at the University of California at San Francisco and co-editor of Molecular & Cellular Proteomics with Alma Burlingame at the same institution. (MCP is published by the American Society for Biochemistry and Molecular Biology.) “If you really want to talk about validation, you have to start piecemeal,” he says, taking each field on its own with its quirks and challenges.

Ioannidis agrees that validation has to be tailored according to the needs of each particular field and the types of measurements available. Just take proteomics. It may have the mission to use large sets of proteins to understand various biological phenomena, but the data come in a variety of forms, ranging from mass spectrometric methods to difference gel electrophoresis. Validation issues for various techniques have to be dealt with in different ways.

As Bradshaw points out, “Validation carries with it the connotation of replication.” He explains that for some -omics fields, such as genomic sequencing, “the replication of the data, both from the terms of technical and biological, is in fact really quite exact.” However, for shotgun proteomics, which identifies by mass spectrometry a large number of proteins from a sample containing millions, “the reproducibility of an experiment, even in the same laboratory on the same sample, is only partial,” says Bradshaw. “You can’t talk about validation [in that case] because of the nature of large-scale mass spectrometry experiments.”

Gilbert Omenn at the University of Michigan, the chairman of the IOM committee on -omics data validation, agrees with Bradshaw. “It’s extremely important to recognize you may not get the same result if you repeat the experiment in the same lab with the same hands with the same samples, because there is a certain stochastic aspect to detection of peptides in mass spectrometry,” he says. But he adds it simply means that there is an even greater need for replication with these types of experiments. While there isn’t a one-size-fits-all procedure for ensuring accuracy of -omics data, Omenn says that no matter the experimental platform, the principles of validation cut across all -omics fields.