It long has been the case that a strong correlative relationship exists between the importance of a scientific publication and the number of times it is cited in subsequent research publications. If a discovery is significant, it gets referenced. If the discovery is in a hot field, citations may come quickly. Alternatively, there are examples of discoveries far ahead of their time that come to be recognized and cited only once their significance is appreciated. Up until recently, measures of citation impact have represented a reasonable correlate to the value of a scientific contribution.
Regrettably, the value of this commodity has been recognized and biomedical researchers have begun to game the system. Likewise, diminished attention to the scholarly process of referencing has eroded the foundational underpinnings of the citation impact system of measuring the value of research publications.
Starting with the latter of these problems, I give a humorous example of how things have gone haywire. Several years ago my colleagues Jian Wang, Peter Alexander and I reported our discovery that mouse embryonic stem cells consume threonine as a hydrocarbon fuel. Mouse ES cells grow at an incredibly rapid clip. Their mitotic doubling time is only four to five hours in duration, exceeding that of the most rapidly growing cancer cells and approaching the doubling time of laboratory strains of yeast. Hypothesizing that this might reflect the possibility that ES cells exist in an unusual metabolic state, we measured the levels of scores of metabolites as a function of ES cell differentiation. When cued to differentiate upon withdrawal of leukemia inhibiting factor and administration of retinoic acid, the growth rate of ES cells slows from a four- to five-hour doubling time to 24 hours.
By observing profound changes in the levels of metabolites associated with one carbon metabolism, we stumbled over the fact that pluripotent ES cells express the threonine dehydrogenase, or TDH, enzyme at a 1,000-fold higher level than any mouse tissue or cell line that we tested, and they use the enzyme to consume threonine as a metabolic fuel. Depletion of threonine from the culture medium killed mouse ES cells, and specific inhibitors of the TDH enzyme kill mouse ES cells while having no effect on any other cultured cells tested to date. We recognized this to be analogous to the metabolic state of rapidly growing bacterial and yeast cells, and we made reference to microbiologists upon whose shoulders we stood.
Three years later, the Harvard University labs of Lewis Cantley and George Daley published a nice follow-up paper in the journal Science. My colleagues and I were delighted to see the replication and extension of our earlier work, yet I was mortified to see that the Harvard manuscript made reference to a paper that I had entirely missed. Yes, the Cantley/Daley paper did cite our paper, yet it also attributed the discovery of threonine dependence of mouse ES cells to a paper from the laboratory of Eric Lander of the Broad Institute. The latter paper was published a full year before our 2009 Science paper.
I immediately downloaded the Broad paper and scoured it from end to end. To my confusion, I could not find the word threonine in the entire paper, much less threonine dehydrogenase or anything to do with the metabolic state of mouse ES cells. Inquiries to Cantley and Daley led me to a postdoctoral fellow who instructed me to view the supplemental data included in the Broad paper. Sure enough, the study listed thousands of genes selectively expressed in undifferentiated ES cells, and the gene encoding threonine dehydrogenase was embedded within the list. This was an “Alice in Wonderland” moment for me, showing that an entirely new era of attribution had evolved. Any discovery that stems from any gene listed in the supplemental data of the Broad manuscript can now be attributed to that paper! Instead of thinking “Are you kidding me?”, I recognize that we are now adapting to the new reality: Citations are no longer your father’s Oldsmobile.
Years ago I was asked to provide “key words” to accompany the publication of one of our papers. Being unusually dense, it did not occur to me why this was of any utility. In the meantime, I began to notice that the titles of more and more papers were littered with trendy names or terms – sirtuins, epigenetics, NFkB, TGFb, p53, mTor and so forth. These key words and littered manuscript titles are, of course, the beacons for search engines that have grown to dominate how our work is packaged for the digital age. I am reminded of the Canon camera commercial featuring Andre Agassi, which said – “Image is everything.” Those who have learned to game the system operate at a distinct advantage – allowing them to sport stratospheric H-factor scores. The idiots who ignore this stuff do so at our peril; what a pathetic, devolved state of affairs we find ourselves in.
How is a scientist taught the proper manner in which to reference prior work? The answer is simple: We teach this to our trainees one at a time just as our mentors taught us. When Andrew Pieper discovered pro-neurogenic chemicals in an in vivo screen carried out in my lab some years ago, we went back to the original work of Joseph Altman who discovered adult neurogenesis in the 1960s. When Gelin Wang and Ting Han discovered that our pro-neurogenic chemicals function by activating the rate limiting enzyme in NAD salvage from nicotinamide , we went back to the original discovery of NAD salvage by Phillip Handler in the 1950s. In aspiring to teach my trainees the scholarly manner of referencing that Oscar Miller, Joe Gall and Donald Brown taught me, am I training my students straight toward failure? Unless things change in a substantive manner, the answer to this question is unquestionably affirmative.
I close with an account of a disconcerting event I encountered this past year. As chairman of the University of Texas Southwestern Medical Center at Dallas biochemistry department, I am charged with nominating members of our faculty for promotion. I work in concert with our senior faculty members to properly time the promotion of assistant professors to the rank of tenured associate professor, as well as promotion of associate professors to full professorship. I then work with the promotion candidate to put together his or her nomination package.
We hold high standards in our department, so approval for promotion by the promotions and tenure committee of our institution has been uniformly positive over the past two decades. This year, however, I ran into a potential roadblock. After having submitted the nomination package, a member of the committee raised the concern that our nominee may not have an adequately strong “national reputation.” I admitted that there was some truth to this: The nominee is a quiet scientist who seldom attends national or international meetings. My retort was simple: Instead of asking whether a scientist has a national reputation, let’s ask whether his or her discoveries have a national reputation. My instructions were simple: Go look at the most recent editions of the textbook "Biochemistry" by Jeremy M. Berg, John L. Tymoczko and Lubert Stryer or the textbook "Molecular Biology of the Cell" by Bruce Alberts, Alexander Johnson, Julian Lewis and Martin Raff. If you find the work of the candidate referenced in the textbooks, this means the science is of clearly confirmed national reputation. The candidate’s work was indeed found in the textbooks, and all was well that ended well.
I would far rather my science to be of national acclaim than for me to be of national acclaim. We should consider how it might be possible to perform evaluations at all levels in just this way. If we can find the work of a scientist in the textbooks, is this not the single best way to assess value with respect to the most essential measures of how our enterprise operates?
Steven McKnight (steven.mcknight@ utsouthwestern.edu
) is president of the American Society for Biochemistry and Molecular Biology and chairman of the biochemistry department at the University of Texas-Southwestern Medical Center at Dallas.