Evaluating a University Scientist

One of the more common questions I get is "How do you evaluate university technologies?" A lot of time is spent on just this topic at any gathering of business development types. Evaluating the technology is important, but if you do not also evaluate the scientists behind the technology you have not done a thorough job.


Why is evaluating the scientist important? University-stage technologies require a LOT of further development and you will need the scientist to do at least some of that development themselves. There are very few technologies that come to a university TCO where I can say, "I'll take it from here, boys!" These technologies are also way too early stage to interest companies or VCs. Most medical technologies from universities will need another 2-3 years (at a minimum) of further development before we can attract commercial interest. Even after a "handoff" to the corporate partner, the university scientist will likely still have a large role in the technology's development.

Still, why does the scientist matter? They matter because someone is going to have to fund them to do this future development. Few university TCOs have the resources to fund much, if any, development so they will need to raise funding from external sources. Then, when it is time to find a commercial partner, you will need your scientist to be credible with the company.

Unfortunately, doing this evaluation can lead into personal areas and that is why evaluating scientists is not often discussed in polite company.

So, what are we looking at with university scientists? An important thing to keep in mind as you go through the list below is that you have to evaluate junior faculty and senior faculty differently (for obvious reasons). Senior faculty are evaluated more on their ability to attract and retain top-flight people below them.

• Where did they go to school? This doesn't tell you everything and there are good scientists produced by many schools that are not considered elite, but I refuse to accept that a scientist's educational history means nothing. If I see someone who got their undergrad at Harvard, Ph.D. from MIT and did their post-doctoral studies at Stanford, what does that say? It probably says they were intelligent to get into Harvard in the first place. They probably did fairly well at Harvard to get into graduate school at MIT and they probably did well at MIT to get a nice post-doctoral position at Stanford. Simply, put....this scientists if probably top-shelf from an intelligence and work ethic standpoint. What about someone who got their undergrad at Podunk U. and did their Ph.D. at MIT? While there is doubtless a story behind why they went to Podunk U., they clearly worked their tail off to get into MIT because MIT doesn't accept many students from Podunk U. What about the reverse? Undergrad at MIT and Ph.D. from Podunk U.? That screams to me that they are smart, but had bad grades in college; not necessarily damning, but something to be aware of.
  
• How much do they publish? In the medical field, this is simple. Go to PubMed (http://www.ncbi.nlm.nih.gov/PubMed) and type their name in using the following format: Dean F. Stell would be entered, "Stell DF". See what comes up. For a junior faculty member in their 30's you probably won't see more than ~10 publications. A senior faculty member could have several hundred. Don't look so much a raw volume. There are SOOOOOO many newer, fringe journals now that pumping the raw number of publications is not as hard as it used to be. What you want to look at is recent activity? Are they publishing consistently? Have they published recently? Are there gaps?
  
• Where do they publish? This is becoming more and more important. Go back to the list you got from PubMed and scan through the journal titles. The first thing I'm looking for are the BIG names: Science, Nature, Cell, New England Journal of Medicine (NEJM), Journal of the American Medical Assoc (JAMA), Lancet, Procedings of the National Academy of Scientists (PNAS), Journal of Biological Chemistry (JBC), Nature Genetics, Nature Medicine, Nature Biotechnology, etc. It is important to understand that these are the prestige journals and academic scientists will strive to get their work published in the best journals possible. Most scientists will not have many big journal names on their list. That's just the nature of the game: most science simply isn't THAT cool. But, it is a big plus to see some of these. I would rather see less publications in top-notch journals than more publications in lesser journals. Each individual field of science will have speciality journals that are considered the best. I can't list them all here, but the easiest way to find out what they are is to call/email a few faculty in the field and ask "What are the best journals in your field?" What you are also looking for is padding. Do they have a lot of publications in mediocre journals? A rule of thumb I use (that has some bad flaws, but I use it nonetheless) is that more names in the journal title is often bad. Cell = Good. Central European Annals of Spinal Cord Injuries = Bad. No scientist sets out to publish in bad journals. They only end up in bad journals if science couldn't get accepted by good journals. Another thing to look out for is the word "Letters" in the title. "Letters" typically mean that the publication was not peer reviewed. [One note on the journal PNAS: PNAS is a prestigious journal, but there are two ways to get published. One is to have your paper run the peer-review gauntlet and get accepted. The other is to have it submitted by a member of the National Academy of Sciences. Members are allowed to publish a certain number of manuscripts every year without much peer review. It certainly has happened that a mediocre scientist has included as an author a National Academy member who has stopped trying very hard for the sole purpose of getting the manuscript published in PNAS. Clearly this has less prestige associated with it.]
  
• Where does your scientist appear in the list of authors? With medical/basic science publications the first author is usually who did the work and wrote the paper. Often this is a graduate student or a junior faculty member. The last author is usually the lab head who got the money that allowed the work to go forward. The folks in the middle could have done a lot of work, or they might have merely reviewed a draft of the paper. You just don't know. However, when you are evaluating the technology itself, you do need to talk to the first authors. It is important to recognize the senior faculty members for getting the grants that make the science possible, but they MAY have a fairly shallow understanding of the technology. Involvement of the last author really varies from intimate involvement with the research to merely sitting on top of a Ponzi scheme that gets their name on a lot of papers.
  
• How is your scientist funded? The ideal situation is to see a blend of funding sources: government, non-profit and commercial. You do want to see some peer-reviewed govenmental funding. The presence of this funding means that other scientists felt the scientist had worthy ideas. Non-profit funding is often not as rigorously peer-reviewed as governmental funding, but it is often funds more translational studies. Commercial funding is an indicator that corporate entities have found the scientist useful to work with. With funding, you are really looking for red flags. No peer-reviewed govermental funding: Does that mean that the science is lacking in merit? Commercial entities ending or scaling back funding relationships: Does that mean the scientist is hard to work with or doesn't produce? All funding from non-peer reviewed sources: Perhaps the scientist is just a pretty face who tells a good story?
  
• Does the scientist have a stable track record? Here you are looking at 5-10 year windows of time and this is mostly relevant with senior faculty. Have they maintained long term scientific collaborations? Do their government grants continually get renewed? Are the groups that funded their work 5-10 years ago still involved? If they moved from another university, do they maintain collaborations at their former university? Are they still collaborating with people they met in graduate school decades ago? If you see a lot of stability that is a very good thing. If you see a completely new cast of characters every 5 years, that is cause for concern. Some turnover is normal, but a complete turnover combined with moving universities every 5-7 years could mean they are just wearing out their welcome everywhere.
  
• Can you take them to dinner with a company? This is mostly a personal judgement. Are they presentable in public? Do they tell off color jokes? Do they have bad manners? Do they have extreme political leanings that they share with everyone? A scientist who passes these tests is often referred to as "road show enabled" meaning you can take them out to pitch their own technology. Most scientists are not road show enabled, but are not going to scare a company. A small minority are just embarassing and you are going to need a plan to pitch the technology without the scientist's help AND a plan for telling the scientist why they aren't invited to dinner with the company.
  
• Do they share credit? The days of the singular scientist triumphing on their own are long past. When you meet with the senior faculty member, have they brought in the graduate student who did the work? Or, do they make it appear that they had all the ideas and the lab staff are merely interchangeable parts? Scientists who hog the limelight could indicate a personality problem, but it can also present a practical problem when you need to speak to the people who actually did the experiements.
• Are the punctual? Can they show up for meetings on time? Are they 15 minutes late for everything in their life? Most scientists who show up for meetings on-time with a well organized plan for what they want to talk about have pretty orderly science. Unfortunately, the reverse is also sometimes true.
  
• What do former colleagues say about them? This may seem invasive, but if you are considering a large investment in a scientist, perhaps touch base with some of their former colleagues. You don't want to ask for a list of references. Go back to that list of PubMed publications and see who else is on their publications and give them a call. You obviously need to be very discrete when doing this because chances are that these folks will contact your scientist and say that you were checking up on them, but if you are considering a large investment (or if the scientist is asking you to make a large investment) this is a reasonable step.
• Are they members of any societies? The big three are the National Academy of Sciences, National Academy of Engineering and the Institute of Medicine. There are many outstanding scientists who are not members of these societies, but if a scientist is being touted as "world-class" it is worth wondering why he/she is not held in similiar esteem by their peers.
  
  

In the end, there is no right or wrong answer. What you are trying to do is paint a complete picture of the scientist. You want to have some idea because this scientist is vitally important during the early stages of development. All things being equal, the "good" scientists have more success not to mention that they are more pleasant to work with.

*Articles are reproduced with permission from Dean Stell, the owner of Technologycommercialization.blogspot.com