Archive

Archive for October, 2011

On the Quantity vs. Quality of Research Output

October 16, 2011 Leave a comment

ᄅ JASON VARNEY|VARNEYPHOTO.COM

Recently one of my Facebook friends has posted the picture above with the comment: “Must be some weird area.” Of course, this picture is a spoof produced for an article in the Scientist magazine about the shortage of open positions that require a Ph.D. Because my friend is an accomplished academic, his post prompted the usual wittiness contest, with fellow academics and graduate students commenting on this photo. I commented as well saying: “4 years to get a Ph.D. plus 3*2 years of postdoc = 10 years of scholarly productivity to publish 6 papers. This is 3/5th of a paper a year! I would not hire this person even for a postdoc position.” My statement must have touched the nerve, as several people started arguing with me vehemently that one cannot judge the quality of a researcher based on the number of his or her papers.

A similar argument has come up in the Research Methods in CS class I am teaching this semester. To help students plan their Ph.D. studies, I asked them how many papers and in which venues they plan to publish before they complete their Ph.D. studies. Some students objected to my asking this question, stating that one cannot correlate publication numbers with the quality of one’s research. One student, in particular, was adamantly defending his position that as long as he did something great for his Ph.D., he could easily report the results in a single publication. Although I appreciated the student’s candor, I do not believe that his strategy is feasible.

German philosopher Georg Wilhelm Friedrich Hegel was the first to point out the phenomenon of quantity transforming into quality. This is a powerful idea that applies to multiple fields. At the physical level, one example of this transformation is that as soon as enough heat is applied to water, it turns into steam. Recently, I encountered a fascinating example of this transformation described eloquently by Maxwell Gladwell in his book “Outliers: The Story Of Success.” In particular, Gladwell is describing the so-called 10,000 hours principle. It takes on average 10,000 hours of deliberate practice in a cognitively demanding field for an individual to achieve remarkable results. In other words, 10,000 hours of practice transform a capable person into an outlier in pursuits as diverse as chess, music, computer programming, business, and math. For me, this is the ultimate application of the Hegelian transition from quantity into quality.

How does this principle apply to the quantity and quality of research output? Is it possible for a Ph.D. student to write just one paper that is going to become highly influential? I am sure there are some rare examples of that, but in general I don’t think this is possible. A highly influential paper must report on a real breakthrough. Publishing truly innovative ideas is hard, as one must overcome the resistance of the established understanding and views in the area. To be able to publish such a research paper, one must develop the ability to present research results effectively. The only way to learn this skill is by practicing it incessantly for many years. To develop this ability, a researcher should have published multiple papers having learned how to deal with rejection in the process. Therefore, I believe that researchers also transition from quantity into quality.

Once someone asked me why I was publishing papers in a whole variety of venues. Wouldn’t it make sense just to focus on publishing in top tier venues? I quickly replied that I had a wide range of ideas, and only some of which deserved to be published in top tier venues. Nevertheless, I strongly believe in evaluating and publishing all promising ideas. This practice is necessary to build my research evaluation and presentation skills, so that when I have a truly great idea I’d have the required skills to evaluate and publish it.

Going back to the original example, six publications for someone with a Ph.D. and three postdocs seems unreasonably few, which to me indicates low research productivity or poor work ethics. I also do not believe that someone can publish only six papers, all of which would be in top tier venues. To get accepted to these venues, one needs to build up his or her research skills by publishing other papers, albeit in lower-ranked venues. So I stand by my original Facebook comment on that fake picture.

A Modest Feature Request

October 16, 2011 Leave a comment

The Au Bon Pain in our student center has recently started using an iPad application for customers to enter their sandwich orders. In the past, you would order a sandwich by filling out a paper form. You’d pencil in your name and sandwich selection and deposit the form to a tray, from where your order would be picked up and processed by the next available sandwich chef. The new system eliminates the paper forms completely, as the iPad transmits the orders electronically.

During the first week, it was an Au Bon Pain manager who would take orders on the iPad. She has explained that it was a way to “squelch all the bugs” before the customers start using the application. After ordering my favorite Mediterranean Wrap, I inquired “Wouldn’t be nice to know what the approximate wait time is?” “What a great idea! Let me write this down.” Well, as a researcher, I come up with new ideas for a living. I also well-realize that not all of my ideas are worth implementing. I also teach courses in software design and engineering on a regular basis, and it immediately occurred to me that this feature request can make a useful teachable moment.

So, as I was waiting for my sandwich to be prepared, I started thinking about what it would take for the developers of this sandwich ordering application to fulfill my feature request, adding the functionality that would display the expected wait time upon completing an order. It immediately occurred to me that implementing this feature may not be trivial. First of all, to make any wait predictions, the system must timestamp when the sandwiches are ordered and done. Even if this functionality is already there, the client part of the application (running on the iPad) may not have the information about the length of the sandwich queue at the server. The client part is likely to simply accept orders and send them the server, which maintains the queue. So if the client is to be able to display any wait information on the iPad, the server interface would have to be extended to provide this information.

One can think that calculating the average wait time is easy. Just calculate N*T, where N is the length of the sandwich order line and T is the average time to prepare a sandwich. However, this formula is unlikely to produce an accurate estimate. Several sandwich chefs usually work in parallel, with different levels of productivity that depends on various factors such as their natural ability and experience. Also, not all sandwiches are created equal–some of them may take longer to prepare than the others. From time to time, sandwich chefs may need to interrupt their work to bring over various sandwich ingredients from somewhere in the back storage. The system probably does not keep track of how many sandwich chefs are present and what their individual productivity is. Adding all this information to the system is likely to significantly complicate its implementation.

To predict the wait time more or less accurately would require using queuing theory, which may require encoding some statistical formulas. Overall, adding this deceptively simple-sounding feature is likely to require substantial changes to the system’s design and implementation. Also, testing the new feature once it is deployed would also take additional resources.

Now the big question is whether all this extra effort would be worth it? Would Au Bon Pain sell more sandwiches if their customers are given an accurate estimate of the time it’d take to make a sandwich? To answer this question would require thinking through representative use cases. Let us walk through a use case that I encounter quite often. Say I have a class to teach in 30 mins. I feel hungry and having a sandwich sounds like a great idea. I also know that it takes me about 10 minutes to get from the Au Bon Pain store to the classroom. If the system informs me that it will take anything longer than 10 minutes to prepare my sandwich, I would definitely cancel my order, lest I be late for class. This use case shows that this feature may in fact cause Au Bon Pain to lose potential customers!

Looking back, it no longer looks like my suggestion to add  the expected wait time feature to the new iPad sandwich ordering system is such a good one. It is not trivial to implement properly, and it may not help the store’s bottom line. A better alternative may be to add a simpler feature that shows the length of the sandwich order line. I often wonder whether all these students hanging around the store are just socializing with their friends or are waiting for their sandwich to be made. Implementing this feature is quite straightforward, and optimistic, short-on-time customers may still choose to wait for their sandwich when given the number of pending orders rather than the expected wait time.

Categories: Uncategorized

On the Prudence of Pessimism

October 9, 2011 Leave a comment

I am in beautiful Detroit, MI, sitting at the gate in the airport, having missed my flight by about 10 minutes. The next flight is in 5 hours. Nevertheless, I feel fortunate. No, I don’t need to have my head examined. I am fortunate indeed to have a confirmed seat on the next flight, which is not just full but oversold. My fellow travelers who missed the same connection were not so fortunate–they may have to spend the night here or fly to a “nearby” airport a 5-6 hours drive away from their final destination.

How did I luck out like that? I am a pessimist by nature, and always prepare for the worst possible outcome. When I was waiting to board my first flight this morning, a cheerful announcement notified me that the flight would be slightly delayed. “We have a minor maintenance issue, but should be out of here in 30 minutes. Please ask the agent to rebook your connecting flight only if your layover is less than 30 minutes.” Well, my layover was longer than one hour and thirty minutes, but nevertheless I immediately proceeded to rebook my connecting flight. “But sir, you should be able to make your flight comfortably–we are expected to be only 30 minutes late.” “Lady, just please rebook me for a later flight just in case. If I make my original flight, I just won’t use that ticket.” Lo and behold, the 30 minutes delay turned into an hour and forty minutes delay. If the next flight were delayed by as little as 10 minutes, I would have probably made it. Now I have a pleasant 5 hour wait in the airport, but as it turned out, I am the fortunate holder of the last seat on the next and last flight of the day.

This realization makes the wait sweeter. It also makes me ponder about the value of pessimism. Not getting back home today would have been disastrous for me. I not only have to teach tomorrow early afternoon, but a senior colleague is scheduled to come by my class to conduct a peer teaching evaluation, crucial for my tenure application. Yes, I will be very tired when I get home, but nothing that a good night sleep would not cure. Well, I would have to break my promise to grade that assignment by tomorrow; Sorry, folks!–I hope you understand.

Is preparing for the worst possible outcome always prudent? Harvard economist, John Kenneth Galbraith, claims that “We all agree that pessimism is a mark of superior intellect.” I don’t know about superior intellect, but this strategy has worked for me well in different settings. For example, whenever I submit a manuscript for review, I immediately start making back up plans for the case if the paper gets rejected. I don’t just make abstract plans, but I usually ask a student co-author to reformat the paper for the next submission target. I don’t just want to be prepared for the possibility of rejection psychologically, I want to know what kind of editing I would have to do to resubmit the paper. If the next publication venue has a smaller page limit, I want to know exactly how many pages we should be prepared to cut. If the opposite is the case, I want to plan what new material we can include to strengthen the paper.  What if your paper is accepted for publication right away? Well, I’d be pleasantly surprised. But what about all the work you have done to shorten or lengthen the paper? This work will never go to waste. Whenever reporting on a research project, I have always found myself needing to write about it in different forms: short, long, and everything in between. These shortened or lengthened manuscripts will serve as building blocks for future publications on the same topic.

Well, it looks like I am going to be able to squeeze in some grading after all. Nothing shortens a wait so productively as grading assignments. Thanks to cloud-base course management systems and 3G tethering, nowadays one can be productive almost everywhere.