Michèle Lamont decided to explore excellence by studying one of the primary mechanisms used by higher education to -- in theory -- reward excellence: scholarly peer review. Applying sociological and other disciplinary approaches to her study, Lamont won the right to observe peer review panels that are normally closed to all outsiders. And she was able to interview peer review panelists before and after their meetings, examine notes of reviewers before and after decision-making meetings, and gain access to information on the outcomes of these decisions.

The result is How Professors Think: Inside the Curious World of Academic Judgment (Harvard University Press), which aims to expose what goes on behind the closed doors where funds are allocated and careers can be made.

 

A statistical model is proposed for the analysis of peer-review ratings of R01 grant applications submitted to the National Institutes of Health. Innovations of this model include parameters that reflect differences in reviewer scoring patterns, a mechanism to account for the transfer of information from an application's preliminary ratings and group discussion to final ratings provided by all panel members and posterior estimates of the uncertainty associated with proposal ratings. Application of this model to recent R01 rating data suggests that statistical adjustments to panel rating data would lead to a 25% change in the pool of funded proposals. Viewed more broadly, the methodology proposed in this article provides a general framework for the analysis of data collected interactively from expert panels through the use of the Delphi method and related procedures.

Page taken from the University of Pennsylvania's PORT webpage. Penn Online Reserach Tutorial (PORT) can be used as an introduction on how to conduct research for a paper.