19 September 2016

Why are evaluations of research reports unreliable? How are corrupt social norms hampering scientific progress? And what can journal editors do to fix these problems? These were the overarching themes in this year’s Erasmus Management Lecture delivered by William H. Starbuck, Professor Emeritus at University of Oregon. At the conclusion of the talk, a panel of ERIM academics with editorial experience at prestigious research journals provided insights regarding the model of science they felt organizational scholars should follow. 

Text Krishnan Nair |  Photos Chris Gorzeman

Professor Starbuck started his lecture by discussing the main challenge associated with the evaluation of research reports. Reviewers agree in principle that research articles should be well-written, interesting, and conducted using appropriate methodologies. However, this agreement often does not translate to evaluations of specific manuscripts. Assessments by different reviewers are rarely correlated except in the case of exceptional manuscripts.   

Why is this the case? Professor Starbuck highlighted three main reasons. First, human minds are limited in their ability to process complex information. Although individuals are only able to handle two or three variables at a time, reviewers are expected to assess research reports on a number of dimensions that do not have a clear hierarchy of importance. Second, reviewers rely on symbols of social approval when assessing manuscripts. For example, they tend to show a clear bias against unknown researchers from less prestigious universities. Third, reviewers are less likely to accept radically new ideas that challenge existing assumptions. This tendency is strengthened when research findings contradict reviewers’ own published findings. 

Dishonesty in Research

Professor Starbuck also discussed the existence of dishonest research practices that are prevalent in the field. Researchers sometimes formulate hypotheses after the results are known, decide to collect additional data if their hypotheses fail to achieve “statistical significance”, and suppress null findings. Some of these practices take place with the implicit or explicit consent of journal reviewers. Although statisticians have been tackling some of these issues for over 80 years, their success has been very limited.    

Potential Solutions

While admitting that change will be difficult, Professor Starbuck provided a number of ways in which journal editors can improve the review process and foster more honest research practices. Editors can improve the evaluation of manuscripts by focusing reviewers on simpler assignments involving a few relevant issues, training reviewers through formal programs, and assigning the assessment of methodology and writing style to specialist reviewers. Editors can promote greater honesty in research by looking critically at studies that support a high percentage of hypothesized effects, insisting that researchers disclose unexpected discoveries made during data analysis, and requiring that papers attribute the addition of hypotheses to editors and reviewers. Editors who are hesitant to change established practices can team-up with fellow editors at other journals to reduce risks.

With Professor Starbuck’s talk fresh in their minds, a panel of ERIM scholars with editorial experience weighed in on the research model they felt organizational researchers should pursue. The discussion was moderated by Professor Joep Cornelissen, associate editor of the Academy of Management Review.

Professor Patrick Groenen criticized the overemphasis in the “classical” experimental paradigm and argued that journal editors should be more open to exploratory research. Professor Groenen also highlighted the importance of replicating existing research for the advancement of science. Professor Pursey Heugens argued that a contingency model rather than a “classical” experimental model should be pursued. Professor Heugens also discussed the role of meta-analyses in the progress of scientific knowledge and the importance of examining effect sizes. Professor Stefan Stremersch pointed out the trade-offs between scholarly and practical relevance, readability, and rigor in management research. Professor Stremersch ended his commentary by advising junior scholars to challenge existing ideas in their pursuit of a research career. 

The audience also helped fuel further discussion. Notably, Professor Michael Jensen (U. Mich.) asked the panel whether journal editors can effectively improve research practices without university departments making corresponding changes in scholars’ incentives. While some of the panelists reemphasized their view that journal editors are important in inducing reforms, all agreed that changes at the departmental level are also a crucial piece of the puzzle if we are to promote better research practices.