Pakistan Journal of Medical Sciences

Published by : PROFESSIONAL MEDICAL PUBLICATIONS

ISSN 1681-715X

HOME   |   SEARCH   |   CURRENT ISSUE   |   PAST ISSUES

-

ORIGINAL ARTICLE

-

Volume 22

April - June 2006

Number 2


 

Abstract

PDF of this Article

Characteristics of Reviewers and Quality of Reviews:
A retrospective study of Reviewers at
Pakistan Journal of Medical Sciences

Shaukat Ali Jawaid1, Masood Jawaid2, Maqbool H. Jafary3

Abstract

Objective: To study the characteristics, evaluate the performance of reviewers and quality of their reviews as being helpful for the authors and the editors to make a final decision on the manuscripts.

Material and Methods: Information was retrieved from 404 peer review proformas filled in by the Reviewers involving 377 manuscripts from July 2003 to November 2005.The information collected included age, sex, academic affiliation, number of publications, training local or overseas, punctuality, hand written notes or typed, participation in peer review workshops etc. Rating scale of 1-5 (1 meant casual and 5 detailed comments) was used to evaluate their comments regarding originality of manuscripts, comments regarding strong and weak points, tables, usefulness of the manuscript and interpretation of results.

Results: A total of sixty eight reviewers were evaluated. Majority of the reviewers 46(67.7%) were between the age of 40-60 years. Only three were female. Four (5.9%) reviewers had more than hundred publications to their credit while twelve (17.6%) had 51-100, thirty two (47%) had between 25-50 and twenty (29.4%) had less than twenty five publications. Fifty one (75%) were affiliated with academic institutions. Those who attended peer review workshops (38 out of 68) did a better review. Eleven (16.2%) did not wish to disclose their identity for various reasons. Based on their consistency and quality of reviews, fifteen reviewers were rated excellent by the Editors which included eight retired medical teachers, thirty three good and twenty as average reviewers.

Conclusions and suggestions: Performance of reviewers and quality of their reviews was mostly dependent on their interest in academics. Best reviewers were retired medical teachers personally known to the editors, those in private sector over 60 years of age and those who were exposed to any training programme in peer review system. There is a general tendency among the Editors to overuse efficient, punctual reviewers which must be checked to avoid burn out syndrome which ultimately affects the quality of their reviews. Editors should be consistently on the look out for good quality reviewers to expand their Reviewers Database which will eventually help improve the overall quality of the manuscripts and standard of the journal.

Key Words: Peer Review, Characteristics of reviewers, Quality of reviews.

Pak J Med Sci April - June 2006 Vol. 22 No. 2 101 -106


1. Mr. Shaukat Ali Jawaid
Managing Editor

2. Dr. Masood Jawaid
Assistant Editor

3. Dr. Maqbool H. Jafary
Chairman Editorial Board

1-3: Pakistan Journal of Medical Sciences,
Karachi. Pakistan.
E-Mail: pjms@pjms.com.pk

Correspondence:
Shaukat Ali Jawaid
E-Mail: shaukat@pjms.com.pk


INTRODUCTION

The quality and usefulness of any medical journal depends on the quality of manuscripts it attracts from the authors, the competence and critique of its editorial staff including reviewers besides judgment of the Editor. The reviewers have a very important role in peer review system. The Editors are responsible for creating, maintaining and sustaining an experienced, knowledgeable and competent group of reviewers. Potential reviewers are identified through personal contact at professional meetings, personal acquaintances, looking at editorial boards of various publications and from amongst those who contribute their manuscripts to the journal. In some cases those selected to serve as Reviewers may not have met or be known to the Editor except through the quality of their articles and publications.

A medical journal is as good as its Editorial Board and reviewers. The reviewer’s guidance to the editor can be extremely helpful. Based on their comments the editors most often take the final decision to accept, reject or return the manuscript for revision. As such the success of peer review is totally dependent on the expertise, dedication and fair mindedness of reviewers.

When National Library of Medicine created Index Medicus in 1960s, peer review was not a requirement for inclusion of any journal although it was a highly weighted factor which remains even today.1

World Association of Medical Editors (WAME) was formed in 1995 to improve the quality of medical journals particularly those which has limited resources. It conducts its activities on internet and is a very useful forum for education and training of Editors as well as reviewers. Recently there was an excellent debate on peer review on WAME LISTSERVE. Being reviewer is viewed differently by different reviewers. Some do it because their employer or academic institutions they are affiliated with, give them some points; some do it as it provides them an opportunity to learn and update themselves while there are others who will review the manuscripts as a favour to the editor and still others find it an honour to be on the list of reviewers as they get recognized as experts in their specialty. The Editors particularly in developing, Third World countries sometimes find themselves in a fix when they receive a manuscript on a subject in which they do not have a reviewer in the database who is competent in that particular discipline. This leads to un-necessary delay in review process. If the editors are honest enough to tell such difficulties to the authors, they have the risk of losing them, encouraging them to submit their manuscripts to other journals. This can undermine image of the journal and discourage authors from submitting manuscripts in future. These are the real dilemmas faced by the editors and it also emphasizes the importance of peer review system.2

It was in this background that we decided to conduct this study to find out the characteristic’s of our reviewers, evaluate their performance and assess the quality of their reviews as being helpful for the authors and the editors to make a final decision on the manuscripts. It was an observational study.

MATERIALS AND METHODS

We use a comprehensive peer review proforma designed in line with the uniform requirements for manuscripts submitted to biomedical journals as recommended by the International Committee of Medical Journal Editors3 (ICMJE). It is routinely sent to the reviewers along with the manuscript. We assessed 404 such proformas filled in by the reviewers involving 377 manuscripts from July 2003 to November 2005. In addition to reviewing the quality of the contents, the reviewers are also supposed to grade the manuscript as poor, satisfactory, good and excellent. If approved, whether they would like it to be published immediately or processed in routine. If not approved, they must give reasons or guide the authors how to revise and improve the manuscript.

Reviewer’s professional and demographic profile was retrieved from their Curriculum Vitae in our database. The information collected included age, sex, number of publications, and training (local or overseas), affiliation with academic institution, retired faculty members or in private practice. It was also ascertained whether the reviewers sent hand written notes or typed, information about misplacement of manuscripts, punctuality, participation in peer review workshops, whether they corrected the language or left it to be done by the editors. A rating scale of 1-5 (1 meant casual and 5 detailed comment) was used to evaluate the comments by the reviewers regarding originality of manuscripts, strong and weak points, tables, and usefulness of the manuscript besides interpretation of results. Overall tone of the reviewers was also assessed as harsh or courteous. The data was analyzed by using SPSS Version 10.

RESULTS

A total of sixty eight reviewers were included in the evaluation. Two reviewers (2.9%) were less than forty years of age and twenty (29.4%) were over the age of sixty years. However, majority of the reviewers 46(67.7%) were between 40-60 years of age. Only three were female. Four reviewers had more than hundred publications to their credit while twelve had between 51-100, thirty two had between 25-50 and twenty had less than twenty five publications. Fifty one (75%) was affiliated with academic institutions whereas 17 (25%) were in private practice. Mean number of manuscripts reviewed by each reviewer during the review period was 5.95±6.97. Fifty one (75%) had their postgraduate training overseas while the remaining 17 had local postgraduate qualifications. Thirty eight (55.9%) attended the peer review workshop as against thirty (44.1%) who were not exposed to any training in peer review system. Thirty six (52.9%) wished the editors to correct the language of the authors. Thirty six (52.9%) failed to return manuscripts in time and needed reminders while eighteen (26.5%) misplaced the manuscripts and had to be sent duplicate copies. Eleven (16.2%) did not wish to disclose their identity for various reasons. Comments on 27 manuscripts did not help the editors to make a final decision and necessitated second opinion from another reviewer. Majority of the reviewers (80%) sent their comments in the form of hand-written notes while about 20% sent typed comments along with detailed analysis of the results and usefulness or otherwise of the manuscript. According to this rating scale, majority of our reviewers (mean score 2.3) did not comment on originality of the manuscript and usefulness of the study. (Mean score 2.1)(Table-I).

Based on their consistency and quality of reviews, fifteen reviewers (22.05%) were rated excellent by the Editors while thirty three (48.52%) were rated as good and twenty (29.41%) as average reviewers.

Most of the retired medical teachers who had their training overseas, were some how exposed to medial writing. They proved to be good reviewers whereas some of those in private practice but interested in academics or had part time affiliations with some academic institutions also did better reviews. However, they had the draw back of not having any secretarial assistance, hence most of their reviews, comments were hand written instead of being typed.

Training of the reviewers, we found definitely improve the quality of their reviews and they were also keen to attend such workshops. Hence we organized workshops on Peer Review in collaboration with Pakistan Medical Journalists Association at Karachi, Lahore and Rawalpindi-Islamabad which were largely attended by Editors and Reviewers4-6 BMJ also provides training material to the reviewers which they have developed to evaluate the effects of training on quality of reviews. They routinely run four workshops every year charging nominal fee. The reviewers do find it useful and interesting.7,8

Overall tone of the reviewers, we found, was good and courteous. Affiliation with academic institutions did not have much impact on the quality of reviews. A few reviewers mostly from overseas used e-mail to convey their comments. On-line review of manuscript is not practiced by most of our reviewers as yet though it is time saving and cost effective.

DISCUSSION

This is the first study from Pakistan reviewing the reviewers, evaluating their performance and the quality of their reviews. There have been many studies on this subject overseas but our findings are quite in contrast to the findings in those studies conducted in the West. Study by Kliewer et al found that reviewers from academic institutions were better than those in private practice.9 However, our study shows that affiliation with academic institution did not have much impact on the quality of their Reviews. Study by Siegelman and colleagues observed that young reviewers may be more enthusiastic in reviewing the manuscripts and do more detailed review. They do it so because they are keen to seek recognition from the editors. In some cases they may also be better educated in study designs, biostatistics and new imaging techniques. As the reviewers get older, their enthusiasm for such academic activities fades, hence young reviewers need to be encouraged. The authors concluded that best reviewers were younger individuals from academic institutions.10 In our study those rated as excellent reviewers were over sixty years of age, retired medical teachers or those in private practice interested in academics. However since most of them were personally known to the editorial team for over three decades, they were most obliging. This might have influenced our results to a great extent as such response cannot be expected from retired medical teachers as a routine.

Gilbert et al, in their study observed that success of the peer review system is mostly dependent on the Editors who are supposed to keep track of reviewers performance, protect authors from the reviewers particularly from reviewers who are not so kind, are not so helpful and at times very critical.11 Observations in our study are almost similar and we always try to be helpful to the authors. Not only that our reviewers are also requested to encourage the authors giving them useful suggestions to improve their manuscript if they have certain deficiencies. At times if the study has useful data, some of our reviewers even do not mind re-writing certain portion of the manuscript to encourage authors.

Many other studies have also supported the view that younger reviewers produce good reviews.12-15 Which are again in contrast to our findings although a few young reviewers did a better job but these were exceptions. Another study found that male reviewers were more likely to give extreme scores on manuscripts as compared to female reviewers but such difference did not affect the ultimate disposition of the manuscript.15 Since in our study we had only three female reviewers, any difference in their comments is difficult to compare.

Some studies have highlighted the inadequacies of poor review and their limitations to improve the quality of the manuscripts. However, there are very few studies which have evaluated the effects of interventions to improve peer review and effects of training.16 Our study showed that those who attended peer review workshops (38 out of 68) did a better review.

We practice open peer review which offers many benefits.17 It ensures increased accountability, fairness and transparency. It leads to better quality of review. Study by Rooyen SV and colleagues highlighted that authors surveyed were in favour of open peer review. These authors felt that open peer review is feasible and it won’t be detrimental to the quality of reviews. They also used a review quality instrument.18 We used a slightly modified instrument as a rating scale to assess our reviewers and the quality of their reviews.

Stossel and colleagues also found that reviewers with lower rank and academic qualifications did better review than highly placed academicians and they were also less likely to refuse to review manuscripts.12 We had similar observations as regards keenness of the reviewers with lower qualifications from academic institutons but did not involve them without any training and exposure to peer review process.

Another study conducted in 1993 showed that good peer reviewers were under the age of forty years, they were affiliated with academic institutions, they were known to the editors which is again in contrast to our findings.13 However yet another study in 1998 found that good reviewers were between the age of 40-60 years. Moreover those who had their training in North America, were trained in epidemiology and statistics were better reviewers.15 Our study findings are similar to the extent that those who had some training in epidemiology and biostatistics proved to be better as regards the quality of their reviews.

Findings in our study differ from findings of most of the above studies because most of our faculty members write under compulsion just to get selected or for promotion, hence the quality of their manuscripts is not so good. Most of them start writing when they are already over forty years of age. One has to be a good writer to become a good reviewer; hence most often it is difficult to find competent young reviewers from the academic institutions though exceptions particularly in basic sciences are there. We had only three female reviewers. This number is low again because we have very few female specialists and faculty members. Even among those female specialists, only a few are interested in academics that have some contributions to medical literature. Hence, we could involve only those who were interested and capable of reviewing the manuscripts.

Suggestions

1. A proactive approach by the Editors can improve their reviewers’ database.

2. The editors should nurture new peer reviewers by offering training courses inviting those persons who are interested in peer reviewing.

3. Peer reviewers may not know as to what is expected of them, hence it is important to provide some guidelines to the reviewers so that their comments are constructive and helpful for the authors as well as the editors.

4. Journal’s reviewer’s database must be constantly scrutinized, updated and replenished.

5. The editors should avoid overusing efficient, punctual reviewers to avoid burn out syndrome which ultimately may affect the quality of their reviews.

6. Editors should not have fixed views as to what kind of reviewers might turn out to be good reviewers because these characteristics could vary from one setting to another. However, editors should continue to evaluate their reviewers but it is difficult to predict their future performance.

Limitations of the study: Pakistan Journal of Medical Sciences is a general medical journal. Size of the reviewers is rather small. The results may have been different if a similar study was conducted in a specialty journal. No informed consent was taken from the reviewers who were included in this study as it was a retrospective observational study and data was retrieved from the Peer Review Proforma’s. It is also a subjective assessment and evaluation of the performance of the reviewers and the quality of their reviews by the editors. We are in the process of expanding our Reviewers database inducting some reviewers from overseas based on the quality of their manuscripts which they have been contributing. They are not at all known to the editors and have been selected just on merit and their willingness to review the manuscripts. We intend to undertake a prospective study after obtaining informed consent from the reviewers. A combined study about the quality of the reviews involving a few leading peer reviewed journals regularly published from Pakistan, which we intend to do, will give a national perspective. We have also observed that some of the reviewers which were evaluated as excellent, being over the age of sixty years can review only a few manuscripts. That has necessitated increase in reviewers database. Hence the findings of our next study may be different than those being reported from this study.

CONCLUSIONS

In our study performance of reviewers and quality of their reviews was mostly dependent on their interest in academics although affiliation with academic institution did not have much impact. Best reviewers were retired medical teachers and those in private practice and most of them were over sixty years of age.

REFERENCES

1. Bordage G, Caelleigh SA. A tool for reviewer’s criteria for research manuscripts. Academic Medicine 2001; 76:904-8.

2. Jawaid SA. Problems faced by editors of peer review medical journals. Saudi Med J 2004; 25, Suppl 1: 447-51.

3. Uniform requirements for manuscripts submitted to biomedical journals as approved by ICMJE (International Committee of Medical Journal Editors) revised in October 2004. www.icmje.org

4. Jawaid SA. Proceedings of workshop on Peer Review System. Pak J Med Sci 2002; 18(4) 328-33.

5. Pakistan Medical Journalists Association (PMJA) workshop on peer review at Fatima Jinnah Medical College, Lahore, Pakistan. Pulse International 2005; 6(7) 1-9.

6. Workshop on Peer Review System at Pakistan Institute of Medical Sciences, Islamabad, Pakistan. Pulse International 2005; 6(10) 1-9.

7. Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: Randomized controlled trial. BMJ 2004; 328:673-5.

8. Schroter S, Groves T. BMJ training in peer reviewers (Editorial) BMJ 2004; 328:658.

9. Kliewer MA, Freed KS, DeLong DM, Pickhard PJ, Provenzale JM. Reviewing the reviewers; comparison of review quality and reviewers characteristics at the American Journal of Roentgenology 2005; 184:1731-5.

10. Sieglman SS. Assassins and Zealots; Variations in peer review. Radiology 1991; 178:637-42.

11. Gilbert JR, Williams Es, Lundberg GD. Is there gender bias in JAMA peer review process? JAMA 1994; 272:139-42.

12. Stossel TP. Reviewer status and review quality. Experience of the Journal of Clinical Investigations, New Eng J Med 1985; 312:658-9.

13. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good quality reviews. J Gen Int Med 1993; 8:422-8.

14. Nylenna M, Riis P, Kdarlsson Y. Multiple blinded reviews of the same two manuscripts, JAMA 1994; 272:149-51.

15. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for general medical journal? JAMA 1998; 280:231-3.

16. Rennie D. Editorial Peer Review: its development and rationale In: Godliee F; Jefferson T. Eds. Peer Review in Health Sciences, London BMJ Books 1999.

17. Jawaid SA, Jafary MH. Peer Review to improve quality of manuscripts: Are we ready for open peer review? (Editorial) Pak J Med Sci 2002; 18(3) 173-80.

18. Rooyen SV, Godlee F, Evans S, Black N, Smith R. Effects of open peer review and on reviewers recommendations: a randomized trial. BMJ 1999; 318:23-7.


HOME   |   SEARCH   |   CURRENT ISSUE   |   PAST ISSUES

Professional Medical Publications
Room No. 522, 5th Floor, Panorama Centre
Building No. 2, P.O. Box 8766, Saddar, Karachi - Pakistan.
Phones : 5688791, 5689285 Fax : 5689860
pjms@pjms.com.pk