Exercise: Peer Review

The foundation of the scientific literature in all disciplines is peer review. All published work is critiqued by other researchers and evaluated on the basis of validity, quality, innovation, and relevance. The process varies by publication and institution, but frequently involves both overall scoring as well as specific feedback and recommended changes.

The process is by no means perfect. Plenty of errors and redundancy make it through to publication, and the system depends upon a vast quantity of volunteer labor supporting a commercial industry. But by and large the discipline of writing for others to critique creates a community standard for due diligence in research.

In this exercise, you will play the part of an individual researcher asked to review a conference paper. Normally, all the papers submitted to a conference are distributed through a network of editors and associate editors, and so the review request would come from an individual editor tasked with finding several reviewers (aka referees) for each paper in their allocation. That editor would combine your review with others to form an overall assessment of acceptance priority. Your comments would be anonymously forwarded to the author to suggest revisions for the final version of the paper. Some conferences are double-blind and so the author identities are also stripped before forwarding to the referees.

The hardest reviews to write are the middling-quality papers: reasonable research premises, promising directions, but incomplete process or poorly argued conclusions. These take a lot of thought to critique constructively.

For the purposes of this exercise, you’ll choose your own paper to review. Obviously, this poses a selection bias since it will by definition be a published paper, but you may still find room to give it a conditional pass or outright rejection. But more importantly , you’ll have to consider possible critiques and suggested improvements. And even for the best papers a reviewer needs to argue positively for acceptance.

Objectives

  1. Become familiar with the process of peer review for conference papers.

  2. Practice in critical reading of a research paper.

  3. Practice writing a peer review.

Steps

  1. Locate a conference paper of interest related to soft robotics which is not either from the course bibliography or one you previously posted. You might try a lateral search from a paper you’ve noted, or exploring references and citations of familiar work, or a fresh search.

  2. Try locating the author and reviewer guidelines for the conference, either from the actual year of publication or the current iteration. If you can’t find the specific guidelines, some representative guideline links appear below. Please read the guidelines to familiarize yourself with the overall process.

  3. I also suggest reading the first 2.5 pages of the Ken Hinckley essay recommended to CHI reviewers for some perspective on the process. (The latter half of this essay shifts more toward the tasks of the editors.)

  4. Read the paper carefully through for content, noting down any questions or ambiguities which occur to you.

  5. Please look up any critical references and at minimum read the abstracts. This will help you gauge relevance and novelty.

  6. Write a review following the review criteria and form below. Unless you are already experienced in the research area, I don’t expect you to identify specific technical errors. But instead you can identify strengths and weaknesses in research process and argument. Be sure to articulate both strengths and weaknesses; a weak paper needs critique, a strong paper still needs a clear argument for acceptance.

General Review Criteria

The following form is a synthesis of typical review criteria, although strongly influenced by the HRI reviewer notes.

  1. Originality. All papers must present original work.

  2. Novelty. We welcome big idea and provocative papers, even if they may not be perfectly implemented. We also welcome papers that bring together different disciplines and approaches.

  3. Relevance. All papers must be relevant to the field.

  4. Soundness. A paper needs to be technologically and/or methodologically sound based on the criteria generally used for that technology or method within a given field.

  5. Technical detail. Please provide adequate details to indicate what was done, how the data were collected, sample size and characteristics, what type of robot was involved, etc. Authors should use correct terminology for their methods to avoid being evaluated against the incorrect set of criteria.

  6. Accessibility. All papers must be written to be accessible for a broad, interdisciplinary/multidisciplinary audience.

Please note by that by completing this review you assert the following:

  1. I agree to keep the submissions I review confidential.

  2. I agree to write constructive and courteous reviews.

  3. I agree to disqualify myself from performing a review if there is a conflict of interest.

  4. I agree to return reviews on time, and to respond to requests from the Chairs or Program Committee should they require more information on or discussion of a particular submission.

  5. I understand that my identity will be known by the Chairs and Committee Members of the categories for which I am reviewing, and that my identity may be disclosed to others who are reviewing the same submission.

  6. I agree to keep the names of other reviewers confidential, should I become aware of them.

Referee Form

The following form is a synthesis of typical review forms, although strongly influenced by the HRI process.

  1. Do you have any conflict of interest in reviewing this paper? A “conflict of interest” is defined as follows:

    1. Ph.D. thesis advisor or advisee

    2. Postdoctoral advisor or advisee

    3. Collaborators or co-authors for the past 48 months

    4. Any other individual or institution with which the investigator has financial ties

    Yes/no. If yes, please disqualify yourself instead of proceeding.

     

  2. Expertise. Provide your expertise in the topic area of this paper.

    • 4 - Expert

    • 3 - Knowledgeable

    • 2 - Passing Knowledge

    • 1 - No Knowledge

    Single choice.

     

  3. Summary. Please summarize what you believe are the paper’s main contributions to the field of soft robotics.

    Please write a short paragraph.

     

  4. Strengths and Weaknesses. What are the main strengths and weaknesses of this work? Does the paper have strengths in originality and novelty?

    Please write a short paragraph.

     

  5. Soundness. Are the ideas, algorithms, results or studies technologically/methodologically sound?

    Please write a short paragraph arguing for the strengths and weaknesses of the work.

     

  6. Related Work. Does the paper adequately describe related and prior work?

    Please write a sentence or short paragraph.

     

  7. Presentation. Is the paper well organized, well written and clearly presented?

    Please write a sentence or short paragraph.

     

  8. Suggestions. Do you have suggestions for improving this paper?

    Please write several paragraphs detailing specific points of the paper which merit reconsideration. Be sure to address the text, figures and tables, mathematics, and grammar and spelling.

     

  9. Comments to Committee (Hidden from authors). Does the paper have enough originality and importance to merit publication? Is the paper relevant to the field? These comments will NOT be sent to the authors:

    Please write one or more paragraphs as needed to justify your review judgement.

     

  10. Overall Rating. Provide your overall rating of the paper (5 is best)

    • 5 - Definite accept: I would argue strongly for accepting this paper.

    • 4 - Probably accept: I would argue for accepting this paper.

    • 3 - Borderline: Overall I would not argue for accepting this paper.

    • 2 - Probably reject: I would argue for rejecting this paper.

    • 1 - Definite reject: I would argue strongly for rejecting this paper.

Deliverables

  1. Blog post with your review.