Proposal Review Process

To provide greater transparency on the peer review process for ASHE 2021, we are outlining the steps in this 2021 peer-review process. As is a regular practice, Program Committee members ensure consistency and clarity in reviews submitted.
 

Assignment of Reviewers

  • Beginning on the submission deadline date, Section and Pre-Conference Chairs will review applications for reviewers to their respective section or pre-conference. They ensure applicants are current ASHE members and meet the requirements to be a reviewer (e.g., have a terminal degree, are a doctoral candidate, or are a doctoral student) and have provided enough information (e.g., recent publications, methodological expertise) to allow them to appropriately match them to a proposal(s).

  • Once volunteers are accepted, chairs will begin to review proposals. This typically begins with a skim of the title, abstract, keywords, and (new for 2021) the proposal’s suggested reviewer pool. Chairs will begin to match reviewers to proposals. The system shows chairs the number of reviews each person has volunteered to complete across all sections and pre-conferences and limits the total number of assignments to the maximum the person signed up for or 12 if they volunteered for more than 12.

  • If a chair is unable to match a proposal to a reviewer in the applicant pool, they may reach out to someone who is qualified to be a reviewer to ask them to serve. This often happens with niche methods or research topics.

  • The guidance provided to chairs regarding reviewer assignment is: “While having all three reviewers with methodological and topical expertise for each proposal is ideal, it’s seldom possible. We suggest that you assign at least one person who has methodological expertise and one person who has topical expertise to the proposal. If these qualifications are met, your third reviewer may be more of a “generalist.””

  • For 2021, the reviewer assignment criteria are as follows:

    • Interactive Symposia, Self-Designed Paper Sessions, Research Papers, Scholarly Papers, and PVDS sessions should have at least 2 doctoral/terminal degree holders (e.g. PhD, EdD, JD, etc.) and may have one doctoral candidate.

    • Roundtables and Posters sessions should have at least one doctoral/terminal degree holders and can have up to two graduate students.

  • Once chairs have completed reviewer assignments (over the course of about a two-week period), the Program Committee Co-Chairs and ASHE Staff review assignments. This is a check to ensure reviewers meet the established criteria and no reviewer is assigned to more reviews than they volunteered for and no more than 12 overall.

  • Also during this process, if a section chair or pre-conference chair is listed on a proposal in their section, the proposal is assigned to the Program Committee Co-Chairs for reviewer assignment in accordance with the ASHE Conflict of Interest Policy. In this instance, Program Committee Co-Chairs will also make an accept/reject a recommendation to the chairs based on reviewer feedback; the ASHE Staff will mask the reviews in the Conference Portal so that the respective chair will only have access to the feedback but not information about the reviewers.

Peer Review

  • Once reviews are sent out to reviewers, reviewers will once again (as in 2020) have the opportunity to return reviews that they either feel they are not qualified to review (e.g., there was a mismatch in the reviewer assignment) and/or they believe they have a conflict of interest (e.g., they recognize the work or writing of a close colleague).

  • Chairs monitor reviewer progress throughout the review period of about three weeks. When reviewers are unable to complete reviews, chairs tap into “back up” reviewers as needed. These reviewers must also be qualified to review.

Making Accept/Reject Decisions

  • When the review period is closed, chairs begin to review the reviews. This includes determining alignment in a single review between the scores, comments, and recommendation as well as between reviews. If there is a mismatch within a single review or between reviewers, the chair may either accept the recommendation of two of the three reviews or assign a fourth reviewer.

  • If in reviewing comments, chairs are made aware of potential plagiarism or inappropriate comments, this is immediately referred to the ASHE Executive Director for investigation. In cases of the latter, the review may be deleted and assigned to a fourth reviewer; the other reviews of this reviewer will also be examined.

  • The guidance provided to chairs is: “You should strongly consider proposals that receive a score of 4.5 and higher for acceptance. Proposals that receive scores of 3.5 and lower should be strongly considered for rejection. Proposals which receive scores of 3.51 to 4.49 should be evaluated more closely.” In instances when a chair wants to reject a proposal with an average score 4.0 or accept a proposal with an average score of 3.5, they must consult with the Program Committee Co-Chairs.

  • Section chairs are provided with an allocation based on the number of submissions to their section. For example, if the International Higher Education section receives 6% of the submissions, they will be allocated 6% of sessions within the general conference. The number of sessions is determined by the number of rooms available (for an in-person conference) and the number of time slots. Pre-Conference Chairs work with the ASHE Staff on their allocation based upon room availability and time. In both instances, chairs are granted the ability to create sessions as they deem appropriate. For example, a session may be one interactive symposium or three research/scholarly papers. Posters and roundtables have a different allocation based on space and time.

  • After initial accept and reject decisions are made, chairs begin to create sessions from individual submissions (i.e., research papers, scholarly papers, roundtables). In some instances, an individual submission that would otherwise be accepted may not fit into a created session or there may not be space for a session. When this happens, the chairs provide this information to the Program Committee Co-Chairs in the rare case that space becomes available. This is when chairs also assign discussants and session chairs based on those who volunteered and (new for 2021) recommendations of submitters.

  • In instances when a research or scholarly paper proposal has indicated they would be considered for a poster and/or roundtable, chairs may accept the proposal as this type depending on space availability. As such, a paper proposal may have received lower scores or a reject decision based on the paper rubric, but is well suited for a poster or roundtable.

  • ​After chairs have made their accept/reject decisions and created sessions, the Program Committee Co-Chairs and ASHE Staff review any outliers (e.g., proposals above 4.0 that were rejected and proposals below 3.5 that were accepted), ensure allocations were met, and review proposals that chairs wanted to accept but did not have space. 

  • In early July, notifications are sent out by the ASHE Office to submitters and assigned chairs and discussants.

Scheduling

  • Once decisions are sent out, the ASHE Staff begins to build the General Conference schedule. This is a month-long process that involves ensuring presenters are not double-booked and that there is topical variety within time blocks within the space and time available. With over 150 sessions and over 1,000 presenters, this is a detailed process and as such we are unable to accept requests for presentation times.

  • Pre-Conference Chairs create their schedule with assistance from the ASHE Staff. Pre-Conference Chairs typically create their schedule prior to accept/reject decisions being sent out based on room and time availability. Just as with the General Conference, we are unable to accept requests for presentation times for Pre-Conferences.