Proposal Review Process

For 2020, the Program Committee was charged with reviewing and deciding on 1,187 proposals. To help make these decisions, 511 ASHE members completed 3,580 reviews.

This year, the Program Committee made many changes to enhance this process to ensure a quality, constructive, reliable, and equitable process:

  • We provided resources to reviewers including an updated reviewer guide, a webinar focused on best practices, and examples of reliable and constructive reviews.

  • The Program Committee worked to ensure a match between proposals and reviewers. This included assigning proposals based on content and/or methodological expertise.

  • We reduced the number of reviews assigned to reviewers, assigning no more than 12 proposals to any reviewer in the first round of assignments. 

  • To increase our pool of qualified reviewers we required current ASHE members with a terminal degree who submitted a research or scholarly paper to be available to review at least 3 proposals.

  • Once reviewers received their proposals to review, they could return them to the Program Committee if they felt unqualified to complete the review to ensure that reviewers had the appropriate expertise for each proposal.

  • In the midst of COVID19 as well as nationwide protests, we extended review deadlines and reassigned reviews when necessary.

  • The PC co-chairs served as an additional check of proposals with inconsistent scores or scores on the margins for acceptance.

While these steps undoubtedly enhanced the review process, we also want to be fully transparent about how process works overall:

  1. Once proposals were submitted, the Program Committee worked over a 11 day period to assign reviewers who best match proposals based on methodological and/or content expertise.

  2. Reviewer assignments were sent out on May 14 to reviewers.

  3. The Program Committee monitored the process and re-assigned reviews when there was a mismatch or a conflict of interest (e.g., someone knows who submitted the proposal although it is masked, likely a colleague, co-worker, or student).

  4. On June 1, the Program Committee began to ensure all proposals had at least three complete reviews; if not, proposals were assigned to back-up reviewers.

  5. Once a proposal had three reviews, the Program Committee (by section and by pre-con) reviewed the rubric scores, feedback to the author, feedback to the association, and accept/reject recommendation. In this review, section chairs may have determined an additional review was needed and/or comments were unreliable, unconstructive, or otherwise harmful. In this process, we had given guidance to reviewers to consider an average score of 4 or more for accepted proposals. However, when co-chairs made decisions for their sections, average quantitative scores for accepted proposals were also considered with the qualitative comments as well as space available.

  6. Once these decisions were made, section chairs began to create and slate sessions based on topical matches as well as the set allocation per section (which was based on the overall number of submissions to that section and the total breakout sessions). In this process, section chairs also reached out to other sections to create sessions and/or trade proposals that were better matched in another section. In some instances, proposals were deemed acceptable but there weren't readily available matches; this may have led to a proposal being placed with other proposals which also warranted acceptance, but may not be a topical match.

  7. Once all sessions were created, the Program Committee Co-Chairs and ASHE Staff conducted a final review of the sessions, ensuring consistency across sections, that each paper session has a chair and discussant, and that no participant exceed the limit of 5 appearances on the program during the general conference.