How To FAccT Review

Related: HOW TO FAccT Area Chair

Why do I need to write a review?

Your review is important for two reasons:

1. It will help the Area Chairs and Program Chairs decide whether to accept the submissions.


2. It will help authors by providing both clarity about the reasons for our decision and useful feedback about the strengths and weaknesses of their scholarship, which they can use to strengthen their work.

Please make your review as detailed and informative as possible -- short, superficial reviews that venture uninformed opinions or guesses are worse than no review, since they may result in the rejection of a high-quality submission.

FAccT is a community, and every paper submitted to the conference is submitted out of a desire to be part of that community. Reviewers are already de facto part of the FAccT community. Please be welcoming to those who are trying to join you, or who are already here. Reviews can convey all the information necessary to inform a considered editorial judgment without making anyone feel bad about themselves or their work. Please re-read your reviews before you submit them, and make sure that the tone is constructive and kind, even if the verdict is critical.

What belongs in my review?

Please describe and consider the strengths of the submission. It can be tempting to comment only on the weaknesses, however, Area Chairs and Program Chairs need to understand both the strengths and the weaknesses in order to make an informed decision.

1. Summary of contribution: Summarize the main ideas of the submission and explain any contributions to the literature, especially in relation to previous work at FAT*/FAccT as well as other archival conferences and discipline-specific journals. This section helps to ground reviews in the full details of the paper, reminds us why the paper matters, and is invaluable to Area Chairs and Program Chairs.
2. Quality: FAccT is committed to strengthening the disciplinary and interdisciplinary areas it touches by maintaining a high standard of quality in the submissions it accepts. Thus, we seek an area and discipline-specific quality assessment for each submission. Please see Quality Assessment Criteria for more guidance on this criterion.
3. Clarity: A superbly written paper provides enough information for an expert reader to reproduce its results or reach similar conclusions based on the analysis, and for a novice reader to understand the basics of the work. Please see Clarity Assessment Criteria for detailed suggestions.
4. Originality: Strong work makes an original contribution. Please see Originality Assessment Criteria for more information.
5. Scholarship: Strong work cites relevant prior work, and makes clear how it relates to that work. For more guidance, see Scholarship: Detailed Criteria.
6. Significance/Impact: Strong work makes a contribution that is important to the FAccT academic field. See Significance: Detailed Criteria.
7. Relevance: If you believe that a submission is out of scope for FAccT, then please justify this judgement appropriately. In formulating this judgment, you may find it helpful to review the Areas of Interest found in the Call for Papers. Note that this list of topics was not meant to be all-inclusive. We welcome submissions that address other important problems surrounding the fairness, accountability and transparency of socio-technical systems.

Overall Score

Please provide an “Overall Score” for each submission.

Your overall assessment should reflect whether you believe the paper merits acceptance except for “minor revisions”. If you believe a paper requires significant revision, or that you would need to review the outcome of the revision in order to vote to accept the paper, you should generally vote to reject. There is one key exception. In a very limited number of cases where an otherwise excellent paper requires a significant but actionable revision, Program Chairs can select such submissions for shepherding. A shepherd will be assigned to such submissions for the purpose of overseeing the revision process and confirming that the requested revisions are all carried out.

The Area Chairs and Program Chairs will interpret Overall Scores via the following scale:

• 3: Would be a top 10% accepted paper at venues I highly respect. A “must accept”. I will fight for accepting this submission.
• 2: Would be a top 50% accepted paper at venues I highly respect. A very good submission, clear accept. I vote and argue for accepting this submission.
• 1: Would be an accepted paper at a highly respected venue in my discipline. A good submission; an accept. I vote for accepting this submission, but would not be upset if it were rejected.
• 0: Marginally above acceptance threshold; an accept. I tend to vote for accepting this submission, but rejecting it would not be that bad.
• -1: Marginally below acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad.
• -2: A clear reject. I vote and argue for rejecting this submission.
• -3: A “must reject”. Paper is fundamentally flawed (E.g., major results are trivial, wrong, already known, etc.) — I will fight for rejecting this submission.

The focus here is whether the paper’s level of quality allows it to make a significant contribution to key conversations in FAccT related fields. Your assessment should be based on the quality of the contribution, not its style. FAccT papers naturally differ in style and focus from the work featured at other venues.

You should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The “Overall Score” for each submission should reflect your assessment of the submission’s contributions.

Confidence Score

Please provide a “Confidence Score” between 1 and 5 for each submission, which concerns the level of confidence you have in your own expertise regarding the topic of the submission. The Area Chairs and Program Chairs will interpret these scores via the following scale:

• 5: You are absolutely certain about your assessment. You are very familiar with the related work.
• 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
• 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
• 2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
• 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Note: If you feel that your confidence rating is likely to be a 1 at the end of a review due to your lack of expertise in the given subject area, you should notify the Area Chair as early as possible in the process. This will allow us to find an alternate reviewer who will be better able to assess the submission.