FAccT 2022 Reviewing Guide

Why do I need to write a review?

Your review is important for two reasons:

1. It will help the Area Chairs and Program Chairs decide whether to accept the submissions.


2. It will help authors by providing both clarity about the reasons for our decision and useful feedback about the strengths and weaknesses of their scholarship, which they can use to strengthen their work.

Please make your review as detailed and informative as possible -- short, superficial reviews that venture uninformed opinions or guesses are worse than no review, since they may result in the rejection of a high-quality submission.

FAccT is a community, and every paper submitted to the conference is submitted out of a desire to be part of that community. Reviewers are already de facto part of the FAccT community. Please be welcoming to those who are trying to join you, or who are already here. Reviews can convey all the information necessary to inform a considered editorial judgment without making anyone feel bad about themselves or their work. Please re-read your reviews before you submit them, and make sure that the tone is constructive and kind, even if the verdict is critical.

Due dates for return of reviews

Reviews are due by March 21 11:59 pm AoE.

Area Chairs should return their meta-reviews by April 5 11:59 pm AoE.

Ethics Review

Papers at FAccT should be grounded in high ethical standards. If you think there are reasons for closer scrutiny of the submission on ethical grounds, then please flag that in the ethical review section of the reviewer form. You might consider whether the methods producing the research met appropriate ethical standards ('upstream' questions of research ethics), and whether the research has the potential to be used, misused or abused, in ways that might subsequently lead to individual or collective harm ('downstream' questions of ethics proper).

Identification of Authors

Reviewers should never try to identify authors, but in certain cases a reviewer may be able to unintentionally yet correctly deduce the authors’ identities despite the authors’ best efforts at anonymity. If this happens, please do not divulge the identities to anyone, but do tell the Area Chair that this has happened and make a note of this in the “Confidential Comments to Program Chairs” text field when you submit your review. Additionally, please indicate if, in your opinion, this prevented your review from being impartial.

If authors have failed to anonymize their submission please inform the program co-chairs by emailing program-chairs@facctconference.org.

What belongs in my review?

Please describe and consider the strengths of the submission. It can be tempting to comment only on the weaknesses. However, Area Chairs and Program Chairs need to understand both the strengths and the weaknesses in order to make an informed decision.




1. Summary of contribution

Please summarize the main ideas of the submission and explain any contributions to the literature, especially in relation to previous work at FAT*/FAccT as well as other archival conferences and discipline-specific journals. This section helps to ground reviews in the full details of the paper, reminds us why the paper matters, and is invaluable to Area Chairs and Program Chairs.



2. Quality

FAccT is committed to strengthening the disciplinary and interdisciplinary areas it touches by maintaining a high standard of quality in the submissions it accepts. Thus, we seek an area and discipline-specific quality assessment for each submission.


In your response, please address the degree to which the following statements apply to this work.

• The work presented is comprehensive. 
• Claims are well supported by theoretical analysis, evidence, argument, and/or experimental results.
• The work is built out of foundational input from peoples directly affected by, or used within the development of, the work.
• The work incorporates foresight on populations most likely to bear disproportionate negative effects directly related to this work.
• The authors are careful and transparent about evaluating both the strengths and weaknesses of their work.



3. Clarity

A superbly written paper provides enough information for an expert reader to reproduce its results or reach similar conclusions based on the analysis, and for a novice reader to understand the basics of the work.


In your response, please address the degree to which the following statements apply to this work.

• The submission is clearly written and well organized.
• The submission adequately informs the reader about the research.
• The claims/arguments are explored in sufficient depth.



4. Originality

Strong work makes an original contribution.



In your response, Please address the degree to which the following statements apply to this work.

• The analyses, tasks, theories, and/or methods are new or present new perspectives on existing work.
• The work is a novel combination of well-known research.



5. Scholarship

Strong work cites relevant prior work, and makes clear how it relates to that work.

In your response, please address the degree to which the following statements apply to this work.

• The work is a novel combination of well-known research.
• The work situates itself in the context of related work.
• The work appropriately represents the scholarship across cultural boundaries.
• It is clear how this work differs from previous contributions.
• Related work is adequately cited.



6. Significance/Impact

Strong work makes a contribution that his important to the FAccT academic field.

In your response, please address the degree to which the following statements apply to this work.

• The results or conclusions are important for the FAccT academic field.
• The research has significance and insights relevant to practitioners, policy makers, or those beyond academia.
• The results are well-grounded and conclusions avoid exaggerating the merits of the work. 
• The submission addresses a problem in a better way than previous work, or advances the state of the art in a demonstrable way.
• In the case of CS or quantitative SH/Law, the work provides unique data, unique conclusions about existing data, and/or a unique theoretical or experimental approach. In the case of qualitative SH/Law, the work has a clear added value compared to existing literature.




7. Relevance

If you believe that a submission is out of scope for FAccT, then please justify this judgement appropriately. In formulating this judgment, you may find it helpful to review the Areas of Interest found in the Call for Papers . Note that this list of topics was not meant to be all-inclusive. We welcome submissions that address other important problems surrounding the fairness, accountability and transparency of socio-technical systems.
• The submission has relevance to FAccT. [Scale from Strongly Agree … Strongly Disagree]
• [Optional] Additional text comments on relevance .



8. Overall score

Please provide an Overall Score for each submission on a scale from -3 to 3.

Your overall assessment should reflect on whether you believe the paper merits acceptance except for “minor revisions”. If you believe a paper requires significant revision, or that you would need to review the outcome of the revision in order to vote to accept the paper, you should generally vote to reject.

There is one key exception. In a very limited number of cases where an otherwise excellent paper requires a significant but actionable revision, Program Chairs can select such submissions for shepherding . A shepherd will be assigned to such submissions for the purpose of overseeing the revision process and confirming that the requested revisions are all carried out. 

The Area Chairs and Program Chairs will interpret Overall Scores via the following scale:

• 3: Would be a top 10% accepted paper at venues I highly respect. A “must accept”. I will fight for accepting this submission.
• 2: Would be a top 50% accepted paper at venues I highly respect. A very good submission, clear accept. I vote and argue for accepting this submission.
• 1:Would be an accepted paper at highly respected venue in my discipline. A good submission; an accept. I vote for accepting this submission, but would not be upset if it were rejected.
• 0: Marginally above acceptance threshold; an accept. I tend to vote for accepting this submission, but rejecting it would not be that bad.
• -1: Marginally below acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad.
• -2: A clear reject. I vote and argue for rejecting this submission.
• -3: A “must reject”. Paper is fundamentally flawed (E.g., major results are trivial, wrong, already known, etc.) — I will fight for rejecting this submission.



The focus here is whether the paper’s level of quality allows it to make a significant contribution to key conversations in FAccT related fields. Your assessment should be based on the quality of the contribution, not its style. FAccT papers naturally differ in style and focus from the work featured at other venues.

You should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The “Overall Score” for each submission should reflect your assessment of the submission’s contributions.

Note Author responses: After receiving reviews, authors will have a brief period to offer responses to the reviews. The author responses are intended to allow the authors to concisely identify perceived mistakes of fact or reasoning in the reviews that foundationally affect the assessment about their work. Authors are not required or expected to submit a response. Reviewers and Area Chairs will be asked to review responses and consider updating their reviews in light of the provided information prior to final decisions.




9. Confidence score

Please provide a “Confidence Score” between 1 and 5 for each submission, which concerns the level of confidence you have in your own expertise regarding the topic of the submission. The Area Chairs and Program Chairs will interpret these scores via the following scale:

• 5: You are absolutely certain about your assessment. You are very familiar with the related work.
• 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
• 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
• 2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
• 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.



Note : If you feel that your confidence rating is likely to be a 1 at the end of a review due to your lack of expertise in the given subject area, you should notify the Area Chair as early as possible in the process. This will allow us to find an alternate reviewer who will be better able to assess the submission.