For instructions on using HOTCRP please refer to this document .

Why do I need to write a review?

Your review is important for two reasons:

  1. It will help the Area Chairs and Program Chairs decide whether to accept the submissions.

  2. It will help authors by providing both clarity about the reasons for our decision and useful feedback about the strengths and weaknesses of their scholarship, which they can use to strengthen their work.

Please make your review as detailed and informative as possible -- short, superficial reviews that venture uninformed opinions or guesses are worse than no review, since they may result in the rejection of a high-quality submission.

What belongs in my review?

Please describe and consider the strengths of the submission. It can be tempting to comment only on the weaknesses. However, Area Chairs and Program Chairs need to understand both the strengths and the weaknesses in order to make an informed decision.


    1. Summary of contribution

    Please summarize the main ideas of the submission and explain any contributions to the literature, especially in relation to previous work at FAT*/FAccT as well as other archival conferences and discipline-specific journals. This section helps to ground reviews in the full details of the paper, reminds us why the paper matters, and is invaluable to Area Chairs and Program Chairs. [ Open text response.]

    2. Quality

    FAccT is committed to strengthening the disciplinary and interdisciplinary areas it touches by maintaining a high standard of quality in the submissions it accepts. Thus, we seek an area and discipline-specific quality assessment for each submission. [Open text response.]

    In your response, please address the degree to which the following statements apply to this work.

    • The work presented is comprehensive. 
    • Claims are well supported by theoretical analysis, evidence, argument, and/or experimental results.
    • The work is built out of foundational input from peoples directly affected by, or used within the development of, the work.
    • The work incorporates foresight on populations most likely to bear disproportionate negative effects directly related to this work.
    • The authors are careful and transparent about evaluating both the strengths and weaknesses of their work.

    3. Clarity

    A superbly written paper provides enough information for an expert reader to reproduce its results or reach similar conclusions based on the analysis, and for a novice reader to understand the basics of the work. [Open text response.]

    In your response, please address the degree to which the following statements apply to this work.

    • The submission is clearly written and well organized.
    • The submission adequately informs the reader about the research.
    • The claims/arguments are explored in sufficient depth.

    4. Originality

    Strong work makes an original contribution. [Open text response.]

    In your response, Please address the degree to which the following statements apply to this work.

    • The submission is clearly written and well organized.
    • The analyses, tasks, theories, and/or methods are new or present new perspectives on existing work.
    • The work is a novel combination of well-known research.

    5. Scholarship

    Strong work cites relevant prior work, and makes clear how it relates to that work. [Open text response.]

    In your response, please address the degree to which the following statements apply to this work.

    • The work is a novel combination of well-known research.
    • The work situates itself in the context of related work.
    • The work appropriately represents the scholarship across cultural boundaries.
    • It is clear how this work differs from previous contributions.
    • Related work is adequately cited.

    6. Significance/Impact

    Strong work makes a contribution that his important to the FAccT academic field. [ Open text response. ]

    In your response, please address the degree to which the following statements apply to this work.

    • The results or conclusions are important for the FAccT academic field.
    • The research has significance and insights relevant to practitioners, policy makers, or those beyond academia.
    • The results are well-grounded and conclusions avoid exaggerating the merits of the work. 
    • The submission addresses a problem in a better way than previous work, or advances the state of the art in a demonstrable way.
    • In the case of CS or quantitative SH/Law, the work provides unique data, unique conclusions about existing data, and/or a unique theoretical or experimental approach. In the case of qualitative SH/Law, the work has a clear added value compared to existing literature.


    7. Relevance

    If you believe that a submission is out of scope for FAccT, then please justify this judgement appropriately. In formulating this judgment, you may find it helpful to review the Areas of Interest found in the Call for Papers . Note that this list of topics was not meant to be all-inclusive. We welcome submissions that address other important problems surrounding the fairness, accountability and transparency of socio-technical systems.
    • The submission has relevance to FAccT. [Scale from Strongly Agree … Strongly Disagree]
    • [Optional] Additional text comments on relevance .

    8. Overall score

    Please provide an Overall Score for each submission on a scale from 1-7.

    Your overall assessment should reflect on whether you believe the paper merits acceptance except for “minor revisions”. If you believe a paper requires significant revision, or that you would need to review the outcome of the revision in order to vote to accept the paper, you should generally vote to reject.

    There is one key exception. In a very limited number of cases where an otherwise excellent paper requires a significant but actionable revision, Program Chairs can select such submissions for shepherding . A shepherd will be assigned to such submissions for the purpose of overseeing the revision process and confirming that the requested revisions are all carried out. 

    The Area Chairs and Program Chairs will interpret Overall Scores via the following scale:

    • 7: Would be a top 10% accepted paper at venues I highly respect. A “must accept”. I will fight for accepting this submission.
    • 6: Would be a top 50% accepted paper at venues I highly respect. A very good submission, clear accept. I vote and argue for accepting this submission.
    • 5:Would be an accepted paper at highly respected venue in my discipline. A good submission; an accept. I vote for accepting this submission, but would not be upset if it were rejected.
    • 4: Marginally above acceptance threshold; an accept. I tend to vote for accepting this submission, but rejecting it would not be that bad.
    • 3: Marginally below acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad.
    • 2: A clear reject. I vote and argue for rejecting this submission.
    • 1: A “must reject”. Paper is fundamentally flawed (E.g., major results are trivial, wrong, already known, etc.) — I will fight for rejecting this submission.

    The focus here is whether the paper’s level of quality allows it to make a significant contribution to key conversations in FAccT related fields.s. Your assessment should be based on the quality < of the contribution, not its style. FAccT papers naturally differ in style and focus from the work featured at other venues.

    You should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The “Overall Score” for each submission should reflect your assessment of the submission’s contributions.

    Note : FAccT 2021 will allow authors to study the peer reviews/cross-disciplinary reviews of their paper and submit a short (500 word) rebuttal in the case where authors perceive severe misunderstandings. 


    9. Confidence score

    Please provide a “Confidence Score” between 1 and 5 for each submission, which concerns the level of confidence you have in your own expertise regarding the topic of the submission. The Area Chairs and Program Chairs will interpret these scores via the following scale:

    • 5: You are absolutely certain about your assessment. You are very familiar with the related work.
    • 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
    • 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

    Note : If you feel that your confidence rating is likely to be a 1 at the end of a review due to your lack of expertise in the given subject area, you should notify the Area Chair as early as possible in the process. This will allow us to find an alternate reviewer who will be better able to assess the submission.