If you still have questions, you can email: ad-ae-appendices@info.supercomputing.org

Artifact Description (AD) Appendices are required for all papers submitted to the Technical Program at SC19. Artifact Evaluation (AE) Appendices are optional, but strongly encouraged.

AD and AE Appendix Requirements

Review Process

Impact of AD and AE Appendices

ACM Artifacts Available and Artifacts Evaluated Badges

AD and AE Appendix Requirements

Q1. Are AD and AE appendices required in order to submit to SC19?

A1. An AD appendix is required for all Technical Program submissions. An AE appendix is optional but strongly encouraged.

Q2. Do I need to make my software open source in order to complete the AD appendix?

A2. No. You are not asked to make any changes to your computing environment in order to complete the appendix. The AD appendix is meant to describe the computing environment in which you produced your results. Any author-created software does need to be open source, however, to be eligible for the ACM Artifacts Available badge (see below).

Q3. How should I format my AD Appendix

A3. You don’t need to worry about formatting the Appendices. You will be presented with an online form during the paper submission with questions that you will answer directly on the submission site. After answering the questions, the system will automatically generate a PDF of the Appendix for you.

Q4. What information do I need to provide in the AD/AE Appendix online form?

A4. A printout of the questions included in the AD/AE Appendix online form is provided in the Author-Kit repository. Be sure to familiarize yourself with these before writing your paper, and ideally before or while you are producing your results.

Review Process

Q5. Who will review my appendices?

A5. The AD & AE Appendices will be reviewed with your paper by the Technical Program committee, but the artifact URLs will be removed from the version they review, as a precaution in support of double-blind review. In addition, the AD/AE Appendices Committee will review the unredacted appendices, and will check that artifacts are indeed available in the URLs provided. They will also help authors improve their appendices, in a double-open arrangement.

Q6. How will review of appendices interact with the double-blind review process?

A6. The AD appendix should describe the data, software and hardware artifacts involved in producing the results. Reviewers could discover the author’s identity if they embark on an online search, but they will be asked not to, in support of double-blind review. Author-provided artifact URLs will be redacted from the appendices provided to the reviewers.

Impact of AD and AE Appendices

Q7. What’s the impact of an Artifact Description appendix on scientific reproducibility?

A7. Reproducibility depends on, as a first step, sharing the provenance of results with transparency, and the AD appendix is an instrument of documentation and transparency. A good AD appendix helps researchers document their results, and helps other researchers build from them.

Q8. What’s the impact of an Artifact Evaluation appendix on scientific reproducibility?

A8. An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the supercomputing community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). The various forms of Artiface Evaluation improve confidence that computational results from these special platforms are correct.

ACM Artifacts Available and Artifacts Evaluated Badges

By the ACM Badging definitions:

ACM Artifacts Available badge: This badge is applied to papers in which associated artifacts have been made permanently available for retrieval.

The AD/AE Appendix form (new for SC19) will automatically determine eligibility for an ACM Artifacts Available badge on the basis of the answers to questions about the availability of author-created software, hardware or data products. The conditions of eligibility are:

  • All author-created software artifacts are maintained in a public repository under an OSI- approved license.
  • All author-created hardware artifacts are available and comply with the Open Source Hardware Definition.
  • All author-created data artifacts are maintained in a public repository with a stable identifier, such as a DOI.

Artifacts Evaluated badge: This badge is applied to papers whose associated artifacts have successfully completed an independent audit.