ISPASS runs an optional artifact evaluation (AE) process for authors of accepted papers. Participating is voluntary and has no bearing on the acceptance decision, which has already been made by the time AE begins.
Badges
The availability and quality of artifacts is denoted by badges that appear on the first page of the paper in the digital library. Because ISPASS is an IEEE-sponsored conference, it follows the IEEE Xplore reproducibility badge scheme. ISPASS awards three of them:
- Available — the code and/or datasets, including associated data and documentation, are reasonable and complete, and can potentially be used to support reproducibility of the published results.
- Reviewed — the code and/or datasets are reasonable and complete, run to produce the outputs described, and can support reproducibility of the published results.
- Reproducible — an additional step was taken to certify that an independent party regenerated computational results using the author-created research objects, methods, code and conditions of analysis. Reproducible assumes the research objects were also Reviewed.
Artifacts may be code or datasets; one paper may be accompanied by both.
IEEE Xplore also defines a fourth badge, Replicated, for replication studies performed by other authors. It is not awarded as part of this process.
What a submission consists of
An artifact submission has two parts.
1. The paper and a two-page appendix
Prepare the appendix using the artifact appendix template. Papers that pass artifact evaluation may include an updated version of this appendix in the camera-ready version; the two-page artifact appendix does not count towards the regular page limit.
The appendix is expected to contain:
- an abstract
- an itemized meta-information list
- access to the artifact
- system requirements and dependencies
- experiment workflow
- steps for evaluation
- results
2. The artifact itself
Provide access to the code and/or data, with enough documentation that a reviewer who has never seen the project can run it.