PERCOM 2020 will, for the first time, provide the opportunity for authors of accepted papers to participate in an Artifact Evaluation (AE) process. Authors who choose to participate can provide supplementary materials as artifacts (tools, applications, data sets, etc.) that will be reviewed solely for functional completeness (e.g., does provided code compile and run as expected; do experimental results match expected results, etc.). Participating authors will be allotted two additional pages in a separate Artifacts section of the proceedings to describe the nature and contents of the artifact package. Submission of artifacts packages will have no bearing on paper acceptance or publication. However, papers that have verified artifacts or data will be associated with badges in the conference program and proceedings. Two badges will be available: Artifact-Certified and Result-Certified. Upon submission, participating authors should indicate which badge(s) the committee should consider for the submission.
Artifacts of interest include (but are not limited to) the following:
- Software, including implementations or simulations of systems or algorithms.
- Data repositories, including, e.g., logging data, system traces, survey raw data.
- Frameworks, including tools and services illustrating new approaches to pervasive computing systems and application development.
This list is not exhaustive; authors with questions about the suitability of an artifact should reach out to the chairs.
The submission process is straightforward and follows the following steps:
- Package software and datasets. Use a freely available tool, such as VirtualBox; Docker container image; git/mercurial/svn repositories; or zip/tar files.
- Create an artifact guide. The guide can be up to 2 pages and should be prepended to a preliminary version of the accepted paper (as a single PDF file). The guide’s contents will vary based on the artifact but it should describe minimal software and hardware requirements, how to obtain and deploy the artifact, and what a reviewer should expect (e.g., the expected experimental results, or steps to navigate an application).
Complete submission details will be available on the website after the paper acceptance notification date. More details about the steps will be provided once the acceptance notifications are communicated to the authors.
The Review Process
The review process will have three phases:
- Initial Review. The reviewers will follow the instructions given in the artifact’s guide to ensure that artifacts can be deployed and run as described. Any bugs or questions will be documented as feedback.
- Artifact Revision. During the second phase, authors will be able to correct any bugs in their artifacts/guides to make sure reviewers can un-package and run the artifact before the deeper evaluation.
- Final evaluation. Reviewers perform their final evaluation to ensure that each artifact runs as expected and/or contains the expected data.
Note that the first two phases may be intertwined (several rounds) if authors and reviewers are quick their communication; every submission is guaranteed at least one opportunity for revision.
Best Artifact Award
A PERCOM 2020 Best Artifact Award will be presented during the conference.
- Registration of intent to submit an artifact: January 3, 2020
- Initial artifact submission deadline: January 10, 2020
- Artifact feedback and revision period: January 10-27, 2020
- Artifact evaluation notification (badge allotment): January 29, 2020
- Artifact guide camera ready deadline: January 31, 2020
For further information about the artifact submission and evaluation, please contact the Co-Chairs, Vaskar Raychoudhury (firstname.lastname@example.org), Md Osman Gani (email@example.com), and Christian Becker (firstname.lastname@example.org).