need to write a report for the conference

[timm]

This is a pace for the PC chairs to dribble out notes on what words, does not work for artifacts

Random notes at submission time

Low numbers. Why:

• ?low community interest in artifacts.
• ? hard to distinguish from tool track.

Poor conformance to the format we offered for artifact submission.

• Was the format wrong? In terms of light to heavyweight, it was on the lightweight side.
• Is the community still divided on "right format"? The folks with a VM just ignored our notes and offered a VM. after a 3 minute download and a 4 minute "import to virtual box" session, we had their code working perfectly on a reviewer machine. should we just demand VMs all the time?

Working in Github

v.cool. interaction with artifact authors is much more complex than supported by, say, easychair. debates on what to download, what to keep at central site

Writing artifacts

• Keep it short: 11BG (unzipped)? Wild. Maybe artifacts should be "teasers". Things we can fast download and peek out and taste before swallowing.
  • Note, some of the bigger downloads from Google Drive just timed out without downloading. Internet still needs debugging.
• Art to authoring "cliff notes" (i.e. short intro). Need some support text in the repo. e.g https://github.com/researchart/fse18/blob/master/submissions/pattern-fuzzing/README.md. Note that these are the notes that a web surfer would use to decide if they want to use this or not.

[timm]

Download issues

need to change CFP to ensure artifact can download

• we are having problems downloading your artifact. very slow downloads that never seem to terminate. can you place it on another server? is there anyway we can use plain old wget to retreive it?

[timm]

artifacts are getting BIG

e.g. 13GB and 11GB zipped

[timm]

lesson of the journal papers

journal papers are very unifrorm. hence, we can build tools like easychair and cyperchair

when you leave that sanitized world behind, you're up for a whole menagerie of artifacts. and its wild and compelx world

[timm]

double blind

.. is a myth. too much information in the repos.

[timm]

some disappointment that available, which is a higher rank that reusable, does not sound as cool as reusable.

[timm]

some thanks from the authors for our innovative interactive cooperative review process

really dont understand how to do artifacts otherwise-- certain install problems can be fixed with one very quick patch from the authors.

[timm]

Are we artifact review boards replication service? If an artifact is posted and reviewers can get the same scripts running as the authors and the same numbers dribbling out, then can the very fact of submitting something to an artifact review board make a paper "replicated"?

[timm]

not hero peer reviewing of insightful ideas, but certifications of artifacts. this is an engineering, process. paradigm support.

[timm]

Some issues with definitions.

• Are all available, reusable artifacts also functional?
  - We said yes
• Does available apply resuable?
  - We said yes-- unless the artifact documentation was weak or reviewers hit too many bugs.
• If an artifact is replicated by no one, yet, and the artifact PC team can run the scripts and get the same output as seen in the paper, is that a "replications"?
  • In the end, we said no since if we said "yes", that would place the onus on the reviewers to understand the semantics of the paper and the full meaning of the output. This seemed an excessive additional burden since our reviewers spent much time struggling with install scripts and other systems issues.
  • That said, we think our artifact evaluation PCs might might allow for "replication by review board", if they carefully warn their reviewers that such replications will be asked of them.
  • On the other hand, for a lightweight, more scalable process, we caution against "replication by review board".