Update submitting.md#1554
Conversation
dstansby
left a comment
dstansby
left a comment
There was a problem hiding this comment.
Other than a minor suggestion, I think these tweaks are good 👍
|
I think there is a case here that we don't really discuss but is a really important one. That is of research software that's being used for a major project, but is currently still in its proprietary phase. Publishing software ahead of results, I think, is something that we should encourage to enable maximum transparency. The alternative is waiting for proprietary data and results to be released, and then acquiring citations, and then being 'able' to publish the software in JOSS. That leaves a significant amount of time where the software goes unpublished. Indeed we can argue that they can make the software open source in this time period, but isn't the whole point of JOSS to make sure that this software is ready for use (and hence ready for duplication and reproduction of results) by the community? A plausible example here would be some high energy physics project, let's call it the Big Collider; they are running a new analysis pipeline that is as-of-yet-unpublished. Big Collider's science runs began on 24 July 2025 and they will end one year later. They look to publish results in Nature on 19 September 2026, demonstrating the discovery of 'left' and 'right' quarks! Alongside this, they will release their data for re-analysis and their entire toolchain, but they have a key analysis package, QuarkFinder, that they've been using. The developers of QuarkFinder come to us on 2 May 2026 and ask to publish their software, noting that it's part of Big Collider and that they want to publish before results for maximum transparency. Do we review QuarkFinder? I, personally, would say that we should even though it's not been documented (because data and methods are proprietary at this stage), and because it's never been used or cited externally (because the papers only come out in September). Crucially, I think the rules as written would say that we shouldn't publish it. |
I think we would review this currently. We would just need evidence that the project is using the software, which doesn't have to be a paper with a citation. |
| **2. Demonstrated research impact** | ||
|
|
||
| There must be evidence that the software is being used for research — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. | ||
| There **must be evidence that the software is being used for research** — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. |
There was a problem hiding this comment.
| There **must be evidence that the software is being used for research** — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. | |
| There **must be evidence that the software is being used for research** — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. Adoption in currently proprietary workflows (e.g. before an experiment's first papers and/or data releases) is acceptable but must be demonstrated to the editorial team. |
There was a problem hiding this comment.
Can we find another word than "proprietary"? This sounds too commercial to me. Maybe we could say "Use in active research projects (e.g. before an experiment's first papers and/or data releases) may be acceptable but must be demonstrated to the editorial team."
There was a problem hiding this comment.
Happy to find another word, I used 'proprietary' because it's the standard word in astronomy for a time when a group has exclusive access to data, usually from an external observatory through a fixed-term 'proprietary period'.
There was a problem hiding this comment.
"Un-seen adoption in workflows during an exclusive use period (e.g. before an experiment's first papers and/or data releases) is acceptable but must be demonstrated to the editorial team."?
There was a problem hiding this comment.
Would "closed-source" be sufficient?
Though I guess that is still thinking in terms of software end dependencies...
Just "closed workflows"?
There was a problem hiding this comment.
The software has to be developed in the open in any case, so I'm not sure saying "closed" is useful.
There was a problem hiding this comment.
The software under submission to JOSS, yes, but the application of it in another place could be closed, no?
E.g. I know some groups who have their core software (which would be submitted to JOSS) as open source, but then the analysis on a particular dataset/simulation, and the work towards a specific publication facilitated by that software takes place in a private repository.
This private repository where the software is used to generate results was what I understood as "workflow" in this context?
I also think @JBorrow's "unseen" would work.
| **2. Demonstrated research impact** | ||
|
|
||
| There must be evidence that the software is being used for research — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. | ||
| There **must be evidence that the software is being used for research** — at minimum by the developers themselves, and ideally by others. Acceptable signals include: references in published papers or preprints, DOIs linking to the software, documented adoption by other research groups, or clear integration into research workflows. |
There was a problem hiding this comment.
DOIs linking to the software
Anyone can just make a DOI to link to their software, like we have people do from Zenodo, so this doesn't seem like a good signal.
Also do presentations count? I have seen people note those.
I've also had people vaguely say a research project is using it currently but without anything specific to back that up, in which case I haven't thought it really "counts".
There was a problem hiding this comment.
I agree with this point and was just writing a similar comment. I think we need to be clearer what does and does not count, even if that is internally, so it can be applied consistently across tracks and reviews.
E.g. for the anecdote about the "Big Collider" above what evidence that they are using it in research to prepare a publication would be acceptable, other than being a big well-known project and we trust their word? I think criteria should be something that can be applied consistently to any piece of software no matter the size/prestige of the submitting author/group.
5 participants
No description provided.