Ten Simple Rules of Credible Practice/Mathematical and Computational Sciences Team
Top Ten Simple Rules Ranked:
- Use competition of multiple implementations to check and balance each other
- Document your code and make your code readable
- Explicitly identify experimental scenarios illustrating when, why, and how the model is false \ Explicitly list your limitations
- Make it easy for anyone to repeat and/or falsify your results \ Make sure your results are reproducible
- Use traceable data that can be traced back to the origin
- Use version control
- Define your evaluation metrics in advance
- Practice Verification / Validation / Uncertainty Quantification \ Attempt verification within context
- Define the use context for which the model is intended
- Develop with the end user in mind
Draft of Rules in Original Order + Score Before Ranking:
Here are the Raw Rules in original order with alternatives noted after \ according to our team (Score according to our discussion – based on responses by everyone in the forum who responded).
- Use version control (12 = +2 Pras, +10 Jacob)
- Include credible/pedigreed implementations alongside the other implementations \ Use credible solvers (2 = +2 Tony)
- Explicitly identify experimental scenarios illustrating when, why, and how the model is false\ Explicitly * list your limitations (16 = +12 Joy, +2 Pras, +2 Tony)
- Define the use context for which the model is intended (10 = +10 Pras)
- Define your evaluation metrics in advance (12 = +2 Pras, + 10 Tony)
- Use appropriate data (input, validation, verification)
- Attempt validation within context (2= +2 Pras)
- Practice Verification / Validation / Uncertainty Quantification \ Attempt verification within context (11 = +11 Pras)
- Attempt uncertainty (error) estimation
- Perform appropriate level of sensitivity analysis within context of use
- Disseminate whenever possible (source code, test suite, data, etc) \ Practice Open Science (2= +2 Pras)
- Report early and often, in the form of disseminated models and results \ Report appropriately (4= +2 Pras, +2 Tony)
- Use consistent terminology or define your terminology
- Entice others to explore, criticize, repeat, and falsify your results. \ Get it reviewed by independent users/developers/members
- Learn from discipline-independent examples (2=+2 Tony)
- Be very concrete and particular first, generalize second \ Be a discipline-independent/specific example (2=+2 Tony)
- Choose the best solutions based on a clear project formulation \ Follow discipline-specific guidelines
- Prespecify use cases and then choose standards and methods based on the context \ Conform to discipline- * specific standards (2 = +2 Tony)
- Document your code and make your code readable (22 = +11 Pras, +11 Tony)
- Develop with the end user in mind (5 = +5 Joy)
- Provide user instructions whenever possible and applicable (2 =+2 Pras)
- Do not merely "practice what you preach", make your work easy for others to challenge \ Practice what you preach
- Make it easy for anyone to repeat and/or falsify your results \ Make sure your results are reproducible (14 = +10 Joy, +2 Pras, +2 Tony)
- Provide examples of use ( 2= +2 Pras)
- Use traceable data that can be traced back to the origin (13 = +2 Pras, +11 Jacob)
- Use competition of multiple implementations to check and balance each other (35 = +11 Joy, +12 Tony, +12 Jacob)
- Rely only on wet-lab data for which provenance is established.
Notes:
The Rule Scoring and Ranking can be found in the forum at: https://simtk.org/forums/viewtopic.php?f=848&t=4333
This ranking is only for the Mathematical and Computational Sciences Team. The ranking for all committee teams can be found in the parent wiki in the following link: http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice