-
Notifications
You must be signed in to change notification settings - Fork 0
R Development Practices
To ensure that the software we're developing is of a consistent quality, here are some guidelines to follow.
Automated testing is one of the most important practices for producing high quality software. Although many people think of testing as some cumbersome necessity and not something interesting (or even useful) at all, it can save you and others a lot of time and effort. By writing automated tests you can easily pinpoint errors when some part of your project breaks. The tests can give an insight on how a change in one place of the code affects the rest of the software. That is, sometimes a seemingly small change in a function in one place could have major effects on a different part of the software depending on it. Even worse, it could be that external packages depend on your function. Writing proper automated tests could help to prevent these problems.
In this document I'll try to provide an overview of different types of tests and the goal of those types of tests. I will briefly discuss unit testing and integration testing, and provide some insight about the differences between them. After that I will give a short introduction to the practice of test driven development (or TDD), and some practices that I find useful to apply in scientific projects.
Note that these are mere guidelines. For more information about tests and test-driven development the book Test-Driven Development, By Example by Kent Beck is a good read.
The general idea of automated testing is to see whether your code, or a small section of your code behaves as you would expect it to behave. That is, given some R function, and given a set of predefined inputs, you know what the outcome would be when applying your function to the set of inputs. This outcome is something you can check programmatically. If your function gets changed and the outcome differs from the pre specified output, your test function will detect this and will raise an error.
One of the most commonly used test frameworks for testing R packages is testthat. Testthat is a package by Hadley Wickham that allows you to create test files and assert the outcomes of the code you are testing. See the testthat github page, the testthat R-journal article, or the chapter on R testing for more information.
Setting up the testthat framework in your package is straightforward. Assuming you have the devtools package installed, and assuming that you are testing an R-package, you can just run devtools::use_testthat() from R (in the directory of you package), and the necessary dependencies and folders are added to your project directory. You can now start and add your tests to the folder /tests/testthat/.
Unit testing is the most general form of testing. In unit testing you test a small portion (or unit) of the software. In practice this could be a small module, class, or function. The goal of a unit test is to check whether your unit of code performs the way it should, neglecting the package as a whole. The nice thing about unit testing is that you can test at the lowest level without having to worry about the global structure of the application or package you are creating. An example of a very basic unit test in R using the testthat package is provided below:
# The function you are testing
multiply <- function(a, b) {
if(is.null(a) || is.null(b) || !is.numeric(a) || !is.numeric(b)){
throw('Two values should be provided')
}
return(a * b)
}
test_that("it should multiply two numbers", {
expected <- 10
result <- multiply(5, 2)
expect_equal(result, expected)
})The unit test shown above tests the case in which the code receives exactly what you would expect it to receive, that is, two numeric values which it can use to multiply. This is also known as the happy flow of the function. However, generally you also want to test the other different paths in the code. That means, you'd also like to test whether the if condition works:
test_that("it should throw an error if only one parameter is provided", {
expected <- 'Two values should be provided'
expect_error(multiply(5, NULL), expected)
expect_error(multiply(NULL, 2), expected)
})
Writing tests for all different paths in a function can be a lot of work. As a general rule of thumb, if you feel like your code has too many paths (which is generally a sign that you should do some refactoring, but nevertheless), try to hit every line in the function at least once, and explicitly test the critical combinations of these if statements.
In descending priority:
- Test the happy flow
- Test the critical secondary paths in the code
- Try to hit all lines in the code at least once
- Test all paths your code could take
One very powerful tool for writing unit tests is so called mocking. Mocking is the practice of stubbing (or mocking) a part of the software with some fake piece of code, of which you specify exactly what it does and what it returns. By mocking you allow your unit of code to be really separately tested from the rest of the code. Furthermore, by mocking code you can check whether a call to a function inside of the function you are testing is actually made with the correct parameters. Consider the following code snippet as an example. In this example the identical function from the base class gets mocked with a new function that always returns true, regardless of its input.
test_that("it should mocking", {
# Without the mock, 4 != 9. Note the expect_FALSE.
expect_false(identical(4, 9))
with_mock(
`base::identical` = function(...) TRUE,
# With this specific mock everything is identical. Note the expect_TRUE
expect_true(identical(4, 9))
)
})Although mocking is a very powerfull tool, use it wisely. Keep in mind that by mocking a function you do not test the internals of that function anymore (which is essentially the whole point). If you use mocking you assume that the function that is being mocked is tested elsewhere, in another unit test. A framework that might be useful for mocking is the mockery package. The mockery framework also allows you to check the number of calls made to a function, and do more in-depth analysis on the code.
If you are using some of the object oriented features of R (e.g., R6), you might encounter so called private functions. Private functions are only accessible from the inside of the class, or (in the case of R6) from any of the subclasses. Generally you don't want to test these functions, just the public functions calling them. By not testing the private functions you allow your code to be flexible, so you can swap out pieces of code in your private methods, and extract code from a public function to a private function without having to rewrite all of your tests. There is some discussion about this, but the general opinion seems to be that this is considered the best practice. If you have the feeling that you have a lot of code in a private method, and you would really like to test it, it usually is a sign that your private function is becoming to complex, and deserves to be in a separate class as a public function. This allows you to test the function, and is probably a better separation of concerns for your software package.
Integration tests focus on the software package as a whole. Where unit tests focus on small units of code, in integration tests this focus shifts to the publicly facing interface of the software. Even though you might have tested every single function in unit tests, it does not ensure you that the combination of all of those units of code also works. The goal of integration testing is to solve exactly that problem, by testing the application as a whole, usually from the point of view as the end user. In this case the end user is yet another developer / researcher using the package that we are developing. What you want to test in integration tests is if all of the functions you export (so functions that are accessible from outside of the package). For example, if you have written a function to fit a linear model (e.g., the lm() function), what you want to do is call that specific function with different arguments, and see if the function responds as you would expect it to respond.
Generally for integration tests you do not want to make use of any mocked functions, only if it is absolutely necessary (e.g., if you are calling an external API). Recall, in unit testing the focus is on the separate parts, and therefore mocking of other parts of the software is fine, in integration tests it is actually the point that all of these functions are tested.
Test-driven development is a practice where you first write a failing test and then write the actual implementation of your program to make the test pass. This might sound weird, but it can be a very useful and straightforward thing to do. Whenever you are writing code you are usually already doing this in a very informal sense. For example, if you write code you test it by running (pieces of) it in an R console, and check if the outcome is equal to what you expected it to be (generally with the help of print statements). If you would formalize this step, you are already there.
If such a strict first-test-then-code method does not work for you, you could do it in an iterative fashion, where you first create a placeholder for the test, write some code, write the test, write some code again, and iterate until everything works. This is of course not anywhere near as strong as formal TDD, but at least beats not writing tests at all. Furthermore, by writing your code this way you don't have to write tests afterwards.
The following script might help when doing TDD (or when writing R code in general). You run it from a terminal (Linux or macOS), and it will watch your tests and code for changes. Whenever one of them changes, it runs the test suite (or part of it) again. This way you don't have to start your tests manually. Add the following piece of code to your .bashrc, .bash_profile, or .zshrc (or whatever terminal you are using):
rtest () {
echo -e "library(mockery)\ntestthat::auto_test_package('.')" | R --no-save
}Then open a new (regular, not an R) terminal, browse to the top level directory of the package you want to test and type rtest. It should start R and run your tests. Note that this piece of code does expect testthat to be installed.
Testing is a rather straightforward task, but a very important one. Adding tests to your project will dramatically increase its stability and might help preventing bugs, especially when the project grows to a large scale.
For unit tests I would suggest to create one test file for every single R file you have in your R directory. Furthermore, I would prepend test files with test-, so you can easily tell them apart from other files. So, if you would have an R file called multiply.R, you would also have an test-multiply.R file in the /tests/testthat/ directory of your project.
For integration tests I would create separate test files, and appending their names with an i (testi-<filename.R>). It is a bit of inbound signalling, but it does help separating the integration tests from the unit tests. [CK: this won't work because with testthat test files need to begin with "test". Could do "testi-" though.] [FB: Good point, I've changed it. Feel free btw to change anything I wrote, it was just a suggestion what I think are good practices. I didn't do real research before writing any of this. Perhaps it could be a good idea to set up a slack / gitter, so we have a place to discuss things? Big downside of using these wiki pages is that you don't really have PR's or somewhere to make comments.]
In order to keep your tests and the results thereof legible, use the context() and test_that function with an understandable description. Whenever a test fails, it will give you an immediate idea of what went wrong and how you might be able to fix it.
Lastly, it is generally not a good idea to include randomness in your tests. You want your tests to be deterministic, so you reduce the chance of flaky falling tests (even though you are setting a seed). If you need to use data in your test, you can use the inst directory for storing this data. [CK: is there a source for this, and/or can the rationale be elaborated upon? With a properly set seed a data simulation is perfectly deterministic. Of course when testing parallelization it can be more complex to get the seed correctly set.] [FB: I've put this in because of personal experiences and apparently Oleg is also not the biggest fan. I couldn't find strong guidelines why using randomness would be good / bad, and we could discuss it further if your prefer it? Actually, several projects we wrote started with some kind of random sampling in it, with the idea to make the tests more reliable (this was in Ruby though, although that shouldn't mind). In the end we removed all the randomness because of weird and hard to debug failures. But again, please change anything you'd like, it is probably highly opinionated and could definitely use a second opinion!]