process exits with exit code 0 when tests fail

[radfahrer]

I have two gulp tasks, one that sets up istanbul with gulp-istanbul:

gulp.src(['lib/**/*.js'])
		.pipe(istanbul(istanbulOptions))
		.pipe(istanbul.hookRequire());

and the other that runs the tests

return gulp.src(['test/**/' + serviceName + '.js'])
		.pipe(jasmine(jasmineOptions))
		.pipe(istanbul.writeReports());

the second "depends" on the first

When I run this combination and tests fail, the exit code is 0

[ssoloff]

I just ran into this, as well. I don't know the root cause, but it has something to do with piping the results of jasmine() into a subsequent stream. For example, the following task body will cause Gulp to fail (i.e. exit code != 0) when a test fails:

return gulp.src('spec/**/*.spec.js')
  .pipe(jasmine())

However, simply piping the results into any other stream will not cause Gulp to fail (i.e. exit code = 0) when a test fails:

return gulp.src('spec/**/*.spec.js')
  .pipe(jasmine())
  .pipe(gutil.noop())

I worked around this with my Jasmine/Istanbul integration by promisifying the task body so that I can isolate the Jasmine execution such that nothing needs to be piped downstream of jasmine():

const streamToPromise = require('stream-to-promise')

gulp.task('test', () => {
  return streamToPromise(
    gulp.src('lib/**/*.js')
    .pipe(istanbul())
    .pipe(istanbul.hookRequire())
  )
  .then(() => streamToPromise(
    gulp.src('spec/**/*.spec.js')
      .pipe(jasmine())
  ))
  .then(() => streamToPromise(
    gulp.src([])
      .pipe(istanbul.writeReports())
  ))
})

The third promise is a bit of a hack, as I had to use gulp.src([]) to create a dummy empty stream that I can pipe in order to write the Istanbul reports.

I'm pretty sure the same thing could be done without promises by simply creating a series of three dependent tasks (instead of the normal two the gulp-istanbul plugin suggests). For example (please forgive the useless and misleading task names):

gulp.task('pre-pre-test', () => {
  return gulp.src('lib/**/*.js')
    .pipe(istanbul())
    .pipe(istanbul.hookRequire())
})

gulp.task('pre-test', ['pre-pre-test'], () => {
  return gulp.src('spec/**/*.spec.js')
    .pipe(jasmine())
})

gulp.task('test', ['pre-test'], () => {
  return gulp.src([])
    .pipe(istanbul.writeReports())
})

[ssoloff]

After looking into this a bit more, I'm less convinced it's a gulp-jasmine problem, but simply an artifact of how Node streams handle errors. This post by the author of gulp-plumber was enlightening.

So another possible solution would be (from here):

return gulp.src('spec/**/*.spec.js')
  .pipe(jasmine())
  .on('error', (err) => {
    gutil.log(err)
    process.exit(1)
  })
  .pipe(istanbul.writeReports())

Note that gulp-plumber's default error handler does not exit the process, so if you want a nonzero exit code, you still have to pass it a custom error handler that will explicitly exit the process.

One more observation that made me think this was originally a gulp-jasmine problem... If I use gulp-mocha instead of gulp-jasmine, I get the behavior I want of Gulp exiting with a nonzero status when a test fails. Looking at the implementation of gulp-mocha, it uses Node domains to keep track of errors that may occur through a chain of streams, similar to what gulp-plumber does. I suppose gulp-jasmine could be written in such a way, as well, so that it Just Works in the scenario discussed in this issue. However, I don't know if this would limit other use cases where the user wants more flexibility on how plugin errors should be handled (see this article for an example).

[sagarthecook]

if you want to exit forcefully use below code
gulp.doneCallback = function(err) {
process.exit(err ? 1 : 0);
};