We’ve been trying to fix an issue with our continuous integration where Capybara tests were failing, but the CI process reported success. Rather annoying for CI!
We started by adding some logging to our test Rake task to see the error code returned from RSpec. This shows the PID and the exit code, explicitly failing when the exit status is either false or a non-zero error code.
rc = system("bundle exec rspec spec")
logger.info "rspec spec #{$?}"
fail "rspec spec failed with exit code #{$?.exitstatus}" if (rc.nil? || ! rc || $?.exitstatus != 0)The culprit turned out to be a bug in Ruby (#4400, present in 1.9.2-p180) where nested at_exit calls are not unwound in reverse order of registration. There’s a long thread ( capybara #178) that discusses it. What happens is that the Capybara driver registers an at_exit block to terminate the browser, which executes after the RSpec at_exit block and causes the calling process to loose a previously set exit code. There’s a neat workaround that was committed to minitest (diff) and applying the same workaround to rspec-core (pull #410, diff, or just try my fork) resolves this issue. I hope rspec-core takes my pull request, it looks pretty innocent.
There’s also a fun part to the story: @febuiles who I had the pleasure to work with for a bit, tweeted this:
That’s what I love about open-source, it’s a small world that enables quick and meaningful impact.