Previous incidents

We have scaled services to address the issue.

We are changing the way scaling works for test runs. This requires us to migrate our database and have a short window of downtime for test runs. Test runs will resume after maintenance.

Due to problems with database backups, we are taking a short downtime to switch to some newer database infrastructure. Runs that were started before the downtime may fail but should succeed when rerun.

We will be providing a post mortem by the end of the week.

This is completed

Test runs are scheduling again, we are still actively monitoring to make sure there are not additional issues.

app.qawolf.com and all test runs will be down for planned maintenance for approximately 30-60 minutes. Scheduled runs will run after the downtime. Deployment notifications may receive error responses and will need to be resent after the downtime.

We identified the cause and runs are happening normally now

This partial run outage is now resolved. For approximately 2 hours, some test runner nodes were unable to pull images from DockerHub, leading to scheduled runs failing to start. We took several steps to mitigate the issue but we believe it to be a temporarily issue with the DockerHub API that is resolved. At this point, all runs seem to be succeeding normally.