JobStaq - Notice history

This incident has been resolved.

During the release of a change intended to enhance resilience of our scheduling system and update our routing database (which previously had a single point of failure on one server), a configuration error was made which resulted in auto-scheduling functionality to become unavailable. This error has now been corrected, and as a result of the changes that were deployed, should be significantly less likely to occur again in the future.

A majority of the servers forming our database cluster went offline, forcing the database into read-only mode, and bringing the site offline. The database cluster has now been restored to handle writes, and we're monitoring to ensure all servers in the database cluster are healthy.

An update is currently being released which fixes sender IDs for users who are using their mobile number as the sender.

We apologies for any issues this issue has caused. As a result of this incident, JobStaq is now much more fault tolerant to SMS provider issues, and we've got several options to quickly switch to if any issues are identified going forwards.

An update is currently rolling out to transition messages sent from alpha-numeric sender IDs to a new provider, which should be complete in the next few minutes.

Messages configured to be delivered with a mobile number a sender will continue to be sent from a pool of random numbers until a replacement provider has been identified.

We've identified an alternative messaging provider to support alphanumeric sender IDs, and will begin working to transition SMSs from these senders over to their platform. We're still figuring out the best path forwards for users who have the SMS sender as their mobile number.

All systems are back to regular operations.

Root cause:

A scheduled automated task to apply the latest TLS security certificates to our load balancer began at 2:45AM. At 2:51AM, the changes were applied to the load balancer, however, failed to save correctly, which resulted in all HTTPS traffic failing to be served. Automatic alerts did not reach an engineer due to iOS sleep mode being incorrectly configured to silence these notifications, resulting in a delay to notice the issue.

On discovering the issue, traffic was immediately redirected away from the load balancer to a single backend server, to reduce user-facing impact of this issue. As of 8:55AM, all user requests were being served as usual. We encountered some delays in getting services fully restored, due to lacking error messaging from our infrastructure provider's interface. The load balancer HTTPS components were deleted and re-deployed from scratch, which remedied the issue.

A temporary fix has been put in place to bring things online while we identify the root cause of the problem. At this stage, all user-facing impact should be resolved, although the affected components are under "degraded performance" as our automatic fault tolerance is no longer in effect, until the load balancer is healthy again.