Duration: 1 day, 7 hours and 1 minute

Affected Components: Email Sending

Oct 12, 12:31:00 GMT+0
Resolved - .

Oct 12, 12:00:00 GMT+0
Identified - An update deployed over the weekend resulted in issues tracking the status of emails sent through one of our communication providers. All impacted emails will have shown up in a "processing" state despite having been sent. When we were alerted of this error, we attempted to re-deliver all emails which were believed to be stuck in the processing state, however later discovered that these in fact had already been sent out, and the issue was tracking related. All emails are now being routed via the email provider which was not impacted by this issue. Some customers may have received duplicate copies of emails from the last 3 days in the last hour (1-2x). No further duplicate emails will be received, and we're working on a fix to ensure this does not happen again..

Duration: 4 minutes

Affected Components: Web interace, Customer portals, API

Oct 10, 21:29:31 GMT+0
Resolved - This downtime was caused by a reboot to apply configuration changes to the load balancer..

Oct 10, 21:25:06 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 11 minutes

Affected Components: Mailchimp, Letter Sending, SMS Sending, Email Sending, Xero, QuickBooks, Stripe

Sep 29, 10:47:04 GMT+0
Resolved - .

Sep 29, 10:36:14 GMT+0
Identified - A bug resulted in a large number of tasks being added to the background processing queue, causing delays to the processing of tasks (eg. communication delivery, notifications, integration sync). We're currently removing these tasks from the queue as fast as possible to allow important tasks to resume..

Duration: 13 minutes

Affected Components: Web interace, API, Customer portals

Aug 8, 13:18:38 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Aug 8, 13:05:51 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 7 minutes

Affected Components: Web interace, API, Customer portals

Jun 9, 09:48:26 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Jun 9, 09:41:07 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 4 minutes

Affected Components: Web interace, API, Customer portals

May 3, 14:06:03 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

May 3, 14:01:53 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 3 minutes

Affected Components: Web interace, API, Customer portals

Apr 13, 19:40:45 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Apr 13, 19:38:06 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 22 minutes

Affected Components: Web interace, API, Customer portals

Apr 9, 15:17:44 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Apr 9, 14:56:08 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 10 minutes

Affected Components: Web interace, API, Customer portals

Mar 30, 19:35:29 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Mar 30, 19:25:53 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 5 minutes

Affected Components: Web interace, API, Customer portals

Feb 15, 11:40:31 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Feb 15, 11:35:07 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 13 minutes

Affected Components: Web interace, API, Customer portals

Jan 25, 20:43:45 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Jan 25, 20:30:51 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 6 minutes

Affected Components: Web interace, API, Customer portals

Jan 17, 20:20:25 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Jan 17, 20:14:06 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 1 minute

Affected Components: Web interace, API, Customer portals

Jan 7, 21:38:26 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Jan 7, 21:37:06 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 4 minutes

Affected Components: Web interace, API, Customer portals

Nov 29, 22:45:30 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Nov 29, 22:41:05 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 3 hours and 12 minutes

Affected Components: Web interace, Customer portals

Nov 18, 14:42:00 GMT+0
Resolved - Cloudflare is reporting this incident is now resolved, and we expect that access to the web interface and customer portals should be back to normal now. We are continuing to monitor the situation, and work on post-incident remediation tasks to minimise the impact of a similar incident in future..

Nov 18, 14:26:58 GMT+0
Monitoring - We've updated the error page being returned to users due to this incident to provide more details, and link to a temporary alternative interface we've setup that does not rely on Cloudflare, at <https://jobstaq-web-v2.netlify.app/> We are experiencing intermittent issues updating this status page, as the provider also relies on Cloudflare for some services, so updates may be delayed. Work has begun to setup a fallback for the web interface when Cloudflare is offline, to reduce the impact of future outages of this kind, to essential functionality..

Nov 18, 13:00:00 GMT+0
Identified - Issues have continued after a short period of functionality..

Nov 18, 12:21:00 GMT+0
Resolved - This incident has been resolved..

Nov 18, 11:30:00 GMT+0
Investigating - Cloudflare, one of our hosting providers, is currently experiencing issues with their network, causing issues loading the JobStaq web interface and customer portals. <https://www.cloudflarestatus.com/>.

Duration: 16 hours and 11 minutes

Affected Components: Letter Sending, Web interace, API, Email Sending, SMS Sending

Oct 24, 23:11:25 GMT+0
Resolved - All background jobs have been processed. We are working on improving monitoring & alerting around background jobs to ensure this does not happen again..

Oct 24, 23:06:55 GMT+0
Monitoring - The backlog of queued background messages is currently recovering, and should catch up shortly..

Oct 24, 07:00:00 GMT+0
Investigating - Background jobs are failing to execute in a timely manner..

Duration: 1 minute

Affected Components: Customer portals, Web interace, API

Sep 27, 19:37:37 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Sep 27, 19:37:06 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Affected Components: Customer portals, Web interace, API

Jun 30, 21:39:30 GMT+0
Resolved - Our automated monitoring has detected this outage is now resolved..

Jun 30, 21:39:08 GMT+0
Investigating - Our automatic monitoring has detected a full outage affecting our API, web interface and customer portals. Our technical team have been notified and will investigate promptly and post updates here as appropriate. Our mobile app will continue to function, and syncing will resume once this issue is resolved..

Duration: 12 minutes

Affected Components: Customer portals, Web interace, API

Jun 1, 20:57:00 GMT+0
Resolved - This incident has been resolved. During the release of a change intended to enhance resilience of our scheduling system and update our routing database (which previously had a single point of failure on one server), a configuration error was made which resulted in auto-scheduling functionality to become unavailable. This error has now been corrected, and as a result of the changes that were deployed, should be significantly less likely to occur again in the future..

Jun 1, 20:45:00 GMT+0
Investigating - We're aware of an issue impacting automatic scheduling, and working on a fix..

Duration: 42 minutes

Affected Components: Web interace, API

May 19, 09:49:27 GMT+0
Resolved - This incident has been resolved..

May 19, 09:19:00 GMT+0
Monitoring - A majority of the servers forming our database cluster went offline, forcing the database into read-only mode, and bringing the site offline. The database cluster has now been restored to handle writes, and we're monitoring to ensure all servers in the database cluster are healthy..

May 19, 09:07:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 23 hours and 39 minutes

Affected Components: SMS Sending

May 5, 17:56:51 GMT+0
Resolved - An update is currently being released which fixes sender IDs for users who are using their mobile number as the sender. We apologies for any issues this issue has caused. As a result of this incident, JobStaq is now much more fault tolerant to SMS provider issues, and we've got several options to quickly switch to if any issues are identified going forwards..

May 5, 13:53:54 GMT+0
Identified - An update is currently rolling out to transition messages sent from alpha-numeric sender IDs to a new provider, which should be complete in the next few minutes. Messages configured to be delivered with a mobile number a sender will continue to be sent from a pool of random numbers until a replacement provider has been identified..

May 5, 10:50:39 GMT+0
Identified - We've identified an alternative messaging provider to support alphanumeric sender IDs, and will begin working to transition SMSs from these senders over to their platform. We're still figuring out the best path forwards for users who have the SMS sender as their mobile number..

May 4, 18:18:08 GMT+0
Investigating - Our SMS provider has made changes to their sending interface without any notification to us, which has resulted in all messages being sent from a shared pool of random UK mobile numbers, rather than the configured sender name or mobile number. It appears this issue started on April 1st, 2025\. We're investigating the best steps forwards from here to get SMS messages back to being sent from the intended senders..

Duration: 18 hours and 34 minutes

Affected Components: Customer portals, Web interace, API

May 1, 20:25:29 GMT+0
Resolved - All systems are back to regular operations. Root cause: A scheduled automated task to apply the latest TLS security certificates to our load balancer began at 2:45AM. At 2:51AM, the changes were applied to the load balancer, however, failed to save correctly, which resulted in all HTTPS traffic failing to be served. Automatic alerts did not reach an engineer due to iOS sleep mode being incorrectly configured to silence these notifications, resulting in a delay to notice the issue. On discovering the issue, traffic was immediately redirected away from the load balancer to a single backend server, to reduce user-facing impact of this issue. As of 8:55AM, all user requests were being served as usual. We encountered some delays in getting services fully restored, due to lacking error messaging from our infrastructure provider's interface. The load balancer HTTPS components were deleted and re-deployed from scratch, which remedied the issue..

May 1, 07:55:26 GMT+0
Identified - A temporary fix has been put in place to bring things online while we identify the root cause of the problem. At this stage, all user-facing impact should be resolved, although the affected components are under "degraded performance" as our automatic fault tolerance is no longer in effect, until the load balancer is healthy again..

May 1, 01:51:00 GMT+0
Investigating - HTTPS traffic towards our API, web interface, and customer portals is currently not being served..

Duration: 5 minutes

Affected Components: Customer portals, API, Web interace

Mar 20, 14:38:00 GMT+0
Resolved - This incident has been resolved. This impacted anyone with a logo configured in their company's account which is included on the PDFs. The cause of this issue was due to recent changes to our infrastructure security configuration to prevent malicious bots causing impact on services was preventing our PDF generation system from loading logo images to include on PDFs. .

Mar 20, 14:33:00 GMT+0
Investigating - PDF generation for quotes, invoices and forms are currently failing..

Duration: 1 hour and 16 minutes

Affected Components: Customer portals, API, Web interace

Mar 12, 20:14:21 GMT+0
Resolved - This incident has been resolved..

Mar 12, 18:58:00 GMT+0
Identified - Our hosting provider's storage system is currently experiencing an outage, causing issues for any JobStaq requests that need to read or write from stored files. We are monitoring for status updates to determine our best course of action. .

Duration: 22 minutes

Affected Components: Customer portals, API, Web interace

Oct 14, 16:51:59 GMT+0
Completed - Maintenance has completed successfully..

Oct 14, 16:30:00 GMT+0
Identified - In response to today's outage, we're performing maintenance on our servers this evening to increase resource allocations to avoid a repeat of the cause of today's issue. If all goes to plan, you shouldn't notice any major disruption to your usage of JobStaq..

Duration: 27 minutes

Affected Components: Customer portals, API, Web interace

Oct 14, 11:30:19 GMT+0
Resolved - The cause of this outage has been identified, and is due to database instances having crashed due to insufficient resources. JobStaq is served from a database setup consisting of 3 servers, and the system continues to operate without issues if one server crashes. At 2024-10-14T10:50:35.559404Z, server 3 of our UK cluster crashed due to insufficient resources, and at 2024-10-14T10:50:35.566518Z this was identified, with server 2 taking over responsibility of the primary (leader) database server. However, shortly after at 2024-10-14T10:51:51.217720Z, this server went offline before server 3 was able to recover - again due to running out of resources, which caused the database to be in a state where manual input is required to bring the system back online. We'll be performing upgrades on all database servers this evening to increase the resources avaialble, to ensure this doesn't occur again..

Oct 14, 11:15:56 GMT+0
Monitoring - The database has been brought back online, we're continuing to investigate what happened to ensure the issue is resolved and prevent this occurring again..

Oct 14, 10:51:00 GMT+0
Investigating - We've run into an issue with our database. This is being actively investigated, and updates will be provided as we know more about the issue..

Duration: 1 day, 22 hours and 12 minutes

Affected Components: Customer portals, Web interace, API

Mar 4, 22:11:53 GMT+0
Completed - Over the course of today, we've gradually transitioned services over to the new cluster of servers, and as of writing, the old server has been turned off, soon to be decomissioned once we are confident there are no issues with the new setup. There were a couple of minor outages lasting a few minutes over the course of today - apologies for any inconvenience, though going forwards, the new setup should increase reliability and allow us to perform maintenance on systems without any noticable impact to users..

Mar 3, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 3, 00:00:00 GMT+0
Identified - In response to recent performance and reliability issues, we're making some large changes to our infrastructure to improve the speed and reliability of the software going forwards. During this maintenance window, access to the JobStaq web interface, customer portals, and API may be unavailable..

Duration: 55 minutes

Affected Components: Web interace, API

Feb 18, 16:22:23 GMT+0
Completed - Everything is operating again, and the maintenance window has finished. Thanks for your patience, we'll continue to monitor things to make sure everything continues working as intended..

Feb 18, 15:27:00 GMT+0
Identified - We've put the API and server into maintenance mode while we work on creating backups and migrating data over to the new database..

Feb 18, 12:00:00 GMT+0
Identified - We're making changes to our database infrastructure to increase resilience, in response to the brief outage experienced earlier this week. As a result, our API and web interface will be offline for a short period of time during this upgrade. The mobile apps will continue to work, though syncing will be unavailable until this upgrade has completed..