This incident has been resolved! We have fixed the immediate issue, which has restored functionality to near-normal levels. Systemically, we are still working through a rather large queue, so email times may be a bit slower than normal. Internally, we are working on cleaning some things up which should prevent an issue of this nature from happening again.
That said, there is a possibility that some emails may have been irretrievably lost, especially those sent earlier in the day. Additionally, we are aware that some Formstack Documents deliveries into Sign were similarly impacted. If impacted, recalling and re-doing the affected documents, as outlined here, should address the issue: https://help.insuresign.com/hc/en-us/articles/360031409952-Recall-Redo. Those who have the Sign delivery from Documents who were negatively impacted may need to re-merge those documents in order to push the delivery through again.
Thank you for your patience and understanding!
Posted Oct 07, 2020 - 21:01 UTC
Monitoring
Email functionality has returned, albeit a bit slower than normal while the queue catches up. Additionally, the delivery connection from Formstack Documents is now working again. We are continuing to monitor system performance while working behind the scenes to ensure stability.
Posted Oct 07, 2020 - 19:56 UTC
Update
We are currently working on a comprehensive fix for the degraded performance issues we have been seeing today. At this time, we do not have an ETA on when email functionality and the Formstack Documents delivery connection will be fully restored, however the application itself and document function appear to have returned to normal. Rest assured, it is all hands on deck until we get this resolved.
Posted Oct 07, 2020 - 19:06 UTC
Identified
While we have identified the root cause of the issue, we are still seeing sub optimal performance. At this time, email functionality has been impacted again, and the Formstack Documents delivery option is still down. We will update once functionality has been restored.
Posted Oct 07, 2020 - 17:54 UTC
Monitoring
We have implemented a fix which should restore functionality. Document uploads should be operational, however those using the Formstack Documents delivery option are still currently impacted. Systemically, we are working through a queue of requests and as that backlog clears, performance should continue to improve. We will continue to monitor the situation to ensure the fix is successful.
Posted Oct 07, 2020 - 16:39 UTC
Identified
We have identified the root cause of the service interruption and are working on implementing a fix.
Posted Oct 07, 2020 - 16:11 UTC
Update
We are continuing to investigate this issue.
Posted Oct 07, 2020 - 16:09 UTC
Update
We are continuing to investigate the issue. We are closing in on the root cause of the service interruption and will update once we are confident we've identified the cause.
Posted Oct 07, 2020 - 16:03 UTC
Update
We are continuing to investigate this issue.
Posted Oct 07, 2020 - 15:00 UTC
Investigating
We are currently investigating an issue where users are experiencing delays in uploading new documents and long loading times.
Posted Oct 07, 2020 - 14:00 UTC
This incident affected: Main Application, Documents, and Email System (SendGrid).