[Oct 12 2021 12:30PM]

We have observed, and received reports from over the weekend, of email processing events where outgoing email can be delayed on average between 15-30 minutes, and up to 2 hours in extreme cases.

We are working to identify the issue at hand, and will perform updates as required to resolve it.

Please note that all emails are being successfully sent and no action is required to have emails delivered at this time.

We are continuing to monitor the service for delays, and will update this post with more information as it becomes available.

[Oct 13 2021 11:00AM] 

We have released an update to our email processing systems to help reduce the delayed processing events.

This update has split our email pipeline into multiple processing queues, allowing us to dynamically assign emails into different queues for processing. This gives us the ability to move specific email types to their own queue when and if they’re causing any delays, allowing the system to process the queue in a more efficient manner.

We are closely monitoring this update, and will post more information as it becomes available.

[Oct 15 2021 11:00AM]

Splitting the ClubRunner email processing into multiple queues allowed us to get more diagnostic data and identify the primary cause of the delayed processing events. Yesterday, we released an update to address the processing issues. 

Monitoring after the update, we're now seeing most emails sent out in less than 2 minutes. We are continuing to monitor the performance of the email system. 

We will continue to post more information as it becomes available.

[Oct 18 2021 3:10 PM]

We have not observed any new delayed email processing events over the weekend.  We are continuing to see average processing times under  2 minutes.

We will continue to monitor, and post more information as it becomes available.

[Oct 20 2021 12:00 PM] 

We have not observed  any new emails delays since our update on Friday Oct 20 2021, and are marking this issue as resolved.