Calling Services - Performance Degradation on US-EAST

Incident Report for Sinch

Postmortem

Reason for Outage:

A Sinch deployment on US-East on February 19, 2024 resulted in an ‘out of memory’ situation which also caused connectivity issues and exposed a bug in the media resource layer. As the customers experienced trouble completing calls, the additional load of the call re-attempts resulted in further challenges stabilizing the platform. At production load on Thursday, February 20, 2024, a side effect of the updates caused processes to crash. Once the cause of the crash was identified, a re-deploy was rolled out to resolve the problem.

While it is important to perform maintenance such as this to ensure the quality of our service, Sinch takes all necessary precautions to alleviate any inconvenience to our customers. We are sincerely sorry for disruptions to your service as a result of this event. If you need further details or require the formal RFO, please contact your Sinch Account Manager.

Posted Feb 26, 2024 - 18:26 UTC

Resolved

This incident has been resolved.

Incident start time: 2024-02-19 15:39 UTC
Incident restoration time: 2024-02-20 16:34 UTC
Incident stop time: 2024-02-20 18:24 UTC
Posted Feb 20, 2024 - 21:03 UTC

Monitoring

A fix has been implemented and we are monitoring the results.


Incident start time: 2024-02-19 15:39 UTC
Incident restoration time: 2024-02-20 16:34 UTC
Posted Feb 20, 2024 - 19:03 UTC

Update

We are continuing to investigate this issue.
Posted Feb 20, 2024 - 16:10 UTC

Update

We are continuing to investigate this issue.
Posted Feb 20, 2024 - 14:02 UTC

Investigating

We are writing to inform you that we are currently observing performance degradation for our calling services.

Incident start time: 2024-02-19 15:39 UTC
Incident restoration time: TBA
Incident stop time: TBA

We appreciate your patience and apologize for any inconvenience experienced while this is being worked on.

Please contact support@sinch.com should you require further information or assistance.
Posted Feb 20, 2024 - 12:01 UTC
This incident affected: Voice (Data Calling, Phone Calling (PSTN), Video Calling, SIP Trunk).