After a close monitoring period of 12 hours, we can confirm that the service remains normal with full ability of receiving, processing, and delivering traffic as designed. We again sincerely apologize for the inconvenience caused during the major incident and we would like to express our gratitude for your kind understanding, support extended to all our customers and operators.
Should you need any clarification, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect and then option 1).
Posted May 27, 2023 - 03:46 UTC
Update
Dear Clients,
We appears to have fully restored normal service and our observation suggests traffic is being received, processed, and delivered well. However, We would like to monitor for next few hours before calling for a full-service normalcy.
Should you need any clarification, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect and then option 1)
Posted May 26, 2023 - 22:14 UTC
Monitoring
Dear Clients,
We appears to have fully restored normal service and our observation suggests traffic is being received, processed, and delivered well. We would like to monitor for next two hours before calling for a full-service normalcy.
Should you need any clarification, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect and then option 1)
Posted May 26, 2023 - 16:54 UTC
Update
Dear Clients,
We observe larger part of the service has recovered and serving our customers' traffic without any errors. There's a subset of SMPP customers' traffic we are still queuing, and we are doubling down our efforts in investigation and remediation. We apologize for the inconvenience and thank you for understanding.
Should you need any clarification, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect and then option 1)
Posted May 26, 2023 - 11:55 UTC
Update
Dear Clients,
This is to update that while the majority of the systems and applications were restored by our engineers overnight GMT, some VMs are still unavailable, and our investigations are bringing these up. We will provide further update once the remaining core systems are operational and full traffic processing is restored.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 26, 2023 - 08:31 UTC
Update
The part replacement has been successful so far and the storage cluster is performing as expected for our monitoring. We are gradually recovering the service and therefore, our customers, operators should see their connectivity restored by now. Validation checks are underway now.
As we make more progress, we'll keep you informed.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 26, 2023 - 03:32 UTC
Update
Dear Customers,
The charging of spare units is complete, and the battery backup unit has been replaced. The CMP and Message Manager portals are accessible to the customers.
Our System Engineering team is working to restore the power source in our data center and rebuilding servers at the earliest.
As we make more progress, we'll keep you informed.
If you have any questions, for Operator Services please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 26, 2023 - 00:53 UTC
Update
We are continuing to work on a fix for this issue.
Posted May 25, 2023 - 21:32 UTC
Update
Dear Customers,
The replacement component has arrived from Germany and charging of the units is on-going. Our technicians in the datacenter have started the replacement procedure. It is a sensitive procedure that will take at least 2-3 hours. Meanwhile our efforts in rebuilding some of the critical servers with application is still underway to remediate earlier if possible.
As we make more progress, we'll keep you informed.
If you have any questions, for Operator Services please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 25, 2023 - 21:31 UTC
Update
Dear Customers,
We have a revised part arrival time at 22:00 CEST / 20:00 UTC. Meanwhile our efforts in rebuilding some of the critical servers with application is still underway in an effort to remediate earlier if possible.
On 25 May 2023 at 06:15 GMT, two redundant power supply units (hardware failure) powering Sinch Interconnect storage infrastructure failed affecting our complete ability to receive, process and deliver customers’ messages. This is an extremely rare and remote scenario where both power supply units failed at the same time. It is possible our customers and operators’ connectivity is affected by this major incident. Our data centre provider is currently working hard in sourcing an alternate power supply unit from entire Europe region. We have a potential replacement arriving from Germany and an ETA for that is about 3 hours from now. Meanwhile we understand this is by now a prolonged outage and this has never happened before and we sincerely apologize for this inconvenience. We seek your kind understanding of this rare / remote nature of the incident and we would come back and provide an update on an hourly basis.
Should you need any clarification, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect and then option 1).
Posted May 25, 2023 - 17:26 UTC
Update
Dear Customers,
Our technical experts internally are constantly investigating on priority to bring back the services, we will update further as we progress.
We are writing to give you an update that the power source in our data center has failed leaving our Extreme I/O Storage Cluster without power. This has an impact on the flow of traffic to customers and operators. To quickly replace the power source, we have actively engaged remote engineer from Vendor who will provide ETA following further investigation.
As we make more progress, we'll keep you informed.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 25, 2023 - 09:31 UTC
Update
We are continuing to investigate this issue.
Posted May 25, 2023 - 07:43 UTC
Investigating
Ticket Number :: MNOC-48038 Services:: DI SMS 365(iSMS), DI SMS 365 (SMSx), DI SMS 365 Enterprise HUB,
Description :: We are writing to inform you that one of the host is down on our data center, which will impact delay in delivery of MO, MT and DLR traffic. Internal teams are proactively investigating the issue and will update on progress as soon as possible.
Impact ::
DI SMS 365(iSMS)
Mobile Terminating (MT) Traffic: Impacted Mobile Originated (MO) Traffic: Impacted DR Notification : Impacted
DI SMS 365(SMSx) Mobile Terminating (MT) Traffic: Partially Impacted Mobile Originated (MO) Traffic: Partially Impacted DR Notification : Partially Impacted
DI SMS 365- Enterprise Messaging HUB. Mobile Terminating (MT) Traffic: Impacted Mobile Originated (MO) Traffic: Impacted DR Notification : Impacted
Incident Start Date & Time (GMT) : 25th May 2023, 06:15 Hrs. GMT
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted May 25, 2023 - 06:49 UTC
This incident affected: Sinch DI Services (SMS 365 Operator RoW).