A hardware module (Linecard) was identified as faulty and traffic towards the interface card was rerouted to a redundant switch. This should restore stability to the service and restored normalcy.
As a preventative measure, we would schedule a planned maintenance to replace the old switches and migrate traffic over to a new network hardware. Customers would be duly notified of this planned migration. We sincerely apologize for the inconvenience and thank you for understanding.
Incident Start Date & Time (GMT) : 30 July 2024, 15:05 GMT Hrs. Incident End Date & Time (GMT) : 30 July 2024, 15:55 GMT Hrs
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted Jul 30, 2024 - 16:07 UTC
Identified
Dear Clients,
As part of an emergency network change, we are shutting down one of our data center and and rerouting traffic. There can be a momentary queue/ delay to customers messages as we execute this activity as an emergency change. Customers may also observe a momentary disconnect and reconnect during this activity.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted Jul 30, 2024 - 15:45 UTC
Update
We are writing to inform you that our engineers have located the issue with intermittent delays, and are working with applying an emergency fix.
During the implementation of this emergency fix, customers may observe intermittent connection drops for a few seconds.
Posted Jul 30, 2024 - 15:45 UTC
Monitoring
We are writing to inform you that the queues have cleared and our support teams are actively monitoring the issue. We will update on further progress.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted Jul 30, 2024 - 15:35 UTC
Investigating
Ticket Number :: MNOC-54761 Services:: DI SMS 365-Operator Services
Description :: Our monitoring indicates that mobile terminating (MT ) messages are being held from processing and delivering due to downstream connectivity anomalies and therefore, few of our customers may observe MT, DR delays. Our Support Groups are actively triaging on priority for a restoration / remediation.
As we make more progress, we'll keep you informed
Impact :: Mobile Terminating (MT) Traffic: Delay in Delivery Mobile Originated (MO) Traffic: none DR Notification : Delay in Delivery
Incident Start Date & Time (GMT) : 30 July 2024, 15:05 GMT Hrs.
If you have any questions, please contact the Operator Service NOC immediately at operatorservices.noc@sinch.com or +1 470.300.8394 (select ‘Support’ from the menu, option 2 for Digital Interconnect, and then option 1) and reference this notification.
Posted Jul 30, 2024 - 15:30 UTC
This incident affected: Sinch DI Services (SMS 365 Operator RoW, SMS 365 Operator US).