As at 09:48 GMT 29 June 2023, services were restored by deploying application hosts configurations enabling them to communicate with alternate storage array. Past 24 hours of close monitoring reflects services remain normal and operational without any abnormalities. If you have further questions or concerns, do not hesitate to reach out to essupport.digitalinterconnect@sinch.com.
Posted Jun 30, 2023 - 16:15 UTC
Update
Dear Client,
We can confirm that the incident is restored and all Mobile Originating (MO ) messages that were held in queue have been processed and delivered. We can also confirm that no messages were lost during the incident period. Our groups continue to move affected hosts to alternate storage networks to restore the stability and normalcy. We would continue to keep you posted on the progress.
We are writing to inform you that the third Party vendor determined a corruption in storage arrays causing the incident. As next steps of recovery: We are moving affected application hosts onto an alternate storage array in order to resume processing of the messages held in queue. Meanwhile, we would continue our efforts in rebooting the affected storage array and explore possible restoration of corrupted data from backups. These steps are likely to take next two to three hours and therefore, we would keep you updated after two hours from now.
We are writing to inform you that customers messages via our product Message Manager is not affected and our internal tests have been successful so far. Few customers mobile originating (MO ) messages continues to be held within the service and delivery would be delayed. Our Support Groups & 3rd party engineers continues to triage on priority for a remediation.
Ticket Number :: MNOC-48886 Services:: DI SMS 365-Enterprise Messaging
Description ::
Our monitoring indicates that mobile originating (MO ) messages are being held from processing and delivering due to downstream connectivity anomalies and therefore, few of our customers may observe MO, MT, DR delays. Our Support Groups including 3rd party engineers are actively triaging on priority for a restoration / remediation.
As we make more progress, we'll keep you informed
Impact :: Mobile Terminating (MT) Traffic: Delay in Delivery Mobile Originated (MO) Traffic: Delay in Delivery DR Notification : Delay in Delivery
Incident Start Date & Time (GMT) : 29 June 2023, 00:06 Hrs.