Current Status: Service Restored
Issue/Impact (reported September 26, 5 PM CST)
Update: September 27, 10 PM
Our engineering team has identified the root cause of the incident and is now working to apply remediation measures to restore service back to normal. We will update this advisory once the service is fully restored.
Update: September 27, 8 PM
Our engineering team continues to work with network and SBC vendors to remediate the situation.
We have isolated the problems to certain types of SIP Traffic that connects to one of our Session Border Controllers in Chicago. Certain types of registration strings that don't failover or may be singularly registered to that SBC are experiencing intermittent audio problems. Others have some redundancy built in via DNS FQDN or otherwise. We disabled the affected port while we are troubleshooting which resolved a large majority of those affected because the system fails over as it should.
For those still having trouble, we continue to need your independent reports so that we can identify which numbers are not properly failing over and put in an emergency route in place until the final issue is resolved.
Current experience is outbound calls connection intermittence. Root cause analysis will be provided once we are finished resolving the issue. We have engaged the experts in each area of our network and systems to expedite this resolution and we will be updating accordingly.
Update: September 27, 6 PM
Remediation activities and triage continues and ongoing. Current experience is outbound calls connection intermittence. This advisory will be updated with further news or when resolution is confirmed.
Update: September 27, 4 PM
Our
engineering team continues with remediation activities. Service intermittence
continues primarily with outbound calls. This advisory will be updated with
further news, or when resolution is confirmed.
Service intermittence continues with outbound calls, and the engineering team continues to work to ensure a prompt resolution. This Advisory will be updated upon further news, or the service being fully restored.
Update: September 27, 12 PM CST
We
continue to see intermittent issues with outbound calls only, and only for some
users. Our team is still working to address these remaining incidents and we
will update this Advisory promptly once we have news or reach a resolution.
Update: September 27, 7:45 AM CST
Update: September 27, 5:25 AM CST
After extensive testing we have confirmed that the service has been restored, we will continue monitoring and please report any outgoing issues placing or receiving calls to support@teamventi.com. RCA will be provided as soon as it becomes available.