The root cause of this issue was addressed through a software release on 2024-05-29.
Posted May 30, 2024 - 17:35 EDT
Resolved
This incident has been resolved.
Posted May 15, 2024 - 12:22 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 14, 2024 - 22:37 EDT
Update
We are continuing to work on a fix for this issue.
Posted May 14, 2024 - 22:35 EDT
Identified
Analysis identified an exceptionally high database load exceeding acceptable parameters. This surge likely stemmed from the rapid expansion of server resources for data ingestion and call handling implemented the previous day.
Clients experienced slow and incomplete page loads during the outage window.
The following actions were taken to address the performance issues and prevent recurrence:
1. Increased Resource Capacity: The core database was allocated twice the CPU and RAM resources to alleviate immediate capacity constraints. 2. Enhanced Monitoring: An additional monitor for the database load factor was incorporated into the dashboards and operational alarms for proactive identification of future load spikes. 3. Optimized Code: Frequency of automated page reloads, a significant contributor to the database load factor, was reduced.
Posted May 14, 2024 - 19:29 EDT
Investigating
Omnistream database performance degradation resulting in slow and incomplete page loads for clients.