Due to AWS Operation issues, namely "Network services" breakdown since 11.45 PST Sep 19 2023, we were facing outage in our Payroll and Payroll Calendar features for our PX and HRX portals
The root cause communicated from AWS is that their network service performance encountered high latencies due to high load issues in underlining systems.
There were some partial impact to other services randomly if any new connections set up , as per the nature of the incident from AWS.
Around 15:43 PST, AWS notified that there is a workaround being applied step-wise and sooner it would cover more broader systems that impacted our services.
Around 19:15 PST, the fix from AWS restored the underlying services which resolved the related issues. All our systems tested fine and services restored.
Sep 18, 21:07 PDT