Remote Access service disruption on alero.io data center was identified

Incident Report for CyberArk Remote Access

Postmortem

Following planned maintenance work on Sunday, March 19th 2023, customers in the Remote Access US Data Center have experienced difficulties connecting to applications via Remote Access. The interruption started at 6am EST and ultimately resolved at approximately 1pm EST.

During the maintenance work we have issued an update to all SSH tunnels in order to apply additional security and performance enhancements.
As a result a large number of tenant configuration sets were automatically propagated at the same time, which caused unexpected stability problems on our database instance.

After troubleshooting and debugging by the Remote Access engineers and developers, along with AWS engineers, the issue was finally resolved by making configuration changes to our backend systems order that reduced the load on our database instance.

Mitigation steps:

Working to investigate with our DB provider why it could not stand the original load
Continue operating with the changes made that will prevent the same issue from recurring in future updates
Introducing additional load testing to our QA process

Posted Mar 22, 2023 - 09:35 EDT

Resolved

The issue affecting application access via Remote Access (formerly Alero) is resolved. We thank you for your patience and apologize for any inconvenience.

Posted Mar 19, 2023 - 13:15 EDT

Monitoring

A fix has been implemented. We continue to monitor the service closely as functionality returns to normal.

Posted Mar 19, 2023 - 13:07 EDT

Update

Our engineers are still working on resolving the issue.
We will report back here as soon as more information is available.

Please do not hesitate to contact support by email at support@cyberark.com or by visiting the customer community, https://cyberark-customers.force.com/

Posted Mar 19, 2023 - 12:42 EDT

Update

We appreciate your patience while we our engineers are working on resolving the issue.
We will report back here as soon as more information is available. Please do not hesitate to contact support by email at support@cyberark.com or by visiting the customer community, https://cyberark-customers.force.com/

Posted Mar 19, 2023 - 09:28 EDT

Update

We are continuing to work on a fix for this issue.

Posted Mar 19, 2023 - 07:49 EDT

Identified

CyberArk has identified a possible service disruption and we are working to resolve this. This impacts accessing Remote Access applications via the alero.io portal. Other regions are not impacted. We thank you for your patience at this time.

Posted Mar 19, 2023 - 07:47 EDT

This incident affected: Remote Access US (Authentication Service, Connector Service, Mobile Service, Web Portal).