At GCA, we are our client’s trusted partners. In times of crisis, we work hard to ensure they can rely on us to resolve any incidents quickly. Our top resources react fast in high-pressure situations and we keep an open line of communication with key stakeholders to keep them aware of progress.
Let’s dive into one example of our crisis resolution.
Background
One of GCA’s clients is a major healthcare organization in the Southeast United States with more than 70,000 employees. For years, GCA has been continuously engaged in working with the client’s IDM to maintain and enhance their environment.
In 2019, this client started using Google Workspace and implemented a BYOD policy. Fast forward to the summer of 2021 (long after the project was live) when an unexpected issue occurred.
The problem with Identity Management in general, or rather what makes it challenging, is that it is at the mercy of many connected systems. Even if every corner use case is accounted for, IDM systems are going still have issues when a DBA changes a service account password or the networking team changes a configuration.
The Problem
In this instance, Google fixed a bug in their product which triggered a massive outage for doctors, nurses and C-Suite executives.
On a regular Thursday morning, 18,000 employees at the healthcare organization woke up to find all their Google tools (like Gmail, Calendar, Chat, Docs etc.) removed from their cell phones overnight. Can you imagine waking up as a CIO or as a doctor and not being able to view your calendar for the day?
How It Happened
When this client rolled out the Google Workspace, they also defined a BYOD policy. In this policy, it stated that the pin used to unlock phones needed to be changed every 90 days.
In the Google admin tool, this was properly set. But there was a bug. The Google software wasn’t enforcing the pins’ 90-day expiration, so it never asked end-users to change their pin. Because the devices weren’t forcing a pin change, nobody proactively changed their pins.
This went unnoticed until Google fixed their bug. With the bugs fixed, all of sudden 18,000 BYOD phones were now out of compliance. And the policy for a BYOD that is out of compliance is to wipe the phone of all company data and
Google Workspace apps.
This is where the right team in IT comes into play. This was full crisis mode. Meetings were canceled and all focus from a wide team was set to resolve this business-impacting issue. Very quickly, our client realized some of the best resources to pull in were the GCA team.
The Timeline
Acting as an extension of their organization, GCA reorganized work priorities and schedules within 15 minutes. Here is how the issue got resolved:
08:20 – The client contacts GCA
08:35 – GCA begins root cause analysis.
09:00 – GCA has the issue identified and gets on a call with the client’s security directors to facilitate communications with Google.
10:00 – GCA works with clients to run tests and review Google API to determine options.
11:30 – GCA and client have a strategy identified. The list of affected users is collected from logs and GCA begins the development of a script to re-approve all affected devices.
1:30 – GCA meets with the client and tests the script with a small batch of 10 devices. Tests are successful.
2:00 – Emergency change window is opened, and script is executed on all devices.
5:00 – Script completes with a 99.99% success rate. The remaining outliers were found to be C-level employees who had IT resolve them manually or people who went through the process of re-enrolling, changing pins and
re-downloading data manually.