Disaster Recovery exercises are what DR Managers and practitioners live for. It is the chance to show that all the planning that occurs over the course of months or years will allow us to restore our data centers and our applications in the event a true disaster ever occurs. We hold these exercises either monthly, quarterly, semi-annually, or annually to provide the organization with a feeling of security. But what does an exercise look like during Covid?
In the past, many organizations would gather their infrastructure resources together and sequester them to an off-site facility to perform a DR exercise. Back in the day, that probably meant having these resources at the colocation, so they had direct access to tapes, servers, storage, and many other necessary items. Over the past several years, it may have meant getting those same individuals together at a hotel or off-site facility in town where the exercise could be performed.
But what about a time in history where people should not be sequestered with other individuals? With covid ravaging the globe in 2020, this has created the need to shift and adapt how we perform our DR exercises. We have turned what was once a strategic group event with the necessary engineers on-site to one where every aspect of the recovery is handled remotely.
What are the impacts on the overall exercise? I know that my preference has always been in having the infrastructure resources sequestered in the same facility and room for ease of communication. When a discussion needed to occur, a group of infrastructure resources, or maybe the entire team, could either talk in the room, or retreat to another room in the facility to discuss items or importance. And, while technologies continue to improve in online meeting facilitation, it is still not as good as those meetings in allowing for the best forms of communication.
Many things are lost in running in-person meetings, and therefore in a DR exercise online through these meeting facilitation apps. Interruptions are going to happen far more than if the people are in-person and can see the person talking and the others around them. While you can use meeting facilitation apps that allow you to see all the attendees, you are not able to read all the cues that you would be able to in-person. One of the things that this could impact the most is the ability to have these larger group discussions and accomplish what is needed in as timely a fashion as would happen in person.
I am in no way saying that you cannot run these discussions through meeting facilitation apps. I am saying that it could and probably will impact how well the group discussion happens. But are there other things to also consider when running a DR exercise completely remote?
What about having the ability to find or get in contact with a certain engineer quickly? If they were at the DR exercise facility, you would have a good understanding for where they are, whether that was to take a break of some kind, or they went on a short walk to clear her/his head. You have the reasonable expectation that they will return within a set timeframe.
When all those resources become remote, access to each of them becomes more of a challenge. Any person involved in the recovery process can turn off the meeting facilitation software and “step away” for a time. Yes, they probably have a cell phone, but does that guarantee that you, as the DR Manager, will be able to get them to respond? The responsibility shifts to having resources involved who will stay connected during the exercise no matter what.
What about faulty technology or communication issues for one of the resources? While it is much easier to guarantee accessibility and communication while everyone is in the same room, when each of these resources connects from their own networks, there is a chance that one or more could be impacted by some sort of communication or technological issue. Planning needs to be accomplished to verify that this will not occur during the exercise, and for that matter during a true disaster, and that mitigation efforts are predetermined if they do occur. Whether this is some sort of backup technology or additional resources queued up to take over, this needs to be determined prior to any exercise and should become part of the DR Plan.
Many things can impact the successful completion of a DR exercise. And even more can impact a successful recovery during a true DR Event. Making sure that you are ready for any potential event will assist you in being prepared. So, do you know what your DR exercise will look like during Covid, and how you will prepare to handle the issues that arise?
