I have been recovering from Covid19. Not the way I wanted to spend the early weeks of 2021, coming on the heals of a disastrous 2020. However, having Covid did provide some time to think about Disaster Recovery and always having the right personnel available. Therefore, it is of utmost importance to have the correct DR personnel ready if something happens to your data center.
When we consider DR exercises, we usually plan for this. Yes, sometimes we will have an impromptu exercise where we have not told people that they will need to be ready or who will be involved. However, unless you work in an environment where doing this is possible, planning for certain unknown issues rarely becomes part of the process.
Case in point, my getting Covid. If the organization that I worked for had a disaster, would we be prepared with the reality that the DR Manager would not be available to run the recovery? That is a question that many organizations need to ask themselves. Is there a backup DR Manager, or someone who has enough experience within the DR recovery process to at least help get you through? What about other personnel?
This example goes to show why it is crucial to cross-train your DR resources. From the DR Manager to every technical resource required in a recovery, having a person who can perform the necessary steps to recover your data center is the most important tool at your disposal. If only one DBA is trained to recover the necessary databases and perform this within your DR colocation, knowing what servers are used, what naming schemes are employed, and even what needs to be done from a multi-node implementation becomes incredibly important to the overall recovery process.
The same holds true for networking, servers, storage, middleware, and application development, in addition to many more. You name it, every part of your infrastructure and application environments are crucial to the overall recovery process. If your organization only has one application developer knowledgeable in setting up a certain critical application in the DR environment, you had better make sure that a backup is adequately trained to handle the work if the primary is not available. This could include such things as configurations and files necessary for the application to be recovered properly.
Planning for a DR is more than just knowing what needs to be recovered. It requires having the necessary resources available no matter what the circumstances are. Cross-training can help alleviate many issues you may run into if resources have Covid, are injured in the disaster, or are not able to get access to communications or networks to perform their tasks. Having the correct DR personnel ready in the event of any disaster is of vital importance in setting up a quality DR plan.
