Editor’s note: Mike Trudnak is executive director of IT services, and Daryl Porter is chief technology officer at Clinipace Worldwide, a global digital clinical research organization based in Morrisville. As the recent tornadoes and the recent outages of the Amazon “cloud” network and the Sony PlayStation network all clearly indicate, having disaster recovery and backup plans are essential for any business that relies on the Internet.
MORRISVILLE, N.C. – We’d planned for years, preparing for what happened this past weekend. We didn’t want it to happen, of course, but we were ready when it did. And because we planned, we were able to easily roll with the punches that Mother Nature doled out.
As a global digital clinical research organization specializing in fully integrated clinical research services for biopharmaceutical and medical device firms, Clinipace Worldwide works with clinical data that’s closely regulated by the federal government. Losing that data could have an enormous impact on our business. Because we are federally regulated, we have a legal obligation. If were found to be negligent in the handling of our data, the FDA or our clients could criminally charge us. At a minimum, we might lose a client; there are many clinical research organizations that are no longer in existence because they mismanaged a client’s data. Clinical trials are too complex and, in many cases, too expensive to allow the data to be compromised and risk the trial.
Monitors and clinical research associates travel around the country and world for months to years at a time working with sites (doctor’s offices, hospitals, etc.) to collect data in a regulated fashion. Many of these trials are run to develop medications for very sick people, so simply going back and recollecting it would never be possible. The data is analyzed to the nth degree; working with incomplete data sets is simply not an option.
Since Clinipace is federally regulated, we are required to establish a comprehensive disaster recovery plan. That plan includes provisions where we have to regularly test the plan every 60 days to make sure what is supposed to work will, in fact, work if and when we need it. We decided a long time ago that our company needed a trusted partner to help us implement our disaster recovery plan, but to also help us run our networking infrastructure. We chose Windstream Hosted Solutions because its data centers are secure from both a physical and cyber perspective. They also proved to us that they were capable of staying up and running, even if businesses all around them could not. And we knew that they had other data centers, including two in Charlotte, which could easily be provisioned as a backup to our backups if the worst occurred.
So, when Raleigh got hit last weekend with tornadoes, we were ready.
Tornadoes ripped through the region, coming within several miles of our homes. From our perspective, however, everything was totally transparent. We received an email from Windstream about 6:00 pm Saturday night, informing us that its Raleigh data center was running on backup power. Then, we received another email Sunday morning, letting us know the data center was back on primary power. That’s all we needed to know that our data was safe and secure. We’d never needed to activate our DR plan before, but we came very close to having to do so this time.
There are, of course, lessons from this storm that any business can implement, regardless of its size:
• Get a DR plan in place. Amazingly, numerous studies show about one-third of all companies don’t even have an idea of what they’ll do when a disaster occurs. These businesses are almost asking for a catastrophic failure, because they’ll lose their data in the process.
• Test the plan. Simply having the plan in place doesn’t mean it’s going to work. People leave an organization and new ones join. New facilities are added. Your DR plan must be regularly updated to ensure that all of these changing factors are taken into account. We’ve heard it said that companies that have a plan but don’t test it are actually in worse shape that those that have no plan at all, because they’re relying on a false sense of security which could easily doom their firm.
• Make the testing a regular part of your business processes. If the testing is regarded as something of an afterthought, it’ll be left behind all too often. And, as outlined above, that’s begging for trouble. (Incorporating DR into our business processes has also proven to be a good selling point for prospects.)
• Separate your primary and backup data centers. Windstream Hosted Solutions hosts our primary production site in Raleigh as well as our backup site at its centers in Charlotte, which are two hours apart. This enables Clinipace to keep its backup system separated enough so that if a disaster hits one site, it will not hit the other. We considered separating them further, but agreed internally that part of our business processes would include regular inspection of both sites. We didn’t want to have to fly across the country in order to do so.
• Find a provider who understands your specific needs. Our consultants at Windstream Hosted Solutions have worked with us over the years, understanding that our requirements have changed as we’ve grown, and helping us develop specific strategies to meet them. Your needs are different from ours…and your needs next year will be different that what they are today. Finding a trusted, long-term advisor who can work with you to craft solutions that meet your challenges is vital.
We may never need to fully implement our DR plans, but we came far closer this month than we ever have before. And the knowledge that we were ready allowed us to sleep far more soundly that Saturday night.
Get the latest news alerts: Follow WRAL Tech Wire at Twitter.
