Revised August 2020

RTO and RPO overview

This article explains how to use RTO and RPO within your Business Continuity Planning (BCP).

Have you ever wondered what you would do if you if you had to recover a failed system and all its data? It is not always easy to put these theories into practice if you don’t have an identical lab to test your Disaster Recovery (DR) strategy.

One way to help you make an informed decision when designing your DR plan is to use RTO and RPO metrics. RTO and RPO means (Recovery Time Objective), and (Recovery Point Objective) respectively.

If you want to know if you can recover a system after a failure which has seen loss of data, you will need to make some basic calculations.

The industry standard for this uses the metrics RTO and RPO. By providing some information on what you already know, and what you want to obtain, you will have a clear idea of what to expect following a disaster.

RTO and RPO explained

You only need to consider RTO and RPO for the systems you want to keep alive!!

RTO and RPO explained

Two of the most important metrics every organisation and IT Directors must understand are RTO (recovery time objective), and RPO (recovery point objective). Together, these allow organisations to design and implement a robust disaster recovery strategy with a logical backup process which will allow them to restore every failed IT system within their targeted downtime parameters.

On their own, RTO and RPO values aren’t much use, and they must be used together. If you are responsible for planning and implementing a disaster recovery plan, these two metrics will help you get there.

A DR plan using RTO and RPO can, and should be as simple as possible. A basic example could be a laptop which a sole proprietor needs to run their business from, or alternatively a more complex example of an ISP with 1,000 servers in a data centre. Both are businesses which are severely impacted when business continuity fails. 

Each should examine their own RTO and RPO and check if they are achievable.

How will RTO and RPO help my organisation?

These two metrics when used together will specify how long a system can be offline for, and how much data can be lost from it before your business processes suffer significant harm.

When you agree on your metrics, you can then benchmark your systems and processes to see if they comply with what you need.

Both will help you calculate how often your backups should run, and what your acceptable restore recovery time will be. From there you will know how your data recovery process should work.

How we protect your data

  • Encryption explained
  • What happens to your data at rest?
  • Know your data can't be compromised

RTO (Recovery Time Objective)

Recovery Time Objective

RTO is the time it takes to restore a previously working system and this must be within a previously defined Service Level Agreement (SLA). Returning a previous working system to normality after an outage is often referred to as Business Continuity.

Scenario
A call centre (let’s call it Buddy Care) with 1,000 operators uses a customer management system to handle all customer enquiries. The system runs from a cluster of Linux servers and the company can’t deal with any customer queries or orders whilst it is offline.

Permitted System Downtime
Senior management have estimated they lose £10,000 in sales and £5,000 in customer churn when they can’t service their customers’ requirements for the first hour.
Customer churn cost doubles every hour when those customers become more frustrated. These values are affected further if the system is offline during a promotional period or peak time.

RTO calculation
From these figures, the company decides their SLA to recover from any system outage is 1 hour.
It is clear that a 1 hour SLA is probably unachievable, therefore the system will need improving by way of adding clustering, replication or a live-failover. 

We have covered these services at the bottom of this article.

RTO conclusion
Now they know their existing system cannot be recovered 
within one hour following a disaster, they have started their DR planning. Simply by looking at RTO they now have a business-case and budget to improve their systems so they can maintain their 1 hour RTO.

RTO: one hour

RPO (Recovery Point Objective)

Recovery Point Objective

Scenario
At Buddy care they decide they can lose a week of customer service calls, but can’t lose any order details because these contain legal agreements and customer commitments.

The RPO statement says “we need all data recovered“. That is a perfectly normal response, however the business continuity SLA (RTO) of 1 hour dictates the RPO. If they need 10 hours to recover all data, that cannot be done with an RTO of 1 hour.

Permitted Data Loss
In an environment like Buddy Care’s, there will always be some data lost after a restore because the transactions are happening every few seconds, but the backup only runs once every hour. It is commonplace for the live backup sets to contain the most recent data and for historical data to be stored in another data set which is infrequently accessed. After the live data has been restored, it is expected historical data (6 months or more) will take longer to restore. The durability of that data isn’t in question, it is simply the persistence of the service which serves that data. This is the difference between the persistence of a service (is it available now?), and the durability of data (can we restore all the data?).

RPO is a very significant figure and should be impressed on senior management that it is a way of mitigating and managing data loss after the event. They should be aware of what type of data won’t be initially restored and whether it will be restored at a later data.

Let’s not be technical here though. Even though we are talking tech and maths here, the conundrum is the same and for senior management.

RPO calculation (which data can be lost)
One week of customer service calls.
No order details can be lost.

RPO conclusion
One week of customer service calls.
No order details can be lost.

RPO: one week of customer service calls. No order details can be lost.

Analysis delegates

Because this is a critical business decision, the RTO and RPO metrics should be decided by Senior management.

The IT Team will be required to confirm what is possible with the existing systems and recommend what is required to achieve the RTO and RPO.

To recap

This must be thorough and regularly tested. 

If a backup takes 2 hours, don’t assume the restore will take 2 hours. During a restore, you might need to procure and replace hardware or rebuild a system from scratch before a restore can start. These processes can take days if you haven’t envisaged what is required or don’t have access to replacement hardware.
What if your server goes offline at 06:00 on Sunday? Can you buy another one before Monday, what about restoring the operating system before you can restore your databases and applications?

Without a thorough restore test, your DR plan will be nothing more than a dream. Ideally, you want to be doing test restores rather than live restores. If you are doing a live restore when a system is down, it is fair to say someone has messed up somewhere. EVERY PC and server you install nowadays will have a whole lot of diagnostic and error checking going on. More than 80% of the DR situations we get involved with are because of faulty hard disks which have been reporting impending errors for days or years. A disk in a RAID array fails and the RAID fault tolerance level is managing things nicely until another disk fails and the array dies. We most commonly see RAID, 5,6 and 10 in use. These are expensive systems and can easily be kept healthy by swapping disks during a first-instance failure.

Motherboards and memory failures are different. With memory, we will normally see some CRC errors starting before they fail. Regarding the others, try dual power supplies, keeping spare parts or even a spare server. Your RTO and RPO calculations will dictate your budget, i.e. do you keep a spare server, cluster your servers, or have live failover replication between your premises and data centres?

What else should be considered?

Now we have the hardware DR plan written, lets automate things.

Backups can easily be automated and the results should be monitored by a competent member of staff. The action of staff monitoring backups should be logged somewhere so that management know the logs are being checked fully and not just given lip service.

Test restores are very important and I can’t overstate that. It is not uncommon for an IT Admin to move data within a server or to another server when extra storage is required, and this needs to be communicated to the team who are responsible for data backups so the backup sets can be altered.

Some other issues we have seen when a restore is needed are:
The encryption password is wrong or has been lost (without these, data cannot be restored).

T
he system architecture is unknown when rebuilding a new server (what disk sizes and partitioning did we use on the server before it crashed?).

The media where our data is stored isn’t available (this used to be physical tapes, but nowadays is more likely to be insufficient bandwidth download for a cloud restore). 

The only way to deal with this is through regular restores and benchmarking. With virtualisation and low cost hardware, it is fairly easy to recover an entire server to a restore lab so you can check if your RTO is maintained.

Deliver RTO and RPO in steps

Calculate your RTO and RPO values and have these approved by senior management.

Remember, these are business decisions and not solely IT decisions. Lets assume these are set in stone.

Decide if the RTO and RPO can be achieved with the existing systems and backups. If yes, then test it in your lab.

Identify other risks, power, substations, networking, etc. Your systems might be 100% bullet-proof, however you might be let down by single points of failure from your power or connectivity providers.

Don't rely on RTO and RPO to fix your problems

RTO and RPO are calculations to assist you in calculating your DR strategy. Of course, you hope you never need to put them to the test.

With the correct planning, support from management, and a good IT budget, your IT system should be able to self-heal and preferably not fail in the first place.

Even with never-fail redundancy, data corruption can still occur. When it does, the corruption is simply spread amongst all the nodes. Every techie knows this, however it is more commonplace nowadays with increasing cybercrime and ransomware.

One response to “RTO and RPO explained”

  1. Marcus says:

    I have been asked to look into RTO by my boss. I was wondering how many IT admins do both?

Leave a Reply

Your email address will not be published. Required fields are marked *

BOBcloud.net
The Old Sorting Office, Corsham, Wiltshire SN13 9AA
Tel: 0800 907 8238 https://www.bobcloud.net/wp-content/themes/bobcloud/images/logo.png