What really limits backup restore speed?

When "that moment" comes and you have to restore data and systems from backup, you quickly discover that your two-hour RTO has little to do with reality. What's more, the cause of a slow restore rarely lies where you would intuitively expect it. The daily routine of running backups is abruptly interrupted and you face a new challenge: the restore itself. Interestingly, for many teams this is the first real test of the whole process, and often a very painful lesson.

A restore is a process, not a single operation#

When you think about fast restores or plan to lower your RTO, remember that a restore is not just "copying a file from the repository to the target server". It is a chain of consecutive operations, and the throughput of the whole process equals the throughput of its slowest link. To show how complex a restore from backup is, here is a simplified sequence:

1. Reading metadata. The backup software (Veeam in this example) reads the metadata file and identifies which files make up the selected restore point: the last full backup plus all the incrementals that must be applied to reach the requested point in time.
2. Rehydration. Data in the repository is not a ready-to-copy machine image. It consists of compressed and deduplicated blocks spread across many files. Before it can return to the production server as a virtual machine (VM), it has to be decompressed and assembled into a logical image. This stage is rehydration, and this is where the bottleneck most often hides.
3. Transfer through "intermediaries" (proxies). With Veeam, two processes handle the transfer: one on the repository side (reads and rehydrates) and one on the target side (writes data to production storage through the hypervisor). What we measure as "restore speed in MB/s" is the effective speed of this stream, not the repository read speed or the link bandwidth.
4. Writing to target storage. Write speed on the production array depends on the transport mode (Direct SAN, Hot-Add, NBD), because each has different performance characteristics.
5. Registering and starting the VM. Once all blocks are written, the machine is registered in the hypervisor inventory on the production cluster and can be powered on.

Where is the bottleneck most often?#

Now that we know what a restore looks like, it is easy to see that a high-throughput network is not enough: poor performance of the repository or the virtualization cluster will drag restore speed down completely, and that is not the whole story either.

Below are the typical bottlenecks of a restore from backup, ordered from the most common.

1. Repository (source). Restoring from backup is both a disk-bound and a compute-bound operation, so the device's I/O and CPU matter a lot. A repository on slow SATA disks or with an undersized CPU can put a hard ceiling of 200–300 MB/s on the restore, regardless of what you have elsewhere.
2. Target storage. Under restore load, the production array and virtualization cluster often turn out to be slower than the vendor's datasheet promised. Especially when they have to serve other virtual machines or several concurrent restore sessions at the same time.
3. Proxy. The Veeam proxy receives data from the source, processes it and passes it to the target. In practice the proxy becomes the bottleneck less often than the repository or target storage. It can still happen when it gets too little compute for many parallel sessions, or too many restore jobs at once.
4. Network. Only at this stage do we reach the limits of network interfaces and bandwidth. The link becomes the bottleneck mainly when the restore runs over a wide area network (WAN), for example when the source is an off-site backup in a data center. On a local network (LAN), bandwidth is usually not a problem.

Why is the repository so often the culprit?#

Because of rehydration. Data in the backup repository is not a ready machine image; it is compressed and deduplicated blocks. Rehydration is the reverse process: decompressing those blocks and assembling them back into a full, logical image that the hypervisor (target) can run.

To do this, the repository server has to do three things at once:

• decompress blocks: the compression algorithm (LZ4, zlib or another one chosen when the backup job was created) has to be reversed for every data block,
• locate physical blocks: deduplication means one physical block can be referenced by many machines or many restore points, and the process has to "untangle" that,
• assemble the chain: starting from the last full backup, it applies the subsequent incrementals up to the selected point in time, block by block.

The result is a stream of "ready" data flowing to the production cluster. And that stream is exactly what we measure as restore speed.

So rehydration needs compute power and a capable disk subsystem. Yet a common mistake teams make is building the backup repository on an old box that has been retired from the production cluster.

Where to start looking for the bottleneck?#

I suggest a three-step analysis based on the answers to these questions.

1. Is the problem there from the start, or does it build up? If the restore is slow from the first minute, the problem may be on the repository or proxy side. If it starts well but slows down along the way, the target storage may not keep up with the growing load.
2. How many restore jobs run in parallel? Every parallel session competes for the same resources. Sometimes running fewer sessions gives each one more resources, and the total restore time is shorter.
3. How old is the restore point? The further back you go, the more incrementals Veeam has to put together. That directly increases the load on the repository and the restore time.

Test restores: the cure for surprises#

We rarely check whether something works unless we are forced to. In day-to-day operations, restoring from backup usually means recovering specific files or folders, mailboxes or database objects. Restoring most of your systems is an extremely rare scenario.

But when you do have to restore an entire environment after a security incident and the business is at a standstill, everyone is watching the IT team. There is no room for tuning infrastructure then, because every hour means real financial losses. So my advice is: test your restores regularly, at least once a quarter. It is also worth using automation; with Veeam that is SureBackup. It will tell you whether you still meet your target RTO, which takes some pressure off an already stressful recovery. Remember: the point of a backup system is not making backups, it is restoring from them!