Disaster Recovery Using Bareos
General
When disaster strikes, you must have a plan, and you must have prepared in advance, otherwise the work of recovering your system and your files will be considerably greater. For example, if you have not previously saved the partitioning information for your hard disk, how can you properly rebuild it if the disk must be replaced?
Unfortunately, many of the steps one must take before and immediately after a disaster are very much dependent on the operating system in use. As a consequence, this chapter will discuss in detail disaster recovery only for selected operating systems.
Important Considerations
Here are a few important considerations concerning disaster recovery that you should take into account before a disaster strikes.
If the building which houses your computers burns down or is otherwise destroyed, do you have off-site backup data?
Disaster recovery is much easier if you have several machines. If you have a single machine, how will you handle unforeseen events if your only machine is down?
Do you want to protect your whole system and use Bareos to recover everything? Or do you want to try to restore your system from the original installation disks, apply any other updates and only restore the user files?
Steps to Take Before Disaster Strikes
Create a rescue or CDROM for your systems. Generally, they are offered by each distribution, and there are many good rescue disks on the Web
Ensure that you always have a valid bootstrap file for your backup and that it is saved to an alternate machine. This will permit you to easily do a full restore of your system.
If possible copy your catalog nightly to an alternate machine. If you have a valid bootstrap file, this is not necessary, but can be very useful if you do not want to reload everything.
Ensure that you always have a valid bootstrap file for your catalog backup that is saved to an alternate machine. This will permit you to restore your catalog more easily if needed.
Try out how to use the Rescue CDROM before you are forced to use it in an emergency situation.
Make a copy of your Bareos .conf files, particularly your bareos-dir and your bareos-sd configuration files, because if your server goes down, these files will be needed to get it back up and running, and they can be difficult to rebuild from memory.
Bare Metal Recovery of Bareos Clients
A so called “Bare Metal” recovery is one where you start with an empty hard disk and you restore your machine.
Generally, following components are required for a Bare Metal Recovery:
A rescue boot image (e.g. CDROM or network boot) containing a copy of your OS and including the Bareos File daemon, including all libraries
The Bareos client configuration files
Useful: a copy of your hard disk information
A full Bareos backup of your system
Linux
From the Relax-and-Recover web site (https://relax-and-recover.org):
Relax-and-Recover is a setup-and-forget Linux bare metal disaster recovery solution. It is easy to set up and requires no maintenance so there is no excuse for not using it.
Relax-and-Recover (ReaR) is quite easy to use with Bareos.
Installation
For Bareos >= 22, ReaR > 2.7 is required. If ReaR > 2.7 is not part of your distribution, check the download section on the ReaR website. The current (2024) stable release of ReaR is 2.7, therefore until a newer version is released, a ReaR snapshot package must be used, see Installing a ReaR Snapshot Package.
Configuration
For ReaR to work with Bareos, the following packages must be installed:
rear > 2.7 (currently only via a ReaR snapshot package)
bareos-fd
bareos-bconsole
Make sure you have a working Bareos File Daemon configuration on the system and already did a backup of all disks you want to restore (the fileset has to include all data required to fully restore the system. A good starting point is LinuxAll (Dir->Fileset)).
You also need the Bareos Console installed and configured,
so that it can connect to the Bareos Director.
Ideally the Bareos Console is configured in a way that it can only access this client,
see Configure a Named Console for ReaR.
To configure the Bareos backend, ReaR uses the variables
BAREOS_CLIENT, BAREOS_FILESET and BAREOS_RESTORE_JOB.
It tries to detect them automatically,
however it is recommend to set them explicitly to avoid surprises in case the Bareos Director configuration changes.
All required settings can be specified in a ReaR configuration file,
normally /etc/rear/local.conf.
Please refer to the ReaR documentation for details.
Assuming the system you want to protect with ReaR is referenced by Bareos as bareosclient-fd, you use LinuxAll as backup fileset and the default restore job (RestoreFiles) is sufficient for you, your ReaR configuration will look like this:
/etc/rear/local.conf (minimal)BACKUP=BAREOS
BAREOS_CLIENT=bareosclient-fd
BAREOS_FILESET=LinuxAll
BAREOS_RESTORE_JOB=RestoreFiles
If you want to create an ISO image and store it on an NFS server with the IP address 192.168.10.1, you can use the following configuration:
/etc/rear/local.conf (with OUTPUT_URL)# This is default:
#OUTPUT=ISO
# Where to write the iso image
# You can use NFS, if you want to write your iso image to a nfs server
# If you leave this blank, it will
# be written to: /var/lib/rear/output/
OUTPUT_URL=nfs://192.168.10.1/rear
BACKUP=BAREOS
BAREOS_CLIENT=bareosclient-fd
BAREOS_FILESET=LinuxAll
BAREOS_RESTORE_JOB=RestoreFiles
Create Rescue Image
If you have installed and configured ReaR on your system, type
root@host:~# rear -v mkrescue
to create the rescue image. If you used the configuration example above, you will get a bootable ISO image file, containing all the relevant programs and tools to do a Bareos restore over the network.
Warning
This will not create a Bareos backup on your system! You will have to do that by other means, e.g. by a regular Bareos backup schedule. Also rear mkbackup will not create a backup. In this configuration it will only create the rescue ISO (same as the rear mkrescue command).
Note
The rescue image is created from the running system. It contains the required programs (including bareos-fd and bconsole), the relevant configuration, information about attached storage devices and also the kernel of the running system. You should therefore regularly update your rescue images, see Storing and Retrieving the ReaR Rescue Image, followed by Update ReaR Rescue Image Regularly.
Recovery
In case, you want to recover your system, boot it using the generated ReaR recovery ISO. After booting log in as user root and type
RESCUE bareosclient:~# rear recover
ReaR detects the attached harddisks and, after confirmation, recreated the filesystem on them. For the actual data restore, it again asks for confirmation:
RESCUE bareosclient:~# rear recover
[...]
The system is now ready for a restore via Bareos.
When choosing 'automatic' a Bareos restore without user interaction
will be started with following options:
restore client=bareosclient-fd restorejob="RestoreFiles" fileset="LinuxAll" where=/mnt/local select all done yes
When choosing 'manual', bconsole will be started
and let you choose the restore options yourself.
Keep in mind, that the new root is mounted under '/mnt/local',
so use where=/mnt/local on restore.
The bconsole history contains the preconfigured restore command.
Choose restore mode:
1) automatic
2) manual
(default '1' timeout 300 seconds)
Once selected, the automatic restore mode will run the displayed command via bconsole. This will restore the most recent Bareos backup (from this client with the given fileset) to your system.
The manual restore mode just starts the bconsole program and lets you choose the restore command. For comfort, the bconsole command line history is already filled with some useful commands, especially with the restore command that the automatic restore mode would have selected.
When exiting bconsole, ReaR tries to determine the restore job and waits until this job is finished, while displaying some progress information, like:
Waiting for restore job 113 to finish.
Start: [2024-06-28 14:25:24], Duration: [00:00:22], Status: [R], Restored: [2.1G]
and finally:
Waiting for restore job 113 to finish.
Start: [2024-06-28 14:25:24], Duration: [00:00:27], Status: [T], Restored: [2.7G] OK
Information about finished job:
[...]
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
jobid: 113
job: RestoreFiles.2024-06-28_14.25.22_15
name: RestoreFiles
purgedfiles: 0
type: R
level: F
clientid: 4
client: bareosclient-fd
jobstatus: T
schedtime: 2024-06-28 14:25:22
starttime: 2024-06-28 14:25:24
endtime: 2024-06-28 14:25:51
realendtime: 2024-06-28 14:25:51
duration: 00:00:27
jobtdate: 1,719,584,751
volsessionid: 1
volsessiontime: 1,719,583,621
jobfiles: 63,889
jobbytes: 2,716,580,999
joberrors: 0
jobmissingfiles: 0
poolid: 0
poolname:
priorjobid: 0
filesetid: 0
fileset:
Restored 2.7G
Restore job finished successfully.
Bareos restore finished.
Created SELinux /mnt/local/.autorelabel file : after reboot SELinux will relabel all files
Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
[...]
Updated initramfs with new drivers for this system.
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/vda - installing GRUB2 there
Finished 'recover'. The target system is mounted at '/mnt/local'.
Exiting rear recover (PID 499) and its descendant processes ...
Running exit tasks
RESCUE bareosclient:~ #
The restored system can be found under the /mnt/local directory.
After restoring the files, ReaR restores the bootloader.
The recovery is now completed and the system can be rebooted.
The ReaR Bareos backend also works in ReaRs Automatic Recover mode, selectable at boot, which recreates the system without manual interaction.
Advanced ReaR Bareos Backend Configuration
Installing a ReaR Snapshot Package
The current (2024) stable release of ReaR is 2.7, which does not support Bareos > 22. Therefore until a newer version is released, a ReaR snapshot package must be used:
Download the ZIP file that best matches your distribution from https://github.com/rear/rear/releases/tag/snapshot
Unpack the ZIP file and install the package on the client.
For details, refer to https://relax-and-recover.org/download/
Configure a Named Console for ReaR
The default bconsole configuration on the Bareos Director offers full access to the Bareos Director. This is normally not wanted from a client systems.
Instead it is recommended to configure a Console (Dir) for every Bareos client using ReaR,
limiting the access to only one system and the required console commands.
As all these consoles share much of their configuration,
best start with a common Profile (Dir) for them.
rear-client-profile (Dir->Profile)
is a good starting point for such a profile:
bareos-dir.d/profile/rear-client-profile.confThis Profile (Dir)
limits the access to the required commands.
Access to all clients is denied.
This permission must be given individually in the consoles.
Next, create a Console (Dir)
for every Bareos client using ReaR:
bareos-dir.d/console/bareosclient-console.confCreate the file on the Bareos Director and reload the configuration. Of course an individual password should be used.
The corresponding /etc/bareos/bconsole.conf on the ReaR client will look like this.
/etc/bareos/bconsole.confAddress (Console->Director): must be set to the DNS name or IP address of the Bareos Director.Password (Console->Director): will be ignored and can be empty.Name (Console->Console): must correspond toName (Dir->Console).Password (Console->Console): must correspond toPassword (Dir->Console).
For more details, see Using Named Consoles.
Make sure (via filesystem permissions) that the file is only readable for required users. By default that is the user root and the members of the group bareos. Access to the rescue image should also be limited to the relevant persons.
Storing and Retrieving the ReaR Rescue Image
A rescue image is only useful,
if it is available in case of emergency.
ReaR itself offers flexible ways to distribute the rescue image to other systems
when creating a new image.
The default configuration will store the image locally
in the directory /var/lib/rear/output/.
As a full system backup will also backup this directory, it might be sufficient to just store it there and let Bareos back it up. In case the rescue image is needed, you have to restore it to another system.
Assuming the following settings,
system protected by ReaR is
bareosclient-fd (Dir->Client)with filesetLinuxAll (Dir->Fileset)system to restore the rescue image:
client_with_cd_burner-fd (Dir->Client)
the bconsole command would be this:
*restore client=bareosclient-fd fileset=LinuxAll current fileregex=/var/lib/rear/output/rear-.* restoreclient=client_with_cd_burner-fd
or if your prefer:
*restore client=bareosclient-fd fileset=LinuxAll current fileregex=/var/lib/rear/output/rear-.* restoreclient=client_with_cd_burner-fd regexwhere=!/var/lib/rear/output/!/tmp/rear-rescue-images/!
Make sure to test this thoroughly, especially if you are using Data Encryption, as restoring to another system needs special preparations.
Update ReaR Rescue Image Regularly
While a ReaR Rescue Image is quite flexible, it works best if it matches the current configuration of your system. Any configuration change (hardware, software or environment/network) can require an updated rescue image.
This can be automated using various methods, among others:
periodically (cron)
on system restart (as a configuration change, like a new kernel version, often requires a reboot)
Bareos backup job
Doing this via Bareos backup job has two advantages:
the rescue image corresponds perfectly to your backup
your backup report immediately tells if somethings went wrong with your backup or your restore image.
However, the size of a rescue image is around 200 MB. Adding this to a daily incremental backup is often unnecessary.
So it often makes sense to wrap the creation of a new rescue image by a script to limit the number of times it gets regenerated.
Limitations could be made based on the backup level or the age of the existing rescue image,
like implemented in the following script with the corresponding Job (Dir) configuration:
/usr/local/sbin/update-rear-rescue.sh#!/bin/bash
set -e
set -u
backuplevel="${1:-}"
REAR_OUTPUT_PATH=/var/lib/rear/output/
if [ "$backuplevel" = "Full" ]; then
echo "backup level = $backuplevel: recreating rescue image"
else
days=7
current_rescue_images=( $( find "$REAR_OUTPUT_PATH" -name "rear-*" -mtime -$days ) )
if [ ${#current_rescue_images[@]} -gt 0 ]; then
echo "SKIPPING 'rear mkrescue', as images newer than $days days exists: ${current_rescue_images[@]}"
exit 0
else
echo "no current rescue image found. recreating it"
fi
fi
rear mkrescue
exit $?
Copy the script to /usr/local/sbin/update-rear-rescue.sh on the Bareos Director,
make it executable
and add the Run Script definition to the job intended to backup the data for a ReaR restore.
Restoring a Bareos Server
Above, we considered how to recover a client machine where a valid Bareos server was running on another machine. However, what happens if your server goes down and you no longer have a running Director, Catalog, or Storage daemon? There are several solutions:
Move your server to another machine.
Use a Hot Spare Server on another Machine.
Consider the following steps:
Install the same database server as on the original system.
Install Bareos and initialize the Bareos database.
Ideally, you will have a copy of all the Bareos conf files that were being used on your server. If not, you will at a minimum need create a bareos-dir.conf that has the same Client resource that was used to backup your system.
If you have a valid saved Bootstrap file as created for your damaged machine with WriteBootstrap, use it to restore the files to the damaged machine, where you have loaded a static Bareos File daemon using the Rescue disk). This is done by using the restore command and at the yes/mod/no prompt, selecting mod then specifying the path to the bootstrap file.
After the Catalog is restored, it should be located in (
./var/lib/bareos/bareos.sql) relative to the restore location. After stopping the bareos-dir service, this file then needs to be loaded into the PostgreSQL database. (For example by runningsudo -u bareos psql bareos bareos -f ./var/lib/bareos/bareos.sqlor`su postgres -c "psql bareos -f ./var/lib/bareos/bareos.sql)If you have successfully used a Bootstrap file, you should now be back up and running, if you do not have a Bootstrap file, continue with the suggestions below.
Using bscan scan the last set of backup tapes into your catalog database.
Start Bareos, and using the Console restore command, restore the last valid copy of the Bareos database and the Bareos configuration files.
Move the database to the correct location.
Start the database, and restart Bareos. Then use the Console restore command, restore all the files on the damaged machine, where you have loaded a Bareos File daemon using the Rescue disk.
For additional details of restoring your database, please see the Restoring When Things Go Wrong chapter.