General

General#

What is Restore Verification?#

Performing backups alone is not enough to protect data against loss. It is possible to have terabytes of backed-up data and still be unable to restore the data you actually need. For this reason, system administrators periodically perform restore tests to make sure that backups can be successfully recovered. Without such tests, a backup system is like a fire alarm that has never been tested and may fail when it is needed most.

Testing backup restores is an essential part of the data protection process. However, it can also be tedious and time-consuming. It requires a suitable test environment, administrator time, and careful verification of the restored data.

The Restore Verification feature in Bacularis was created to automate and significantly simplify the restore testing process. It provides tools for building a workflow that automatically selects an appropriate backup, restores and verifies the data, and records the results of the test.

Restore Verification does not try to answer the question, “Did the backup succeed?” Instead, it answers, “Can I actually recover the data I need from this backup?” To do this, Bacularis performs a real restore in a test environment and then verifies the restored data using the selected verification method.

How it works#

../_images/bacularis_restore_verification_workflow.png

In short, the automated Restore Verification process works as follows:

  • A backup matching the criteria defined in the Restore Test is selected.

  • The data is restored to the configured Restore Destination.

  • The restored data is checked using the selected verification method.

  • The test result is recorded and made available for further analysis and notifications.

Verification methods#

Restore Verification supports two methods for checking restored data:

  • Rule-based Verification – Bacularis restores the data and checks it using Verification Rules and modular checker plugins.

  • Native Bacula Verify – verification is performed using the native Bacula Verify mechanism with the DiskToCatalog level.

How restored data is verified#

The way restored data is checked depends on the selected verification method.

Rule-based data verification#

The selected backup is treated as a restore point. Bacularis reconstructs the required backup chain leading to that point, allowing the data to be checked in the state corresponding to the selected point in time.

The restored data is checked using modules called checkers. Each checker evaluates specific conditions on the restored data. Sets of rules that define which data should be checked and which checkers and criteria should be applied are called Verification Rules.

For example, checkers can be used to:

  • verify that a file or directory exists,

  • verify that a file size meets defined criteria,

  • verify that a checksum matches the value stored in the Bacula Catalog,

  • perform many other types of checks.

../_images/bacularis_restore_verification_rule_based_verification_workflow.png

More information: Rule-based Verification

Native Bacula Verify#

With Native Bacula Verify, verification is performed against a single Bacula backup job (JobId) rather than against a complete backup chain. Each test therefore compares the restored data with the metadata stored for a specific backup job. For this reason, Restore Verification currently supports only Full backups when using the Native Bacula Verify method.

The actual verification is performed by a Bacula Verify job. The set of attributes to be checked is defined in the Bacula FileSet configuration using the Verify directive.

../_images/bacularis_restore_verification_native_bacula_verify_workflow.png

More information: Native Bacula Verify

Test results and reporting#

Restore Verification results are recorded in the Bacula job log. This makes test information available in the same places as other Bacula job information, both in the Bacularis interface and through bconsole.

Restore Verification also integrates the test result with the Bacula job status. If verification detects a failure, the job running the test receives an appropriate error status. This allows existing mechanisms such as:

  • monitoring,

  • reporting,

  • notifications,

  • job lists,

  • dashboards

to automatically detect and present failed restore tests.

The contents of the report also depend on the selected verification method.

Rule-based data verification#

The report contains information about each tested location, the applied criteria, the operator, the expected value, and the actual value.

Each test case has an explicit result indicating whether the check passed or failed. If at least one test fails, the Bacula job running Restore Verification receives an appropriate error status.

Native Bacula Verify#

Native Bacula Verify produces the standard report generated by Bacula Verify jobs. It reports differences between restored data in the file system and the metadata stored in the Bacula Catalog. The report also indicates the type of difference that was detected.

If differences are detected, the job status is set to Verify Differences.

What Restore Verification can test#

Restore Verification can be used both for simple checks that confirm whether restored files exist and have the expected properties, and for more advanced application-level tests performed by extensible checker plugins.

Checkers can validate many different types of data. Some theoretical examples include:

  • a file exists – verify that a specific file has been restored successfully,

  • a checksum is correct – verify that the checksum of a restored file matches the value stored in the Bacula Catalog,

  • a database dump can be imported – verify that a restored database dump can be successfully imported into a test database server,

  • SQL queries work after import – verify that expected queries can be executed after the database has been restored and that they return correct results.

Thanks to the extensible checker architecture, Restore Verification can validate not only file properties, but also the behavior of applications and data after they have actually been recovered.

More information: Checker Plugins

Main Restore Verification objects#

Restore Verification is built around four main configuration elements:

  • Restore Test – defines what should be tested,

  • Restore Policy – defines how and when the test should be run,

  • Restore Destination – defines the environment and location to which the data should be restored,

  • Verification Rules – define what data should be checked and how it should be verified when using Rule-based Verification.

Video guide#


Next steps#

  • Getting Started - configure and run your first Restore Verification test.

  • Building Blocks - learn how Restore Tests, Restore Policies, Restore Destinations, and Verification Rules work together.

  • Verification Methods - compare Rule-based Verification and Native Bacula Verify.

Contents