.. _MigrationChapter:

Migration and Copy
==================

.. index::
   single: Migration
   single: Copy

The term Migration, as used in the context of Bareos, means moving data from one Volume to another.
In particular it refers to a Job (similar to a backup job) that reads data that was previously
backed up to a Volume and writes it to another Volume. As part of this process, the File catalog
records associated with the first backup job are purged. In other words, Migration moves Bareos Job
data from one Volume to another by reading the Job data from the Volume it is stored on, writing it
to a different Volume in a different Pool, and then purging the database records for the first Job.

The Copy process is essentially identical to the Migration feature with the exception that the Job
that is copied is left unchanged. This essentially creates two identical copies of the same backup.
However, the copy is treated as a copy rather than a backup job, and hence is not directly available
for restore. If Bareos finds a copy when a job record is purged (deleted) from the catalog, it will
promote the copy as *real* backup and will make it available for automatic restore.

Copy and Migration jobs do not involve the File daemon.

Jobs can be selected for migration based on a number of criteria such as:

-  a single previous Job

-  a Volume

-  a Client

-  a regular expression matching a Job, Volume, or Client name

-  the time a Job has been on a Volume

-  high and low water marks (usage or occupation) of a Pool

-  Volume size

The details of these selection criteria will be defined below.

To run a Migration job, you must first define a Job resource very similar to a Backup Job but with
:config:option:`dir/job/Type = Migrate`\ . One of the key points to remember is that the Pool that is
specified for the migration job is the only pool from which jobs will be migrated. In addition, the
Pool to which the selected Job or Jobs will be migrated is defined by the
:config:option:`dir/pool/NextPool = ...`\  in the Pool resource specified for the Migration Job.

Bareos permits Pools to contain Volumes of different Media Types. However, when doing migration,
this is a very undesirable condition. For migration to work properly, you should use Pools
containing only Volumes of the same Media Type for all migration jobs.

A migration job can be started manually or from a Schedule, like a backup job. It searches for
existing backup Jobs that match the parameters specified in the migration Job resource, primarily a
:config:option:`dir/job/SelectionType`\ . If no match was found, the Migration job terminates
without further action. Otherwise, for each Job found this way, the Migration Job will run a new Job
which copies the Job data to a new Volume in the Migration Pool.

Normally four jobs are involved during a migration/copy:

-  The migration/copy **control job**: This job checks for jobs that need to be copied/migrated and
   starts a copy/migrate worker job for each of these jobs.

-  The migration/copy **worker jobs**: They copy the data of one original job to the resulting
   destination job. The worker jobs are removed from the database when the destination job (the job
   that is the result of the copying) is pruned from the catalog database, or, in case of a copy,
   when the new copy is upgraded to become a backup.

-  The **previous (original) Backup job** (*already run and being copied*): The File records of this
   job are purged when the migration/copy job terminates successfully. The data remain on the volume
   until it is recycled.

-  The **new (destination) Backup job** (*the new copy*) that holds the data from the previous
   Backup job in the new Volume. If you subsequently do a restore, the data will be read from
   this job.

In case of a migration, the original job is purged from the catalog database immediately after the
copy is completed. In a copy job, both original and new backup jobs remain in the catalog database
and will be pruned with the standard Bareos mechanisms (i.e. after the expiration of the retention
times).

If the migration/copy control job finds more than one existing job to migrate, it creates one
migration/copy job for each of them. This may result in a large number of jobs. Please note that
migration/copy doesn’t scale too well if you migrate data off of a large Volume because each job
must read the same Volume, hence the jobs will have to run consecutively rather than simultaneously.

Important migration Considerations
----------------------------------

.. index::
   single: migration; Important migration Considerations

-  Each Pool into which you migrate Jobs or Volumes must contain Volumes of only one
   :config:option:`dir/storage/MediaType`\ .

-  Migration takes place on a **JobId by JobId** basis. That is each JobId is migrated in its
   entirety and independently of other JobIds. Once the Job is migrated, it will be on the new
   medium in the new Pool, but for the most part, aside from having a new JobId, it will appear with
   all the same characteristics of the original job (start, end time, ...). The column **RealEndTime**
   in the catalog **Job** table will contain the time and date that the Migration terminated, and by
   comparing it with the **EndTime** column you can tell whether or not the job was migrated. Also,
   the **Job** table contains a **PriorJobId** column which is set to the original **JobId** for
   migrated jobs. For non-migrated jobs this column is zero.

-  After a Job has been migrated, the File records are purged from the original Job. Moreover, the
   Type of the original Job is changed from `B` (backup) to `M` (migrated), and another Type `B` job
   record is added which refers to the new location of the data. Since the original Job record stays
   in the bareos catalog, it is still possible to restore from the old media by specifying the
   original JobId for the restore. However, no file selection is possible in this case, so one can
   only restore all files this way.

-  A Job will be migrated only if all Volumes on which the job is stored are marked `Full`, `Used`,
   or `Error`. In particular, Volumes marked `Append` will not be considered for migration which
   rules out the possibility that new files are appended to a migrated Volume. This policy also
   prevents deadlock situations, like attempting to read and write the same Volume from two jobs
   at the same time.

-  Migration works only if the Job resource of the original Job is still defined in the current
   Director configuration. Otherwise you’ll get a fatal error.

-  Setting the :config:option:`dir/pool/MigrationHighBytes`\  watermark is not sufficient for
   migration to take place. In addition, you must define and schedule a migration job which looks
   for jobs that can be migrated.

-  Bareos currently does only minimal Storage conflict resolution, so you must take care to ensure
   that you don’t try to read and write to the same device or Bareos may block waiting to reserve a
   drive that it will never find. A way to prevent problems is that all your migration pools contain
   only one :config:option:`dir/storage/MediaType`\ , and that you always migrate to a pool with a
   different Media Type.

-  The :config:option:`dir/pool/NextPool = ...`\  directive must be defined in the Pool referenced
   in the Migration Job to define the Pool into which the data will be migrated.

-  Migration has only be tested carefully for the **Job** and **Volume** selection types. All other
   selection types (**time**, **occupancy**, **smallest**, **oldest**, ...) are experimental
   features.

-  To figure out which jobs are going to be migrated by a given configuration, choose a debug level
   of 100 or more. This activates information about the migration selection process.


Configure Copy or Migration Jobs
--------------------------------

The following directives can be used to define a Copy or Migration job:

Job Resource
''''''''''''

-  :config:option:`dir/job/Type`\  = Migrate|Copy

-

   :config:option:`dir/job/SelectionType`\

-

   :config:option:`dir/job/SelectionPattern`\

-  | :config:option:`dir/job/Pool`\
   | Defines the Pool that will be examined for finding JobIds to migrate.
   | This Pool also sets the storage for reading the volumes containing the jobs.

-

   :config:option:`dir/job/PurgeMigrationJob`\

-  | :config:option:`dir/job/MaximumConcurrentJobs`\
   |  > 1 is needed if you want to have multiple migrate/copy jobs running at the same time



Pool Resource
'''''''''''''

-  | :config:option:`dir/pool/NextPool`\
   | to what pool Jobs will be migrated

-  | :config:option:`dir/pool/MigrationTime`\
   | if :config:option:`dir/job/SelectionType`\  = PoolTime

-  | :config:option:`dir/pool/MigrationHighBytes`\
   | if :config:option:`dir/job/SelectionType`\  = PoolOccupancy

-  | :config:option:`dir/pool/MigrationLowBytes`\
   | optional if :config:option:`dir/job/SelectionType`\  = PoolOccupancy is used

-  | :config:option:`dir/pool/Storage`\
   | if Copy/Migration involves multiple Storage Daemon, see
   | :ref:`section-CopyMigrationJobsMultipleStorageDaemons`


Example Migration Jobs
~~~~~~~~~~~~~~~~~~~~~~

.. index::
   single: Example; Migration Jobs

Assume a simple configuration with a single backup job as described below.

.. code-block:: bareosconfig
   :caption: Backup Job

   # Define the backup Job
   Job {
     Name = "NightlySave"
     Type = Backup
     Level = Incremental                 # default
     Client = bareos-fd
     FileSet = "Full Set"
     Schedule = "WeeklyCycle"
     Messages = Standard
     Pool = Default
   }

   # Default pool definition
   Pool {
     Name = Default
     Pool Type = Backup
     AutoPrune = yes
     Recycle = yes
     Next Pool = Tape
     Storage = File
     LabelFormat = "File"
   }

   # Tape pool definition
   Pool {
     Name = Tape
     Pool Type = Backup
     AutoPrune = yes
     Recycle = yes
     Storage = LTOTape
   }

   # Definition of File storage device
   Storage {
     Name = File
     Address = bareos-sd.example.com
     Password = "secret"
     Device = "File"          # same as Device in Storage daemon
     Media Type = File        # same as MediaType in Storage daemon
   }

   # Definition of DLT tape storage device
   Storage {
     Name = LTOTape
     Address = bareos-sd.example.com
     Password = "secret"
     Device = "tapedrive-0"       # same as Device in Storage daemon
     Media Type = LTO      # same as MediaType in Storage daemon
   }

Note that the backup job writes to the :config:option:`dir/pool = Default`\  pool, which corresponds
to :config:option:`Dir/Storage = File`\  storage.
There is no :config:option:`dir/pool/Storage`\  directive in the Job resource while the two
:config:option:`Dir/Pool`\  resources contain different :config:option:`dir/pool/Storage`\  directives.
Moreover, the :config:option:`dir/pool = Default`\  pool contains a
:config:option:`dir/pool/NextPool`\  directive that refers to
the :config:option:`dir/pool = Tape`\  pool.

In order to migrate jobs from the :config:option:`Dir/Pool = Default`\  pool to the
:config:option:`Dir/Pool = Tape`\  pool we add the following Job resource:

.. code-block:: bareosconfig
   :caption: migrate all volumes of a pool

   Job {
     Name = "migrate-volume"
     Type = Migrate
     Messages = Standard
     Pool = Default
     Selection Type = Volume
     Selection Pattern = "."
   }

The :config:option:`dir/job/SelectionType`\  and :config:option:`dir/job/SelectionPattern`\  directives
instruct Bareos to select all volumes of the given pool (:config:option:`dir/pool = Default`\ )
whose volume names match the given regular expression (:strong:`"."`), i.e., all volumes. Hence
those jobs which were backed up to any volume in the :config:option:`dir/pool = Default`\  pool will
be migrated. Because of the :config:option:`dir/pool/NextPool`\  directive of the
:config:option:`dir/pool = Default`\  pool resource, the jobs will be migrated to tape storage.

Another way to accomplish the same is the following Job resource:

.. code-block:: bareosconfig
   :caption: migrate all jobs named \*Save

   Job {
     Name = "migrate"
     Type = Migrate
     Messages = Standard
     Pool = Default
     Selection Type = Job
     Selection Pattern = ".*Save"
   }

This migrates all jobs ending with :strong:`Save` from the :config:option:`dir/pool = Default`\  pool
to the :config:option:`dir/pool = Tape`\  pool, i.e., from File storage to Tape storage.

.. _section-CopyMigrationJobsMultipleStorageDaemons:

Multiple Storage Daemons
^^^^^^^^^^^^^^^^^^^^^^^^

Beginning from Bareos :sinceVersion:`13.2.0: Copy and Migration Jobs between different Storage Daemons`,
Migration and Copy jobs are also possible from one Storage daemon to another Storage Daemon.

Please note:

-  the director must have two different storage resources configured (e.g. storage1 and storage2)

-  each storage needs an own device with a different Media Type (e.g. File1 and File2) and
   an individual pool (e.g. pool1, pool2)

-  each pool is linked to its own storage via the storage directive in the pool resource

-  to configure the migration from pool1 to pool2, the :config:option:`dir/pool/NextPool`\  directive
   of pool1 has to point to pool2

-  the copy job itself has to be of type copy/migrate
   (exactly as already known in copy- and migration jobs)

Example:

.. code-block:: bareosconfig
   :caption: bareos-dir.conf: Copy Job between different Storage Daemons

   #bareos-dir.conf

   # Fake fileset for copy jobs
   Fileset {
     Name = None
     Include {
       Options {
         Signature = XXH128
       }
     }
   }

   # Fake client for copy jobs
   Client {
     Name = None
     Address = localhost
     Password = "NoNe"
   }

   # Source storage for migration
   Storage {
      Name = storage1
      Address = sd1.example.com
      Password = "secret1"
      Device = File1
      Media Type = File1
   }

   # Target storage for migration
   Storage {
      Name = storage2
      Address = sd2.example.com
      Password = "secret2"
      Device = File2
      Media Type = File2   # Has to be different than in storage1
   }

   Pool {
      Name = pool1
      Storage = storage1
      Next Pool = pool2    # This points to the target storage
   }

   Pool {
      Name = pool2
      Storage = storage2
   }

   Job {
      Name = CopyToRemote
      Type = Copy
      Messages = Standard
      Selection Type = PoolUncopiedJobs
      Spool Data = Yes
      Pool = pool1
   }
