Tag: bacula-dir

bacula – Device not in SD Device resources or no matching Media Type

If you happen to have such error in your bacula backup server:

20-Jun 00:19 bkp-srv-dir JobId 23741: No prior Full backup Job record found.
20-Jun 00:19 bkp-srv-dir JobId 23741: No prior or suitable Full backup found in catalog. Doing FULL backup.
20-Jun 00:19 bkp-srv-dir JobId 23741: Start Backup JobId 23741, Job=srv1-img.2018-06-20_00.19.09_56
20-Jun 00:19 bkp-srv-sd JobId 23741: Fatal error: Device reservation failed for JobId=23741: 
20-Jun 00:19 bkp-srv-dir JobId 23741: Fatal error: 
     Storage daemon didn't accept Device "srv1-img" because:
     3924 Device "srv1-img" not in SD Device resources or no matching Media Type.
20-Jun 00:19 bkp-srv-dir JobId 23741: Error: Bacula bkp-srv-dir 7.0.5 (28Jul14):
  Build OS:               x86_64-pc-linux-gnu ubuntu 16.04
  JobId:                  23741
  Job:                    srv1-img.2018-06-20_00.19.09_56
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "srv1-img" 7.4.0 (16Jan16) x86_64-pc-linux-gnu,ubuntu,12.04
  FileSet:                "srv1-img" 2018-04-13 23:05:00
  Pool:                   "srv1-img-full" (From Job FullPool override)
  Catalog:                "DefaultCatalog" (From Client resource)
  Storage:                "srv1-img" (From Job resource)
  Scheduled time:         20-Jun-2018 00:19:06
  Start time:             20-Jun-2018 00:19:11
  End time:               20-Jun-2018 00:19:11
  Elapsed time:           0 secs
  Priority:               10
  FD Files Written:       0
  SD Files Written:       0
  FD Bytes Written:       0 (0 B)
  SD Bytes Written:       0 (0 B)
  Rate:                   0.0 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         
  Volume Session Id:      3105
  Volume Session Time:    1521814321
  Last Volume Bytes:      0 (0 B)
  Non-fatal FD errors:    1
  SD Errors:              0
  FD termination status:  
  SD termination status:  
  Termination:            *** Backup Error ***

First thing to check is the status of Storage Devices you have with bconsole:

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*status storage
Automatically selected Storage: bkp-srv
Connecting to Storage daemon bkp-srv at 78.140.166.26:9103

bkp-srv-sd Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
Daemon started 20-Jun-18 00:43. Jobs: run=1, running=0.
 Heap: heap=135,168 smbytes=302,033 max_bytes=478,209 bufs=243 max_bufs=262
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
===================================================================
 23736  Incr          3    383.9 K  OK       19-Jun-18 14:05 srv1-dns
 23737  Incr          3    859.9 K  OK       19-Jun-18 14:05 srv2-dns
 23738  Incr        485    301.8 M  OK       19-Jun-18 14:05 srv3-dns
 23739  Incr         12    91.44 M  OK       19-Jun-18 14:05 srv4-dns
 23740  Full          1    1.358 G  OK       19-Jun-18 14:14 BackupCatalog
 23741  Full          0         0   Error    20-Jun-18 14:19 srv1-img
====

Device status:

Device "localstorage" (/mnt/storage/backups/localstorage) is not open.
==

Device "srv1-dns" (/mnt/storage/backups/srv1-dns) is not open.
==

Device "srv2-dns" (/mnt/storage/backups/srv2-dns) is not open.
==

Device "srv3-dns" (/mnt/storage/backups/srv3-dns) is not open.
==

Device "srv4-dns" (/mnt/storage/backups/srv4-dns) is not open.
==

Device "bkp-srv" (/mnt/storage/backups/bkp-srv) is not open.
==
====

Used Volume status:
====

====

As you can see there is no storage Device with name “srv1-img” as it expected from the error above.
You should check two files:

bacula-dir.conf

and

bacula-sd.conf

In file “/etc/bacula/bacula-dir.conf” you should have section like this:

Storage {
  Name = srv1-img
  Address = 10.10.10.10
  SDPort = 9103
  Password = mysecretpass
  Device = srv1-img
  Media Type = File
  Maximum Concurrent Jobs = 10
}

And in file “/etc/bacula/bacula-sd.conf”:

Device {
   Name = srv1-img
   Media Type = File
   Archive Device = /mnt/storage/backups/srv1-img
   LabelMedia = yes;
   Random Access = Yes;
   AutomaticMount = yes;
   RemovableMedia = no;
   AlwaysOpen = no;
 }

The important parts for this particular error (“3924 Device “srv1-img” not in SD Device resources or no matching Media Type.”) are highlighted:

  1. “Name” – the name of the storage
  2. “Media Type” – the type of the storage

You may also check the path of the storage – /mnt/storage/backups/srv1-img.

The two parameters must be the same! And if you verify these two configurations exist in the files and the above two parameters are the same and still get the error you MUST restart your SD (storage device) daemon, a “reload” command from the bconsole won’t work!

bkp-srv bacula # systemctl restart bacula-sd.service
bkp-srv bacula # systemctl status bacula-sd.service
* bacula-sd.service - LSB: Start Bacula Storage Daemon at boot time
   Loaded: loaded (/etc/init.d/bacula-sd; bad; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-20 00:43:22 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 25249 ExecStop=/etc/init.d/bacula-sd stop (code=exited, status=0/SUCCESS)
  Process: 25259 ExecStart=/etc/init.d/bacula-sd start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/bacula-sd.service
           `-25269 /usr/sbin/bacula-sd -c /etc/bacula/bacula-sd.conf -u bacula -g tape

Jun 20 00:43:22 bkp-srv systemd[1]: Starting LSB: Start Bacula Storage Daemon at boot time...
Jun 20 00:43:22 bkp-srv bacula-sd[25259]:  * Starting Bacula Storage daemon... bacula-sd
Jun 20 00:43:22 bkp-srv bacula-sd[25259]:    ...done.
Jun 20 00:43:22 bkp-srv systemd[1]: Started LSB: Start Bacula Storage Daemon at boot time.

And verify it restarted successfully.

Then you the storage device named “srv1-img” should appear in bconsole status storage command:
First thing to check is the status of Storage Devices you have with bconsole:

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*status storage
Automatically selected Storage: bkp-srv
Connecting to Storage daemon bkp-srv at 78.140.166.26:9103

bkp-srv-sd Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
Daemon started 20-Jun-18 00:43. Jobs: run=1, running=0.
 Heap: heap=135,168 smbytes=302,033 max_bytes=478,209 bufs=243 max_bufs=262
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
===================================================================
 23736  Incr          3    383.9 K  OK       19-Jun-18 14:05 srv1-dns
 23737  Incr          3    859.9 K  OK       19-Jun-18 14:05 srv2-dns
 23738  Incr        485    301.8 M  OK       19-Jun-18 14:05 srv3-dns
 23739  Incr         12    91.44 M  OK       19-Jun-18 14:05 srv4-dns
 23740  Full          1    1.358 G  OK       19-Jun-18 14:14 BackupCatalog
 23741  Full          0         0   Error    20-Jun-18 14:19 srv1-img
====

Device status:

Device "localstorage" (/mnt/storage/backups/localstorage) is not open.
==

Device "srv1-dns" (/mnt/storage/backups/srv1-dns) is not open.
==

Device "srv2-dns" (/mnt/storage/backups/srv2-dns) is not open.
==

Device "srv3-dns" (/mnt/storage/backups/srv3-dns) is not open.
==

Device "srv4-dns" (/mnt/storage/backups/srv4-dns) is not open.
==

Device "bkp-srv" (/mnt/storage/backups/bkp-srv) is not open.
==

Device "srv1-img" (/mnt/storage/backups/srv1-img) is not open.
==
====

Used Volume status:
====

====

Extras – rerun the job and verify the backup was executed.

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*run
Automatically selected Catalog: DefaultCatalog
Using Catalog "DefaultCatalog"
A job name must be specified.
The defined Job resources are:
     1: BackupCatalog
     2: srv1-dns
     3: srv2-dns
     4: srv3-dns
     5: srv4-dns
     6: srv1-img
Select Job resource (1-33): 6
Run Backup job
JobName:  srv1-img
Level:    Incremental
Client:   srv1-img
FileSet:  srv1-img
Pool:     Default (From Job resource)
Storage:  srv1-img (From Job resource)
When:     2018-06-20 00:43:59
Priority: 10
OK to run? (yes/mod/no): yes
Job queued. JobId=23742
You have messages.
*messages 
20-Jun 00:44 bkp-srv-dir JobId 23742: No prior Full backup Job record found.
20-Jun 00:44 bkp-srv-dir JobId 23742: No prior or suitable Full backup found in catalog. Doing FULL backup.
20-Jun 00:44 bkp-srv-dir JobId 23742: Start Backup JobId 23742, Job=srv1-img.2018-06-20_00.44.01_00
20-Jun 00:44 bkp-srv-dir JobId 23742: Created new Volume="srv1-img-full0394", Pool="srv1-img-full", MediaType="File" in catalog.
20-Jun 00:44 bkp-srv-dir JobId 23742: Using Device "srv1-img" to write.
20-Jun 00:44 bkp-srv-sd JobId 23742: Labeled new Volume "srv1-img-full0394" on file device "srv1-img" (/mnt/storage/backups/srv1-img).
20-Jun 00:44 bkp-srv-sd JobId 23742: Wrote label to prelabeled Volume "srv1-img-full0394" on file device "srv1-img" (/mnt/storage/backups/srv1-img)
*messages
20-Jun 00:44 bkp-srv-dir JobId 23742: Volume used once. Marking Volume "srv1-img-full0394" as Used.
*messages
You have no messages.

bacula fatal error – Unable to connect to Storage daemon

Bacula is an open software enterprise backup system! Check out the official site here
Complex but useful software, which could automate the whole backup process of all your servers.
Some errors are easy to track some are not, so here is one error with a misleading error message if you do not know or forget the details of how the daemons works.

Here is the error extracted from the logs:

01-Sep 00:45 backup01-de-dir JobId 8789: No prior Full backup Job record found.
01-Sep 00:45 backup01-de-dir JobId 8789: No prior or suitable Full backup found in catalog. Doing FULL backup.
01-Sep 00:45 backup01-de-dir JobId 8789: Job srv123us.2017-09-01_00.45.28_34 waiting 103 seconds for scheduled start time.
01-Sep 00:47 backup01-de-dir JobId 8789: Start Backup JobId 8789, Job=srv123us.2017-09-01_00.45.28_34
01-Sep 00:47 backup01-de-dir JobId 8789: Using Device "web" to write.
01-Sep 00:51 srv123us-fd JobId 8789: Warning: bsock.c:112 Could not connect to Storage daemon on 1.1.1.1:9103. ERR=Connection timed out
01-Sep 01:17 srv123us-fd JobId 8789: Fatal error: bsock.c:118 Unable to connect to Storage daemon on 1.1.1.1:9103. ERR=Interrupted system call
01-Sep 01:17 srv123us-fd JobId 8789: Fatal error: job.c:1893 Failed to connect to Storage daemon: 1.1.1.1:9103
01-Sep 01:17 backup01-de-dir JobId 8789: Fatal error: Bad response to Storage command: wanted 2000 OK storage
01-Sep 01:17 backup01-de-dir JobId 8789: Error: Bacula backup01-de-dir 7.0.5 (28Jul14):
 Build OS:               x86_64-pc-linux-gnu ubuntu 16.04
  JobId:                  8789
  Job:                    srv123us.2017-09-01_00.45.28_34
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "srv123us" 7.0.5 (28Jul14) x86_64-pc-linux-gnu,ubuntu,16.04
  FileSet:                "web" 2017-11-07 17:19:45
  Pool:                   "web-full" (From Job FullPool override)
  Catalog:                "ucdn" (From Client resource)
  Storage:                "web" (From Job resource)
  Scheduled time:         01-Sep-2018 00:47:11
  Start time:             01-Sep-2018 00:47:11
  End time:               01-Sep-2018 01:17:23
  Elapsed time:           30 mins 12 secs
  Priority:               10
  FD Files Written:       0
  SD Files Written:       0
  FD Bytes Written:       0 (0 B)
  SD Bytes Written:       0 (0 B)
  Rate:                   0.0 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         
  Volume Session Id:      4719
  Volume Session Time:    1510075534
  Last Volume Bytes:      0 (0 B)
  Non-fatal FD errors:    2
  SD Errors:              0
  FD termination status:  Error
  SD termination status:  Waiting on FD
  Termination:            *** Backup Error ***

But when we check the status of client from “bconsole” (Bacula’s management Console), everything seems OK, the backup server (Director daemon = bacula-dir) connects and get the report from the client daemon (Bacula File service = bacula-fd) in the server, even when you run a backup job, the status report is OK, the backup is running on the client, here is the output:

srv@local ~ # bconsole
Connecting to Director localhost:9101
1000 OK: 1 backup01-de-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*status
Status available for:
     1: Director
     2: Storage
     3: Client
     4: Scheduled
     5: All
Select daemon type for status (1-5): 3
The defined Client resources are:
     1: srv1us
     2: srv2us
     3: srv123us
Select Client (File daemon) resource (1-3): 3
Connecting to Client srv123us at 108.61.250.36:9102
srv123us-fd Version: 7.0.5 (28 July 2014)  x86_64-pc-linux-gnu ubuntu 16.04
Daemon started 23-Feb-17 00:43. Jobs: run=1 running=0.
 Heap: heap=98,304 smbytes=571,344 max_bytes=571,361 bufs=97 max_bufs=97
 Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
 Plugin: bpipe-fd.so 

Running Jobs:
JobId 8789 Job srv123us.2017-09-01_00.45.28_34 is running.
    Incremental Backup Job started: 01-Sep-17 00:45
    Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0 Errors=0
    Bwlimit=0
    Files: Examined=5 Backed up=0
    SDReadSeqNo=6 fd=5
Director connected at: 01-Sep-17 01:10
====

Terminated Jobs:
====

As you can see, everything seems OK of the status, there was a running job in the client server and it seemed the backup process had been running without errors for more then 20 minutes, but then suddenly got Fatal error (the first log):

01-Sep 00:51 srv123us-fd JobId 8789: Warning: bsock.c:112 Could not connect to Storage daemon on 1.1.1.1:9103. ERR=Connection timed out
01-Sep 01:17 srv123us-fd JobId 8789: Fatal error: bsock.c:118 Unable to connect to Storage daemon on 1.1.1.1:9103. ERR=Interrupted system call
01-Sep 01:17 srv123us-fd JobId 8789: Fatal error: job.c:1893 Failed to connect to Storage daemon: 1.1.1.1:9103
01-Sep 01:17 backup01-de-dir JobId 8789: Fatal error: Bad response to Storage command: wanted 2000 OK storage

And the problem is that, the Director (backup server) connects to the File Service of the client (the daemon on the client), but the opposite connection is not possible! When the backup is ready, the client daemon bacula file service connects to the bacula storage service (which could be on the same server with the director, but it could be on another server) to send the backup files and here is the problem! Client could not connect to the storage! So always check the two way connections: backup server -> client server-port:9102 and backup server-port:9103 (or storage server) <- client.
In the world of bacula:

bacula-dir -> bacula-fd:9102

bacula-sd:9103 -> bacula-fd

Misleading error on causal look it seems like bacula-sd is returning error to bacula-fd (which would mean that bacula-fd could connect to bacula-sd after all), but in reality bacula-dir received and logged that bacula-fd did not connect to bacula-sd resulting in Fatal error.

In our situation the firewall of the backup server was denying the connections from the client, but it could be a DNS resolve issue or another network problem. Most common problems are firewall or DNS resolve issues. The solution – just add accept rule for the IP of the client to connect to port 9103 of the backup (storage) server.