Bacula – show configuration, status and information with bconsole tool

The following list of commands could be used to get a brief or detailed view of a Bacula backup server from the management utility

bconsole

. These commands are extremely useful for getting information on the backup process and policy and Bacula troubleshooting – could be used for fast debugging of an error, problems or misconfiguration.

the following commands give information for

  • Jobs

    list jobtotals Lists stats for all jobs, it also shows all jobs’ names
    show jobs Lists all jobs with their full configurations – show all jobs and for each job show detail explanation of what represent. The detail output includes full configuration of a job including client, catalog, fileset,schedule,pool,message. This will show all relationships between the different components of bacula system, how and which clients,storages,pools,schedules,filesets relate to. You’ll a thoroughly view of how let’s say a server is made the backup.
    show job=[job_name] shows full configuration for a job, the name could be taken by the two above commands
    list jobs Lists all jobs’ status – ID, StartTime, Type (backup?), Level (Full, increment, differential?), Files and bytes processed and the status of the job (Terminated normally, Running,Fatal error and so on)
    list files jobid=[ID] which files were included in the backup? Lists all paths and files included in the backup, not a configuration set but real physical path and filenames.
    status dir if you want to find all scheduled jobs for the next day (or more if you add a parameter). This will show the status of the director process.
  • Storages

    status storage list storage devices and their status – you can see the physical path on the filesystem where the Devices will put backup files
  • Clients

    show client show all clients’ names and backup policy
    status client=[client_name] show client status and what is doing, check the network connection between the director and the client, last terminated jobs and their status.
  • Filesets

    show fileset show all filesets, a fileset is a set with files and directories to include or exclude from a backup.
  • Schedule

    show schedule shows all registered schedulers and details for each one (Run Level=Full,Differential,Incremental), months, days, minutes.
    show schedule=[scheduler_name] shows details for a schedule with name scheduler_name (Run Level=Full,Differential,Incremental), months, days, minutes. It’s like the schedule backup plan of a server
  • Director

    message shows the last message of the backup process. If empty all logs of the backup process could be found in “/var/log/bacula/bacula.log”
    reload the director will re-read its all configuration files. Should be used when adding configuration files.

And here is the example output of the above commands with a little bit of explanation:

  • Jobs

    1) Get the stats and the jobs’ names, the names could be used in many other commands!

    srvbkp@local # bconsole 
    Connecting to Director localhost:9101
    1000 OK: 1 srvbkp-dir Version: 7.0.5 (28 July 2014)
    Enter a period to cancel a command.
    *list jobtotals
    Automatically selected Catalog: allbackup
    Using Catalog "allbackup"
    +------+-----------+-------------------+--------------------------------+
    | Jobs | Files     | Bytes             | Job                            |
    +------+-----------+-------------------+--------------------------------+
    |   90 |        90 |   123,665,584,337 | BackupCatalog                  |
    |    5 |         5 |   281,593,737,603 | RestoreFiles                   |
    |   13 | 1,232,316 |   118,480,634,434 | srv1-media                     |
    |   32 |        12 |             3,674 | srv1-dns                       |
    |   32 |        10 |             3,064 | srv2-dns                       |
    |   32 |        10 |             3,064 | srv3-dns                       |
    |   32 |        10 |             3,086 | srv4-dns                       |
    |   32 |        10 |             3,084 | srv5-dns                       |
    |   26 | 3,837,536 |   587,812,183,466 | srv1-images                    |
    +------+-----------+-------------------+--------------------------------+
    +-------+------------+-------------------+
    | Jobs  | Files      | Bytes             |
    +-------+------------+-------------------+
    | 1,474 | 14,925,321 | 5,475,024,028,957 |
    +-------+------------+-------------------+
    *
    

    2) Get all configurations for all jobs, here are included only two for clarity. All needed information for taking a backup of a server. You can see the files, which will be included (or excluded), where the backup will be stored when will happen in time, and how many different types of backup will be done – full, incremental, and differential. And this whole information is for all clients (servers).

    *show jobs
    Job: name=srv1-dns JobType=66 level= Priority=10 Enabled=1
         MaxJobs=1 Resched=0 Times=0 Interval=1,800 Spool=0 WritePartAfterJob=1
         Accurate=0
      --> Client: name=srv1-dns address=192.168.0.100 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
      --> FileSet: name=bind
          O MZ6
          N
          I /var/lib/named
          N
      --> Schedule: name=bind
      --> Run Level=Full
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Differential
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Incremental
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Storage: name=bind address=192.168.0.10 SDport=9103 MaxJobs=10
          DeviceName=bind MediaType=File StorageId=17
      --> Pool: name=Default PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=100 auto_prune=1 VolRetention=1 year 
          VolUse=0 secs recycle=1 LabelFormat=*None*
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=53687091200
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-full PoolType=Backup
          use_cat=1 use_once=1 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=2 months 
          VolUse=0 secs recycle=1 LabelFormat=bind-full
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-incr PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=7 days 
          VolUse=23 hours  recycle=1 LabelFormat=bind-incr
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-diff PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=1 month 1 day 
          VolUse=0 secs recycle=1 LabelFormat=bind-diff
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Messages: name=Standard
          mailcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: %t %e of %c %l" %r
          opcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: Intervention needed for %j" %r
    Job: name=srv2-dns JobType=66 level= Priority=10 Enabled=1
         MaxJobs=1 Resched=0 Times=0 Interval=1,800 Spool=0 WritePartAfterJob=1
         Accurate=0
      --> Client: name=srv2-dns address=192.168.0.101 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
      --> FileSet: name=bind
          O MZ6
          N
          I /var/lib/named
          N
      --> Schedule: name=bind
      --> Run Level=Full
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Differential
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Incremental
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Storage: name=bind address=192.168.0.10 SDport=9103 MaxJobs=10
          DeviceName=bind MediaType=File StorageId=17
      --> Pool: name=Default PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=100 auto_prune=1 VolRetention=1 year 
          VolUse=0 secs recycle=1 LabelFormat=*None*
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=53687091200
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-full PoolType=Backup
          use_cat=1 use_once=1 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=2 months 
          VolUse=0 secs recycle=1 LabelFormat=bind-full
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-incr PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=7 days 
          VolUse=23 hours  recycle=1 LabelFormat=bind-incr
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-diff PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=1 month 1 day 
          VolUse=0 secs recycle=1 LabelFormat=bind-diff
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Messages: name=Standard
          mailcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: %t %e of %c %l" %r
          opcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: Intervention needed for %j" %r
    

    3) You can get the full configuration information of a job (the information is the same as above, but for a given job name, which could be taken from the first command above, it is not necessary to output all the configurations every time):

    srvbkp@local # bconsole 
    Connecting to Director localhost:9101
    1000 OK: 1 srvbkp-dir Version: 7.0.5 (28 July 2014)
    Enter a period to cancel a command.
    *show jobs=srv2-dns
    Job: name=srv2-dns JobType=66 level= Priority=10 Enabled=1
         MaxJobs=1 Resched=0 Times=0 Interval=1,800 Spool=0 WritePartAfterJob=1
         Accurate=0
      --> Client: name=srv2-dns address=192.168.0.101 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
      --> FileSet: name=bind
          O MZ6
          N
          I /var/lib/named
          N
      --> Schedule: name=bind
      --> Run Level=Full
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Differential
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Incremental
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Storage: name=bind address=192.168.0.10 SDport=9103 MaxJobs=10
          DeviceName=bind MediaType=File StorageId=17
      --> Pool: name=Default PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=100 auto_prune=1 VolRetention=1 year 
          VolUse=0 secs recycle=1 LabelFormat=*None*
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=53687091200
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-full PoolType=Backup
          use_cat=1 use_once=1 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=2 months 
          VolUse=0 secs recycle=1 LabelFormat=bind-full
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-incr PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=7 days 
          VolUse=23 hours  recycle=1 LabelFormat=bind-incr
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Pool: name=bind-diff PoolType=Backup
          use_cat=1 use_once=0 cat_files=1
          max_vols=0 auto_prune=1 VolRetention=1 month 1 day 
          VolUse=0 secs recycle=1 LabelFormat=bind-diff
          CleaningPrefix=*None* LabelType=0
          RecyleOldest=0 PurgeOldest=0 ActionOnPurge=0
          MaxVolJobs=0 MaxVolFiles=0 MaxVolBytes=0
          MigTime=0 secs MigHiBytes=0 MigLoBytes=0
          JobRetention=0 secs FileRetention=0 secs
      --> Messages: name=Standard
          mailcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: %t %e of %c %l" %r
          opcmd=/usr/sbin/bsmtp -h localhost -f "(Bacula) <%r>" -s "Bacula: Intervention needed for %j" %r
    

    4) Lists all jobs’ status – ID, StartTime, Type (backup?), Level (Full, increment, differential?), Files and bytes processed and the status of the job (Terminated normally, Running,Fatal error and so on). Found out if you have backups of a server or the backup process failed!

    srvbkp@local # bconsole 
    Connecting to Director localhost:9101
    1000 OK: 1 srvbkp-dir Version: 7.0.5 (28 July 2014)
    Enter a period to cancel a command.
    *list jobs
    +-------+--------------------------------+---------------------+------+-------+-----------+-----------------+-----------+
    | JobId | Name                           | StartTime           | Type | Level | JobFiles  | JobBytes        | JobStatus |
    +-------+--------------------------------+---------------------+------+-------+-----------+-----------------+-----------+
    |   128 | srv1-test                      | 2016-12-04 23:05:00 | B    | F     |    17,506 |      52,116,400 | T         |
    |   178 | srv1-test                      | 2016-12-09 23:05:01 | B    | I     |        13 |           1,509 | T         |
    |   188 | srv1-test                      | 2016-12-10 23:05:01 | B    | I     |        13 |           1,509 | T         |
    .........................................................................................................................
    | 8,927 | srv2-images                    | 2018-03-04 20:00:00 | B    | F     |         0 |               0 | f         |
    | 8,928 | srv1-media                     | 2018-03-04 20:00:00 | B    | F     |         3 |             978 | T         |
    | 8,930 | srv1-dns                       | 2018-03-04 20:00:01 | B    | F     |         6 |           1,843 | T         |
    | 8,932 | srv2-dns                       | 2018-03-04 20:00:01 | B    | F     |         6 |           1,837 | T         |
    | 8,931 | srv3-dns                       | 2018-03-04 20:00:03 | B    | F     |         5 |           1,542 | T         |
    | 8,933 | srv4-dns                       | 2018-03-04 20:00:04 | B    | F     |         4 |           1,258 | T         |
    | 8,934 | srv5-dns                       | 2018-03-04 20:00:04 | B    | F     |         4 |           1,258 | T         |
    +-------+--------------------------------+---------------------+------+-------+-----------+-----------------+-----------+
    

    5) Which files were included in a backup job? Lists all paths and files included in the backup job (the ID is from the above command):

    *list files jobid=8934
    +----------+
    | Filename |
    +----------+
    | /var/lib/named/ |
    | /var/lib/named/root.cache |
    | /var/lib/named/sec |
    | /var/lib/named/sec/example.com.db |
    | /var/lib/named/sec/example2.net.db |
    | /var/lib/named/pri |
    +----------+
    +-------+----------+---------------------+------+-------+----------+----------+-----------+
    | JobId | Name     | StartTime           | Type | Level | JobFiles | JobBytes | JobStatus |
    +-------+----------+---------------------+------+-------+----------+----------+-----------+
    | 8,934 | srv5-dns | 2018-03-04 20:00:20 | B    | F     |        6 |    1,851 | T         |
    +-------+----------+---------------------+------+-------+----------+----------+-----------+
    

    6) Get the scheduled jobs. Which jobs will be executed and when:

    *status dir
    Scheduled Jobs:
    Level          Type     Pri  Scheduled          Job Name           Volume
    ===================================================================================
    Incremental    Backup    10  06-Mar-18 20:00    srv1-dns           *unknown*
    Incremental    Backup    10  06-Mar-18 20:00    srv2-dns           *unknown*
    Incremental    Backup    10  06-Mar-18 20:00    srv3-dns           *unknown*
    Incremental    Backup    10  06-Mar-18 20:00    srv4-dns           *unknown*
    Incremental    Backup    10  06-Mar-18 20:00    srv5-dns           *unknown*
    ...................................................................................
    Incremental    Backup    10  06-Mar-18 23:05    srv1-media         *unknown*
    Full           Backup    10  06-Mar-18 23:05    srv1-images        *unknown*
    ====
    Running Jobs:
    Console connected at 06-Mar-18 13:36
    No Jobs running.
    ====
    Terminated Jobs:
     JobId  Level    Files      Bytes   Status   Finished        Name
    ====================================================================
      9006  Incr        472    296.8 M  OK       05-Mar-18 23:05 srv1-dns
      9007  Incr     10,547    194.8 M  Error    05-Mar-18 23:05 srv2-dns
      9002  Incr         37    133.0 M  OK       05-Mar-18 23:05 srv3-dns
      8995  Incr         57    372.2 M  OK       05-Mar-18 23:06 srv4-dns
      9000  Incr        391    1.195 G  OK       05-Mar-18 23:07 srv5-dns
      9008  Full        832    7.139 G  OK       05-Mar-18 23:49 srv1-images
      9009  Full          1    1.493 G  OK       05-Mar-18 23:50 srv1-media
      9011  Full    315,027    121.6 G  OK       06-Mar-18 03:44 srv2-images
      9012  Full    314,804    93.85 G  OK       06-Mar-18 04:18 srv2-media
    ====
    *
    
  • Storages

  • Where are the backup files on your system? Trace the bacula media devices to the real path of your backup files.

    *status storage
    Automatically selected Storage: bind
    Connecting to Storage daemon bind at 192.168.0.10:9103
    srvbkp-sd Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
    Daemon started 06-Nov-17 17:25. Jobs: run=5025, running=0.
     Heap: heap=135,168 smbytes=2,231,640 max_bytes=5,264,027 bufs=439 max_bufs=2,152
     Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0
    Running Jobs:
    No Jobs running.
    ====
    Jobs waiting to reserve a drive:
    ====
    Terminated Jobs:
     JobId  Level    Files      Bytes   Status   Finished        Name
    ===================================================================
      9071  Incr          0         0   OK       06-Mar-18 20:00 srv1-dns
      9074  Incr          0         0   OK       06-Mar-18 20:00 srv2-dns
      9073  Incr          0         0   OK       06-Mar-18 20:00 srv3-dns
      9075  Incr          5    2.043 K  OK       06-Mar-18 20:00 srv4-dns
      9078  Incr          5    2.042 K  OK       06-Mar-18 20:00 srv5-dns
      9077  Incr          0         0   OK       06-Mar-18 20:00 srv1-media
      9079  Incr          0         0   OK       06-Mar-18 20:00 srv1-images
      9076  Incr          0         0   OK       06-Mar-18 20:00 srv2-images
      9057  Full          0         0   Other    06-Mar-18 20:31 srv2-media
    ====
    Device status:
    Device "localstorage" (/mnt/storage1/bacula-storage/local) is not open.
    ==
    Device "media" (/mnt/storage1/bacula-storage/media) is not open.
    ==
    Device "bind" (/mnt/storage1/bacula-storage/bind) is not open.
    ==
    Device "image" (/mnt/storage1/bacula-storage/image) is not open.
    ==
    ====
    Used Volume status:
    ====
    Attr spooling: 0 active jobs, 34,753 bytes; 120 total jobs, 34,753 max bytes.
    ====
    *
    
  • Clients

    1) Show all client names in the bacula system, this could be useful to link a client name with a server and then to use the name in the next command (below in 2):

    *show client
    Client: name=srvbkp-fd address=192.168.0.5 FDport=9102 MaxJobs=1
          JobRetention=3 months  FileRetention=2 months  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv1-dns address=192.168.0.100 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv2-dns address=192.168.0.101 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv3-dns address=192.168.0.103 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv4-dns address=192.168.0.104 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv5-dns address=192.168.0.105 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv1-media address=192.168.0.106 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    Client: name=srv1-images address=192.168.0.107 FDport=9102 MaxJobs=1
          JobRetention=1 month  FileRetention=1 month  AutoPrune=1
      --> Catalog: name=allbackup address=localhost DBport=0 db_name=bacula
          db_driver=*None* db_user=bacula MutliDBConn=0
    *
    

    2) Show status of a client. We can use this command to check what is going on with the client at the moment of issuing the command, the last terminated backup jobs and their status. In addition we can check the connection between the Director daemon and the client daemon, because the Director connects at the moment we issue the command, so it is useful for debugging purposes:

    *status client=srv1-dns
    Connecting to Client srv1-dns at 192.168.0.100:9102
    
    srv1-dns-fd Version: 7.0.5 (28 July 2014)  x86_64-pc-linux-gnu ubuntu 16.04
    Daemon started 23-Feb-18 00:43. Jobs: run=8 running=0.
     Heap: heap=98,304 smbytes=188,701 max_bytes=571,361 bufs=64 max_bufs=97
     Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
     Plugin: bpipe-fd.so 
    
    Running Jobs:
    Director connected at: 06-Mar-18 22:51
    No Jobs running.
    ====
    
    Terminated Jobs:
     JobId  Level    Files      Bytes   Status   Finished        Name 
    ===================================================================
      1832  Full          4    1.333 K  OK       01-Mar-18 23:48 srv1-dns
      1836  Incr          0         0   OK       02-Mar-18 00:01 srv1-dns
      1864  Incr          0         0   OK       02-Mar-18 20:30 srv1-dns
      1907  Incr          0         0   OK       03-Mar-18 20:00 srv1-dns
      1950  Full          4    1.333 K  OK       04-Mar-18 20:01 srv1-dns
      1994  Incr          0         0   OK       05-Mar-18 20:00 srv1-dns
      1037  Incr          0         0   OK       06-Mar-18 20:00 srv1-dns
    ====
    *
    
  • Filesets

    Which files will be included or excluded from the backup process. The lines starting with “I” mean “include”, the lines starting with “E” mean exclude.

     *show fileset
    FileSet: name=Full Set
     O M
     N
     I /usr/sbin
     N
     E /var/lib/bacula
     E /proc
     E /tmp
     E /sys
     E /.journal
     E /.fsck
     N
    FileSet: name=Catalog
     O M
     N
     I /var/lib/bacula/bacula.sql
     N
    FileSet: name=images
     O MfZ6
     N
     I /
     N
     E /proc
     E /tmp
     E /run
     E /dev
     E /sys
     N
    FileSet: name=bind
     O MZ6
     N
     I /var/lib/named
     N
    FileSet: name=media
     O MfZ6
     N
     I /
     N
     E /proc
     E /tmp
     E /run
     E /dev
     E /sys
     N
    
  • Schedule

    1) The all timeline plans for taking backups

    *show schedule
    Schedule: name=WeeklyCycle
      --> Run Level=Full
          hour=23 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=5
      --> Run Level=Differential
          hour=23 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=5
      --> Run Level=Incremental
          hour=23 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=5
    Schedule: name=WeeklyCycleAfterBackup
      --> Run Level=Full
          hour=23 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=10
    Schedule: name=images
      --> Run Level=Full
          hour=10 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Differential
          hour=10 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Incremental
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
    

    2) The schedule plan for one client or groups of clients (server/s):

    *show schedule=bind
    Schedule: name=bind
      --> Run Level=Full
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=0 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Differential
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=0 
          wom=1 2 3 4 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
      --> Run Level=Incremental
          hour=20 
          mday=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 
          month=0 1 2 3 4 5 6 7 8 9 10 11 
          wday=1 2 3 4 5 6 
          wom=0 1 2 3 4 5 
          woy=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 
          mins=0
    
  • Director

    1) Show the last messages of the backup processes. As you can see there is an error in one of the jobs, this error means that the client did not connect to the backup daemon (probably the same server with the Director), the problem was the firewall did not allow connections from this client (IP):

    *message
    06-Mar 20:00 srvbkp-dir JobId 1075: Start Backup JobId 1075, Job=srv1-bind.2018-03-06_20.00.01_05
    06-Mar 20:00 srvbkp-dir JobId 1075: Using Device "bind" to write.
    06-Mar 20:00 srvbkp-sd JobId 1066: Elapsed time=00:00:10, Transfer rate=0  Bytes/second
    06-Mar 20:00 srvbkp-dir JobId 1066: Bacula srvbkp-dir 7.0.5 (28Jul14):
      Build OS:               x86_64-pc-linux-gnu ubuntu 16.04
      JobId:                  1066
      Job:                    srv1-media.2018-03-06_20.00.00_56
      Backup Level:           Incremental, since=2018-03-06 20:00:14
      Client:                 "srv1-media" 7.0.5 (28Jul14) x86_64-pc-linux-gnu,ubuntu,16.04
      FileSet:                "bind" 2017-11-07 17:19:45
      Pool:                   "bind-incr" (From Job IncPool override)
      Catalog:                "allbackup" (From Client resource)
      Storage:                "bind" (From Job resource)
      Scheduled time:         06-Mar-2018 20:00:00
      Start time:             06-Mar-2018 20:00:15
      End time:               06-Mar-2018 20:00:26
      Elapsed time:           11 secs
      Priority:               10
      FD Files Written:       0
      SD Files Written:       0
      FD Bytes Written:       0 (0 B)
      SD Bytes Written:       0 (0 B)
      Rate:                   0.0 KB/s
      Software Compression:   None
      VSS:                    no
      Encryption:             no
      Accurate:               no
      Volume name(s):        
      Volume Session Id:      1025
      Volume Session Time:    1509989534
      Last Volume Bytes:      4,618 (4.618 KB)
      Non-fatal FD errors:    0
      SD Errors:              0
      FD termination status:  OK
      SD termination status:  OK
      Termination:            Backup OK
    06-Mar 20:01 srvbkp-dir JobId 1057: Using Device "bind" to write.
    06-Mar 20:05 srv2-dns-fd JobId 1057: Warning: bsock.c:112 Could not connect to Storage daemon on 192.168.0.5:9103. ERR=Connection timed out
    Retrying ...
    06-Mar 20:31 srv2-dns-fd JobId 1057: Fatal error: bsock.c:118 Unable to connect to Storage daemon on 192.168.0.5:9103. ERR=Interrupted system call
    06-Mar 20:31 srv2-dns-fd JobId 1057: Fatal error: job.c:1893 Failed to connect to Storage daemon: 192.168.0.5:9103
    06-Mar 20:31 srvbkp-dir JobId 1057: Fatal error: Bad response to Storage command: wanted 2000 OK storage
    , got 2902 Bad storage
    06-Mar 20:31 srvbkp-dir JobId 1057: Error: Bacula srvbkp-dir 7.0.5 (28Jul14):
      Build OS:               x86_64-pc-linux-gnu ubuntu 16.04
      JobId:                  1057
      Job:                    srv2-dns.2018-03-06_20.00.00_47
      Backup Level:           Full (upgraded from Incremental)
      Client:                 "srv2-dns" 7.0.5 (28Jul14) x86_64-pc-linux-gnu,ubuntu,16.04
      FileSet:                "bind" 2017-11-07 17:19:45
      Pool:                   "bind-full" (From Job FullPool override)
      Catalog:                "allbackup" (From Client resource)
      Storage:                "bind" (From Job resource)
      Scheduled time:         06-Mar-2018 20:00:00
      Start time:             06-Mar-2018 20:00:00
      End time:               06-Mar-2018 20:31:22
      Elapsed time:           31 mins 22 secs
      Priority:               10
      FD Files Written:       0
      SD Files Written:       0
      FD Bytes Written:       0 (0 B)
      SD Bytes Written:       0 (0 B)
      Rate:                   0.0 KB/s
      Software Compression:   None
      VSS:                    no
      Encryption:             no
      Accurate:               no
      Volume name(s):        
      Volume Session Id:      1201
      Volume Session Time:    1509989534
      Last Volume Bytes:      1 (1 B)
      Non-fatal FD errors:    2
      SD Errors:              0
      FD termination status:  Error
      SD termination status:  Waiting on FD
      Termination:            *** Backup Error ***
    

    2) reload – reload the configuration files of bacula system. The daemons will re-read all configuration files in “/etc/bacula”. Unfortunately there is no output:

    *reload
    *
    

Send coins in Ethereum network with geth console

Here are the steps needed to make a secure transfer of your funds under linux distro of Ubuntu. To be able to transfer money safely without any 3rd party involved (no online wallet) just funds from your personal Ethereum wallet address on your computer to another Ethereum wallet address we need the official command line interface for running a full ethereum node:

geth

STEP 1) Install and run an Ethereum node with geth

Here you can follow this tutorial – “Running an Ethereum node from source under Ubuntu 16 LTS”.
The Ethereum node must be running and synced with the Ethereum network.

STEP 2) Attach to the console of geth tool

ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.2-unstable-b574b577/linux-amd64/go1.9.4
coinbase: 0x30d8810bb1a74e808b46788b2316bd93cf056517
at block: 5172179 (Wed, 28 Feb 2018 15:59:17 UTC)
 datadir: /home/ubuntu/.ethereum
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

>

STEP 3) Check how many funds you have in your Ethereum wallet address

While you are in the geth console paste the following:

> web3.fromWei(eth.getBalance('30d8810bb1a74e808b46788b2316bd93cf056517'), "ether")
0.105
> 

You can see there are 0.105 Ethers in the Ethereum wallet address 30d8810bb1a74e808b46788b2316bd93cf056517, replace the address with your public address.

STEP 3) Unlock your account

As you remember your wallet private key is encrypted with a password, so before you can use the private key to send funds you must decrypt the key with:

> personal.unlockAccount('30d8810bb1a74e808b46788b2316bd93cf056517', '<your_password>',300)
true

So your key will stay dectrypted for 300 seconds, this is the purpose of the third parameter. It can be omitted.

STEP 4) Send the amount you like to the Ethereum wallet address you like

, BE CAREFUL the operation is irreversible, if you put wrong address or unknown or unexciting one, you’ll lose your Ether coins!!
Every transaction has a cost, which is computed by the miners. That’s why the gas price is dynamic. The price is determined by the miner of a block and it can change from block to block. You can check the gas price here on http://ethgasstation.info. Determining and computing the price in Ether is very difficult there could be at least 3 new howtos about that. The best and the easiest way is just to check a site (like http://ethgasstation.info) for the average gas price in dollars and when executing a transaction of sending Ethereum funds make sure the same amount of Ether in dollars will stay in your wallet address to be used for the gas.

> eth.sendTransaction({from: '30d8810bb1a74e808b46788b2316bd93cf056517', to: '9009a960fb38c379813a4a2e26c5b94909dc8599', value: web3.toWei(0.104, "ether")})
"0x31b4f213f88f2831756bd9cc37b12e80c6dfe413a3b3423659ca91be352961fc"

If you see the transaction ID, it means the you have executed the command successfully and you can check the status of the transaction.

STEP 5) Check the transaction status.

The status of your transaction could be checked in sites offering Ethereum block explorer sites like

https://etherscan.io/

Here is the check URL for our transaction:

https://etherscan.io/tx/0x31b4f213f88f2831756bd9cc37b12e80c6dfe413a3b3423659ca91be352961fc

Here is the screenshot of the status page:

main menu
Transaction details in https://etherscan.io

And verify the other address has the coins:

ubuntu@ip-172-31-30-78:~/go-ethereum/build/bin$ ./geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.2-unstable-b574b577/linux-amd64/go1.9.4
coinbase: 0x30d8810bb1a74e808b46788b2316bd93cf056517
at block: 5195693 (Sun, 04 Mar 2018 15:40:58 UTC)
 datadir: /home/ubuntu/.ethereum
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

> web3.fromWei(eth.getBalance('9009a960fb38c379813a4a2e26c5b94909dc8599'), "ether")
0.104
>

And how many coins remained at “30d8810bb1a74e808b46788b2316bd93cf056517”:

> web3.fromWei(eth.getBalance('30d8810bb1a74e808b46788b2316bd93cf056517'), "ether")
0.000811

So as you see the Gas was 0.000189 Ethers as you can see in the block explorer page.

Here is the whole output of the “geth” tool:

> ubuntu@ip-172-31-30-78:~/go-ethereum/build/bin$ ./geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.2-unstable-b574b577/linux-amd64/go1.9.4
coinbase: 0x30d8810bb1a74e808b46788b2316bd93cf056517
at block: 5193986 (Sun, 04 Mar 2018 08:48:41 UTC)
 datadir: /home/ubuntu/.ethereum
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

> web3.fromWei(eth.getBalance('30d8810bb1a74e808b46788b2316bd93cf056517'), "ether")
0.105
> personal.unlockAccount('30d8810bb1a74e808b46788b2316bd93cf056517', '<the_password_of_the_address>',300)
true
> eth.sendTransaction({from: '30d8810bb1a74e808b46788b2316bd93cf056517', to: '9009a960fb38c379813a4a2e26c5b94909dc8599', value: web3.toWei(0.104, "ether")})
"0x31b4f213f88f2831756bd9cc37b12e80c6dfe413a3b3423659ca91be352961fc"
> web3.fromWei(eth.getBalance('30d8810bb1a74e808b46788b2316bd93cf056517'), "ether")
0.000811
> 

Install netdata monitoring in CentOS 7

netdata became a great tool for admins to monitor in real time their servers!
At first it was just an additional not mandatory tool to check what’s going on with the servers for the last hour or so, but it evolved to really handy and informative monitoring server tracking every seconds what is going on with the server and server’s most used services like database, web, application service.
Today in version 1.9 (this installation howto is for netdata 1.9) it could track activity at least of this services:

apache          hddtemp          postgres
beanstalk       haproxy          rabbitmq
ceph            isc_dhcpd        retroshare
bind_rndc       ipfs             redis
couchdb         memcached        sensors
chrony          mdstat           samba
cpufreq         mongodb          squid
dns_query_time  nginx            springboot
dnsdist         mysql            smartd_log
elasticsearch   nsd              tomcat
dovecot         nginx_plus       web_log
exim            ovpn_status_log  varnish
example         ntpd             fronius
freeradius      postfix          named
fail2ban        phpfpm           snmp
go_expvar       powerdns         stiebeleltron

And some of these plugins support multiple programs and services, for example web_log supports the access/error logs of the major web servers at the moment.

The installation is really simple netdata includes a script to facilitate the installation process.
Here are the minimal steps to install this great software:

STEP 1) Install dependencies, because will pull it from the official repository we also need git command

yum install -y git gcc make autoconf automake pkgconfig zlib-devel libuuid-devel curl nodejs freeipmi freeipmi-devel elfutils-libelf cmake openssl-devel libuv-devel

As you can see there is a nodejs packet, which depends on additional repository (you could skip this, just the modules, which depends on nodejs won’t work, as if now only the plugins using nodejs are located in “/etc/netdata/node.d/” and they are not so many).

yum -y install epel-release
yum -y install nodejs

STEP 2) Clone the netdata repository

cd
git clone https://github.com/firehol/netdata

STEP 3) Instal netdata

cd netdata
CFLAGS="-march=native -O2 -msse3 -fomit-frame-pointer -pipe" ./netdata-installer.sh --install /usr/local/netdata

Install the netdata software in a separate directory and if you clean the system, just delete this directory. The example above uses

/usr/local/netdata

all files will be installed there.
As you can see the installation output the path of your files

   - the daemon     at /usr/local/netdata/netdata/usr/sbin/netdata
   - config files   in /usr/local/netdata/netdata/etc/netdata
   - web files      in /usr/local/netdata/netdata/usr/share/netdata
   - plugins        in /usr/local/netdata/netdata/usr/libexec/netdata
   - cache files    in /usr/local/netdata/netdata/var/cache/netdata
   - db files       in /usr/local/netdata/netdata/var/lib/netdata
   - log files      in /usr/local/netdata/netdata/var/log/netdata
   - pid file       at /usr/local/netdata/netdata/var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

STEP 4) USE firewall and open the port 19999 of your server to check e able to load the monitoring page

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="<YOURIP>" port protocol="tcp" port="19999" accept"
firewall-cmd --add-rich-rule="rule family="ipv4" source address="<YOURIP>" port protocol="tcp" port="19999" accept"

Because firewalld is the default firewall under CentOS 7 we used it ot show you how to let your IP access netdata web – replace with your current trusted IP.

* The installation process creates start/stop unit files for systemd and tells you how to update it (even you can run it automatically in a cron job)

To stop netdata run:
  systemctl stop netdata
To start netdata run:
  systemctl start netdata
 
Uninstall script generated: ./netdata-uninstaller.sh
Update script generated   : ./netdata-updater.sh
 
netdata-updater.sh can work from cron. It will trigger an email from cron
only if it fails (it does not print anything when it can update netdata).
Run this to automatically check and install netdata updates once per day:
 
sudo ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater

* Here is the output of an installation help menu – it also hints the dependencies it may need:

[root@srv.local netdata]# ./netdata-installer.sh --help

  ^
  |.-.   .-.   .-.   .-.   .-.   .  netdata                          .-.   .-
  |   '-'   '-'   '-'   '-'   '-'   installer command line options  '   '-'  
  +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->


./netdata-installer.sh <installer options>

Valid <installer options> are:

   --install /PATH/TO/INSTALL

        If you give: --install /opt
        netdata will be installed in /opt/netdata

   --dont-start-it

        Do not (re)start netdata.
        Just install it.

   --dont-wait

        Do not wait for the user to press ENTER.
        Start immediately building it.

   --auto-update | -u

        Install netdata-updater to cron,
        to update netdata automatically once per day
        (can only be done for installations from git)

   --enable-plugin-freeipmi
   --disable-plugin-freeipmi

        Enable/disable the FreeIPMI plugin.
        Default: enable it when libipmimonitoring is available.

   --enable-plugin-nfacct
   --disable-plugin-nfacct

        Enable/disable the nfacct plugin.
        Default: enable it when libmnl and libnetfilter_acct are available.

   --enable-lto
   --disable-lto

        Enable/disable Link-Time-Optimization
        Default: enabled

   --zlib-is-really-here
   --libs-are-really-here

        If you get errors about missing zlib,
        or libuuid but you know it is available,
        you have a broken pkg-config.
        Use this option to allow it continue
        without checking pkg-config.

Netdata will by default be compiled with gcc optimization -O2
If you need to pass different CFLAGS, use something like this:

  CFLAGS="<gcc options>" ./netdata-installer.sh <installer options>

For the installer to complete successfully, you will need
these packages installed:

   gcc make autoconf automake pkg-config zlib1g-dev (or zlib-devel)
   uuid-dev (or libuuid-devel)

For the plugins, you will at least need:

   curl, bash v4+, python v2 or v3, node.js

* netdata in action

main menu
All real-time monitoring plugins, System Overview opened

main menu
Memory details

main menu
PHP-FPM local details

* And here is the output of an installation process:

[root@lsrv3 netdata]# CFLAGS="-march=native -O2 -msse3 -fomit-frame-pointer -pipe" ./netdata-installer.sh --install /usr/local/netdata

  ^
  |.-.   .-.   .-.   .-.   .  netdata                                        
  |   '-'   '-'   '-'   '-'   real-time performance monitoring, done right!  
  +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->


  You are about to build and install netdata to your system.

  It will be installed at these locations:

   - the daemon     at /usr/local/netdata/netdata/usr/sbin/netdata
   - config files   in /usr/local/netdata/netdata/etc/netdata
   - web files      in /usr/local/netdata/netdata/usr/share/netdata
   - plugins        in /usr/local/netdata/netdata/usr/libexec/netdata
   - cache files    in /usr/local/netdata/netdata/var/cache/netdata
   - db files       in /usr/local/netdata/netdata/var/lib/netdata
   - log files      in /usr/local/netdata/netdata/var/log/netdata
   - pid file       at /usr/local/netdata/netdata/var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

  This installer allows you to change the installation path.
  Press Control-C and run the same command with --help for help.

Press ENTER to build and install netdata to '/usr/local/netdata/netdata' > 

 --- Run autotools to configure the build environment --- 
[/root/netdata]# ./autogen.sh 
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --force
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
autoreconf: Leaving directory `.'
 OK   

[/root/netdata]# ./configure --prefix=/usr/local/netdata/netdata/usr --sysconfdir=/usr/local/netdata/netdata/etc --localstatedir=/usr/local/netdata/netdata/var --with-zlib --with-math --with-user=netdata CFLAGS=-march=native\ -O2\ -msse3\ -fomit-frame-pointer\ -pipe 
checking whether to enable maintainer-specific portions of Makefiles... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to create a pax tar archive... gnutar
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for __attribute__((returns_nonnull))... no
checking for __attribute__((malloc))... yes
checking for __attribute__((noreturn))... yes
checking for __attribute__((noinline))... yes
checking for __attribute__((format))... yes
checking for __attribute__((warn_unused_result))... yes
checking for struct timespec... yes
checking for clockid_t... yes
checking for library containing clock_gettime... none required
checking for clock_gettime... yes
checking for sched_setscheduler... yes
checking for sched_get_priority_min... yes
checking for sched_get_priority_max... yes
checking for nice... yes
checking for recvmmsg... yes
checking for int8_t... yes
checking for int16_t... yes
checking for int32_t... yes
checking for int64_t... yes
checking for uint8_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for inline... inline
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... yes
checking for _Generic... no
checking for __atomic... yes
checking size of void *... 8
checking whether sys/types.h defines makedev... yes
checking for sys/types.h... (cached) yes
checking for netinet/in.h... yes
checking for arpa/nameser.h... yes
checking for netdb.h... yes
checking for resolv.h... yes
checking for sys/prctl.h... yes
checking for linux/netfilter/nfnetlink_conntrack.h... yes
checking for accept4... yes
checking operating system... linux
checking if compiler needs -Werror to reject unknown flags... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for sin in -lm... yes
checking if libm should be used... yes
checking for ZLIB... yes
checking if zlib should be used... yes
checking for UUID... yes
checking for memory allocator... system
checking for mallopt... yes
checking for mallinfo... yes
checking for LIBCAP... no
checking if libcap should be used... no
checking if apps.plugin should be enabled... yes
checking for IPMIMONITORING... yes
checking for
        ipmi_monitoring_sensor_readings_by_record_id,
        ipmi_monitoring_sensor_readings_by_sensor_type,
        ipmi_monitoring_sensor_read_sensor_number,
        ipmi_monitoring_sensor_read_sensor_name,
        ipmi_monitoring_sensor_read_sensor_state,
        ipmi_monitoring_sensor_read_sensor_units,
        ipmi_monitoring_sensor_iterator_next,
        ipmi_monitoring_ctx_sensor_config_file,
        ipmi_monitoring_ctx_sdr_cache_directory,
        ipmi_monitoring_ctx_errormsg,
        ipmi_monitoring_ctx_create
     in -lipmimonitoring... yes
checking ipmi_monitoring.h usability... yes
checking ipmi_monitoring.h presence... yes
checking for ipmi_monitoring.h... yes
checking ipmi_monitoring_bitmasks.h usability... yes
checking ipmi_monitoring_bitmasks.h presence... yes
checking for ipmi_monitoring_bitmasks.h... yes
checking if freeipmi.plugin should be enabled... yes
checking for NFACCT... no
checking for LIBMNL... no
checking if nfacct.plugin should be enabled... no
checking for setns... yes
checking if cgroup-network can be enabled... yes
checking whether C compiler accepts -flto... yes
checking if -flto builds executables... yes
checking if LTO should be enabled... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating charts.d/Makefile
config.status: creating conf.d/Makefile
config.status: creating netdata.spec
config.status: creating python.d/Makefile
config.status: creating node.d/Makefile
config.status: creating plugins.d/Makefile
config.status: creating src/Makefile
config.status: creating system/Makefile
config.status: creating web/Makefile
config.status: creating diagrams/Makefile
config.status: creating makeself/Makefile
config.status: creating contrib/Makefile
config.status: creating tests/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
 OK   

 --- Cleanup compilation directory --- 
 --- Compile netdata --- 
[/root/netdata]# make -j8 
make  all-recursive
make[1]: Entering directory `/root/netdata'
Making all in charts.d
make[2]: Entering directory `/root/netdata/charts.d'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/charts.d'
Making all in conf.d
make[2]: Entering directory `/root/netdata/conf.d'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/conf.d'
Making all in diagrams
make[2]: Entering directory `/root/netdata/diagrams'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/diagrams'
Making all in makeself
make[2]: Entering directory `/root/netdata/makeself'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/makeself'
Making all in node.d
make[2]: Entering directory `/root/netdata/node.d'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/node.d'
Making all in plugins.d
make[2]: Entering directory `/root/netdata/plugins.d'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/netdata/plugins.d'
Making all in python.d
make[2]: Entering directory `/root/netdata/python.d'
if sed \
        -e 's#[@]localstatedir_POST@#/usr/local/netdata/netdata/var#g' \
        -e 's#[@]sbindir_POST@#/usr/local/netdata/netdata/usr/sbin#g' \
        -e 's#[@]sysconfdir_POST@#/usr/local/netdata/netdata/etc#g' \
        -e 's#[@]pythondir_POST@#/usr/local/netdata/netdata/usr/libexec/netdata/python.d#g' \
        python-modules-installer.sh.in > python-modules-installer.sh.tmp; then \
        mv "python-modules-installer.sh.tmp" "python-modules-installer.sh"; \
else \
        rm -f "python-modules-installer.sh.tmp"; \
        false; \
fi
make[2]: Leaving directory `/root/netdata/python.d'
Making all in src
make[2]: Entering directory `/root/netdata/src'
gcc -DHAVE_CONFIG_H -I. -I..  -DVARLIB_DIR="\"/usr/local/netdata/netdata/var/lib/netdata\"" -DCACHE_DIR="\"/usr/local/netdata/netdata/var/cache/netdata\"" -DCONFIG_DIR="\"/usr/local/netdata/netdata/etc/netdata\"" -DLOG_DIR="\"/usr/local/netdata/netdata/var/log/netdata\"" -DPLUGINS_DIR="\"/usr/local/netdata/netdata/usr/libexec/netdata/plugins.d\"" -DRUN_DIR="\"/usr/local/netdata/netdata/var/run/netdata\"" -DWEB_DIR="\"/usr/local/netdata/netdata/usr/share/netdata/web\""          -march=native -O2 -msse3 -fomit-frame-pointer -pipe -pthread -flto -MT apps_plugin.o -MD -MP -MF .deps/apps_plugin.Tpo -c -o apps_plugin.o apps_plugin.c
make[2]: Leaving directory `/root/netdata'
make[1]: Leaving directory `/root/netdata'
 OK   

 --- Restore user edited netdata configuration files --- 
 --- Fix generated files permissions --- 
[/root/netdata]# find ./system/ -type f -a \! -name \*.in -a \! -name Makefile\* -a \! -name \*.conf -a \! -name \*.service -a \! -name \*.logrotate -exec chmod 755 \{\} \; 
 OK   

 --- Add user netdata to required user groups --- 
Adding netdata user group ...
[/root/netdata]# groupadd -r netdata 
 OK   

Adding netdata user account with home /usr/local/netdata/netdata ...
[/root/netdata]# useradd -r -g netdata -c netdata -s /usr/sbin/nologin --no-create-home -d /usr/local/netdata/netdata netdata 
 OK   

Group 'docker' does not exist.
Adding netdata user to the nginx group ...
[/root/netdata]# usermod -a -G nginx netdata 
 OK   

Group 'varnish' does not exist.
Adding netdata user to the haproxy group ...
[/root/netdata]# usermod -a -G haproxy netdata 
 OK   

Adding netdata user to the adm group ...
[/root/netdata]# usermod -a -G adm netdata 
 OK   

Group 'nsd' does not exist.
Group 'proxy' does not exist.
Group 'squid' does not exist.
Group 'ceph' does not exist.
 --- Install logrotate configuration for netdata --- 
[/root/netdata]# cp system/netdata.logrotate /etc/logrotate.d/netdata 
 OK   

[/root/netdata]# chmod 644 /etc/logrotate.d/netdata 
 OK   

 --- Read installation options from netdata.conf --- 

    Permissions
    - netdata user     : netdata
    - netdata group    : netdata
    - web files user   : netdata
    - web files group  : netdata
    - root user        : root

    Directories
    - netdata conf dir : /usr/local/netdata/netdata/etc/netdata
    - netdata log dir  : /usr/local/netdata/netdata/var/log/netdata
    - netdata run dir  : /usr/local/netdata/netdata/var/run
    - netdata lib dir  : /usr/local/netdata/netdata/var/lib/netdata
    - netdata web dir  : /usr/local/netdata/netdata/usr/share/netdata/web
    - netdata cache dir: /usr/local/netdata/netdata/var/cache/netdata

    Other
    - netdata port     : 19999

 --- Fix permissions of netdata directories (using user 'netdata') --- 
[/root/netdata]# mkdir -p /usr/local/netdata/netdata/var/run 
 OK   

[/root/netdata]# chown -R root:netdata /usr/local/netdata/netdata/etc/netdata 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/etc/netdata -type f -exec chmod 0640 \{\} \; 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/etc/netdata -type d -exec chmod 0755 \{\} \; 
 OK   

[/root/netdata]# chown -R netdata:netdata /usr/local/netdata/netdata/usr/share/netdata/web 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/share/netdata/web -type f -exec chmod 0664 \{\} \; 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/share/netdata/web -type d -exec chmod 0775 \{\} \; 
 OK   

[/root/netdata]# chown -R netdata:netdata /usr/local/netdata/netdata/var/lib/netdata 
 OK   

[/root/netdata]# chown -R netdata:netdata /usr/local/netdata/netdata/var/cache/netdata 
 OK   

[/root/netdata]# chown -R netdata:netdata /usr/local/netdata/netdata/var/log/netdata 
 OK   

[/root/netdata]# chmod 755 /usr/local/netdata/netdata/var/log/netdata 
 OK   

[/root/netdata]# chown netdata:root /usr/local/netdata/netdata/var/log/netdata 
 OK   

[/root/netdata]# chown -R root /usr/local/netdata/netdata/usr/libexec/netdata 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/libexec/netdata -type d -exec chmod 0755 \{\} \; 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/libexec/netdata -type f -exec chmod 0644 \{\} \; 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/libexec/netdata -type f -a -name \*.plugin -exec chmod 0755 \{\} \; 
 OK   

[/root/netdata]# find /usr/local/netdata/netdata/usr/libexec/netdata -type f -a -name \*.sh -exec chmod 0755 \{\} \; 
 OK   

[/root/netdata]# chown root:netdata /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/apps.plugin 
 OK   

[/root/netdata]# chmod 0750 /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/apps.plugin 
 OK   

[/root/netdata]# setcap cap_dac_read_search\,cap_sys_ptrace+ep /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/apps.plugin 
 OK   

[/root/netdata]# chown root:netdata /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/freeipmi.plugin 
 OK   

[/root/netdata]# chmod 4750 /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/freeipmi.plugin 
 OK   

[/root/netdata]# chown root:netdata /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/cgroup-network 
 OK   

[/root/netdata]# chmod 4750 /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/cgroup-network 
 OK   

[/root/netdata]# chown root /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh 
 OK   

[/root/netdata]# chmod 0550 /usr/local/netdata/netdata/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh 
 OK   

[/root/netdata]# chmod a+rX /usr/local/netdata/netdata/usr/libexec 
 OK   

[/root/netdata]# chmod a+rX /usr/local/netdata/netdata/usr/share/netdata 
 OK   

 --- Install netdata at system init --- 
Installing systemd service...
[/root/netdata]# cp system/netdata.service /etc/systemd/system/netdata.service 
 OK   

[/root/netdata]# systemctl daemon-reload 
 OK   

[/root/netdata]# systemctl enable netdata 
Created symlink from /etc/systemd/system/multi-user.target.wants/netdata.service to /etc/systemd/system/netdata.service.
 OK   

 --- Start netdata --- 
[/root/netdata]# /usr/bin/systemctl stop netdata 
 OK   

[/root/netdata]# /usr/bin/systemctl restart netdata 
 OK   

OK. NetData Started!


-------------------------------------------------------------------------------

Downloading default configuration from netdata...
[/root/netdata]# curl -s -o /usr/local/netdata/netdata/etc/netdata/netdata.conf.new http://localhost:19999/netdata.conf 
 OK   

[/root/netdata]# mv /usr/local/netdata/netdata/etc/netdata/netdata.conf.new /usr/local/netdata/netdata/etc/netdata/netdata.conf 
 OK   

 OK  New configuration saved for you to edit at /usr/local/netdata/netdata/etc/netdata/netdata.conf 

[/root/netdata]# chown netdata /usr/local/netdata/netdata/etc/netdata/netdata.conf 
 OK   

[/root/netdata]# chmod 0664 /usr/local/netdata/netdata/etc/netdata/netdata.conf 
 OK   

 --- Check KSM (kernel memory deduper) --- 
 --- Check version.txt --- 
 --- Check apps.plugin --- 
 --- Generate netdata-uninstaller.sh --- 
 --- Basic netdata instructions --- 

netdata by default listens on all IPs on port 19999,
so you can access it with:

  http://this.machine.ip:19999/

To stop netdata run:

  systemctl stop netdata

To start netdata run:

  systemctl start netdata


Uninstall script generated: ./netdata-uninstaller.sh
Update script generated   : ./netdata-updater.sh

netdata-updater.sh can work from cron. It will trigger an email from cron
only if it fails (it does not print anything when it can update netdata).
Run this to automatically check and install netdata updates once per day:

sudo ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater

 --- We are done! --- 

  ^
  |.-.   .-.   .-.   .-.   .-.   .  netdata                          .-.   .-
  |   '-'   '-'   '-'   '-'   '-'   is installed and running now!  -'   '-'  
  +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->

  enjoy real-time performance and health monitoring...

Persistent connections for PHP sessions stored using Memecached

If you have a highly loaded PHP application server you probably came to conclusion to use memcached to store the PHP sessions, because files are too slow and generate many IOs. And when the session are enabled to be stored in memcached server the server could be overloaded with so many connections to the memcached instance(s) that the server could delay the creation of new connections to the memcached server and the server is in trouble even more than before with the file stored sessions.
The solution is to use persistent connections to the memcached server, so when a connection to the memcached is needed it will check to reuse an existing idle connection stored in a pool of connections so no new connection is created each time PHP needs something from the cache like the sessions. In fact PHP accesses sessions from the cache at least once for every request to the server and it could generate a pretty big number of opened connections.

On top of that there are two PHP modules, which offer objects or functions to use memcached server cache in your PHP code. The old module is called “memcache” (link) and the new one is “memcached” (link) – just to note these are the names of the PHP modules not to be mistaken by the server caching system “memcached”.

To configure the php-fpm or mod_php to store sessions in memcached caching server:

  1. use PHP “memcached”, this is the new and better supported module for using a memcached cache server. It supports the newer versions of PHP like 7+.
  2. use the following configuration added to your php.ini or the module ini file included in your php.ini (it really depends on your configuration, but if you are not sure put phpinfo() function in a PHP file and load through your web server – you’ll see the physical location of your php.ini file);
session.save_path = "PERSISTENT=1 localhost:11212"

Replace localhost with the server IP or leave it if the memcached server is on the localhost. This single configuration could decrease the usage of opened sockets from several tens of thousands (literally 30 000 – 100 000 and above) to couple of hundreds, which could have a great impact on system performance! (You’ll see it like a system usage time in your server graphs…)

*Worth mentioning the PHP configuration (php.ini file) must be edited to use memcached server for storing sessions:

session.save_handler = memcached

Running an Ethereum node from source under Ubuntu 16 LTS

This howto is made to show you how to run an Ethereum node. To run an Ethereum node we need the

geth

program to start and wait for the network synchronization – our Ethereum node will receive all the blockchains of the network!
After a successful Ethereum network synchronization we can send and receive coins securely ourselves! No other online service offers us secure sending and receiving of transcations and creation of wallet, the only way a crypto user could be absolutely sure the private key is seen only from him and is generated on his server is to become part of the Ethereum network and create your wallet with “geth” manually by the user himself.
And also be warned DO NOT use insecure sources when:

  1. installing Ubuntu, install it from a cd/dvd/usb downloaded from the official Ubuntu site here.
  2. installing all the needed dependencies from official sources only, not recommended the PPA reposigories, neither!
  3. install yourself the “geth” program, which will create an Ethereum node and after network synchronization could be used to create an Ethereum node.

Three simple rules could almost eliminate leaking your private key to hackers, which in blockchain world means losing your money…
And before start the installation a performance advice do not attempt to run an Ethereum node in a hard disk –

USE SSD

If you use a hard disk even in RAID configuration it could take several days for network synchronization and probably your machine will be useless during the synchronization.

So here are the steps to create from a scratch an Ethereum node:

STEP 1) Install Ubuntu 16 – our howto will be available soon here.

STEP 2) Building from source an Ethereum node

Check out our howto“Building from source an Ethereum node under Ubuntu 16 LTS”.

STEP 3) Start the Ethereum node

if you have followed the howto in STEP 2) you probably have “geth” installed in “~/go-ethereum/build/bin”, so first running it with:

ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth --rpc --fast --cache=8144 &> ./geth.log

STEP 3) Wait for the blockchain network synchronization

When you run “geth” it will check the current state of the network and the current state of your Ethereum node, so no matter it is started for the first time or not a network synchronization is needed, your node need to be synchronized with the Ethereum network to be used for all kind of actions in the network (create a wallet, transfer funds and more).
Here is the output of just started Ethereum node:

INFO [02-27|19:14:02] Maximum peer count                       ETH=25 LES=0 total=25
INFO [02-27|19:14:02] Starting peer-to-peer node               instance=Geth/v1.8.2-unstable-b574b577/linux-amd64/go1.9.4
INFO [02-27|19:14:02] Allocated cache and file handles         database=/home/ubuntu/.ethereum/geth/chaindata cache=6108 handles=512
INFO [02-27|19:14:02] Writing default main-net genesis block 
INFO [02-27|19:14:03] Persisted trie from memory database      nodes=12356 size=2.34mB time=49.406185ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [02-27|19:14:03] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: <nil> Engine: ethash}"
INFO [02-27|19:14:03] Disk storage enabled for ethash caches   dir=/home/ubuntu/.ethereum/geth/ethash count=3
INFO [02-27|19:14:03] Disk storage enabled for ethash DAGs     dir=/home/ubuntu/.ethash               count=2
INFO [02-27|19:14:03] Initialising Ethereum protocol           versions="[63 62]" network=1
INFO [02-27|19:14:03] Loaded most recent local header          number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [02-27|19:14:03] Loaded most recent local full block      number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [02-27|19:14:03] Loaded most recent local fast block      number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [02-27|19:14:03] Regenerated local transaction journal    transactions=0 accounts=0
INFO [02-27|19:14:03] Starting P2P networking 
INFO [02-27|19:14:05] UDP listener up                          self=enode://4ee1f43cc597fa829a04ec97dee38ad4f2c61f39f0f0e5ebd46e095c660cc95aab71eef55b0db9fd64d0afaeb589a8d9cb3eb86fdc07e32183648bc89dbd4aca@[::]:30303
INFO [02-27|19:14:05] RLPx listener up                         self=enode://4ee1f43cc597fa829a04ec97dee38ad4f2c61f39f0f0e5ebd46e095c660cc95aab71eef55b0db9fd64d0afaeb589a8d9cb3eb86fdc07e32183648bc89dbd4aca@[::]:30303
INFO [02-27|19:14:05] HTTP endpoint opened                     url=http://127.0.0.1:8545 cors= vhosts=localhost
INFO [02-27|19:14:05] IPC endpoint opened                      url=/home/ubuntu/.ethereum/geth.ipc
INFO [02-27|19:14:45] Block synchronisation started 
WARN [02-27|19:14:53] Synchronisation failed, retrying         err="block download canceled (requested)"
WARN [02-27|19:14:56] Node data write error                    err="state node 6aa064…378280 failed with all peers (1 tries, 1 peers)"
WARN [02-27|19:14:56] Synchronisation failed, retrying         err="state node 6aa064…378280 failed with all peers (1 tries, 1 peers)"
WARN [02-27|19:15:07] Node data write error                    err="state node 6aa064…378280 failed with all peers (3 tries, 3 peers)"
WARN [02-27|19:15:07] Synchronisation failed, retrying         err="state node 6aa064…378280 failed with all peers (3 tries, 3 peers)"
INFO [02-27|19:15:20] Imported new block headers               count=192 elapsed=1.115s number=192 hash=723899…123390 ignored=0
INFO [02-27|19:15:20] Imported new block receipts              count=2   elapsed=80.248µs number=2   hash=b495a1…4698c9 size=8.00B  ignored=0
INFO [02-27|19:15:20] Imported new block headers               count=192 elapsed=19.822ms number=384 hash=d3d5d5…c79cf3 ignored=0
INFO [02-27|19:15:21] Imported new block receipts              count=4   elapsed=117.07µs number=6   hash=1f1aed…6b326e size=1.10kB ignored=0
INFO [02-27|19:15:22] Imported new block headers               count=2048 elapsed=146.493ms number=2432 hash=78a9e6…079de3 ignored=0

As you can see the node connects to another nodes and receives blockchains and saves them to its local database. At present (late February 2017) a fully synchronized Ethereum node takes 59 Gigabytes of space, but this number will changed when you run your node.

STEP 3) Check the progress of the Ethereum network synchronization

The progress of Ethereum node synchronization could be checked with the “geth” program. Execute in the console:

ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth --exec "web3.eth.syncing" attach
{
  currentBlock: 388,
  highestBlock: 5159514,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 0
}
ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth --exec "web3.eth.syncing" attach
{
  currentBlock: 401,
  highestBlock: 5159514,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 0
}

As you can see we are at the very beginning here, we’ve received 401 of total 5159514 in the network.

After around 2 hours we are at 4882892 of total 5166333 in the network.

ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth --exec "web3.eth.syncing" attach
{
  currentBlock: 4882892,
  highestBlock: 5166333,
  knownStates: 4806466,
  pulledStates: 4800942,
  startingBlock: 0
}

It is normal the highest block to increment, because the network is used intensively.
STEP 4) Successful full synchronization – how to check and to be sure
When you are almost then like in the last section of the STEP 3) you’ll begin to see lines in the geth output like:

INFO [02-28|01:46:28] Imported new block headers               count=0    elapsed=1.281ms   number=5168639 hash=433308…1a710e ignored=71

and when you execute geth with “web3.eth.syncing”:

ubuntu@srv.local:~/go-ethereum/build/bin$ ./geth --exec "web3.eth.syncing" attach
false

The node will keep on receiving new blocks, so synchronization is triggered again, syncing the received blocks and stops again. So when transferring funds or creating a wallet or any other action there will be triggered a synchronization and then execution of the action. There is a callback for when sync starts and stops. If you cannot see multiple times “false” after the last synchronization check with geth, but your are almost at the highest block, no need to worry everything is OK, you node is almost fully synchronized and can be used.

At present with SSD and 32G RAM on a intel i7 processor and a fast Internet connection 100Mbps the full synchronization took about 8 hours.

Building from source an Ethereum node under Ubuntu 16 LTS

The program, which makes an Ethereum node in your computer, is called

geth

. “geth” is written using the GO language and the latest version depends on GO version 1.7+. Ubuntu 16 LTS has GO language package version 1.6 so to build it from source we need first to install a proper version of GO Language.

STEP 1) Install a proper version of GO Language 1.7+

In our tutorial you will install 1.9 using this howto: “Install GO language in Ubuntu 16 LTS from the official site package”

STEP 2) Update your system and install dependencies

sudo apt-get update -y
sudo apt-get install -y build-essential libgmp3-dev git

STEP 3) Get the “geth” code and compile it

git clone https://github.com/ethereum/go-ethereum
cd go-ethereum
make geth

STEP 4) Check is everything is OK, installed geth version 1.8.2-unstable

srv@local:~/go-ethereum$ cd ./build/bin
srv@local:~/go-ethereum/build/bin$ ./geth version
Geth
Version: 1.8.2-unstable
Git Commit: b574b5776695eb30e034fd8c7a468b3f03d4c6b9
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9.4
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

* Next tutorial is getting the node up and running here!

* Troubleshooting

  • if you get
    ~/go-ethereum$ make geth
    build/env.sh go run build/ci.go install ./cmd/geth
    build/env.sh: 30: exec: go: not found
    Makefile:15: recipe for target 'geth' failed
    make: *** [geth] Error 127
    

    You did not install properly go, check our manual here: “Install GO language in Ubuntu 16 LTS from the official site package”. Check whether you source /etc/profile in your current ssh/console/login session, check if you exported the PATH, your PATH must include “your_goinstallation_path/bin”, it’s like “/usr/local/go”.

  • if you get
    ~/go-ethereum$ make geth
    build/env.sh go run build/ci.go install ./cmd/geth
    ci.go:190: You have Go version go1.6.2
    ci.go:191: go-ethereum requires at least Go version 1.7 and cannot
    ci.go:192: be compiled with an earlier version. Please upgrade your Go installation.
    exit status 1
    Makefile:15: recipe for target 'geth' failed
    make: *** [geth] Error 1
    

    The installation just uses the default installed GO language of your system, which is 1.6 and is not supported by the “geth” Ethereum program. There are two options:

When we talk for crypto currency the best source is always the GIT version (probably the best one is the stable tag, but it is safe to use the unstable, too), it is most secure source you could get. Do not download from 3rd party sites or unofficial repositories, you could lose your money.

* Here is the building output

srv@local:~/go-ethereum$ make geth
build/env.sh go run build/ci.go install ./cmd/geth
>>> /usr/local/go/bin/go install -ldflags -X main.gitCommit=b574b5776695eb30e034fd8c7a468b3f03d4c6b9 -v ./cmd/geth
github.com/ethereum/go-ethereum/crypto/sha3
github.com/ethereum/go-ethereum/common/math
github.com/ethereum/go-ethereum/rlp
github.com/ethereum/go-ethereum/vendor/github.com/go-stack/stack
github.com/ethereum/go-ethereum/common/hexutil
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/comparer
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage
github.com/ethereum/go-ethereum/crypto/secp256k1
github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy
github.com/ethereum/go-ethereum/log
github.com/ethereum/go-ethereum/vendor/gopkg.in/karalabe/cookiejar.v2/collections/prque
github.com/ethereum/go-ethereum/common
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/filter
github.com/ethereum/go-ethereum/vendor/github.com/aristanetworks/goarista/monotime
github.com/ethereum/go-ethereum/common/mclock
github.com/ethereum/go-ethereum/event
github.com/ethereum/go-ethereum/crypto/randentropy
github.com/ethereum/go-ethereum/vendor/github.com/pborman/uuid
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/opt
github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix
github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/pbkdf2
github.com/ethereum/go-ethereum/params
github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/scrypt
github.com/ethereum/go-ethereum/vendor/gopkg.in/fatih/set.v0
github.com/ethereum/go-ethereum/cmd/internal/browser
github.com/ethereum/go-ethereum/metrics
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/errors
github.com/ethereum/go-ethereum/common/fdlimit
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/iterator
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/journal
github.com/ethereum/go-ethereum/vendor/github.com/hashicorp/golang-lru/simplelru
github.com/ethereum/go-ethereum/vendor/golang.org/x/net/context
github.com/ethereum/go-ethereum/vendor/github.com/hashicorp/golang-lru
github.com/ethereum/go-ethereum/vendor/golang.org/x/net/websocket
github.com/ethereum/go-ethereum/vendor/github.com/edsrzf/mmap-go
github.com/ethereum/go-ethereum/common/bitutil
github.com/ethereum/go-ethereum/vendor/github.com/rs/xhandler
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/memdb
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table
github.com/ethereum/go-ethereum/crypto/bn256
github.com/ethereum/go-ethereum/vendor/github.com/rs/cors
github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/ripemd160
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/httpu
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/scpd
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/soap
github.com/ethereum/go-ethereum/rpc
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/ssdp
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify
github.com/ethereum/go-ethereum/vendor/github.com/elastic/gosigar
github.com/ethereum/go-ethereum/vendor/golang.org/x/net/html/atom
github.com/ethereum/go-ethereum/vendor/golang.org/x/net/html
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/internal/identifier
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/transform
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/internal/utf8internal
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/internal/tag
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/language
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/runes
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/internal
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/charmap
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/japanese
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/korean
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/simplifiedchinese
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/traditionalchinese
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/unicode
github.com/ethereum/go-ethereum/ethdb
github.com/ethereum/go-ethereum/vendor/github.com/jackpal/go-nat-pmp
github.com/ethereum/go-ethereum/p2p/netutil
github.com/ethereum/go-ethereum/eth/tracers/internal/tracers
github.com/ethereum/go-ethereum/vendor/gopkg.in/olebedev/go-duktape.v3
github.com/ethereum/go-ethereum/vendor/github.com/golang/protobuf/proto
github.com/ethereum/go-ethereum/vendor/github.com/karalabe/hid
github.com/ethereum/go-ethereum/log/term
github.com/ethereum/go-ethereum/metrics/exp
github.com/ethereum/go-ethereum/vendor/github.com/mattn/go-colorable
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1
github.com/ethereum/go-ethereum/vendor/github.com/prometheus/prometheus/util/flock
github.com/ethereum/go-ethereum/vendor/golang.org/x/text/encoding/htmlindex
github.com/ethereum/go-ethereum/les/flowcontrol
github.com/ethereum/go-ethereum/vendor/golang.org/x/sync/syncmap
github.com/ethereum/go-ethereum/internal/jsre/deps
github.com/ethereum/go-ethereum/vendor/golang.org/x/net/html/charset
github.com/ethereum/go-ethereum/vendor/github.com/mattn/go-isatty
github.com/ethereum/go-ethereum/vendor/github.com/fatih/color
github.com/ethereum/go-ethereum/vendor/gopkg.in/sourcemap.v1/base64vlq
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp
github.com/ethereum/go-ethereum/vendor/gopkg.in/sourcemap.v1
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/token
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/dbg
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/file
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/dcps/internetgateway1
github.com/ethereum/go-ethereum/vendor/github.com/huin/goupnp/dcps/internetgateway2
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/ast
github.com/ethereum/go-ethereum/internal/debug
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/parser
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto/registry
github.com/ethereum/go-ethereum/internal/web3ext
github.com/ethereum/go-ethereum/vendor/github.com/peterh/liner
github.com/ethereum/go-ethereum/vendor/github.com/maruel/panicparse/stack
github.com/ethereum/go-ethereum/vendor/github.com/golang/protobuf/protoc-gen-go/descriptor
github.com/ethereum/go-ethereum/p2p/nat
github.com/ethereum/go-ethereum/vendor/github.com/mattn/go-runewidth
github.com/ethereum/go-ethereum/vendor/github.com/mitchellh/go-wordwrap
github.com/ethereum/go-ethereum/vendor/github.com/naoina/go-stringutil
github.com/ethereum/go-ethereum/vendor/github.com/naoina/toml/ast
github.com/ethereum/go-ethereum/vendor/github.com/robertkrimen/otto
github.com/ethereum/go-ethereum/accounts/usbwallet/internal/trezor
github.com/ethereum/go-ethereum/vendor/github.com/nsf/termbox-go
github.com/ethereum/go-ethereum/vendor/github.com/naoina/toml
github.com/ethereum/go-ethereum/vendor/github.com/gizak/termui
# github.com/ethereum/go-ethereum/vendor/gopkg.in/olebedev/go-duktape.v3
In file included from vendor/gopkg.in/olebedev/go-duktape.v3/api.go:7:0:
vendor/gopkg.in/olebedev/go-duktape.v3/api.go: In function ‘_duk_error’:
vendor/gopkg.in/olebedev/go-duktape.v3/duktape.h:510:127: warning: right-hand operand of comma expression has no effect [-Wunused-value]
  (duk_error_raw((ctx), (duk_errcode_t) (err_code), (const char *) (DUK_FILE_MACRO), (duk_int_t) (DUK_LINE_MACRO), __VA_ARGS__), (duk_ret_t) 0)
                                                                                                                               ^
vendor/gopkg.in/olebedev/go-duktape.v3/api.go:153:2: note: in expansion of macro ‘duk_error’
  duk_error(ctx, err_code, "%s", str);
  ^
github.com/ethereum/go-ethereum/internal/jsre
github.com/ethereum/go-ethereum/crypto
github.com/ethereum/go-ethereum/trie
github.com/ethereum/go-ethereum/crypto/ecies
github.com/ethereum/go-ethereum/p2p/discover
github.com/ethereum/go-ethereum/p2p/discv5
github.com/ethereum/go-ethereum/core/types
github.com/ethereum/go-ethereum/p2p
github.com/ethereum/go-ethereum
github.com/ethereum/go-ethereum/core/state
github.com/ethereum/go-ethereum/core/bloombits
github.com/ethereum/go-ethereum/core/vm
github.com/ethereum/go-ethereum/accounts
github.com/ethereum/go-ethereum/ethclient
github.com/ethereum/go-ethereum/accounts/keystore
github.com/ethereum/go-ethereum/accounts/usbwallet
github.com/ethereum/go-ethereum/consensus
github.com/ethereum/go-ethereum/consensus/misc
github.com/ethereum/go-ethereum/dashboard
github.com/ethereum/go-ethereum/whisper/whisperv5
github.com/ethereum/go-ethereum/consensus/clique
github.com/ethereum/go-ethereum/consensus/ethash
github.com/ethereum/go-ethereum/eth/fetcher
github.com/ethereum/go-ethereum/node
github.com/ethereum/go-ethereum/console
github.com/ethereum/go-ethereum/core
github.com/ethereum/go-ethereum/eth/downloader
github.com/ethereum/go-ethereum/eth/filters
github.com/ethereum/go-ethereum/light
github.com/ethereum/go-ethereum/internal/ethapi
github.com/ethereum/go-ethereum/miner
github.com/ethereum/go-ethereum/eth/gasprice
github.com/ethereum/go-ethereum/eth/tracers
github.com/ethereum/go-ethereum/eth
github.com/ethereum/go-ethereum/les
github.com/ethereum/go-ethereum/ethstats
github.com/ethereum/go-ethereum/cmd/utils
github.com/ethereum/go-ethereum/cmd/geth
Done building.
Run "/home/ubuntu/go-ethereum/build/bin/geth" to launch geth.
srv@local:~/go-ethereum$

Install GO language in Ubuntu 16 LTS from the official site package

Ubuntu 16 LTS comes with too old GO language version, the version in the official repository is

golang-1.6.

Many programs released past couple of years depend on go lang 1.7+, so 1.6 in Ubuntu 16 LTS is really obsolete. At present the latest version is 1.9, so we’ll show you simple steps to install the go lang on your ubuntu 16 lts server

STEP 1) Get root if you want to install it globally to be used from any user!

sudo su
cd

STEP 2) Download the GO language binaries from the official site

https://golang.org/dl/

And copy the URL link of the program from the Linux section, now it has the following text:

Linux 2.6.23 or later, Intel 64-bit processor
go1.10.linux-amd64.tar.gz (114MB)

And download it with wget:

wget https://dl.google.com/go/go1.9.4.linux-amd64.tar.gz

STEP 4) Uncompress the file with “tar”

tar xzvf go1.9.4.linux-amd64.tar.gz

STEP 5) Move the files in “/usr/local/go”

you could put the installation anywhere, but most manual installed programs go to “/usr/local”

mv go /usr/local/

STEP 6) Set properly the environment to use your GO 1.9 installation

    • Create file with path and name

      /etc/profile.d/golang.sh

    • Make the file executable with
      chmod 755 /etc/profile.d/golang.sh
      
    • Insert the following line, which will add to the environment PATH your GO executable binaries
[[ -d "/usr/local/go" ]] && export PATH="$PATH:/usr/local/go/bin"

STEP 7) Reload the environment or just log off/in and check if you could execute the GO tool

root@srv.local:~# source /etc/profile
root@srv.local:~# go version
go version go1.9.4 linux/amd64

* Here are all the commands in one place to install the specific version of GO 1.9.4 AMD64, you can use them in a script:

cd
mkdir go-installation
cd ./go-installation
sudo wget https://dl.google.com/go/go1.9.4.linux-amd64.tar.gz
sudo tar xzvf go1.9.4.linux-amd64.tar.gz
sudo mv go /usr/local/
cat <<END | sudo tee /etc/profile.d/golang.sh 1>/dev/null
[[ -d "/usr/local/go" ]] && export PATH="\$PATH:/usr/local/go/bin"
END
sudo chmod 755 /etc/profile.d/golang.sh
source /etc/profile

And do not forget to

source /etc/profile

in your current shell session. It is mandatory for your current shell if you execute the above command from a script!

* This installation of GO Language 1.9 is from the official source the home site of GO Language. There are other easier ways to do it using, for example there are PPA repositories with Ubuntu packages ready to install, but they are not supported by the official GO team! Use them at your own risk as with all other PPA repositories.

mysql slave reset and fixing relay log read failure

Suddenly your slave server reset without a clean shutdown and when it came up again you saw the error of this kind:

2016-02-26 10:41:50 876 [ERROR] Slave SQL: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave. Error_code: 1594
2016-02-26 10:41:50 876 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.005014' position 152146793

So what we know, our master server is OK, our slave server was reset by unknown issue, so the problem is only in out slave logs. Mysql sever shows the status with:

mysql> SHOW SLAVE STATUS\G;

There are multiple lines of information, but the most important in our situation is these two lines:

Relay_Master_Log_File: mysql-bin.005014
  Exec_Master_Log_Pos: 152146793

This is the place where the slave server stopped at (as you can see from the logs above, newer versions of MySQL print these two values in the log, but older versions do not print them in the log, so check them with the above command!).
The slave server stopped at file mysql-bin.005014 and position 152146793 and could not continue, because its files are corrupted. We can reset the position issuing a CHANGE MASTER command, which will clean up the relay logs and the slave will start the replication from this position – no data will be lost. Before issuing the following commands save the relay log files, they can be useful if you have later errors. Here is the command:

STOP SLAVE;              
CHANGE MASTER TO
         MASTER_HOST='1.1.1.1',
         MASTER_USER='replusr',
         MASTER_LOG_FILE='mysql-bin.005014',
         MASTER_LOG_POS=152146793;
START SLAVE;

There three commands above

  • Stop the replication in the slave, because the replication is still running and the slave is logging the binary log received from the master
  • Change master command to reset the logs with the right position
  • Start the replication in the slave

The replication must continue without errors!

In some cases after we issue the above commands and the replication starts it immediately stops with error of

Duplicate entry

.

Last_SQL_Errno: 1062
Last_SQL_Error: Error 'Duplicate entry '3918722' for key 'PRIMARY'' on query. Default database: 'testdb'. Query: 'INSERT INTO `testtable` (`tabid`, `tabip`, `stat`, `ins`) VALUES ('83908', '2591777309', '1', NOW())'

So we did everything right, but our replication is again broken? The problem is that when there is such reset, it could happen the autoincrement of the table is reserved,but not used, because the server was reset just in the middle of the insert operation or it could be inserted properly, but the server was reset in the middle of updating the replication metadata! So you have two options:

  • Change the autoincrement value of the table if there is no record with ID of the duplicate entry, just select it:
    SELECT * FROM testdb WHERE id=[ID_FROM_THE_ERROR]
    

    If there is no ID with such value, change the autoicrement of the table with

    ALTER TABLE tbl AUTO_INCREMENT = [ID_FROM_THE_ERROR];
    
  • Skip the duplicate entry query with
    STOP SLAVE;
    SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
    START SLAVE;
    

    or parallel replication use

    STOP SLAVE;
    START SLAVE UNTIL sql_after_mts_gaps;
    SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
    START SLAVE;
    

You could trace the problem reading the relay logs at the position it stopped.
Often there is an issue with the last recorded position, so you should examine why you have duplicate entry. Check if the entry is inserted and if it is, just skip it! But then if you hit again a duplicate entry or another error, you should reinitialize the slave dumping the replicated databases from the master!

Here is the full log of status command, when there is a problem with the corrupted mysql relay logs:

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 1.1.1.1
                  Master_User: repluser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.005014
          Read_Master_Log_Pos: 246696051
               Relay_Log_File: mysqld-relay-bin.009911
                Relay_Log_Pos: 152146956
        Relay_Master_Log_File: mysql-bin.005014
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 1594
                   Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 152146793
              Relay_Log_Space: 246698113
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 1594
               Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: ce8a6c29-cf8e-11e5-9d39-000000000001
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: 
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 180226 11:54:51
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
1 row in set (0.00 sec)

network time synchronization status with ntpq

There is a handy program called

ntpstats

, which shows the network time synchronization status of your server under Ubuntu 16, but the older version or other linux distros do not have this program. The output of the program is:

srv@local # ntpstat
synchronised to NTP server (129.250.35.250) at stratum 3 
   time correct to within 133 ms
   polling server every 1024 s

As you can see the time is correct within some interval of milliseconds. In other words your time in the server could be trusted only within these milliseconds! If your server is synchronized with the program ntp your time of the server will be correct within a tiny fractions of a second, closer to a time server much smaller interval of correctness. The interval is computed according to how away the server is and how much the latency of the network of the server is.
ntpstats compute the output with the formula:

rootdisp + rootdelay / 2.0

  • rootdisp – indicating the maximum error relative to the primary reference source. If your server gets the time from an external source (GPS, radio, high precise clock external device), this value is the estimated maximum error of that clock and if your server gets the time from another NTP server (in most cases), this value is the current NTP server’s root dispersion (to which your server is synced) plus the dispersion added by the network link between your server and the NTP server
  • rootdelay – indicating the total roundtrip delay to the primary reference source. Because it is a roundtrip we divide it to 2 to get the single trip delay of the network

You can get these two values with the ntpq program:

srv@local # ntpq -c "rv 0 rootdisp,rootdelay"
rootdelay=202.882, rootdisp=31.816

So the time correct to within 31.816+(202.882/2) = 133.257 ms, which is the output of ntpstat.