Moving existing Elasticsearch and Kibana installation from CentOS 7 to CentOS Stream 9

main menu
install Elasticsearch and Kibana

Despite having only two additional installed software under CentOS 7 it is not a good idea to just try upgrading only CentOS 7 to CentOS Stream 9. There is no clear and supported path for upgrading from CentOS 7 to CentOS Stream 9 and even to the older one CentOS 8 (or CentOS Stream 8). The best way is to just make a clean install of CentOS Stream 9 and copy all the Elasticsearch and Kibana files and this article is how to do it without problems.
Here is the plan to move the existing installation of Elasticsearch and Kibana services from CentOS 7 to CentOS Stream 9:

  1. Make a clean install of CentOS Stream 9
  2. Update the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).
  3. Add Elasticsearch and Kibana repositories to the new system. Tune the system crypto policies.
  4. Install Elasticsearch and Kibana software packages, but do not start the services.
  5. Copy Elasticsearch and Kibana important files such as the index directory and the configuration directories. Check the user and group IDs of the files.
  6. Start the Elasticsearch and Kibana services.

In this example, the installation of the new server is just starting a new LXC container, which will host the Elasticsearch and Kibana services. There is no difference between using a container or a physical machine. With LXC container it is easier to copy the needed files such as the Elasticsearch index files, which may be tens of terabytes or more, and various configuration files.

STEP 1) Make a clean install of CentOS Stream 9

Check out the following article on the purpose – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation or if LXC container is preferred – Run LXC CentOS Stream 9 container with bridged network under CentOS Stream 9.

Creating a LXC container of CentOS Stream 9 is really simple and fast:

[root@srv ~]# lxc-create --template download -n kibana.u1x2.com -- --dist centos --release 9-Stream --arch amd64
The cached copy has expired, re-downloading...
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Centos 9-Stream x86_64 (20230511_19:27) container.

Then tune the network following the above article. It is a good idea when configuring the network to preserve the original UUIDs and network addresses (MAC address, too) of the LXC containers network and the inner container’s interface.
So copy the UUID from /var/lib/lxc/loganalyzer-old/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0 to the CentOS Stream 9 network configuration – /var/lib/lxc/loganalyzer/rootfs/etc/NetworkManager/system-connections/ethernet-eth0.nmconnection, which uses NetworkManager. And the LXC container’s MAC address: the variable lxc.net.0.hwaddr from /var/lib/lxc/loganalyzer/config to /var/lib/lxc/loganalyzer/config.
The last step is to run the newly installed system. No errors in the output signals for a successful start-up of the LXC container with the name loganalyzer.

[root@srv ~]# lxc-start -n loganalyzer
[root@srv ~]# 

STEP 2) Upgrade the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).

For example, if the current Elasticsearch is version 7. It is good to upgrade it to the latest version from 7.x before proceeding with the next steps.
The current installed versions of Elasticsearch and Kibana software are from the branch 77.17.4-1 and the latest version is 7.17.10-1.
Check in the old system with (CentOS 7):

[root@loganalyzer-old ~]# yum list installed|egrep -e "(elasticsearch|kibana)"
elasticsearch.x86_64               7.17.4-1                               @elasticsearch
kibana.x86_64                      7.17.4-1                               @elasticsearch


If they are the latest version from their branch/minor version proceed with the other steps. If they aren’t the latest version it is a good idea to update them first and then copy them to the new system. The important thing is to install the same version in the new system or at least, the same major version (for example, if the old one is 7.17.4-1, install it in the new system the 7.17.X version. The best case scenario is under the old and new systems the Elastisearch and Kibana to be the same versions)!
The upgrade of Elasticsearch and Kibana should be easy and straightforward:

[root@loganalyzer-old ~]# yum update elasticsearch kibana
Last metadata expiration check: 0:00:38 ago on Fri May 12 12:58:13 2023.
Dependencies resolved.
================================================================================================================
 Package                      Architecture          Version                  Repository                    Size
================================================================================================================
Upgrading:
 elasticsearch                x86_64                7.17.10-1                elasticsearch                302 M
 kibana                       x86_64                7.17.10-1                elasticsearch                261 M

Transaction Summary
================================================================================================================
Upgrade  2 Packages

Total download size: 563 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): elasticsearch-7.17.10-x86_64.rpm                                          42 MB/s | 302 MB     00:07    
(2/2): kibana-7.17.10-x86_64.rpm                                                 27 MB/s | 261 MB     00:09    
----------------------------------------------------------------------------------------------------------------
Total                                                                            59 MB/s | 563 MB     00:09     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                        1/1 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                1/1 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                1/4 
  Upgrading        : kibana-7.17.10-1.x86_64                                                                1/4 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                1/4 
  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         2/4 
  Upgrading        : elasticsearch-7.17.10-1.x86_64                                                         2/4 
  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         2/4 
  Running scriptlet: kibana-7.17.4-1.x86_64                                                                 3/4 
Stopping kibana service...OK
  Cleanup          : kibana-7.17.4-1.x86_64                                                                 3/4 
  Running scriptlet: kibana-7.17.4-1.x86_64                                                                 3/4 
  Running scriptlet: elasticsearch-7.17.4-1.x86_64                                                          4/4 
  Cleanup          : elasticsearch-7.17.4-1.x86_64                                                          4/4 
  Running scriptlet: elasticsearch-7.17.4-1.x86_64                                                          4/4 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                4/4 
  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         4/4 
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
  Running scriptlet: elasticsearch-7.17.4-1.x86_64                                                          4/4 
  Verifying        : elasticsearch-7.17.10-1.x86_64                                                         1/4 
  Verifying        : elasticsearch-7.17.4-1.x86_64                                                          2/4 
  Verifying        : kibana-7.17.10-1.x86_64                                                                3/4 
  Verifying        : kibana-7.17.4-1.x86_64                                                                 4/4 

Upgraded:
  elasticsearch-7.17.10-1.x86_64                             kibana-7.17.10-1.x86_64                            

Complete!
....
[root@loganalyzer-old ~]# systemctl restart elasticsearch
[root@loganalyzer-old ~]# systemctl restart kibana

After a successful minor update proceed with the next steps. Bear in mind, upgrading to the latest current branch version is not mandatory! It should work with minor differences in the versions or just install the very same software versions to the new system.

STEP 3) Add Elasticsearch and Kibana repositories to the new system.

Login to the new server and add the Elasticsearch repository. Create a file /etc/yum.repos.d/elasticsearch.repo with the following content:

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And for the Kibana software – /etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

List the available packages with their versions:

[root@loganalyzer ~]# dnf list elasticsearch kibana
Elasticsearch repository for 7.x packages                                        43 MB/s |  50 MB     00:01    
Kibana repository for 7.x packages                                               46 MB/s |  50 MB     00:01    
Last metadata expiration check: 0:00:09 ago on Fri May 12 12:54:29 2023.
Available Packages
elasticsearch.aarch64                                  7.17.10-1                                   elasticsearch
elasticsearch.aarch64                                  7.17.10-1                                   kibana-7.x   
elasticsearch.x86_64                                   7.17.10-1                                   elasticsearch
elasticsearch.x86_64                                   7.17.10-1                                   kibana-7.x   
kibana.aarch64                                         7.17.10-1                                   elasticsearch
kibana.aarch64                                         7.17.10-1                                   kibana-7.x   
kibana.x86_64                                          7.17.10-1                                   elasticsearch
kibana.x86_64                                          7.17.10-1                                   kibana-7.x   

Because these repositories use the old algorithm SHA1 for the package signature, the crypto policy should be tuned or a message of the following may occur (under CentOS Stream 9):

Importing GPG key 0xD88E42B4:
 Userid     : "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>"
 Fingerprint: 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
 From       : https://artifacts.elastic.co/GPG-KEY-elasticsearch
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: elasticsearch-7.17.10-1.x86_64
 GPG Keys are configured as: https://artifacts.elastic.co/GPG-KEY-elasticsearch
Public key for kibana-7.17.10-x86_64.rpm is not installed. Failing package is: kibana-7.17.10-1.x86_64
 GPG Keys are configured as: https://artifacts.elastic.co/GPG-KEY-elasticsearch
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

To solve this error and to be able to install these packages the update-crypto-policies tool is needed to update the system policies, which is part of crypto-policies-scripts.

[root@srv ~]# dnf install -y crypto-policies-scripts
Last metadata expiration check: 0:14:40 ago on Fri May 12 12:56:12 2023.
Dependencies resolved.
================================================================================================================
 Package                           Architecture     Version                              Repository        Size
================================================================================================================
Installing:
 crypto-policies-scripts           noarch           20221215-1.git9a18988.el9            baseos            93 k

Transaction Summary
================================================================================================================
Install  1 Package

Total download size: 93 k
Installed size: 225 k
Downloading Packages:
crypto-policies-scripts-20221215-1.git9a18988.el9.noarch.rpm                    286 kB/s |  93 kB     00:00    
----------------------------------------------------------------------------------------------------------------
Total                                                                            88 kB/s |  93 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                        1/1 
  Installing       : crypto-policies-scripts-20221215-1.git9a18988.el9.noarch                               1/1 
  Running scriptlet: crypto-policies-scripts-20221215-1.git9a18988.el9.noarch                               1/1 
  Verifying        : crypto-policies-scripts-20221215-1.git9a18988.el9.noarch                               1/1 

Installed:
  crypto-policies-scripts-20221215-1.git9a18988.el9.noarch                                                      

Complete!
[root@srv ~]# update-crypto-policies --set DEFAULT:SHA1
Setting system policy to DEFAULT:SHA1
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

First, install and then update the crypto policy as shown above. Of course, after installing these packages the policy might be reverted to the defaults.

STEP 4) Install Elasticsearch and Kibana software packages, but do not start the services.

[root@loganalyzer ~]# dnf install elasticsearch kibana
Last metadata expiration check: 0:17:38 ago on Fri May 12 12:56:12 2023.
Dependencies resolved.
================================================================================================================
 Package                      Architecture          Version                  Repository                    Size
================================================================================================================
Installing:
 elasticsearch                x86_64                7.17.10-1                elasticsearch                302 M
 kibana                       x86_64                7.17.10-1                elasticsearch                261 M

Transaction Summary
================================================================================================================
Install  2 Packages

Total download size: 563 M
Installed size: 1.1 G
Is this ok [y/N]: y
Downloading Packages:
(1/2): kibana-7.17.10-x86_64.rpm                                                 41 MB/s | 261 MB     00:06    
(2/2): elasticsearch-7.17.10-x86_64.rpm                                          34 MB/s | 302 MB     00:08    
----------------------------------------------------------------------------------------------------------------
Total                                                                            63 MB/s | 563 MB     00:08     
Elasticsearch repository for 7.x packages                                        29 kB/s | 1.7 kB     00:00    
Importing GPG key 0xD88E42B4:
 Userid     : "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>"
 Fingerprint: 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
 From       : https://artifacts.elastic.co/GPG-KEY-elasticsearch
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                        1/1 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                1/2 
  Installing       : kibana-7.17.10-1.x86_64                                                                1/2 
  Running scriptlet: kibana-7.17.10-1.x86_64                                                                1/2 
Creating kibana group... OK
Creating kibana user... OK

  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         2/2 
Creating elasticsearch group... OK
Creating elasticsearch user... OK

  Installing       : elasticsearch-7.17.10-1.x86_64                                                         2/2 
  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         2/2 
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

  Running scriptlet: kibana-7.17.10-1.x86_64                                                                2/2 
Created Kibana keystore in /etc/kibana/kibana.keystore

  Running scriptlet: elasticsearch-7.17.10-1.x86_64                                                         2/2 
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore

/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : elasticsearch-7.17.10-1.x86_64                                                         1/2 
  Verifying        : kibana-7.17.10-1.x86_64                                                                2/2 

Installed:
  elasticsearch-7.17.10-1.x86_64                             kibana-7.17.10-1.x86_64                            

Complete!

STEP 6) Move the files and configuration to the new system

First, rename the original directories. Note, the users and the groups – elasticsearch:elasticsearch and kibana:kibana and the fact, the data directories of Elasticsearch and Kibana are empty.

[root@loganalyzer ~]# mv /etc/elasticsearch /etc/elasticsearch-org
[root@loganalyzer ~]# mv /etc/kibana /etc/kibana-org
[root@loganalyzer ~]# ls -altr /var/lib/kibana/
total 8
drwxr-s---.  2 kibana kibana 4096 Apr 22 11:29 .
drwxr-xr-x. 18 root   root   4096 May 12 13:14 ..
[root@loganalyzer ~]# ls -altr /var/lib/elasticsearch/
total 8
drwxr-s---.  2 elasticsearch elasticsearch 4096 Apr 23 05:38 .
drwxr-xr-x. 18 root          root          4096 May 12 13:14 ..
[root@loganalyzer ~]# ls -altr /var/lib/kibana/
total 8
drwxr-s---.  2 kibana kibana 4096 Apr 22 11:29 .
drwxr-xr-x. 18 root   root   4096 May 12 13:36 ..
[root@loganalyzer ~]# mv /var/lib/elasticsearch /var/lib/elasticsearch-org
[root@loganalyzer ~]# mv /var/lib/kibana /var/lib/kibana-org

Second, copy the configurations and data directories from the old system to the new one. Using LXC containers it is a matter of just moving directories locally. On the LXC host move the configuration directories and note the users and groups are different, so chwon the directories accordingly to have elasticsearch and kibana users and groups.

[root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/etc/elasticsearch /var/lib/lxc/loganalyzer/rootfs/etc/
[root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/etc/kibana /var/lib/lxc/loganalyzer/rootfs/etc/
[root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/var/lib/elasticsearch /var/lib/lxc/loganalyzer/rootfs/var/lib/
[root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/var/lib/kibana /var/lib/lxc/loganalyzer/rootfs/var/lib/

Users and groups are with different IDs, so they should be corrected.

[root@loganalyzer ~]# ls -altr /etc/|grep elastic
drwxr-s---.  3 root sgx             4096 May 12 13:00 elasticsearch
drwxr-s---.  3 root elasticsearch   4096 May 12 13:14 elasticsearch-org
[root@loganalyzer ~]# ls -altr /etc/|grep kibana
drwxr-s---.  2 root systemd-oom     4096 May 12 12:59 kibana
drwxr-s---.  2 root kibana          4096 May 12 13:14 kibana-org
[root@loganalyzer ~]# chown -R root:elasticsearch /etc/elasticsearch
[root@loganalyzer ~]# chown -R root:kibana /etc/kibana
[root@loganalyzer ~]# ls -altr /etc/elasticsearch
total 64
-rw-rw----.  1 root elasticsearch  3523 Dec 11  2021 elasticsearch.yml
-rw-r--r--.  1 root elasticsearch   596 Dec 11  2021 passwords
-rw-rw----.  1 root elasticsearch  3323 Jun 14  2022 jvm.options
-rw-rw----.  1 root elasticsearch     0 Apr 23 05:35 users_roles
-rw-rw----.  1 root elasticsearch     0 Apr 23 05:35 users
-rw-rw----.  1 root elasticsearch   197 Apr 23 05:35 roles.yml
-rw-rw----.  1 root elasticsearch   473 Apr 23 05:35 role_mapping.yml
-rw-rw----.  1 root elasticsearch 19304 Apr 23 05:35 log4j2.properties
-rw-rw----.  1 root elasticsearch  1042 Apr 23 05:35 elasticsearch-plugins.example.yml
drwxr-s---.  2 root elasticsearch  4096 Apr 23 05:38 jvm.options.d
-rw-rw----.  1 root elasticsearch   199 May 12 13:00 elasticsearch.keystore
-rw-r--r--.  1 root elasticsearch    76 May 12 13:00 .elasticsearch.keystore.initial_md5sum
drwxr-s---.  3 root elasticsearch  4096 May 12 13:00 .
drwxr-xr-x. 67 root root           4096 May 12 13:27 ..
[root@loganalyzer ~]# ls -altr /etc/elasticsearch-org/
total 60
-rw-rw----.  1 root elasticsearch     0 Apr 23 05:35 users_roles
-rw-rw----.  1 root elasticsearch     0 Apr 23 05:35 users
-rw-rw----.  1 root elasticsearch   197 Apr 23 05:35 roles.yml
-rw-rw----.  1 root elasticsearch   473 Apr 23 05:35 role_mapping.yml
-rw-rw----.  1 root elasticsearch 19304 Apr 23 05:35 log4j2.properties
-rw-rw----.  1 root elasticsearch  3329 Apr 23 05:35 jvm.options
-rw-rw----.  1 root elasticsearch  3431 Apr 23 05:35 elasticsearch.yml
-rw-rw----.  1 root elasticsearch  1042 Apr 23 05:35 elasticsearch-plugins.example.yml
drwxr-s---.  2 root elasticsearch  4096 Apr 23 05:38 jvm.options.d
-rw-rw----.  1 root elasticsearch   199 May 12 13:14 elasticsearch.keystore
drwxr-s---.  3 root elasticsearch  4096 May 12 13:14 .
-rw-r--r--.  1 root elasticsearch    76 May 12 13:14 .elasticsearch.keystore.initial_md5sum
drwxr-xr-x. 67 root root           4096 May 12 13:27 ..
[root@loganalyzer ~]# ls -altr /etc/kibana
total 28
-rw-rw----.  1 root kibana  130 Dec 11  2021 kibana.keystore
-rw-r--r--.  1 root kibana   62 Dec 11  2021 .kibana.keystore.initial_md5sum
-rw-rw----.  1 root kibana 5282 Dec 29  2021 kibana.yml
-rw-r--r--.  1 root kibana  305 Apr 22 11:23 node.options
drwxr-s---.  2 root kibana 4096 May 12 12:59 .
drwxr-xr-x. 67 root root   4096 May 12 13:27 ..
[root@loganalyzer ~]# ls -altr /etc/kibana-org/
total 28
-rw-r--r--.  1 root kibana  305 Apr 22 11:23 node.options
-rw-rw----.  1 root kibana 5243 Apr 22 11:23 kibana.yml
-rw-rw----.  1 root kibana  130 May 12 13:14 kibana.keystore
drwxr-s---.  2 root kibana 4096 May 12 13:14 .
-rw-r--r--.  1 root kibana   62 May 12 13:14 .kibana.keystore.initial_md5sum
drwxr-xr-x. 67 root root   4096 May 12 13:27 ..

Correct the users and groups for the data directories:

[root@loganalyzer ~]# ls -altr /var/lib/|grep elasticsearch
drwxr-s---.  2 elasticsearch elasticsearch 4096 Apr 23 05:38 elasticsearch-org
drwxr-s---.  3           998 sgx           4096 Apr 23 05:38 elasticsearch
[root@loganalyzer ~]# ls -altr /var/lib/|grep kibana
drwxr-s---.  2 kibana        kibana        4096 Apr 22 11:29 kibana-org
drwxr-s---. 14           997 systemd-oom   4096 Apr 22 11:29 kibana
[root@loganalyzer ~]# chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
[root@loganalyzer ~]# chown -R kibana:kibana /var/lib/kibana/

Changing the users and the groups may take some time depending on the index size.

So moving these two directories should be enough to be able to start and use the Elasticsearch and Kibana services on the new system, but it worth mentioning /usr/share/elasticsearch/modules/ and /usr/share/elasticsearch/plugins/ directories, which may contain custom user-made modules or plugins, which should be copied to the new system in the same path, too!

Now, it is time to start the Elasticsearch service:

[root@loganalyzer ~]# systemctl enable elasticsearch
Synchronizing state of elasticsearch.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /usr/lib/systemd/system/elasticsearch.service.
[root@loganalyzer ~]# systemctl start elasticsearch
[root@loganalyzer ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; preset: disabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Fri 2023-05-12 13:54:44 UTC; 11s ago
       Docs: https://www.elastic.co
   Main PID: 492 (java)
      Tasks: 71 (limit: 203034)
     Memory: 6.4G
        CPU: 46.595s
     CGroup: /system.slice/elasticsearch.service
             ├─492 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -Xms6g -Xmx6g -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-2319435107151573081 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m" -XX:MaxDirectMemorySize=3221225472 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
             └─684 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

May 12 13:54:28 kibana.u1x2.com systemd[1]: Starting Elasticsearch...
May 12 13:54:44 kibana.u1x2.com systemd[1]: Started Elasticsearch.

After couple of minutes start the Kibana service.

[root@loganalyzer ~]# systemctl enable kibana
Synchronizing state of kibana.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable kibana
Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service → /etc/systemd/system/kibana.service.
[root@loganalyzer ~]# systemctl start kibana
[root@loganalyzer ~]# systemctl status kibana
● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; preset: disabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Fri 2023-05-12 17:07:49 UTC; 10s ago
       Docs: https://www.elastic.co
   Main PID: 929 (node)
      Tasks: 7 (limit: 203034)
     Memory: 183.8M
        CPU: 1.589s
     CGroup: /system.slice/kibana.service
             └─929 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid "--deprecation.skip_deprecated_settings[0]=logging.dest"

May 12 17:07:49 loganalyzer systemd[1]: Started Kibana.

Logs

Here are the first 100 lines of the Elastisearch log after the first start. And there are no errors or problems.

[root@loganalyzer ~]# head -n 100 /var/log/elasticsearch/elasticsearch.log 
[2023-05-12T13:54:31,720][INFO ][o.e.n.Node               ] [loganalyzer] version[7.17.10], pid[492], build[default/rpm/fecd68e3150eda0c307ab9a9d7557f5d5fd71349/2023-04-23T05:33:18.138275597Z], OS[Linux/5.14.0-171.el9.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.1/20.0.1+9-29]
[2023-05-12T13:54:31,724][INFO ][o.e.n.Node               ] [loganalyzer] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-05-12T13:54:31,724][INFO ][o.e.n.Node               ] [loganalyzer] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -Xms6g, -Xmx6g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2319435107151573081, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=3221225472, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2023-05-12T13:54:33,489][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [aggs-matrix-stats]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [analysis-common]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [constant-keyword]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [frozen-indices]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [ingest-common]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [ingest-geoip]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [ingest-user-agent]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [kibana]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [lang-expression]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [lang-mustache]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [lang-painless]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [legacy-geo]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [mapper-extras]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [mapper-version]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [parent-join]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [percolator]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [rank-eval]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [reindex]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [repositories-metering-api]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [repository-encrypted]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [repository-url]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [runtime-fields-common]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [search-business-rules]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [searchable-snapshots]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [snapshot-repo-test-kit]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [spatial]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [systemd]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [transform]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [transport-netty4]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [unsigned-long]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [vector-tile]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [vectors]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [wildcard]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-aggregate-metric]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-analytics]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-async]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-async-search]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-autoscaling]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-ccr]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-core]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-data-streams]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-deprecation]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-enrich]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-eql]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-fleet]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-graph]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-identity-provider]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-ilm]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-logstash]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-ml]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-monitoring]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-ql]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-rollup]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-security]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-shutdown]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-sql]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-stack]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-text-structure]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-voting-only-node]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService     ] [loganalyzer] loaded module [x-pack-watcher]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService     ] [loganalyzer] no plugins loaded
[2023-05-12T13:54:33,532][INFO ][o.e.e.NodeEnvironment    ] [loganalyzer] using [1] data paths, mounts [[/ (/dev/md3)]], net usable_space [16tb], net total_space [26.9tb], types [ext4]
[2023-05-12T13:54:33,532][INFO ][o.e.e.NodeEnvironment    ] [loganalyzer] heap size [6gb], compressed ordinary object pointers [true]
[2023-05-12T13:54:36,413][INFO ][o.e.n.Node               ] [loganalyzer] node name [loganalyzer], node ID [OJH0IV74RNORXhoJH_2Avg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-05-12T13:54:39,643][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [loganalyzer] [controller/684] [Main.cc@122] controller (64 bit): Version 7.17.10 (Build dd124dd90ff8a4) Copyright (c) 2023 Elasticsearch BV
[2023-05-12T13:54:40,006][INFO ][o.e.x.s.a.Realms         ] [loganalyzer] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-05-12T13:54:40,012][INFO ][o.e.x.s.a.s.FileRolesStore] [loganalyzer] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-05-12T13:54:40,723][INFO ][o.e.i.g.ConfigDatabases  ] [loganalyzer] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2023-05-12T13:54:40,725][INFO ][o.e.i.g.DatabaseNodeService] [loganalyzer] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-2319435107151573081/geoip-databases/OJH0IV74RNORXhoJH_2Avg]
[2023-05-12T13:54:41,174][INFO ][o.e.t.NettyAllocator     ] [loganalyzer] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-05-12T13:54:41,203][INFO ][o.e.i.r.RecoverySettings ] [loganalyzer] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-05-12T13:54:41,234][INFO ][o.e.d.DiscoveryModule    ] [loganalyzer] using discovery type [single-node] and seed hosts providers [settings]
[2023-05-12T13:54:41,617][INFO ][o.e.g.DanglingIndicesState] [loganalyzer] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-05-12T13:54:42,139][INFO ][o.e.n.Node               ] [loganalyzer] initialized
[2023-05-12T13:54:42,140][INFO ][o.e.n.Node               ] [loganalyzer] starting ...
[2023-05-12T13:54:42,241][INFO ][o.e.x.s.c.f.PersistentCache] [loganalyzer] persistent cache index loaded
[2023-05-12T13:54:42,242][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [loganalyzer] deprecation component started
[2023-05-12T13:54:42,311][INFO ][o.e.t.TransportService   ] [loganalyzer] publish_address {192.168.0.2:9300}, bound_addresses {192.168.0.2:9300}
[2023-05-12T13:54:43,963][WARN ][o.e.b.BootstrapChecks    ] [loganalyzer] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2023-05-12T13:54:43,963][WARN ][o.e.b.BootstrapChecks    ] [loganalyzer] Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2023-05-12T13:54:43,964][INFO ][o.e.c.c.Coordinator      ] [loganalyzer] cluster UUID [nkmq-v6CTsa1Yq_ZPWPv_Q]
[2023-05-12T13:54:44,136][INFO ][o.e.c.s.MasterService    ] [loganalyzer] elected-as-master ([1] nodes joined)[{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 13, version: 83315, delta: master node changed {previous [], current [{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw}]}
[2023-05-12T13:54:44,445][INFO ][o.e.c.s.ClusterApplierService] [loganalyzer] master node changed {previous [], current [{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw}]}, term: 13, version: 83315, reason: Publication{term=13, version=83315}
[2023-05-12T13:54:44,480][INFO ][o.e.h.AbstractHttpServerTransport] [loganalyzer] publish_address {192.168.0.2:9200}, bound_addresses {192.168.0.2:9200}
[2023-05-12T13:54:44,480][INFO ][o.e.n.Node               ] [loganalyzer] started
[2023-05-12T13:54:45,387][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-anomalies-] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,389][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-state] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,390][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-notifications-000002] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,391][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-stats] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,443][INFO ][o.e.l.LicenseService     ] [loganalyzer] license [5490b670-5e01-47b8-a871-cf0601623c1a] mode [basic] - valid
[2023-05-12T13:54:45,444][INFO ][o.e.x.s.a.Realms         ] [loganalyzer] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-05-12T13:54:45,444][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [loganalyzer] Active license is now [BASIC]; Security is enabled
[2023-05-12T13:54:45,447][INFO ][o.e.g.GatewayService     ] [loganalyzer] recovered [324] indices into cluster_state
[2023-05-12T13:54:45,556][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-stats] for index patterns [.ml-stats-*]
[2023-05-12T13:54:48,105][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-anomalies-] for index patterns [.ml-anomalies-*]
[2023-05-12T13:54:53,101][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-state] for index patterns [.ml-state*]
[2023-05-12T13:54:55,070][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]

And the Kibana first log lines also shows no errors or problems.

[root@loganalyzer ~]# head -n 100 /var/log/kibana/kibana.log 
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins-service"],"pid":929,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","http","server","Preboot"],"pid":929,"message":"http server running at http://192.168.0.2:5601"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins-system","standard"],"pid":929,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins","taskManager"],"pid":929,"message":"TaskManager is identified by the Kibana UUID: c779c0a3-451a-41eb-af8c-044d9ebb839b"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","reporting","config"],"pid":929,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":929,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","actions"],"pid":929,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","alerting"],"pid":929,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["warning","plugins","reporting","config"],"pid":929,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 9\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 11ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 42ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 33ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 6ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 78ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 164ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:38+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 3395ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:38+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 85ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:39+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 4325ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:39+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:42+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 4328ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:42+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 16ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Starting to process 18 documents."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 95ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 5ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 4267ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 5ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Starting to process 367 documents."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 104ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 35ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 3206ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Processed 18 documents out of 18."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 25ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:50+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 3988ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 5269ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:57+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 1841ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:57+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 423ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:59+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 1540ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:59+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Migration completed after 24368ms"}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 18045ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Processed 367 documents out of 367."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 4ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:03+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 1517ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 3538ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 36ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 2ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:08+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 921ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:08+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 209ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 1001ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Migration completed after 34242ms"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","plugins-system","standard"],"pid":929,"message":"Starting [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":929,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2023-05-12T17:09:10+00:00","tags":["info","http","server","Kibana"],"pid":929,"message":"http server running at http://192.168.0.2:5601"}
{"type":"log","@timestamp":"2023-05-12T17:09:10+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now degraded (was unavailable)"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":929,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","securitySolution"],"pid":929,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","reporting","chromium"],"pid":929,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["warning","plugins","reporting","chromium"],"pid":929,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","securitySolution","endpoint:metadata-check-transforms-task:0","0","1"],"pid":929,"message":"no endpoint metadata transforms found"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed common resources shared between all indices"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.uptime.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.logs.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.metrics.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.apm.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:13+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now available (was degraded)"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.uptime.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.logs.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.metrics.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.apm.alerts"}

Leave a Reply

Your email address will not be published. Required fields are marked *