Despite having only two additional installed software under CentOS 7 it is not a good idea to just try upgrading only CentOS 7 to CentOS Stream 9. There is no clear and supported path for upgrading from CentOS 7 to CentOS Stream 9 and even to the older one CentOS 8 (or CentOS Stream 8). The best way is to just make a clean install of CentOS Stream 9 and copy all the Elasticsearch and Kibana files and this article is how to do it without problems.
Here is the plan to move the existing installation of Elasticsearch and Kibana services from CentOS 7 to CentOS Stream 9:
- Make a clean install of CentOS Stream 9
- Update the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).
- Add Elasticsearch and Kibana repositories to the new system. Tune the system crypto policies.
- Install Elasticsearch and Kibana software packages, but do not start the services.
- Copy Elasticsearch and Kibana important files such as the index directory and the configuration directories. Check the user and group IDs of the files.
- Start the Elasticsearch and Kibana services.
In this example, the installation of the new server is just starting a new LXC container, which will host the Elasticsearch and Kibana services. There is no difference between using a container or a physical machine. With LXC container it is easier to copy the needed files such as the Elasticsearch index files, which may be tens of terabytes or more, and various configuration files.
STEP 1) Make a clean install of CentOS Stream 9
Check out the following article on the purpose – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation or if LXC container is preferred – Run LXC CentOS Stream 9 container with bridged network under CentOS Stream 9.
Creating a LXC container of CentOS Stream 9 is really simple and fast:
[root@srv ~]# lxc-create --template download -n kibana.u1x2.com -- --dist centos --release 9-Stream --arch amd64 The cached copy has expired, re-downloading... Downloading the image index Downloading the rootfs Downloading the metadata The image cache is now ready Unpacking the rootfs --- You just created a Centos 9-Stream x86_64 (20230511_19:27) container.
Then tune the network following the above article. It is a good idea when configuring the network to preserve the original UUIDs and network addresses (MAC address, too) of the LXC containers network and the inner container’s interface.
So copy the UUID from /var/lib/lxc/loganalyzer-old/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0 to the CentOS Stream 9 network configuration – /var/lib/lxc/loganalyzer/rootfs/etc/NetworkManager/system-connections/ethernet-eth0.nmconnection, which uses NetworkManager. And the LXC container’s MAC address: the variable lxc.net.0.hwaddr from /var/lib/lxc/loganalyzer/config to /var/lib/lxc/loganalyzer/config.
The last step is to run the newly installed system. No errors in the output signals for a successful start-up of the LXC container with the name loganalyzer.
[root@srv ~]# lxc-start -n loganalyzer [root@srv ~]#
STEP 2) Upgrade the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).
For example, if the current Elasticsearch is version 7. It is good to upgrade it to the latest version from 7.x before proceeding with the next steps.
The current installed versions of Elasticsearch and Kibana software are from the branch 7 – 7.17.4-1 and the latest version is 7.17.10-1.
Check in the old system with (CentOS 7):
[root@loganalyzer-old ~]# yum list installed|egrep -e "(elasticsearch|kibana)" elasticsearch.x86_64 7.17.4-1 @elasticsearch kibana.x86_64 7.17.4-1 @elasticsearch
If they are the latest version from their branch/minor version proceed with the other steps. If they aren’t the latest version it is a good idea to update them first and then copy them to the new system. The important thing is to install the same version in the new system or at least, the same major version (for example, if the old one is 7.17.4-1, install it in the new system the 7.17.X version. The best case scenario is under the old and new systems the Elastisearch and Kibana to be the same versions)!
The upgrade of Elasticsearch and Kibana should be easy and straightforward:
[root@loganalyzer-old ~]# yum update elasticsearch kibana Last metadata expiration check: 0:00:38 ago on Fri May 12 12:58:13 2023. Dependencies resolved. ================================================================================================================ Package Architecture Version Repository Size ================================================================================================================ Upgrading: elasticsearch x86_64 7.17.10-1 elasticsearch 302 M kibana x86_64 7.17.10-1 elasticsearch 261 M Transaction Summary ================================================================================================================ Upgrade 2 Packages Total download size: 563 M Is this ok [y/N]: y Downloading Packages: (1/2): elasticsearch-7.17.10-x86_64.rpm 42 MB/s | 302 MB 00:07 (2/2): kibana-7.17.10-x86_64.rpm 27 MB/s | 261 MB 00:09 ---------------------------------------------------------------------------------------------------------------- Total 59 MB/s | 563 MB 00:09 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: kibana-7.17.10-1.x86_64 1/1 Running scriptlet: kibana-7.17.10-1.x86_64 1/4 Upgrading : kibana-7.17.10-1.x86_64 1/4 Running scriptlet: kibana-7.17.10-1.x86_64 1/4 Running scriptlet: elasticsearch-7.17.10-1.x86_64 2/4 Upgrading : elasticsearch-7.17.10-1.x86_64 2/4 Running scriptlet: elasticsearch-7.17.10-1.x86_64 2/4 Running scriptlet: kibana-7.17.4-1.x86_64 3/4 Stopping kibana service...OK Cleanup : kibana-7.17.4-1.x86_64 3/4 Running scriptlet: kibana-7.17.4-1.x86_64 3/4 Running scriptlet: elasticsearch-7.17.4-1.x86_64 4/4 Cleanup : elasticsearch-7.17.4-1.x86_64 4/4 Running scriptlet: elasticsearch-7.17.4-1.x86_64 4/4 Running scriptlet: kibana-7.17.10-1.x86_64 4/4 Running scriptlet: elasticsearch-7.17.10-1.x86_64 4/4 Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore Running scriptlet: elasticsearch-7.17.4-1.x86_64 4/4 Verifying : elasticsearch-7.17.10-1.x86_64 1/4 Verifying : elasticsearch-7.17.4-1.x86_64 2/4 Verifying : kibana-7.17.10-1.x86_64 3/4 Verifying : kibana-7.17.4-1.x86_64 4/4 Upgraded: elasticsearch-7.17.10-1.x86_64 kibana-7.17.10-1.x86_64 Complete! .... [root@loganalyzer-old ~]# systemctl restart elasticsearch [root@loganalyzer-old ~]# systemctl restart kibana
After a successful minor update proceed with the next steps. Bear in mind, upgrading to the latest current branch version is not mandatory! It should work with minor differences in the versions or just install the very same software versions to the new system.
STEP 3) Add Elasticsearch and Kibana repositories to the new system.
Login to the new server and add the Elasticsearch repository. Create a file /etc/yum.repos.d/elasticsearch.repo with the following content:
[elasticsearch] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
And for the Kibana software – /etc/yum.repos.d/kibana.repo
[kibana-7.x] name=Kibana repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
List the available packages with their versions:
[root@loganalyzer ~]# dnf list elasticsearch kibana Elasticsearch repository for 7.x packages 43 MB/s | 50 MB 00:01 Kibana repository for 7.x packages 46 MB/s | 50 MB 00:01 Last metadata expiration check: 0:00:09 ago on Fri May 12 12:54:29 2023. Available Packages elasticsearch.aarch64 7.17.10-1 elasticsearch elasticsearch.aarch64 7.17.10-1 kibana-7.x elasticsearch.x86_64 7.17.10-1 elasticsearch elasticsearch.x86_64 7.17.10-1 kibana-7.x kibana.aarch64 7.17.10-1 elasticsearch kibana.aarch64 7.17.10-1 kibana-7.x kibana.x86_64 7.17.10-1 elasticsearch kibana.x86_64 7.17.10-1 kibana-7.x
Because these repositories use the old algorithm SHA1 for the package signature, the crypto policy should be tuned or a message of the following may occur (under CentOS Stream 9):
Importing GPG key 0xD88E42B4: Userid : "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>" Fingerprint: 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 From : https://artifacts.elastic.co/GPG-KEY-elasticsearch Is this ok [y/N]: y warning: Signature not supported. Hash algorithm SHA1 not available. Key import failed (code 2). Failing package is: elasticsearch-7.17.10-1.x86_64 GPG Keys are configured as: https://artifacts.elastic.co/GPG-KEY-elasticsearch Public key for kibana-7.17.10-x86_64.rpm is not installed. Failing package is: kibana-7.17.10-1.x86_64 GPG Keys are configured as: https://artifacts.elastic.co/GPG-KEY-elasticsearch The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: GPG check FAILED
To solve this error and to be able to install these packages the update-crypto-policies tool is needed to update the system policies, which is part of crypto-policies-scripts.
[root@srv ~]# dnf install -y crypto-policies-scripts Last metadata expiration check: 0:14:40 ago on Fri May 12 12:56:12 2023. Dependencies resolved. ================================================================================================================ Package Architecture Version Repository Size ================================================================================================================ Installing: crypto-policies-scripts noarch 20221215-1.git9a18988.el9 baseos 93 k Transaction Summary ================================================================================================================ Install 1 Package Total download size: 93 k Installed size: 225 k Downloading Packages: crypto-policies-scripts-20221215-1.git9a18988.el9.noarch.rpm 286 kB/s | 93 kB 00:00 ---------------------------------------------------------------------------------------------------------------- Total 88 kB/s | 93 kB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : crypto-policies-scripts-20221215-1.git9a18988.el9.noarch 1/1 Running scriptlet: crypto-policies-scripts-20221215-1.git9a18988.el9.noarch 1/1 Verifying : crypto-policies-scripts-20221215-1.git9a18988.el9.noarch 1/1 Installed: crypto-policies-scripts-20221215-1.git9a18988.el9.noarch Complete! [root@srv ~]# update-crypto-policies --set DEFAULT:SHA1 Setting system policy to DEFAULT:SHA1 Note: System-wide crypto policies are applied on application start-up. It is recommended to restart the system for the change of policies to fully take place.
First, install and then update the crypto policy as shown above. Of course, after installing these packages the policy might be reverted to the defaults.
STEP 4) Install Elasticsearch and Kibana software packages, but do not start the services.
[root@loganalyzer ~]# dnf install elasticsearch kibana Last metadata expiration check: 0:17:38 ago on Fri May 12 12:56:12 2023. Dependencies resolved. ================================================================================================================ Package Architecture Version Repository Size ================================================================================================================ Installing: elasticsearch x86_64 7.17.10-1 elasticsearch 302 M kibana x86_64 7.17.10-1 elasticsearch 261 M Transaction Summary ================================================================================================================ Install 2 Packages Total download size: 563 M Installed size: 1.1 G Is this ok [y/N]: y Downloading Packages: (1/2): kibana-7.17.10-x86_64.rpm 41 MB/s | 261 MB 00:06 (2/2): elasticsearch-7.17.10-x86_64.rpm 34 MB/s | 302 MB 00:08 ---------------------------------------------------------------------------------------------------------------- Total 63 MB/s | 563 MB 00:08 Elasticsearch repository for 7.x packages 29 kB/s | 1.7 kB 00:00 Importing GPG key 0xD88E42B4: Userid : "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>" Fingerprint: 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 From : https://artifacts.elastic.co/GPG-KEY-elasticsearch Is this ok [y/N]: y Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: kibana-7.17.10-1.x86_64 1/2 Installing : kibana-7.17.10-1.x86_64 1/2 Running scriptlet: kibana-7.17.10-1.x86_64 1/2 Creating kibana group... OK Creating kibana user... OK Running scriptlet: elasticsearch-7.17.10-1.x86_64 2/2 Creating elasticsearch group... OK Creating elasticsearch user... OK Installing : elasticsearch-7.17.10-1.x86_64 2/2 Running scriptlet: elasticsearch-7.17.10-1.x86_64 2/2 ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Running scriptlet: kibana-7.17.10-1.x86_64 2/2 Created Kibana keystore in /etc/kibana/kibana.keystore Running scriptlet: elasticsearch-7.17.10-1.x86_64 2/2 Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore /usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly. Verifying : elasticsearch-7.17.10-1.x86_64 1/2 Verifying : kibana-7.17.10-1.x86_64 2/2 Installed: elasticsearch-7.17.10-1.x86_64 kibana-7.17.10-1.x86_64 Complete!
STEP 6) Move the files and configuration to the new system
First, rename the original directories. Note, the users and the groups – elasticsearch:elasticsearch and kibana:kibana and the fact, the data directories of Elasticsearch and Kibana are empty.
[root@loganalyzer ~]# mv /etc/elasticsearch /etc/elasticsearch-org [root@loganalyzer ~]# mv /etc/kibana /etc/kibana-org [root@loganalyzer ~]# ls -altr /var/lib/kibana/ total 8 drwxr-s---. 2 kibana kibana 4096 Apr 22 11:29 . drwxr-xr-x. 18 root root 4096 May 12 13:14 .. [root@loganalyzer ~]# ls -altr /var/lib/elasticsearch/ total 8 drwxr-s---. 2 elasticsearch elasticsearch 4096 Apr 23 05:38 . drwxr-xr-x. 18 root root 4096 May 12 13:14 .. [root@loganalyzer ~]# ls -altr /var/lib/kibana/ total 8 drwxr-s---. 2 kibana kibana 4096 Apr 22 11:29 . drwxr-xr-x. 18 root root 4096 May 12 13:36 .. [root@loganalyzer ~]# mv /var/lib/elasticsearch /var/lib/elasticsearch-org [root@loganalyzer ~]# mv /var/lib/kibana /var/lib/kibana-org
Second, copy the configurations and data directories from the old system to the new one. Using LXC containers it is a matter of just moving directories locally. On the LXC host move the configuration directories and note the users and groups are different, so chwon the directories accordingly to have elasticsearch and kibana users and groups.
[root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/etc/elasticsearch /var/lib/lxc/loganalyzer/rootfs/etc/ [root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/etc/kibana /var/lib/lxc/loganalyzer/rootfs/etc/ [root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/var/lib/elasticsearch /var/lib/lxc/loganalyzer/rootfs/var/lib/ [root@srv ~]# mv /var/lib/lxc/loganalyzer-old/rootfs/var/lib/kibana /var/lib/lxc/loganalyzer/rootfs/var/lib/
Users and groups are with different IDs, so they should be corrected.
[root@loganalyzer ~]# ls -altr /etc/|grep elastic drwxr-s---. 3 root sgx 4096 May 12 13:00 elasticsearch drwxr-s---. 3 root elasticsearch 4096 May 12 13:14 elasticsearch-org [root@loganalyzer ~]# ls -altr /etc/|grep kibana drwxr-s---. 2 root systemd-oom 4096 May 12 12:59 kibana drwxr-s---. 2 root kibana 4096 May 12 13:14 kibana-org [root@loganalyzer ~]# chown -R root:elasticsearch /etc/elasticsearch [root@loganalyzer ~]# chown -R root:kibana /etc/kibana [root@loganalyzer ~]# ls -altr /etc/elasticsearch total 64 -rw-rw----. 1 root elasticsearch 3523 Dec 11 2021 elasticsearch.yml -rw-r--r--. 1 root elasticsearch 596 Dec 11 2021 passwords -rw-rw----. 1 root elasticsearch 3323 Jun 14 2022 jvm.options -rw-rw----. 1 root elasticsearch 0 Apr 23 05:35 users_roles -rw-rw----. 1 root elasticsearch 0 Apr 23 05:35 users -rw-rw----. 1 root elasticsearch 197 Apr 23 05:35 roles.yml -rw-rw----. 1 root elasticsearch 473 Apr 23 05:35 role_mapping.yml -rw-rw----. 1 root elasticsearch 19304 Apr 23 05:35 log4j2.properties -rw-rw----. 1 root elasticsearch 1042 Apr 23 05:35 elasticsearch-plugins.example.yml drwxr-s---. 2 root elasticsearch 4096 Apr 23 05:38 jvm.options.d -rw-rw----. 1 root elasticsearch 199 May 12 13:00 elasticsearch.keystore -rw-r--r--. 1 root elasticsearch 76 May 12 13:00 .elasticsearch.keystore.initial_md5sum drwxr-s---. 3 root elasticsearch 4096 May 12 13:00 . drwxr-xr-x. 67 root root 4096 May 12 13:27 .. [root@loganalyzer ~]# ls -altr /etc/elasticsearch-org/ total 60 -rw-rw----. 1 root elasticsearch 0 Apr 23 05:35 users_roles -rw-rw----. 1 root elasticsearch 0 Apr 23 05:35 users -rw-rw----. 1 root elasticsearch 197 Apr 23 05:35 roles.yml -rw-rw----. 1 root elasticsearch 473 Apr 23 05:35 role_mapping.yml -rw-rw----. 1 root elasticsearch 19304 Apr 23 05:35 log4j2.properties -rw-rw----. 1 root elasticsearch 3329 Apr 23 05:35 jvm.options -rw-rw----. 1 root elasticsearch 3431 Apr 23 05:35 elasticsearch.yml -rw-rw----. 1 root elasticsearch 1042 Apr 23 05:35 elasticsearch-plugins.example.yml drwxr-s---. 2 root elasticsearch 4096 Apr 23 05:38 jvm.options.d -rw-rw----. 1 root elasticsearch 199 May 12 13:14 elasticsearch.keystore drwxr-s---. 3 root elasticsearch 4096 May 12 13:14 . -rw-r--r--. 1 root elasticsearch 76 May 12 13:14 .elasticsearch.keystore.initial_md5sum drwxr-xr-x. 67 root root 4096 May 12 13:27 .. [root@loganalyzer ~]# ls -altr /etc/kibana total 28 -rw-rw----. 1 root kibana 130 Dec 11 2021 kibana.keystore -rw-r--r--. 1 root kibana 62 Dec 11 2021 .kibana.keystore.initial_md5sum -rw-rw----. 1 root kibana 5282 Dec 29 2021 kibana.yml -rw-r--r--. 1 root kibana 305 Apr 22 11:23 node.options drwxr-s---. 2 root kibana 4096 May 12 12:59 . drwxr-xr-x. 67 root root 4096 May 12 13:27 .. [root@loganalyzer ~]# ls -altr /etc/kibana-org/ total 28 -rw-r--r--. 1 root kibana 305 Apr 22 11:23 node.options -rw-rw----. 1 root kibana 5243 Apr 22 11:23 kibana.yml -rw-rw----. 1 root kibana 130 May 12 13:14 kibana.keystore drwxr-s---. 2 root kibana 4096 May 12 13:14 . -rw-r--r--. 1 root kibana 62 May 12 13:14 .kibana.keystore.initial_md5sum drwxr-xr-x. 67 root root 4096 May 12 13:27 ..
Correct the users and groups for the data directories:
[root@loganalyzer ~]# ls -altr /var/lib/|grep elasticsearch drwxr-s---. 2 elasticsearch elasticsearch 4096 Apr 23 05:38 elasticsearch-org drwxr-s---. 3 998 sgx 4096 Apr 23 05:38 elasticsearch [root@loganalyzer ~]# ls -altr /var/lib/|grep kibana drwxr-s---. 2 kibana kibana 4096 Apr 22 11:29 kibana-org drwxr-s---. 14 997 systemd-oom 4096 Apr 22 11:29 kibana [root@loganalyzer ~]# chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/ [root@loganalyzer ~]# chown -R kibana:kibana /var/lib/kibana/
Changing the users and the groups may take some time depending on the index size.
So moving these two directories should be enough to be able to start and use the Elasticsearch and Kibana services on the new system, but it worth mentioning /usr/share/elasticsearch/modules/ and /usr/share/elasticsearch/plugins/ directories, which may contain custom user-made modules or plugins, which should be copied to the new system in the same path, too!
Now, it is time to start the Elasticsearch service:
[root@loganalyzer ~]# systemctl enable elasticsearch
Synchronizing state of elasticsearch.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /usr/lib/systemd/system/elasticsearch.service.
[root@loganalyzer ~]# systemctl start elasticsearch
[root@loganalyzer ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; preset: disabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-05-12 13:54:44 UTC; 11s ago
Docs: https://www.elastic.co
Main PID: 492 (java)
Tasks: 71 (limit: 203034)
Memory: 6.4G
CPU: 46.595s
CGroup: /system.slice/elasticsearch.service
├─492 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -Xms6g -Xmx6g -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-2319435107151573081 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m" -XX:MaxDirectMemorySize=3221225472 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
└─684 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
May 12 13:54:28 kibana.u1x2.com systemd[1]: Starting Elasticsearch...
May 12 13:54:44 kibana.u1x2.com systemd[1]: Started Elasticsearch.
After couple of minutes start the Kibana service.
[root@loganalyzer ~]# systemctl enable kibana
Synchronizing state of kibana.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable kibana
Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service → /etc/systemd/system/kibana.service.
[root@loganalyzer ~]# systemctl start kibana
[root@loganalyzer ~]# systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; preset: disabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-05-12 17:07:49 UTC; 10s ago
Docs: https://www.elastic.co
Main PID: 929 (node)
Tasks: 7 (limit: 203034)
Memory: 183.8M
CPU: 1.589s
CGroup: /system.slice/kibana.service
└─929 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid "--deprecation.skip_deprecated_settings[0]=logging.dest"
May 12 17:07:49 loganalyzer systemd[1]: Started Kibana.
Logs
Here are the first 100 lines of the Elastisearch log after the first start. And there are no errors or problems.
[root@loganalyzer ~]# head -n 100 /var/log/elasticsearch/elasticsearch.log
[2023-05-12T13:54:31,720][INFO ][o.e.n.Node ] [loganalyzer] version[7.17.10], pid[492], build[default/rpm/fecd68e3150eda0c307ab9a9d7557f5d5fd71349/2023-04-23T05:33:18.138275597Z], OS[Linux/5.14.0-171.el9.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.1/20.0.1+9-29]
[2023-05-12T13:54:31,724][INFO ][o.e.n.Node ] [loganalyzer] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-05-12T13:54:31,724][INFO ][o.e.n.Node ] [loganalyzer] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -Xms6g, -Xmx6g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2319435107151573081, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=3221225472, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2023-05-12T13:54:33,489][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [aggs-matrix-stats]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [analysis-common]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [constant-keyword]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [frozen-indices]
[2023-05-12T13:54:33,490][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [ingest-common]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [ingest-geoip]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [ingest-user-agent]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [kibana]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [lang-expression]
[2023-05-12T13:54:33,491][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [lang-mustache]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [lang-painless]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [legacy-geo]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [mapper-extras]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [mapper-version]
[2023-05-12T13:54:33,492][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [parent-join]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [percolator]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [rank-eval]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [reindex]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [repositories-metering-api]
[2023-05-12T13:54:33,493][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [repository-encrypted]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [repository-url]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [runtime-fields-common]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [search-business-rules]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [searchable-snapshots]
[2023-05-12T13:54:33,494][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [snapshot-repo-test-kit]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [spatial]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [systemd]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [transform]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [transport-netty4]
[2023-05-12T13:54:33,495][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [unsigned-long]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [vector-tile]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [vectors]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [wildcard]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-aggregate-metric]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-analytics]
[2023-05-12T13:54:33,496][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-async]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-async-search]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-autoscaling]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-ccr]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-core]
[2023-05-12T13:54:33,497][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-data-streams]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-deprecation]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-enrich]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-eql]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-fleet]
[2023-05-12T13:54:33,498][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-graph]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-identity-provider]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-ilm]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-logstash]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-ml]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-monitoring]
[2023-05-12T13:54:33,499][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-ql]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-rollup]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-security]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-shutdown]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-sql]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-stack]
[2023-05-12T13:54:33,500][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-text-structure]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-voting-only-node]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService ] [loganalyzer] loaded module [x-pack-watcher]
[2023-05-12T13:54:33,501][INFO ][o.e.p.PluginsService ] [loganalyzer] no plugins loaded
[2023-05-12T13:54:33,532][INFO ][o.e.e.NodeEnvironment ] [loganalyzer] using [1] data paths, mounts [[/ (/dev/md3)]], net usable_space [16tb], net total_space [26.9tb], types [ext4]
[2023-05-12T13:54:33,532][INFO ][o.e.e.NodeEnvironment ] [loganalyzer] heap size [6gb], compressed ordinary object pointers [true]
[2023-05-12T13:54:36,413][INFO ][o.e.n.Node ] [loganalyzer] node name [loganalyzer], node ID [OJH0IV74RNORXhoJH_2Avg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-05-12T13:54:39,643][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [loganalyzer] [controller/684] [Main.cc@122] controller (64 bit): Version 7.17.10 (Build dd124dd90ff8a4) Copyright (c) 2023 Elasticsearch BV
[2023-05-12T13:54:40,006][INFO ][o.e.x.s.a.Realms ] [loganalyzer] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-05-12T13:54:40,012][INFO ][o.e.x.s.a.s.FileRolesStore] [loganalyzer] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-05-12T13:54:40,723][INFO ][o.e.i.g.ConfigDatabases ] [loganalyzer] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2023-05-12T13:54:40,725][INFO ][o.e.i.g.DatabaseNodeService] [loganalyzer] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-2319435107151573081/geoip-databases/OJH0IV74RNORXhoJH_2Avg]
[2023-05-12T13:54:41,174][INFO ][o.e.t.NettyAllocator ] [loganalyzer] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-05-12T13:54:41,203][INFO ][o.e.i.r.RecoverySettings ] [loganalyzer] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-05-12T13:54:41,234][INFO ][o.e.d.DiscoveryModule ] [loganalyzer] using discovery type [single-node] and seed hosts providers [settings]
[2023-05-12T13:54:41,617][INFO ][o.e.g.DanglingIndicesState] [loganalyzer] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-05-12T13:54:42,139][INFO ][o.e.n.Node ] [loganalyzer] initialized
[2023-05-12T13:54:42,140][INFO ][o.e.n.Node ] [loganalyzer] starting ...
[2023-05-12T13:54:42,241][INFO ][o.e.x.s.c.f.PersistentCache] [loganalyzer] persistent cache index loaded
[2023-05-12T13:54:42,242][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [loganalyzer] deprecation component started
[2023-05-12T13:54:42,311][INFO ][o.e.t.TransportService ] [loganalyzer] publish_address {192.168.0.2:9300}, bound_addresses {192.168.0.2:9300}
[2023-05-12T13:54:43,963][WARN ][o.e.b.BootstrapChecks ] [loganalyzer] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2023-05-12T13:54:43,963][WARN ][o.e.b.BootstrapChecks ] [loganalyzer] Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2023-05-12T13:54:43,964][INFO ][o.e.c.c.Coordinator ] [loganalyzer] cluster UUID [nkmq-v6CTsa1Yq_ZPWPv_Q]
[2023-05-12T13:54:44,136][INFO ][o.e.c.s.MasterService ] [loganalyzer] elected-as-master ([1] nodes joined)[{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 13, version: 83315, delta: master node changed {previous [], current [{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw}]}
[2023-05-12T13:54:44,445][INFO ][o.e.c.s.ClusterApplierService] [loganalyzer] master node changed {previous [], current [{loganalyzer}{OJH0IV74RNORXhoJH_2Avg}{2niecUCPRf6bpqL1KxjhsQ}{192.168.0.2}{192.168.0.2:9300}{cdfhilmrstw}]}, term: 13, version: 83315, reason: Publication{term=13, version=83315}
[2023-05-12T13:54:44,480][INFO ][o.e.h.AbstractHttpServerTransport] [loganalyzer] publish_address {192.168.0.2:9200}, bound_addresses {192.168.0.2:9200}
[2023-05-12T13:54:44,480][INFO ][o.e.n.Node ] [loganalyzer] started
[2023-05-12T13:54:45,387][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-anomalies-] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,389][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-state] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,390][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-notifications-000002] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,391][INFO ][o.e.x.c.t.IndexTemplateRegistry] [loganalyzer] upgrading composable template [.ml-stats] for [ml] from version [7170499] to version [7171099]
[2023-05-12T13:54:45,443][INFO ][o.e.l.LicenseService ] [loganalyzer] license [5490b670-5e01-47b8-a871-cf0601623c1a] mode [basic] - valid
[2023-05-12T13:54:45,444][INFO ][o.e.x.s.a.Realms ] [loganalyzer] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-05-12T13:54:45,444][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [loganalyzer] Active license is now [BASIC]; Security is enabled
[2023-05-12T13:54:45,447][INFO ][o.e.g.GatewayService ] [loganalyzer] recovered [324] indices into cluster_state
[2023-05-12T13:54:45,556][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-stats] for index patterns [.ml-stats-*]
[2023-05-12T13:54:48,105][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-anomalies-] for index patterns [.ml-anomalies-*]
[2023-05-12T13:54:53,101][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-state] for index patterns [.ml-state*]
[2023-05-12T13:54:55,070][INFO ][o.e.c.m.MetadataIndexTemplateService] [loganalyzer] updating index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]
And the Kibana first log lines also shows no errors or problems.
[root@loganalyzer ~]# head -n 100 /var/log/kibana/kibana.log
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins-service"],"pid":929,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","http","server","Preboot"],"pid":929,"message":"http server running at http://192.168.0.2:5601"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","config","deprecation"],"pid":929,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins-system","standard"],"pid":929,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins","taskManager"],"pid":929,"message":"TaskManager is identified by the Kibana UUID: c779c0a3-451a-41eb-af8c-044d9ebb839b"}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","security","config"],"pid":929,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","reporting","config"],"pid":929,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":929,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","actions"],"pid":929,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["warning","plugins","alerting"],"pid":929,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2023-05-12T17:08:33+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["warning","plugins","reporting","config"],"pid":929,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 9\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2023-05-12T17:08:34+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 11ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 42ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 33ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 6ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 78ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:35+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 164ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:38+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 3395ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:38+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 85ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:39+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 4325ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:39+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:42+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 4328ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:42+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 16ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Starting to process 18 documents."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 95ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 5ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 4267ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 5ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Starting to process 367 documents."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 104ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:43+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 35ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 3206ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Processed 18 documents out of 18."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 25ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:46+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:50+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 3988ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 5269ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:55+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:57+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 1841ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:57+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 423ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:59+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 1540ms."}
{"type":"log","@timestamp":"2023-05-12T17:08:59+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana_task_manager] Migration completed after 24368ms"}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 18045ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Processed 367 documents out of 367."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 7ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:01+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 4ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:03+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 1517ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 3538ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 36ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:07+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 2ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:08+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 921ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:08+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 209ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 1001ms."}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","savedobjects-service"],"pid":929,"message":"[.kibana] Migration completed after 34242ms"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","plugins-system","standard"],"pid":929,"message":"Starting [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
{"type":"log","@timestamp":"2023-05-12T17:09:09+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":929,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2023-05-12T17:09:10+00:00","tags":["info","http","server","Kibana"],"pid":929,"message":"http server running at http://192.168.0.2:5601"}
{"type":"log","@timestamp":"2023-05-12T17:09:10+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now degraded (was unavailable)"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":929,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","securitySolution"],"pid":929,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["info","plugins","reporting","chromium"],"pid":929,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2023-05-12T17:09:11+00:00","tags":["warning","plugins","reporting","chromium"],"pid":929,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","securitySolution","endpoint:metadata-check-transforms-task:0","0","1"],"pid":929,"message":"no endpoint metadata transforms found"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed common resources shared between all indices"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.uptime.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.logs.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.metrics.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:12+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installing resources for index .alerts-observability.apm.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:13+00:00","tags":["info","status"],"pid":929,"message":"Kibana is now available (was degraded)"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.uptime.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.logs.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.metrics.alerts"}
{"type":"log","@timestamp":"2023-05-12T17:09:14+00:00","tags":["info","plugins","ruleRegistry"],"pid":929,"message":"Installed resources for index .alerts-observability.apm.alerts"}