Installing Apache ZooKeeper on Solr cluster instances

Install and configure Apache ZooKeeper on all Solr cluster instances.

1. Connect to the target server using your preferred ssh tool (e.g., PuTTY) and log in.
2. Become the root user:

   sudo su -

3. Install Java 11 (e.g., the OpenJDK 11 JRE):
   For Ubuntu:

   apt install software-properties-common
   add-apt-repository ppa:openjdk-r/ppa
   apt update
   apt install openjdk-11-jre-headless

   For RHEL:

   dnf install java-11-openjdk-headless.x86_64

4. Create a new user named zookeeper as follows:
   1. Create the group zookeeper for the user:

      groupadd zookeeper

   2. Create the user zookeeper:
      For Ubuntu:

      adduser --home /var/lib/zookeeper --ingroup zookeeper zookeeper

      For RHEL:

      adduser zookeeper -d /var/lib/zookeeper -g zookeeper
      mkdir /var/lib/zookeeper
      chown zookeeper:zookeeper /var/lib/zookeeper/

5. Download and extract the ZooKeeper archive:

   cd /opt
   wget "https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz"
   tar xzf apache-zookeeper-3.8.4-bin.tar.gz
   chown -R zookeeper:zookeeper apache-zookeeper-3.8.4-bin
   ln -s apache-zookeeper-3.8.4-bin zookeeper

6. Switch the user to zookeeper:
   For Ubuntu:

   sudo su - zookeeper

   For RHEL:

   su -s /bin/bash zookeeper

7. Create a copy of the sample configuration file zoo_sample.cfg:

   cd /opt/zookeeper/conf
   cp zoo_sample.cfg zoo.cfg

8. Edit the copied configuration file zoo.cfg:

   nano zoo.cfg (or vi zoo.cfg)

9. Modify the configuration in the file zoo.cfg as follows:
   1. Update the data directory variable:

      dataDir=/var/lib/zookeeper/data

   2. Add all ZooKeeper server addresses at the end of the file:
      Important: Use IPv4 IPs only since IPv6 isn't supported yet!

      server.1 = <first  server IP>:2888:3888:participant;0.0.0.0:2181
      server.2 = <second  server IP>:2888:3888:participant;0.0.0.0:2181
      server.3 = <third  server IP>:2888:3888:participant;0.0.0.0:2181
      
      e.g.:
      
      server.1 = 192.168.102.21:2888:3888:participant;0.0.0.0:2181
      server.2 = 192.168.102.22:2888:3888:participant;0.0.0.0:2181
      server.3 = 192.168.102.23:2888:3888:participant;0.0.0.0:2181

   3. Comment out the clientPort directive

      # clientPort=2181

   4. Add the following lines to the end of the file:

      standaloneEnabled=false
      reconfigEnabled=false
      4lw.commands.whitelist=mntr,conf,ruok

   5. Save your changes and exit the editor.
10. Create the file java.env in the directory /opt/zookeeper/conf:

    nano /opt/zookeeper/conf/java.env

11. Insert the following settings into the file java.env:

    ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
    ZOO_LOG_DIR="/var/lib/zookeeper/logs"

12. Configure the memory available to the Java JVM according to the memory available on your machine by inserting the following settings into the file java.env:
    1. Maximum amount of memory that can be allocated to the JVM heap (MB):

       ZK_SERVER_HEAP="4096"

    2. Initial amount of memory allocated to the JVM heap:

       export JVMFLAGS="-Xms2048m"

13. Save your changes and exit the editor.
14. Create the ZooKeeper data and log directories:

    mkdir /var/lib/zookeeper/data /var/lib/zookeeper/logs

15. Set the instance ID variable. For the first server set this to "1", to "2" for the 2nd, etc.:

    echo "1" > /var/lib/zookeeper/data/myid

16. Press Ctrl+D or enter exit to go back to the root user.
17. Register ZooKeeper as a systemd service so that it will be started on boot:
    1. Create a service file:
       For Ubuntu:

       nano /etc/systemd/system/zookeeper.service

       For RHEL:

       nano /usr/lib/systemd/system/zookeeper.service

    2. Add the following lines:

       [Unit]
       Description=Zookeeper Daemon
       Wants=syslog.target
       
       [Service]
       Type=forking
       WorkingDirectory=/var/lib/zookeeper
       User=zookeeper
       ExecStart=/opt/zookeeper/bin/zkServer.sh start
       ExecStop=/opt/zookeeper/bin/zkServer.sh stop
       TimeoutSec=30
       Restart=on-failure
       
       [Install]
       WantedBy=multi-user.target

    3. Set up the service:

       systemctl start zookeeper
       systemctl enable zookeeper

       For RHEL:

       systemctl daemon-reload
       systemctl enable zookeeper.service
       systemctl start zookeeper.service