DefinIT

Creating a vRealize Log Insight 2.5 cluster with Integrated Load Balancing

Written by Sam McGeown
Published on 11/2/2015 - Read in about 12 min (2372 words)
Published under VMware #availability #cluster #insight #integrated load balancer #load balancer #log #Security #syslog #vRealize Log Insight #vRealize Log Insight

vRealize Log InsightvRealize Log Insight 2.5 improves on the clustering in previous versions with an Integrated Load Balancer (ILB) which allows you to distribute load across your cluster of Log Insight instances without actually needing an external load balancer. The advantage of this over an external load balancer is that the source IP is maintained which allows for easier analysis.

The minimum number of nodes in a cluster is three, the first node becomes the Master node and the other two become Worker nodes. The maximum number of nodes supported is six, though acording to Mr Log Insight himself, Steve Flanders, the hard limit is more:

The Log Insight appliance comes in four sizes: an Extra Small, for use in labs or PoC, through to Large, which can consume a whopping 112.5GB of logs per day. Those figures scale in a linear fashion for clusters, so a 3-node cluster of large instances can consume 337.5GB per day, from 2,250 syslog connections at a rate of 22,500 events a second. See more on sizing vRealize Log Insight.

Load Balanced Cluster Pre-requisites

  • Configure a minimum of three nodes in a Log Insight cluster.
  • Verify that all Log Insight nodes and the specified Integrated Load Balancer IP address are on the same network.
  • The Log Insight master and worker nodes must have the same certificates. Otherwise the Log Insight Agents configured to connect through SSL will reject the connection. When uploading a CA-signed certificate to Log Insight master and worker nodes, set the Common Name to ILB IP address during certificate generation request. See Generate a Certificate Signing Request.
  • You must synchronize the time on the Log Insight Linux Agent virtual appliance with an NTP server. See Synchronize the Time on the Log Insight Virtual Appliance.

With those in mind, I will deploy 3 nodes, VRLI-01, VRLI-02 and VRLI-03, and load balance them under the URL “loginsight.definit.local”.

Generate a certificate for vRealize Log Insight

I am using a Microsoft Windows Server 2012 Certificate Authority, with the VMware certificate template created as per Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108).

I also have OpenSSL installed to generate the CSR, and will use certreq.exe to submit and retrieve the certificate.

Create a config file for OpenSSL with the following settings. Replace the values in red with your own, ensuring that the Common Name is the IP address you will assign to the load balancer.

[ req ]

default_bits = 2048

default_keyfile = rui.key

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = digitalSignature, keyEncipherment, dataEncipherment

extendedKeyUsage = serverAuth, clientAuth

subjectAltName = DNS:loginsight.definit.local, DNS:vrli-01.definit.local, DNS:vrli-02.definit.local, DNS:vrli-03.definit.local

[ req_distinguished_name ]

countryName = UK

stateOrProvinceName = West Sussex

localityName = Horsham

0.organizationName = DefinIT

organizationalUnitName = Lab

commonName = 192.168.1.26

Use OpenSSL to generate a new private key file, and a Certificate Signing Request:

c:\OpenSSL\bin\openssl.exe req -new -nodes -out z:\Certificates\LogInsight\rui.csr -keyout z:\Certificates\LogInsight\rui.key -config z:\Certificates\LogInsight\loginsight.cfg

This command generates a CSR (rui.csr), a private key (rui.key) and takes the config file as an input (loginsight.cfg).

Next, submit the template using certreq:

Z:\Certificates\LogInsight>certreq -submit -attrib "CertificateTemplate:VMware" rui.csr

Select your Issuing Certificate Authority and then save the certificate that is returned (I called mine LogInsight.cer).

Now create a new text file and paste in, in order, the contents of the private key file (rui.key), the certificate (LogInsight.cer) and the Issuing CA certificate. Save the new file as LogInsight.pem.

This certificate will need to be added to all of the LogInsight instances later.

Deploy the Master vRealize Log Insight node

The OVF deploy is simple:

Select the OVF template
      <p class="wp-caption-text">
        Select the OVF template
      </p>
    </div></td> 
    
    <td valign="top" width="300">
      <p>
        <div style="width: 254px" class="wp-caption alignnone">
          <a href="/images/2015/02/image6.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb6.png" alt="vRealize Log Insight - Accept the OVF additional configuration items" width="244" height="145" border="0" /></a>
          
          <p class="wp-caption-text">
            Accept the OVF additional configuration items
          </p>
        </div></td> </tr> 
        
        <tr>
          <td valign="top" width="300">
            <p>
              <div style="width: 254px" class="wp-caption alignnone">
                <a href="/images/2015/02/image7.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb7.png" alt="vRealize Log Insight - Accept the EULA" width="244" height="145" border="0" /></a>
                
                <p class="wp-caption-text">
                  Accept the EULA
                </p>
              </div></td> 
              
              <td valign="top" width="300">
                <p>
                  <div style="width: 254px" class="wp-caption alignnone">
                    <a href="/images/2015/02/image8.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb8.png" alt="vRealize Log Insight - Select a name and location" width="244" height="145" border="0" /></a>
                    
                    <p class="wp-caption-text">
                      Select a name and location
                    </p>
                  </div></td> </tr> 
                  
                  <tr>
                    <td valign="top" width="300">
                      <p>
                        <div style="width: 254px" class="wp-caption alignnone">
                          <a href="/images/2015/02/image9.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb9.png" alt="vRealize Log Insight - Select the size required" width="244" height="145" border="0" /></a>
                          
                          <p class="wp-caption-text">
                            Select the size required
                          </p>
                        </div></td> 
                        
                        <td valign="top" width="300">
                          <p>
                            <div style="width: 254px" class="wp-caption alignnone">
                              <a href="/images/2015/02/image10.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb10.png" alt="vRealize Log Insight - Select the storage" width="244" height="145" border="0" /></a>
                              
                              <p class="wp-caption-text">
                                Select the storage
                              </p>
                            </div></td> </tr> 
                            
                            <tr>
                              <td valign="top" width="300">
                                <p>
                                  <div style="width: 254px" class="wp-caption alignnone">
                                    <a href="/images/2015/02/image11.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb11.png" alt="vRealize Log Insight - Choose a network" width="244" height="145" border="0" /></a>
                                    
                                    <p class="wp-caption-text">
                                      Choose a network
                                    </p>
                                  </div></td> 
                                  
                                  <td valign="top" width="300">
                                    <p>
                                      <div style="width: 254px" class="wp-caption alignnone">
                                        <a href="/images/2015/02/image12.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb12.png" alt="vRealize Log Insight - Configure the network settings" width="244" height="145" border="0" /></a>
                                        
                                        <p class="wp-caption-text">
                                          Configure the network settings
                                        </p>
                                      </div></td> </tr> 
                                      
                                      <tr>
                                        <td valign="top" width="300">
                                          <p>
                                            <div style="width: 254px" class="wp-caption alignnone">
                                              <a href="/images/2015/02/image13.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb13.png" alt="vRealize Log Insight - Review and deploy" width="244" height="145" border="0" /></a>
                                              
                                              <p class="wp-caption-text">
                                                Review and deploy
                                              </p>
                                            </div></td> 
                                            
                                            <td valign="top" width="300">
                                            </td></tr> </tbody> </table> 
                                            
                                            <p>
                                              Once the node has <strong>completed</strong> it's boot up and is at the screen below you can proceed.
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image14.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb14.png" alt="image" width="244" height="184" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Open your web browser of choice and go to the IP address of the Master node and you should see the startup wizard. For the Master node we start a new deployment:
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image15.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb15.png" alt="image" width="244" height="180" border="0" /></a> <a href="/images/2015/02/image16.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb16.png" alt="image" width="244" height="180" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              <strong><img class=" alignright" style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb17.png" alt="image" width="487" height="106" border="0" />Note</strong>: I<em>f you see "Failed to start new deployment" you can reboot the appliance and try again, however I found that it caused problems later on in the deployment ("Login failed" during the wizard) and required a re-deploy of the appliance. When I redeployed, I rebooted pre-emptively and still saw the "Failed to start new deployment" message. Rebooting again allowed me to proceed - this time without the "Login failed" messages. The login failed errors actually stop you from completing the wizard, so if you see those it's time to redeploy.</em>
                                            </p>
                                            
                                            <p>
                                              After a reboot clicking the "Next" button takes you straight to the next step, which is to set the admin password (This is a different user to the "root" password that we set during the OVF deploy, that's used for SSH access).
                                            </p>
                                            
                                            <div style="width: 254px" class="wp-caption alignnone">
                                              <a href="/images/2015/02/image18.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb18.png" alt="vRealize Log Insight - Configure the Admin user" width="244" height="160" border="0" /></a>
                                              
                                              <p class="wp-caption-text">
                                                <strong>Configure the Admin user</strong>
                                              </p>
                                            </div>
                                            
                                            <div style="width: 254px" class="wp-caption alignnone">
                                              <a href="/images/2015/02/image20.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb20.png" alt="vRealize Log Insight - Add the license key" width="244" height="160" border="0" /></a>
                                              
                                              <p class="wp-caption-text">
                                                Add the license key
                                              </p>
                                            </div>
                                            
                                            <div style="width: 254px" class="wp-caption alignnone">
                                              <a href="/images/2015/02/image21.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb21.png" alt="vRealize Log Insight - Configure notifications" width="244" height="160" border="0" /></a>
                                              
                                              <p class="wp-caption-text">
                                                Configure notifications
                                              </p>
                                            </div>
                                            
                                            <p>
                                              At this point I skipped the NTP and SMTP settings to configure them later (I have found the wizard to be unreliable, NTP would not pass tests and SMTP would not send a test email). Finish the wizard and you should be presented with the login page:
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image22.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb22.png" alt="image" width="644" height="421" border="0" /></a>
                                            </p>
                                            
                                            <h3>
                                              Configure the Master Node
                                            </h3>
                                            
                                            <p>
                                              Log in to the master node using "admin" and the password you set. In the top right hand side of the dashboard view, you can open the Administration page
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image23.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb23.png" alt="image" width="151" height="161" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Select "Time" under the "Configuration" heading. Time is always critical for clustering, as well as for accurate logging. I have a local NTP server set up on my network to which everything syncs - so I configured and tested that:
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image24.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb24.png" alt="image" width="644" height="323" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Similarly, go to the SMTP section and configure the SMTP settings:
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image25.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb25.png" alt="image" width="644" height="380" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              And again, if required, select the Authentication page and configure Active Directory support
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image26.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb26.png" alt="image" width="644" height="344" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              AD Groups can be added via the "Access Control" page
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image27.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb27.png" alt="image" width="644" height="380" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Finally, import the certificate via the SSL page and reboot
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image28.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb28.png" alt="image" width="644" height="421" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image29.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb29.png" alt="image" width="122" height="29" border="0" /></a>
                                            </p>
                                            
                                            <h3>
                                              Deploy the Worker nodes
                                            </h3>
                                            
                                            <p>
                                              Deploy two more instances in the same way as the first - but this time use the "Join Existing Deployment" option, and then specify the first node (for me vrli-01.definit.local) and click "go". You'll then see a message to go and approve the Worker on the Master node (and you'll get an email, if you set that up).
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image30.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb30.png" alt="image" width="244" height="160" border="0" /></a> <a href="/images/2015/02/image31.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb31.png" alt="image" width="244" height="160" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Click allow to join the worker to the cluster.
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image32.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb32.png" alt="image" width="644" height="246" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              The Worker will then show a nice green tick, and the Master will have a whole load of information which can be summed up as "configure DNS and NTP, and think about load balancing" - all of which we have done, or will do.
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image33.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb33.png" alt="image" width="244" height="160" border="0" /></a> <a href="/images/2015/02/image34.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb34.png" alt="image" width="244" height="160" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Click OK on the Worker node and it will take you back to the login page. Log in using the admin credentials and you can see there are very few options available to us, just some general information and the SSL page.
                                            </p>
                                            
                                            <p>
                                              Import the SSL certificate we generated before and restart the Worker
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image35.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb35.png" alt="image" width="644" height="421" border="0" /></a>
                                            </p>
                                            
                                            <h3>
                                              Enabling the Integrated Load Balancer
                                            </h3>
                                            
                                            <p>
                                              Now the cluster has been deployed and the two Worker nodes are registered with the Master, we can enable the Integrated Load Balancer (ILB). So long as the pre-requisites have been met, this is simply a case of ticking the box and entering an IP address for the load balancer:
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image36.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb36.png" alt="image" width="644" height="421" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              Once saved, it will take a few seconds to configure and then the status will go green - "Available"
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image37.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb37.png" alt="image" width="356" height="162" border="0" /></a>
                                            </p>
                                            
                                            <p>
                                              I've created a DNS record for the load balanced IP to use to direct clients, this means any future changes can be implemented easily.
                                            </p>
                                            
                                            <p>
                                              My Log Insight cluster is now available, secured by SSL, on <a href="https://loginsight.definit.local">https://loginsight.definit.local</a>
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image38.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb38.png" alt="image" width="644" height="473" border="0" /></a>
                                            </p>
                                            
                                            <h3>
                                              Post deployment configuration
                                            </h3>
                                            
                                            <p>
                                              Having a load balanced cluster is all well and good, but if the nodes are on the same physical host and that goes down, you could be faced with a loss of data while HA recovers them. Be sure to create an anti-affinity rule to keep them on separate hosts!
                                            </p>
                                            
                                            <p>
                                              <a href="/images/2015/02/image39.png"><img style="background-image: none; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-width: 0px;" title="image" src="/images/2015/02/image_thumb39.png" alt="image" width="644" height="455" border="0" /></a>
                                            </p>
Share this post
Home