elasticsearch logstash and kibana (elk) vm for …...elasticsearch logstash and kibana (elk) vm for...
TRANSCRIPT
Hitachi Hyper Scale-Out Platform (HSP)Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
MK-95HSP033-0009 November 2016
© 2016 Hitachi, Ltd. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying and recording, or stored in a database or retrieval system forcommercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data SystemsCorporation (collectively, “Hitachi”). Licensee may make copies of the Materials provided that any suchcopy is: (i) created as an essential step in utilization of the Software as licensed and is used in no othermanner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials."Materials" mean text, data, photographs, graphics, audio, video and documents.
Hitachi reserves the right to make changes to this Material at any time without notice and assumes noresponsibility for its use. The Materials contain the most current information available at the time ofpublication.
Some of the features described in the Materials might not be currently available. Refer to the most recentproduct announcement for information about feature and product availability, or contact Hitachi DataSystems Corporation at https://support.hds.com/en_us/contact-us.html.
Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicableHitachi agreements. The use of Hitachi products is governed by the terms of your agreements with HitachiData Systems Corporation.
By using this software, you agree that you are responsible for:
1) Acquiring the relevant consents as may be required under local privacy laws or otherwise fromauthorized employees and other individuals to access relevant data; and
2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance withrelevant laws.
Notice on Export Controls. The technical data and technology inherent in this Document may besubject to U.S. export control laws, including the U.S. Export Administration Act and its associatedregulations, andmay be subject to export or import regulations in other countries. Reader agrees tocomply strictly with all such regulations and acknowledges that Reader has the responsibility to obtainlicenses to export, re-export, or import the Document and any Compliant Products.
Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries.
AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eServer, FICON, FlashCopy,IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/OS, z9, z10, z13,z/VM, and z/VSE are registered trademarks or trademarks of International Business Machines Corporation.
Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft,the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQLServer, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo, Windows Azure, WindowsPowerShell, Windows Server, the Windows start button, andWindows Vista are registered trademarks ortrademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission fromMicrosoft Corporation.
All other trademarks, service marks, and company names in this document or web site are properties oftheir respective owners.
Contents
Preface vIntended audience viProduct version viRelease notes viDocument revision level viDocument conventions viiAccessing product documentation viiGetting help viiiComments viii
Quick start 1Before you begin 2Procedure 2
Deploy the ELKVMs onHSP 5Before you begin 6Procedure 6
Deploy the ELK template 7Installation and configuration script 8
Appendix A: Logstash Configuration File 11Logstash default configuration file 12
Appendix B: KVM image details 13ELK VM software bundle 14Licenses used 15
Contents iii
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
iv Contents
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Preface
This document describes and provides instructions for deploying theElasticsearch, Logstash, and Kibana (ELK) VM template on Hitachi HyperScale-Out Platform (HSP).
This Preface includes the following information:
❒ Intended audience
❒ Product version
❒ Document revision level
❒ Document conventions
❒ Accessing product documentation
❒ Getting help
❒ Comments
Preface v
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Intended audience
This document is intended for system administrators, Hitachi Data Systemsrepresentatives, and authorized service providers who need to deploy theELK VM template on Hitachi Hyper Scale-Out Platform (HSP).
Readers of this document should be familiar with the following:
• ELK administration
• Linux operating system and working in a restricted shell environment
• Site-specific network information
Product version
This document applies to Hyper Scale-Out Platform release 1.2.0.037 orlater.
Release notes
The release notes for this product are available on Hitachi Data SystemsSupport Connect: https://support.hds.com/en_us/contact-us.html.Read the release notes before installing and using this product. They maycontain requirements or restrictions that are not fully described in thisdocument or updates or corrections to this document.
Document revision level
Revision Date Description
MK-95HSP033-00 09 November 2016 Initial release
Intended audience
vi Preface
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Document conventionsThis document uses the following typographic conventions:
Convention Description
Bold Indicates:
Text in a window or dialog box, such as menus, menu options,buttons, and labels. For example:On the Add Pair dialog box, clickOK.
Text appearing on screen or entered by the user. For example:The -split option…
<Italic> in anglebrackets
Indicates a variable, which is a placeholder for site- or installation-specific details that you need to provide. For example:copy <source-file> <target-file>
monospace Indicates text that is displayed on the screen or text that you needto enter. For example:
# pairdisplay -g oradb
Also, the name of a directory, folder, or file. For example:
The horcm.conf file…
This document uses the following icons to draw attention to information:
Icon Meaning Description
Important Calls attention to information that is essential to thecompletion of a task.
Caution Warns that failure to take or avoid a specified action couldresult in adverse conditions or consequences (for example,loss of access to data).
Accessing product documentation
Product documentation is available on Hitachi Data Systems SupportConnect: https://knowledge.hds.com/Documents. Check this site for themost current documentation, including important updates that may havebeen made after the release of the product.
Document conventions
Preface vii
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Getting help
Hitachi Data Systems Support Portal is the destination for technicalsupport of products and solutions sold by Hitachi Data Systems. To contacttechnical support, log on to Hitachi Data Systems Support Connect forcontact information: https://support.hds.com/en_us/contact-us.html.
Hitachi Data Systems Community is a global online community for HDScustomers, partners, independent software vendors, employees, andprospects. It is the destination to get answers, discover insights, and makeconnections. Join the conversation today! Go to community.hds.com,register, and complete your profile.
Comments
Please send us your comments on this document:
Include the document title and part number, including the revision (forexample, -01), and refer to specific sections and paragraphs wheneverpossible. All comments become the property of Hitachi Data Systems.
Thank you!
Getting help
viii Preface
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Quick start
This chapter is a summary of the procedure for deploying an ELK cluster onHSP.
This summary is intended as a quick reference for users that are familiarwith manually deploying VMs on HSP.
The full procedure is described starting at Deploy the ELK VMs on HSP
❒ Before you begin
❒ Procedure
Quick start 1
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
1
Before you begin
Verify that you have:
• An initialized HSP cluster running at least HSP 1.2.0.037. Ensure thatyour HSP cluster has its DNS domain name, DNS server and NTP serverconfigured correctly.
• Downloaded copy of the qcow2 file containing the repository-vm for ELK.This file is available from Technical Information Service Center (TISC),and may be called something like ELK_HSP-1.1.1_r1.qcow2
• A Linux client
Procedure
1. Add the ELK template to the HSP cluster.
2. Deploy the ELK VMs. One or more ELK VM can be deployed to form theELK cluster.
3. Log into the first ELK VM instance as root.
ssh <ELK-VM Instance IP address>
Username: rootPassword: smrace1
4. Run the following script to install, setup, and start the ELK services:
/opt/escale/install/install_elk.py -m 3 -d 2
Where:Value for m = sets the number of master nodes in the ELK cluster.In the example above this is set to 3 master nodes.
Value of d = sets the number of data nodes in the ELK cluster.In the example above this is set to 2 data nodes.
2 Quick start
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Before you begin
5. When the script has finished running create a Logstash configurationLogstash default configuration fileat:
/etc/logstash/conf.d/
Note: The default Logstash configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.
6. Check that all of the services are running using the following commands:
service elasticsearch statusservice logstash statusservice kibana status
Quick start 3
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Procedure
4 Quick start
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Deploy the ELK VMs on HSP
This chapter describes the procedure for deploying the ELK cluster. At theend of this procedure the VM instances will be deployed and ready to beused.
❒ Before you begin
❒ Procedure
Deploy the ELK VMs on HSP 5
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
2
Before you begin
Verify that you have:
• An initialized HSP cluster running at least HSP 1.2.0.037. Ensure thatyour HSP cluster has its DNS domain name, DNS server and NTP serverconfigured correctly.
• Downloaded copy of the qcow2 file containing the repository-vm for ELK.This file is available from Technical Information Service Center (TISC),and may be called something like ELK_HSP-1.1.1_r1.qcow2.
• Linux client.
Procedure
First, add the vm-template to the HSP cluster.
1. Mount a share from the HSP cluster on your Linux system:
linux# mount <HSP Cluster Virtual IP Address>:/<myshare> /mnt/hspWhere /myshare is the name of the available share.
2. Copy the qcow2 images from the ISOs to the share:
linux# cp /mnt/iso/ELK_HSP-1.1.1_r1.qcow2 /mnt/hsp
3. On the HSP cluster, add the vm-template:
admin@Node003:~$ hspadm vm-template add --name elk_template1 --image-format qcow2 --image-path <myshare>:/ELK_HSP-1.1.1_r1.qcow2
4. Verify this was successful:
admin@Node003:~$ hspadm vm-template list
6 Deploy the ELK VMs on HSP
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Before you begin
Deploy the ELK template
1. Deploy the vm-template. In this example we are creating a 5 node ELKcluster.
admin@Node003:~$ hspadm vm-instance-group add -–name hsp_elk --num-instances 5 --vm-template elk_template1 --vm-size medium
2. Verify this was successful:admin@Node001:~$ hspadm vm-instance list --template elk_template1
Deploy the ELK VMs on HSP 7
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Procedure
Installation and configuration script
Logging into the first VM instance launches a script that installs, configures,and starts the Elasticsearch, Logstash, and Kibana services on all of the ELKVM instances.
Note: The script installs the Logstash service on a data node but does notstart it. You must create you own Logstash configuration file to start theLogstash service. The default configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.
1. SSH to the first ELK VM instance (in this example 10.129.71.21 isused):
ssh 10.129.71.21 (VM instance address)
Username: rootPassword: smrace1
2. Run the following script to install, setup, and start the ELK services:
/opt/escale/install/install_elk.py -m 3 -d 2
Where:Value for m = sets the number of master nodes in the ELK cluster.In the example above this is set to 3 master nodes.
Value of d = sets the number of data nodes in the ELK cluster.In the example above this is set to 2 data nodes.
8 Deploy the ELK VMs on HSP
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Procedure
3. When the script has finished running create a Logstash configuration at:
/etc/logstash/conf.d/
Note: The default Logstash configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.
4. Check that all of the services are running using the following commands:
service elasticsearch statusservice logstash statusservice kibana status
Deploy the ELK VMs on HSP 9
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Procedure
10 Deploy the ELK VMs on HSP
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Logstash Configuration File
This appendix contains the Logstash default configuration file.
You must create you own Logstash configuration file to start the Logstashservice.
See the www.elastic.co website for details on creating your own Logstashconfiguration file. The configuration file must be created at:
/etc/logstash/conf.d/.
❒ Logstash default configuration file
Appendix A: Logstash Configuration File 11
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
A
Logstash default configuration file
12 Appendix A: Logstash Configuration File
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Logstash default configuration file
KVM image details
Elasticsearch, Logstash, and Kibana (ELK) deployment begins with a KVMbased VM image.
The package consists of one VM image:
• ELK_HSP-1.1.1_r1 ELK VM software bundle
❒ ELK VM software bundle
❒ Licenses used
Appendix B: KVM image details 13
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
B
ELK VM software bundle
• ELK VM with Elasticsearch, Logstash, and Kibana repositories.
¡ CentOS 7.2 x64 - Linux Operating system
¡ Installed software
– Oracle JDK 1.8 jdk-8u60-linux-x64 – Java virtual machine
– Python 2.7.5 – Python interpreter (for scripting language)
¡ Packaged Software
– Elasticsearch 2.3.3
– Logstash 2.3.2
– Kibana 4.5.1
– Lucene 5.5.0
14 Appendix B: KVM image details
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
ELK VM software bundle
Licenses used
This media kit includes software covered under the following licenses:
Apache License, Version 2.0 (the "License")
The Platform Components comprising the Hortonworks Data Platform (HDP)are released under the Apache 2.0 License. Unless required by applicablelaw, software distributed under the Apache License is distributed on an “ASIS” BASIS.
Licensed under the Apache License, Version 2.0 (the "License"); you maynot use this file except in compliance with the License. You may obtain acopy of the Apache License at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS, WITHOUTWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
Optional Add-Ons and 3rd-Party Components
While HDP is provided under the terms of the Apache 2.0 License, it is alsocommonly used with 3rd-Party Components and Optional Add-Ons(example Hive ODBC Driver). When you choose to use those components, itis recommended you read and understand the licensing terms specific toeach of those components. Also note that Add-Ons can be downloadedseparately from HDP and each comes with instructions on how to install andleverage the features alongside HDP.
Trademarks:
Apache and the Apache feather logo are trademarks of The Apache SoftwareFoundation
CentOS is a trademark of Red Hat, Inc.
Hortonworks Data Platform (HDP) is trademark of Hortonworks Inc.
Python is a registered trademark of the Python Software Foundation
Appendix B: KVM image details 15
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Licenses used
16 Appendix B: KVM image details
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide
Hitachi Data Systems
Corporate Headquarters2845 Lafayette StreetSanta Clara, California 95050-2627U.S.A.www.hds.com
Regional Contact Information
Americas+1 408 970 [email protected]
Europe, Middle East, and Africa+44 (0) 1753 [email protected]
Asia Pacific+852 3189 [email protected]
MK-95HSP033-00