state of the art logging. kibana4solr is here!
DESCRIPTION
Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. ConfigureTRANSCRIPT
KIBANA4SOLRMarkus KloseHead of Training & [email protected]
• Me / SHI GmbH
• Motivation of Kibana4Solr
• What is Kibana
• Project
– What is Kibana4Solr
– Implementation / Architecture
– Pitfalls
– Prospect
• Kibana4Solr
– Use cases
– Demo
• Q & A
Agenda - State Of the Art Logging with Kibana4Solr
• Head of Training & Consulting
– Project management
– Requirements Engineering
– Consulting
• LucidWorks Certified Lucene / Solr Trainer
• LucidWorks Certified Lucene / Solr Developer
Markus Klose / SHI GmbH
• History
– Founded in 1994
– Since 1995 focus on Search Technologies
– Since 2000 focus on Open Source Solutions
– Since 2011 Gold Partnership with LucidWorks
– Since 2013 Partnership with elasticsearch
• Complete Software Service Provider
– Consulting, Engineering, Training, Support
– Open Source Projects
– 5 Consultants, 4 Trainers, 20 Developers
• 100+ years of experience in working with search technologies
• 200+ customers out of different branches
SHI GmbH - Company
SHI GmbH – Search Services
Consulting Search Strategy Consulting
Conception of Search Applications
Execution of Proof of Concepts
Search Usability Studies
Architecture & Design Reviews
Sizing & Scaling
Relevancy Tuning
Search Semantics & Lingustics
Realisation Search Implementation
Programming (Extensions, Portals, Specialized Applications, Embedded, …)
Support Development Support (ExpertLink)
Service Level Agreements (SLA)
Providing Search Consultants and Developers
Trainings Classroom and On-site
Apache Solr Basics, Unleashed & Under The Hood
Apache Mahout (planned)
Apache Hadoop (planned)
360° Services
• In customer projects there is always the need of monitoring data/content and userdata
– „what are the most searched terms?“
– „what hit was selected?“
– „are there searches with no hits?“
– „how are my products categorized?“
– „how many of my products are almost not in stock?“
– …
• „…is there nothing out of the box?...“ -> NO not with Solr
Motivation 1
• Use and combine existing tools to offer all our customer a good analyzing infrastructure
• Good experience with kibana in our elasticsearch projects
idea: enable kibana to use a solr index
Motivation 2
• “Kibana is an open source (Apache Licensed), browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch ... Kibana strives to be easy to get started with, while also being flexible and powerful”
visualize logs and time-stamped data
• Seamless integration with Logstash, Apache Flume, Fluentd and others
What is Kibana
KibanaElasticsearchElasticsearchLogstashLogstashlogslogs
• Create custoum Dasbords BY SEARCHING with
– bar, line diagrams
– scatter plots
– pie charts
– maps
What is Kibana
• combines the Power of Kibana and Solr
• Translator elasticsearch <-> solr
• open source
• deployable as war file in all major application servers and servlet containers
What is kibana4solr
What is kibana4solr NOT
• No indexing / crawling tool
• No complete monitoring tool
• 06/2013 - Idea kibana4solr
• 07/2013 - POC
• 08/2013 - Implementation
– Table
• 09/2013 – idea SEMKit
– Refactoring !!!
– Facetting (query…)
• TODAY - kibana4solr/SEMKit is still in progress
• 11/2013 - more features (histogram facets)
• 11/2013 - GitHub
• …
The project – kibana4solr / SEMKit
Today06/2013
basic concept of kibana4solr
solr4kibana
KibanaKibana
SolrSolr
ElasticsearchElasticsearch
• SEMKit – Search Engine Migration Kit
– Framework for smooth search engine migrations
– Connectors are plugable
– More Connectors can be integrated easily
• kibana4solr is one configuration of the SEMKit
– Elasticsearch <-> Solr
the bigger picture
SEMKit
SolrSolr ES
Solr ESES
FASTFAST
FASTFAST
……
……
• Minior
– Settings in Kibana
– Settings in Solr
– Tomcat settings
– To much refactoring because the idea change within project
• Major
– Different syntax and feature spectrum, e.g.
• labels on facets
• …
Pitfalls & Challanges
• Business Aware Monitoring
– Search tracking
– User Tracking - Trends, Conversions, Cancelation
• Perfomance Monitoring
– Backend Tracking
– Network Tracking
• Data Analysis
– Measure the Development of one Product
– Extract knwoledge form social data
– Sediment Analysis
all that in REAL TIME and NOW with one tool and based on a SOLR INDEX
Kibana4Solr Use Cases
• Kibana4Solr
– Logs to Solr
– RealTimeLogging & analysis
• SEMKit
– Search Engine Migrations
• Fast2Solr
• …
– Multi-Channel-Monitoring
• Kibana4MySQL
• Kibana4SAP
• …
Prospects
Demo
• Kibana4Solr/SEMKit Websites
– GitHub
– SHI GmbH
– @SHIEngineers
• Kibana Websites
– http://www.elasticsearch.org/overview/kibana/
– http://rashidkpc.github.io/Kibana/index.html
• Q & A
Resources
KIBANA4SOLRMarkus KloseHead of Training & [email protected]
• Project (10 slides)
• Kibana4solr (prepare demonstration)
• Spellcheck
• Links?
checklist