document: vlastic search manual description ... · document: vlastic search manual description:...

Document: Vlastic Search Manual

Description: Administration and User Guide for installing and using Vlastic Search. The Global Search extension module for vtiger CRM

Revision: 2

Document Status: Published

Written by: Alan Lord

Date Approved: 08/07/2016

© 2016 Libertus Solutions Revision No. 2

Table of ContentsIntroduction........................................................................................................................................................3

Important Information................................................................................................................................3Administration Guide........................................................................................................................................4

Minimum Requirements.............................................................................................................................4Elasticsearch installation or acquisition..................................................................................................4

Installation of Elasticsearch on Linux................................................................................................................ 5Testing your Elasticsearch server....................................................................................................................... 6Installation of Elasticsearch on Windows.........................................................................................................7Elasticsearch as a Service.................................................................................................................................... 8

Installation of the Vlastic Search Extension Module for vtiger..........................................................9Configuring and Managing Vlastic Search............................................................................................12Vlastic Search Settings.............................................................................................................................13

Server and Port................................................................................................................................................... 14Index Name.......................................................................................................................................................... 15Indexed Modules................................................................................................................................................ 15Query Type........................................................................................................................................................... 16Max. in Popup...................................................................................................................................................... 17Max. per Module................................................................................................................................................. 17Max. Results......................................................................................................................................................... 18Max. per Cron...................................................................................................................................................... 18Open in new browser tab.................................................................................................................................. 18Front End Search Enabled................................................................................................................................. 18

Vlastic Search Status Information..........................................................................................................19Total Records Indexed....................................................................................................................................... 19Per Module........................................................................................................................................................... 19Last Cron Index................................................................................................................................................... 19

Vlastic Search Indexing Scheduler..........................................................................................................20Indexing in real-time.................................................................................................................................22

User Guide........................................................................................................................................................23Using Vlastic Search..................................................................................................................................23

Open in new tab.................................................................................................................................................. 25Querying with Vlastic Search..................................................................................................................26

Simple Query String Operators and usage....................................................................................................26Example Searches............................................................................................................................................... 27

Information about Vlastic Search................................................................................................................28


IntroductionVlastic Search is an extension module for the Open Source version of vtiger from Libertus Solutions. Libertus Solutions' Vlastic Search module provides an extremely fast, simple-to-use and comprehensive global search facility to vtiger CRM using the Elasticsearch1 full-text search engine.

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyse big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

Important Information

Vlastic Search requires an accessible Elasticsearch engine. It will not work without one.

Please read the information in the Administration section about the installation of Elasticsearch or purchase of a cloud based solution.

1 https://www.elastic.co/products/Elasticsearch


Administration GuideThis section is primarily intended for the administrators of the vtiger CRM system but other users might find the content interesting or useful.

Minimum RequirementsApart from having an Elasticsearch server, Vlastic Search depends on a php library called elasticsearch-php1 which depends on a minimum version of the PHP library of 5.3.9. You should also have the php curl library installed.

Elasticsearch installation or acquisitionBefore buying and configuring Vlastic Search for vtiger you should already have an Elasticsearchsearch engine configured and available.

At a high level there are basically 2 choices:

1. Install an Elasticsearch search engine either on the same server where your vtiger systemis installed or an alternative server within your domain

2. Acquire a hosted Elasticsearch search engine as a service

If you decide to install Elasticsearch on your own server we recommend you consult the available on-line documentation for your particular requirements and make sure you understand the important configuration parameters, especially when it comes to securing access to the Elasticsearch service.

The Elasticsearch documentation is easy to read and well structured:

https://www.elastic.co/guide/en/Elasticsearch/reference/current/index.html

1 https://github.com/elastic/elasticsearch-php/tree/1.0


https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Installation of Elasticsearch on LinuxInstalling Elasticsearch on most reasonably modern Linux systems is normally just a single command:

Deb based (Debian, Ubuntu 15.04 or later):

sudo apt-get install elasticsearch

Older Ubuntu/Debian:

Use the elastic search repositories as documented in the guidehttps://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

or download the install files and follow the instructionshttps://www.elastic.co/downloads/elasticsearch

RPM based (Redhat, CentOS etc.)

yum install elasticsearch

After installation you should consult the relevant documentation to setup access and configure the service.

If Elasticsearch is installed on the same machine as vtiger and will only be used with vtiger then you should configure Elasticsearch to only listen for connections on localhost.

It is essential that you secure Elasticsearch from unwanted connections, if your Elasticsearch server is available to the Internet then anyone would be able to access all the information youput in it from vtiger.

If Elasticsearch is on a different server to vtiger then you will need to determine the appropriatesettings to allow vtiger to connect to it securely.

By default Elasticsearch listens on port 9200 for its REST API. This port is configurable if necessary.


https://www.elastic.co/downloads/elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

Testing your Elasticsearch server

You can test that it is working from the command line of the server by running

curl -X GET http://localhost:9200/

If it is working you should get a response like this:

{ "status" : 200, "name" : "Apalla", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.5", "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4", "build_timestamp" : "2016-02-02T09:55:30Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search"}

You can also try connecting to it with a normal web browser from another computer with a URL like http://yourserver:9200 but if that works then you have a problem!

To get Elasticsearch to only listen to requests originating on the same server edit the file /etc/elasticsearch/elasticsearch.yml and edit the network.host line so that it only listens to local connections

network.host: 127.0.0.1

If you are installing Elasticsearch on a different server to your vtiger server then you would needto use a firewall to make sure that Elasticsearch is only listening to connections from your vtiger server.

Once Elasticsearch is listening to your vtiger server and not everywhere else, you are ready for the next stage.


http://yourserver:9200/

http://localhost:9200/

Installation of Elasticsearch on WindowsLibertus Solutions doesn't recommend Windows for anything, but the Vlastic Search module hasbeen tested on Windows.

Please see the documentation for installation of Elasticsearch on Windows:

https://www.elastic.co/guide/en/Elasticsearch/reference/current/setup-service-win.html


https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service-win.html

Elasticsearch as a ServiceIf you do not want to install Elasticsearch on your own servers you can purchase Elasticsearch as a service from multiple providers. Here are the first few that we found on Google:

https://www.elastic.co/cloud

https://aws.amazon.com/Elasticsearch-service/

https://qbox.io/

https://bonsai.io/


https://bonsai.io/

https://qbox.io/

https://aws.amazon.com/elasticsearch-service/

https://www.elastic.co/cloud

Installation of the Vlastic Search Extension Module for vtigerVlastic Search requires an accessible installation of the Elasticsearch search engine. Please see above.

The Vlastic Search extension module is installed via the Extension Store which is an application within vtiger itself. This ensures that the version of Vlastic Search installed has been approved and tested by the vtiger team. The Extension Store also informs the administrator when updatesare available, and enables easy upgrading, for any installed modules.

To install Vlastic Search navigate to the CRM Settings area and click the button on the left column called “Extension Store”. This will display the various optional extension modules that are currently available in the vtiger Marketplace1. The extension modules in the Marketplace have been approved and tested by vtiger before being allowed to be installed by end users.

1 https://marketplace.vtiger.com


If this is the first time you have visited the Extension Store you will need to register a new user…

and you may also need to supply a credit card to allow the purchase of any non-free extension modules.


Once you have provided the necessary information you can either browse or search for the Vlastic Search module. Once found click on the “More Details” button to view more information about the module. If you wish to install Vlastic Search you should pay the requested amount via your credit card and then you will be able to install the module simply by clicking on the green “Install” button.


Configuring and Managing Vlastic SearchOnce installed, Vlastic Search needs to be configured before it can be used. Vlastic Search has a single configuration screen accessible from the Module Manager in the CRM Settings area.

All parameters are configured in the “Vlastic Search Settings” area.


Vlastic Search SettingsWhen Vlastic Search is first installed most settings will be set to a default value. The screen is divided into 2 sections. The top section is for configuration of Vlastic Search and the bottom section displays some useful information about the status of the search engine.

The meaning of each configuration parameter is as follows:


Server and PortSpecify the connection parameters to connect to the Elasticsearch server. Vlastic Search communicates with Elasticsearch using a REST API (by default over http). In the most simple system where Elasticsearch is running on the same server as vtiger and is listening on the default port then simply enter:

localhost:9200

where localhost means this server, :9200 means use port 9200, and no scheme means use http://.

Your Elasticsearch server could however be on a remote or cloud based, system and protected by http auth and/or encryption. Depending on your specific conditions, the format of the Server and Port field could be something like:

https://user:[email protected]

If you plan to use https you must use a certificate signed by a Certificate Authority (CA) and your server must have up-to-date root certificates.

Although the elasticsearch-php client library used in Vlastic Search can support self-signed certificates we currently have no way of supporting this via the vtiger administration user interface.

The elasticsearch-php documentation describes the various connection parameters on this page:

https://www.elastic.co/guide/en/Elasticsearch/client/php-api/current/_security.html


https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_security.html

Index NameThis field contains the unique identifier for the Elasticsearch server. All data from this vtiger system will be sent to your Elasticsearch server by Vlastic Search under this Index Name. The Index Name must only contain lower-case letters.

Important: If you have more then one vtiger system and wish to use Vlastic Search on some or all ofthem but want to use a single Elasticsearch server then you must make sure that each Vlastic Search configuration has a unique Index Name.

Caution: If you change this value after Vlastic Search has been running for some time, you will be creating a completely new database in your Elasticsearch server and all your CRM data will need to be re-indexed. You can of course change the setting back again.

If you want to delete all the data in an index you can do this from the command line

curl -XDELETE http://localhost:9200/ your-index-name /

Indexed ModulesThis is a multi-select option where you choose which modules within vtiger you want to be indexed in your Elasticsearch server. The recommended setting here is to configure all modules that are currently enabled and are being used in your vtiger system. When Vlastic Search is first installed the process will look for all enabled Entity Modules within your vtiger system and populate this parameter accordingly.


http://localhost:9200/your-index-name/



Query TypeThe Elasticsearch search engine supports many types of query and syntax. The majority of these are generally more suited to automated “machine-generated” type queries, but some types of supported query are useful for humans to use directly. Vlastic Search currently implements four of the query types supported by Elasticsearch:

• Simple Query String This is the recommended and default setting for Vlastic Search. The Simple Query String query is a more robust form of the Query String query type discussed below. It supports multiple terms, wildcards, and various operators to enhance the quality of your search results. The description of the various operators and syntax structure of the Simple Query String query type is described in the User Guide section. In addition, further information on this query type can be found in the following page of the Elasticsearch manual: https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-simple-query-string-query.html

• Query StringThe Query String query format supports several additional search operators and featuresover the Simple Query String setting but it is less “robust”. Meaning that any invalid text in the query will be cause the query to fail completely and return an empty result set. However, once some proficiency is gained in using the query feature this might be a goodoption as it does support a few extra operators. The full description of this query type is described in the following page:https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-query-string-query.html

• FuzzyThe Fuzzy query uses similarity based on Levenshtein1 edit distance for string fields, and a +/- margin on numeric and date fields. In short this means you can use it search things using even misspelled words. For example typing “hotil” into the search box wouldfind any record that contains a word within 1 Levenshtein edit distance away, so “hotel” would be found as would any other sequence of characters containing up to one character change. If the search term is longer than 5 characters the Fuzzy search will lookup to 2 Levenshtein edit distances away. For numeric and date searches Elasticsearch calculates a range of values for results instead of Levenshtein, so for example searching for a number would yield results within a +/- range of up to 2. Fuzzy search does not support many of the operators supported in the previous two choices consequently its usefulness is somewhat arguable in the context of a user and vtiger. However it is supported by Vlastic Search and further information on its operation can be found here:https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-fuzzy-query.html

• MatchThe Match query type provides a relatively simple boolean type query facility. Using this search type Vlastic Search will return records where the search term matches exactly. It does not support field name prefixes, wildcard characters, or other "advanced" features. For this reason, chances of it failing are very small / non existent, and it provides an excellent behaviour and performance. However the lack of wildcard and other advanced features might make it less useful for a “normal” vtiger user. More information on the Match query type can be found here:

1 https://en.wikipedia.org/wiki/Levenshtein_distance


https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-simple-query-string-query.html

https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-simple-query-string-query.html

https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl-match-query.html

Elasticsearch supports many other query types and features which are not yet supported by Vlastic Search but might be of interest to the reader. You can find out about the various other types of query supported by Elasticsearch here:

https://www.elastic.co/guide/en/Elasticsearch/reference/current/query-dsl.html

Max. in PopupVlastic Search operates in two ways. In the first, you simply type the characters you are looking for into the search box at the top of the page when in any standard module and any results will be presented to you in a small popup display. This field controls the maximum number of resultsthat will be shown in that popup.

Max. per ModuleVlastic Search also has its own dedicated Search View screen where you can examine the results of your search in much more detail. Vlastic Search creates a menu item found under the “Tools” section by default.

This option determines the maximum number of results that will shown within each modules results on this display.

When you enter characters to search in the field at the top, if any results are found the page will load multiple individual vtiger List Views containing the records found for each module. If the query returns more results than this setting's value you will be able to use the normal page forward, back and jump navigation buttons within each module's records to examine the results.


https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

Max. ResultsWhen a search request is made to the Elasticsearch server, it is necessary to provide a maximum number of results you want to get back. This parameter allows you to configure that value. OnceElasticsearch has returned a set of matching records, these records are then validated against the vtiger user's permissions before being displayed.

Max. per CronThis parameter allows the administrator to determine the maximum number of records that can be indexed each time the Vlastic Search Indexing Scheduler runs. The default is 1000 records. If you have a large database then setting this value higher should help to speed up the initial Indexing process, but it may slow your vtiger system down so adjust it carefully and complementany Max. per Cron change with the frequency setting of the Vlastic Search Indexer too. Once themajority of your data has been stored in Elasticsearch however, you can set this to a fairly small number, e.g. 100, as the Event Handler should cater for most changes, deletions and additions inreal-time.

If you have an extremely large database of millions of records then we suggest turning of the Vlastic Search Indexer job in the admin screen (see below) and running the indexer from the command line during quiet periods, e.g. at night or over the weekend. On a Linux server a simplecommand such as:

for i in {1..150}; do php modules/LSVlasticSearch/resources/LSVlasticSearchCron.service; done

run from the root of your vtiger system will run the Vlastic Indexing Scheduler 150 times. If you set the Max. per Cron to “20000” that will index up to 3,000,000 records. Of course you will needto determine the optimum settings based on your own configuration, hardware and server performance.

Open in new browser tabWhen this button is checked, clicking on a record in the Vlastic Search view screen will open it in Detail View in a new tab in your browser, effectively making the search results persistent so you do not need to keep re-searching if you have opened the wrong record.

Front End Search EnabledThis is an important setting! When Vlastic Search is first installed this button will be unchecked.It means that the global search box in vtiger functions as normal. Once Vlastic Search has successfully indexed a sufficient amount of your vtiger CRM data then you can enable the Vlastic Search feature by checking this box. Doing so installs a JavaScript library which overrides the functionality of the Vtiger Global Search box and forwards all text entered to your Elastic Search server for analysis and display of the results. Clearly, if your Elasticsearch server has not yet been setup, has not yet indexed most of your CRM data or becomes unavailable for whatever reason then you should un-check this box to make the Global Search box go back to using the standard vtiger search function.


Vlastic Search Status Information

In the Vlastic Search Engine Status section the information displayed is as follows:

Total Records IndexedShows the total count of all vtiger records that have been indexed in your Elasticsearch server.

Per ModuleThis shows a count of the number of records of each module that have been indexed by Elasticsearch and also the total count of records for that module. For example, if there are 3000 Leads in your vtiger CRM system and Elasticsearch has indexed 2000 of them so far then the display will show Leads (2000 of 3000).

Last Cron IndexThis indicates the last time the Vlastic Search Indexer Scheduled job ran. By default this is configured to run every 30 minutes and will index up to 1000 records per cycle or whatever the Max. per Cron setting is. The operation of the scheduler is described in the next section.


Vlastic Search Indexing Scheduler

Important Note: The vtiger scheduler only works if the vtiger cron feature has already been configured.

When Vlastic Search is first installed, a “scheduled job” is registered in the vtiger CRM scheduler to run once every 30 minutes once it has been Activated.

On initial installation of Vlastic Search this scheduler is set to Inactive. This should be set to Active only after the Vlastic Search Configuration settings have been setup and you have a working connection to your Elasticsearch server.

Once Vlastic Search has been configured, edit the scheduler entry and set the Status to “Active”.


The frequency of the Vlastic Search process may be configured here and the service disabled altogether if necessary. Once the majority of your data has been indexed this process may be set to run less frequently if desired. The Vlastic Search Event Handler should keep most* of yourElasticsearch server data up-to-date once your vtiger CRM data has been initially indexed. Also note the information in the previous section regarding the Max. per Cron setting. These two settings work together.

* Unfortunately, vtiger versions 6.4.0 and below do not have an “Event” trigger for the Lead Conversion process. This means that if a Lead is converted in vtiger to a Contact, Organisation and/or Opportunity, Vlastic Search does not know that the Lead is no longer visible so it will remain in the Elasticsearch server's database. Consequently you should keep the Vlastic Search Indexing Scheduler running as this includes a function that looks for recently converted Leads and removes these records from the Elasticsearch database.


Indexing in real-timeVlastic Search also attempts to Index changes to records in near-real-time as well as using the frequency-based scheduler described above.

Each time any entity record (for a module configured for Indexing) is saved, modified or deleted in vtiger, a vtiger Event Handler1will attempt to update the Elasticsearch database with the appropriate changes.

1 http://community.vtiger.com/help/vtigercrm/developers/vtlib/module-event.html


http://community.vtiger.com/help/vtigercrm/developers/vtlib/module-event.html

User Guide

Using Vlastic SearchThe user-facing side of Vlastic Search allows the viewer to perform full-text searches very rapidly across as much of the vtiger CRM system as has been configured for Vlastic Search.

Vlastic Search is very simple to use. In the search box at the top of all vtiger pages you simply enter the search term you are looking for. As soon as the third character has been entered the term is sent to the Elasticsearch server for evaluation. Each time a new character is entered, or one is removed or changed, the term will be resubmitted to Elasticsearch. There is no need to use the “enter” key to transmit the search term.

Vlastic Search presents the results of the search term in two ways.

The first way presents the user with a simple list of records in a small popup window on top of wherever they happen to be within vtiger. The information displayed is the vtiger “Label” or Entity Name of each record and the module to which it belongs.

As you can see above the search term entered was Hotel Smith. This particular phrase caused Vlastic Search to look for all records that contained the word Hotel or Smith or both. You can click on an entry to open it, or use the cursor keys to scroll down the list and enter to open the selected result.


If you want to examine the search results in more detail you can click the link at the bottom of the popup which says “Go To Full Page Search”. This takes the user to Vlastic Search's dedicated Search View layout which presents all the results, by module, in List Views.


Each modules' list of records has an indicator on the left next to the module name which shows how many records were found for that particular module. If there are more results than the amount configured to be displayed per module then the familiar navigation buttons on the righthand side will be active to enable you to page back and forth to examine more of the results. Each list of results uses the module's standard “All Filter” to determine the columns shown.

The search term can be altered using the same search box when in the Vlastic Search screen and the results will be updated in the lists below rather than in the Popup window described above.

Open in new tabOne final feature of the dedicated Vlastic Search View layout is configurable by the administrator. If enabled, when you click on a result in the list to look at its Detail View, the record will be automatically opened in a new tab of the user's web browser, effectively keeping this page of search results persistent.

Alternatively this setting can be disabled so that clicking on the record in the list will change thistab's display to that record's Detail View page which is the normal behaviour in vtiger list views.


Querying with Vlastic SearchDepending on the configuration of Vlastic Search various parameters may be used to help you narrow down, or broaden, the scope of your search.

Simple Query String Operators and usageThe default search method of Vlastic Search is called Simple Query String and it supports the following special characters:

● + signifies AND operation If you place a + between two words the search result will show only those records where both words exist

● | signifies OR operationIf you place a | between two words the search result will show records where either wordexists

● - negates a single token Putting a minus sign in front of a word means find all records that do not contain that word

● " wraps a number of tokens to signify a phrase for searching If you want to search for a phrase of more than one word enclose the whole phrase in quotation marks

● * at the end of a term signifies a prefix query The asterisk is a wild card. Putting an asterisk at the end of a word or part of word or phrase means find everything beginning with

● ( and ) signify precedence Use brackets to determine the precedence of the search terms – just like in mathematics


Example Searches

Here are some example searches to indicate the use of these special characters:

london+hotelThis will find all records within vtiger where the terms “london” AND “hotel” are present in the same record. Omitting the + (using a space) signifies the default operator which in Vlastic Search's case is the OR operator

“Novotel London”This search term, wrapped in quotation marks will find all records within vtiger that contain this exact phrase. To explain a bit more it would not find a record where the Organisation name is “Novotel Hammersmith London” but it would find a record where the Organisation name was “Hammersmith Novotel London”

0207* | 0208*Find all records in vtiger which contain a number sequence beginning 0207 or 0208. This would be useful if you were looking for all records which have a London telephone number for example.

(020 7* + London) -SmithThis is a more complicated example to show the use of multiple operators. This query term looks for all records which contain a number beginning with “020 7” AND also contain the word “London”. Then remove any records in this result which contain the word “Smith”.

WC1*Find all records in vtiger that have a string of characters beginning with “WC1”. This couldbe used to look for any record containing a London WC1 postcode.

As you may have noticed Vlastic Search is case insensitive.


Information about Vlastic SearchThis release of Vlastic Search makes use of two third party developments:

● The Elasticsearch search engine itself● The Elasticsearch-php library1, the official PHP low-level client for Elasticsearch.

Elasticsearch-PHP is dual licensed under both Apache v2.0 & LGPL v2.1

1 https://github.com/elastic/Elasticsearch-php


https://github.com/elastic/elasticsearch-php

document: vlastic search manual description ... · document: vlastic search manual description:...

Documents