improving drupal search experience with apache solr and ... · improving drupal search experience...

Improving Drupal search experience with Apache Solr and Elasticsearch

Milos Pumpalovic – Web Front-end DeveloperGene Mohr – Web Back-end Developer

Improving Drupal search experience with Apache Solrand Elasticsearch


About Us

2

MilosPumpalovic• FrontEndDeveloper• Drupalthemingandmoduledevelopment• [email protected]

GeneMohr• BackEndDeveloper• FullstackandDrupalmoduledevelopment• [email protected]


What is search engine and why we use it?

3

SearchengineinDrupal

WhatarethedifferencesfromDrupalSearchAPI?

Whyweuseit?


What is Solr?

4

Solr isasuperfast,opensource,standalonesearchserverbuiltonApacheLucene.Someofthefeaturesare:• AdvancedFull-TextSearch• DesignedforHighVolumetraffic• FacetedSearchandFiltering• Scalable,flexible,extensible• RichDocumentParsing– built-inabilitytoindexPDF’s,Worddocumentsandmore• Multiplesearchindexes• QuerySuggestions,SpellingandMore– advancedcapabilityforauto-complete,spell

checking,highlightingandmore


Apache Solr Configuration

5

• Installmodule• CreateanindexonSolr engine• Createaview• Attachfacetfilters


Install the Drupal modules

6

$composerconfig repositories.drupal composerhttps://packages.drupal.org/8

Downloadthemodule SearchAPISolrSearch.

$composerrequiredrupal/search_api_solr

InDrupal8,ComposerisusedtomanagePHPlibraries.TheSearchAPImoduledoesn’tincludeSolarium,sodependenciesneedtobesetviaComposerwiththefollowingcommands:

Install the following modules:

Optional:installSolr SearchDefaultsmodule


Add Solr server

7

Goto Configuration->Searchandmetadata->SearchAPI.Click“AddServer”andconfiguretheServer.Servernameexample:LocalSolr Server


Confirm Solr server is connected

8

Atthisstage,theserver’sstatuspageshouldshowmessage:"TheSolr servercouldbereached"and"TheSolr corecouldbeaccessed"

IfSolr SearchDefaultsmodulewasinstalled,itcanbeuninstalledasit’snolongerrequired.


Add search index

9

Goto Configuration->Searchandmetadata->SearchAPI.Click“AddIndex”

Selectingthe“Content”datasource,optionsarepresentedtoselectwhichbundlesaretobeindexed


Add fields to index

10

Beforesearchcanbeperformed,selectallthefieldsthatshouldbeavailabletosearch.Thatisconfiguredinthe“Fields”tab.


Add processors to the index

11

Laststepistoaddadditional’processors’.

Thisincludesitemssuchas:• Contentaccess• Ignorecase(case-insensitivesearch)• Tokenizer(splitintoindividualwords)


Verify search index is working

12

Oncefieldsandprocessorsaresetup,goingbacktothe”View”tab,willshowthestatusoftheindex,andatthispoint,thecontentisreadytobeindexedifnotalreadysettoindeximmediatelywhentheindexiscreated.

Indexingofcontentisdoneviacron andanynewcontentwillgetindexedthen.


Create a Faceted Solr Search page

13

Finalstepistocreateasearchpage.ThisisdonebycreatingaViewandcreateaPage.

Generalsettingstoconfigure:ViewSettings:Show>Index(createdearlier)PageSettings:• PageTitle• Path:/search/content• DisplayFormat>Unformattedlistof“RenderedEntity”

Theclick“Saveandedit”toconfiguretheview.1. UnderFormat>Show,click“Settings”forRendered

Entityandchange”viewmode”toTeaser2. UnderFilterCriteria,add“Fulltext search”fieldand

exposethefieldforfiltering

Furthersolr configurationcanbedoneintheSolr schema,suchas:

• Searchingpartsofworks,usingSolr’s EdgeNGramFilterFactory,forexample,”info”willmatch”information”

• LowerCaseTokenizer– removewhitespaceandnon-lettersandconvertallletterstolowercase


Add search facets

14

Withthesearchpagesetupnow,wewanttoaddfacetstoletusersfilterdowncontent.NavigatetoConfiguration>Searchandmetadata>Facetsthenclick“Addfacet”

Facetshaveanumberofsettingstoconfigure:- Widget- Showtheamountofresults- Sorting(bycount,displayvalue…)- Operator(ORandAND- ANDfiltersare

exclusiveandnarrowtheresultset.ORfiltersareinclusiveandwidentheresultset.)

- …


Add search facets

15

Laststep,addthenewlycreatedFacetblocksontheBlockLayoutpage

Note:Facetblocksdon’tneedtohavethepagevisibilityset,itwillautomaticallydetectifthepageasearchpage,otherwise,theywillnotbedisplayed


Search Page

16

Finally,viewthesearchpagecreated


What is Elasticsearch?

17

• Alsocapableofprovidingadvancedfull-textsearch• Easiertoinstall• RESTbasedAPIs,asimpleHTTPinterface,andusesschema-freeJSON• DistributedDocumentStore- easytouseJSONdocument-orientedstorage

platform• Loggingdataandanalysis• Visualizationandreal-timedatamonitoring whencoupleofKibana


Elasticsearch Configuration

18

• Installmodule• CreateanindexonElasticsearch engine• Createaview• Attachfacetfilters


Install the Drupal modules

19

$composerrequirenodespark/des-connector:5.x-dev$composerupdate

TheSearchAPImoduleneeds Elasticsearch PHPlibrarywhichprovidestheabstractlayerofElasticsearchConnectormoduleinDrupal. Thiscanbeinstalledthroughcomposer.

Install the following modules:

NOTE:Elasticsearch doesn’thaveSolr SearchDefaults-likemodule.Settingupandcreatingaviewpagehastobedonemanually.


Add Elasticsearch

20

Goto Configuration>Searchandmetadata>Elasticsearch Connector.

Click“AddCluster”andconfiguretheServer.


Add Elasticsearch (continue)

21

Goto Configuration>Searchandmetadata>Elasticsearch Connector.Click“AddCluster”andconfiguretheserver.

Asdefault,itis“elasticsearch.”Ifyouwanttoeditthecluster/nodeinformation,editelasticsearch.ymlfile.


Add search index (same as Solr)

22

Goto Configuration>Searchandmetadata>SearchAPI.Click“AddIndex”

Selectingthe“Content”datasource,optionsarepresentedtoselectwhichbundlesaretobeindexed


Add fields to index (same as Solr)

23

Beforesearchcanbeperformed,selectallthefieldsthatshouldbeavailabletosearch.Thatisconfiguredinthe“Fields”tab.


Add processors to the index (same as Solr)

24

Laststepistoaddadditional’processors’.

Thisincludesitemssuchas:• Contentaccess• Ignorecase(case-insensitivesearch)• Tokenizer(splitintoindividualwords)


Verify search index is working (same as Solr)

25

Oncefieldsandprocessorsaresetup,gobacktothe”View”tab.Itwillshowthestatusoftheindex,andatthispoint,thecontentisreadytobeindexedifnotalreadysettoindeximmediatelywhentheindexiscreated.

Indexingofcontentisdoneviacron andanynewcontentwillgetindexedthen.


Create a view page

26

1.GotoStructure>Addview

2.Provideaviewnameandselectyourindexnameastheviewsource


Create a view page (continue)

27

3.UnderFormat>Show,selectRenderedEntity

Or,youcanselect“Fields”andaddeachfieldyouwouldliketodisplayintheFieldssection.

4.UnderFilterCriteria,add“Fulltext search”fieldandexposethefieldforfiltering

5.AddSortCriteria:Thebestonetouseis“Relevance(desc)”


Add search facets (same as Solr)

28

Withthesearchpagesetupnow,wewanttoaddfacetstoletusersfilterdowncontent.NavigatetoConfiguration>Searchandmetadata>Facetsthenclick“Addfacet”

Facetshaveanumberofsettingstoconfigure:• Widget• Showtheamountofresults• Sorting(bycount,displayvalue…)• Operator(ORandAND- ANDfiltersare

exclusiveandnarrowtheresultset.ORfiltersareinclusiveandwidentheresultset.)


Place facet blocks (same as Solr)

29

Laststep,placethenewlycreatedFacetblocksontheBlockLayoutpage

Note:Facetblocksdon’tneedtohavethepagevisibilityset,itwillautomaticallydetectEachofthepageasearchpage,otherwise,theywillnotbedisplayed.


Search Page

30

Finally,viewthesearchpagecreated.

Improving Drupal search experience with Apache Solr and Elasticsearch 31

Apache Solr or Elasticsearch?

Source:AscenefromAnnieGetYourGun(1950)


When to use Apache Solr

32

• Iftherearefewthousandnodesofcontent(ormore,dependingonserverperformanceandwhetherhostedlocallyorexternally)

• Youdonotwanttoputstressonthedatabase

• Ifsearchingisamajorfeatureofthesite

• Ifyouneedtosearchmultiplefields


When to use Elasticsearch

33

• TheElasticStack(Elasticsearch,Logstash,andKibana)caninteractivelysearch,discover,andanalyzetogaininsightsthatimprovetheanalysisoftime-seriesdata.

• Noneedforupfrontschemadefinition.Schemacanbedefinedpertypeforcustomizationofindexingprocess.

• Hasanedgeinthecloudenvironment- thisisdependuponSolrCloudadvancement.

• Hasadvantagesofsearchforenterpriseorhigher-ed levelwhereanalyticsplaysabiggerrole.


Additional resources

34

Solr:ConfiguringtheSearchAPIserver(D8)https://www.drupal.org/node/2763137

Drupal8andSolr:Google-fastsearchonyourownwebsite|Whyandhowhttps://blog.openlucius.com/en/blog/drupal-8-and-solr-google-fast-search-your-own-website-why-and-how

SetupfacetedApacheSolr searchonDrupal8https://www.jeffgeerling.com/blog/2016/set-faceted-apache-solr-search-on-drupal-8-2016-deprecated

WhentoconsiderApacheSolrhttps://lastcallmedia.com/blog/when-consider-apache-solr

Elasticsearch:IndexingcontentfromDrupal8usingElasticsearchhttps://www.lullabot.com/articles/indexing-content-from-drupal-8-to-elasticsearch

HowtoindexAttachmentsandFilestoElasticsearchhttps://qbox.io/blog/index-attachments-files-elasticsearch-mapper

---------------

Solr vs.Elasticsearch:5FactorstoConsiderhttps://www.cmswire.com/information-management/solr-vs-elasticsearch-5-factors-to-consider/


Questions?

35

improving drupal search experience with apache solr and ... · improving drupal search experience...

Documents