getting started with lucidworks enterprise · 2019-06-14 · getting started with lucidworks...

Getting Started With LucidWorks Enterprise A Lucid Imagination Technical White Paper

Getting Started With LucidWorks Enterprise A Lucid Imagination Technical White Paper • October 2010 Page i

© 2010 by Lucid Imagination, Inc. under the terms of Creative Commons license, as detailed at http://www.lucidimagination.com/Copyrights-and-Disclaimers/. Version 1.6, published 17 December 2010. Solr, Lucene, and their logos are trademarks of the Apache Software Foundation.

http://www.lucidimagination.com/Copyrights-and-Disclaimers/�

http://www.apache.org/�

Getting Started With LucidWorks Enterprise A Lucid Imagination Technical White Paper • October 2010 Page ii

Abstract

LucidWorks Enterprise is the search solution development platform built on the power of Apache Solr/Lucene technology, developed by the enterprise search experts at Lucid Imagination. LucidWorks Enterprise leverages the disruptive innovation of the leading open source search technology to deliver unmatched scalability to billions of documents, with subsecond query and faceting response time. By building and expanding the scalable power of Solr open source technology with vital new features, the search experts at Lucid Imagination have created an integrated platform that simplifies and empowers predictable, reliable search application development.

This document is intended to provide you with a basic working knowledge of the LucidWorks Enterprise search development platform. It provides you with an overview of the software’s functions and an explanation of how to use them from the provided user interface, as opposed to from a programming perspective. You will learn about installation, indexing content from local files, web sites, and databases, and searching, as well as improving the user experience using features such as user alerts, auto-complete and spell-check. This document does not require any previous programming experience.

Getting Started With LucidWorks Enterprise A Lucid Imagination Technical White Paper • October 2010 Page iii

Table of Contents Introduction ............................................................................................................................................................ 2

What You’ll Learn in this Document .......................................................................................................... 2

What This Document Won’t Teach You .................................................................................................... 3

How LucidWorks Enterprise Works .......................................................................................................... 3

Installation ............................................................................................................................................................... 4

Using the Installation Wizard ....................................................................................................................... 4

Installing Via Command Line ........................................................................................................................ 6

Testing the Installation ................................................................................................................................... 7

Basic searching ................................................................................................................................................... 12

Understanding Search Queries ................................................................................................................. 12

Searching individual fields .................................................................................................................... 14

Range queries ............................................................................................................................................. 14

Faceted Searching .......................................................................................................................................... 14

Improving search coverage ............................................................................................................................ 17

Understanding Fields ................................................................................................................................... 17

Indexing the Local Filesystem ................................................................................................................... 20

Indexing HTTP Documents ........................................................................................................................ 21

Indexing Database Records ........................................................................................................................ 23

Indexing Solr Documents ............................................................................................................................ 25

Scheduling Tasks ............................................................................................................................................ 26

Deleting Data Sources .................................................................................................................................. 27

Organizing your data ........................................................................................................................................ 28

Understanding collections .......................................................................................................................... 29

Who sees what? Using search filters ..................................................................................................... 30

Improving the Search Experience ............................................................................................................... 31

User Alerts ........................................................................................................................................................ 31

Getting Started With LucidWorks Enterprise A Lucid Imagination Technical White Paper • December 2010 Page 1

Helping Users Create Their Queries ....................................................................................................... 33

Auto-complete ............................................................................................................................................ 33

Spell-checking ............................................................................................................................................. 34

Find Similar links ....................................................................................................................................... 34

Enabling These Functions ...................................................................................................................... 35

Specifying Fields ........................................................................................................................................ 36

Indexing ........................................................................................................................................................ 37

Improving Relevance ........................................................................................................................................ 37

Synonyms .......................................................................................................................................................... 38

Stopwords ......................................................................................................................................................... 38

Click Scoring ..................................................................................................................................................... 40

Summary ............................................................................................................................................................... 41

Next Steps ............................................................................................................................................................. 42

Next Steps ............................................................................................................................................................. 42


Introduction Welcome to LucidWorks Enterprise, the search platform that takes the power of Solr/Lucene and delivers it to you in one convenient, supported package. This document will take you from the very beginning of installing the software down through some of the things you’ll need to know to make the most of it.

LucidWorks Enterprise has been designed to provide you with the search capabilities and benefits of Solr while still providing the ease of use you need to work efficiently in an environment in which data is everywhere, and you need to get a handle on it. While it does provide some great opportunities for programmers to take control and build powerful search applications using those capabilities, it’s also been designed to take much of the pain out of using such a complex system.

As such, many of the things you can do with LucidWorks Enterprise can be accomplished without any programming at all. The administrative user interface provides a way to index documents for searching, make queries, and even learn about how your system is being used. It also provides a web interface to most of the functions you’ll need to run your system.

What You’ll Learn in this Document This document is meant to give you a running start on getting the most out of LucidWorks Enterprise. It teaches you about:

• How LucidWorks Enterprise works

• Installing the software and indexing your first test collection

• Searching for data, and how to make the most of the individual fields you’ve indexed

• How to use faceted searches to filter results

• How to index local files (such as word processing documents), web pages (such as those on local or remote web sites), and even databases

• How to tell LucidWorks about specific attributes of your data

• How to configure user-friendly features such as user alerts for new content, auto-complete, spell-check, and the ability to find related results that don’t necessarily contain the specified search term


• How to improve the quality of search results by specifying synonyms and stop words, common words that should be left out of a search

• How to provide more relevant search results by analyzing the behavior of your users

What This Document Won’t Teach You This document is intended to get you started using LucidWorks Enterprise; it’s not a programmer’s guide. In addition to numerous ReST-based APIs, LucidWorks Enterprise gives you complete access to the source code underlying the platform, so you have full control over query handling, results relevancy, and other factors. For more information on making use of these capabilities, see the product documentation.

How LucidWorks Enterprise Works LucidWorks Enterprise works by “indexing” data, or breaking it down into individual words or terms, each of which is assigned to a “field”, against which you can later search. A collection of fields is considered a “document”. For example, a PDF on your hard drive might have fields for “author”, “title”, and “text”, and these three fields make up the document.

Because you can control not only the names of these fields, but also how they’re treated by the indexer and the query engine, you have a great deal of flexibility when it comes to indexing your data. For example, you might index product data out of your database, and specify that you want the title and description to be indexed (searchable) and that the id column is to be treated as a unique key, so that if you update the database, each product “document” in the index can be updated appropriately.

Once you’ve indexed your data, it’s ready for searching. The query parser takes the user’s request and compares it to the data stored in the index. If it finds a relevant match (or matches) it then returns information about all of the matching documents.

LucidWorks Enterprise provides a convenient web-based interface for indexing content, and for controlling the types of information to be returned for each document that satisfies a query. You can then build upon the application to decide how to use that data. That said, the LucidWorks Enterprise interface provides everything you need to see the results of a query, with no programming required.


But first, you’ll need to install the software.

Installation LucidWorks Enterprise 1.6 is available for download from http://www.lucidimagination.com/lwe/download, as both a Windows-based installer and a cross-platform *.jar file.

LucidWorks Enterprise provides both a graphical and a command-line option for installation. Both provide the same options, but the command-line version can be used for systems where a graphical user interface isn’t an option. Note that this document is meant to be a guide, and not a comprehensive look at installation. If you need more details, please search the downloaded documentation for “Installation”.

Using the Installation Wizard To install LucidWorks Enterprise using the GUI, perform the following steps:

1) Make sure that you have Java 1.6 (JDK or JRE) installed on your machine. You can download Java from http://www.oracle.com/technetwork/java/javase/downloads/index.html.

2) Double-click the installer to start it. If you are using the *.jar file and double-clicking doesn’t start the installer, go to the command line and type java -jar lucidworks-enterprise-installer-1.6.jar

(Make sure to use the correct filename.)

3) Click Next to go to the system requirements. The typical desktop machine will handle small-scale data collections -- less than 100,000 documents, depending on your specific hardware and existing software, as well as how you intend to use the data. You will need 8GB to 16GB for a large scale deployment. LucidWorks Enterprise runs on Windows (XP or higher), Linux (kernel 2.4 or higher) and MacOS (10.5 or higher).

http://www.lucidimagination.com/lwe/download�

http://www.oracle.com/technetwork/java/javase/downloads/index.html�


4) Click Next. Read the user agreement, make sure it doesn’t contain anything objectionable, and click “I accept the terms of this license agreement”. Click Next again.

5) Choose which components you want to install on this machine, and the addresses from which you want to access them.

The default is to install and activate all components on the local server using ports 8888 and 8989, and for beginning purposes this is just fine. If this configuration conflicts with existing applications, however, feel free to change the port numbers. Keep in mind, however, that when they are installed during the same session, the SearchUI, AdminUI, and Alerts need to use the same port. If you are installing components on different machines or ports, be sure to alter the addresses to point to the appropriate server and port. Click Next to continue.


6) Select the installation path and click Next. (If you’re reinstalling the software, make sure that the previous installation directory has been deleted.) Click OK to let the installer create the new directory.

7) Review the selected options and click Next to install the software. Depending on your system, this may take several minutes. When the progress bar shows that the installation is complete, click Next.

8) LucidWorks Enterprise takes a bit longer than usual to start for the first time, so make sure that Start LucidWorks Enterprise is checked, and then click Next to continue with the installation.

9) Click the Next button when it becomes available.

10) Decide whether to create shortcuts for other users, and where to place them, and then click Next.

11) If all has gone well, you will see a screen telling you LucidWorks Enterprise has been installed, and offering the opportunity to create an installation script. If you are installing on multiple computers, this script will simplify the process by pre-filling the values you chose for this installation. Click Done to dismiss the installer.

Installing Via Command Line The process for installing via the command line is virtually identical to installing via the graphical interface. Follow these steps:

1) Make sure that you have Java 1.6 (JDK or JRE) installed on your machine. You can download Java from http://www.oracle.com/technetwork/java/javase/downloads/index.html.

2) In the directory in which the *.jar file is located, execute the following command: java -jar lucidworks-enterprise-installer.1.6.jar --console

(Again, make sure to use the correct file name.)

3) Follow the steps presented in the installer. For information on specific steps, see the instructions above.

http://www.oracle.com/technetwork/java/javase/downloads/index.html�


Testing the Installation You can test the new installation by pulling up the administration user interface in your browser. To do that, go to:

http://localhost:8989

If everything installed and started properly, you will see the generic search page:

Of course, at this point, you don’t actually have any data indexed, so there’s no point searching. To index data, you will need to log into the admin UI, so click the “login” link at the top of the page. You can find the username and password for the administration user in the README document. (This user is pre-installed; before going to production, you will want to either use the User API or an LDAP directory to manage your users.)

Logging in will bring you to the Quick Start page, where you can index content. Just to have something to search against, click Local Filesystem and enter the path for a small-ish directory of documents.

http://localhost:8989/�


Click Continue.

For now, choose to index this content “immediately”.


Click Finish to go to the Dashboard Summary page.


This page shows you what’s been going on with your system. In this case, we haven’t had any queries, so we can just see the number of documents that have been indexed. This page updates automatically, so by watching it, you’ll know when indexing has been completed. The final data will show under “Recently Completed”.


Finally, we have data to search! Click the Search tab to go to the search page.

Notice that information about the data you’ve indexed already appears on the search page. In this case, you can see document authors, the data source, and the types of documents, along with keywords pulled from your documents. (The more documents you have, the more representative these keywords are likely to be. In this case, each of those words appears only once.) These notations are called facets, and can be used to narrow results. (We’ll talk more about facets later.)

Enter a search term and click Search to see the results.


Now that we know everything’s working properly, we can talk about searching itself.

Basic searching On the surface, searching is pretty simple. Enter a keyword, press “search”, and get results. And that’s true. It is that simple. But it’s also powerful, in that you have the ability to get more out of your searching than a simple keyword search. In this section, we’ll discuss how to get more out of your searches.

Understanding Search Queries The simplest search query involves just a keyword or phrase, such as when I enter “Lucid” in the search box:


We can also combine terms into a single query. For example, I can find documents that contain the terms “indexing” and “delete” with

indexing AND delete

LucidWorks Enterprise provides a default AND operator unless you specify something else, such as

indexing OR delete

which finds documents that contain either term.

In fact, LucidWorks Enterprise includes support for a whole range of operators, including comparative operators such as < and proximity operators such as NEAR, BEFORE, or AFTER. See the LucidWorks Enterprise User’s Guide for the list.


Searching individual fields Sometimes, however, you want to be more specific. For example, I might want to find all of my PDF files. If I did a search for just

application/pdf

I’d get no results, because that information isn’t stored in the default search field. Instead, I could search for

mimeType:application/pdf

This tells LucidWorks Enterprise to search for documents that have a value of “application/pdf” in the mimeType field.

Range queries You also have the option to search for a range of values. For example, I can find all of the documents in my index that have 50 pages or less with:

pageCount:[0 TO 50]

Or if I wanted to be even more specific, I could find all PDF files in that range: mimeType:application/pdf AND pageCount:[0 TO 50]

One place you often find this kind of query is in faceted searches.

Faceted Searching The use of facets is one of the great recent advances in searching. Search facets enable users to “narrow down” their search by a variety of factors. For example, if we go back to the original keyword search for “Lucid”, you can see a number of different options down the right-hand side of the page:


Notice that each entry includes not just a description of what it is, but also how many relevant documents there are. So I can see that this data source has 2 PDF documents that mention “Lucid”, and one that was authored by Grant Ingersoll. If I wanted to narrow my search to, say, OpenOffice presentations, I could click that link under Type.


This action narrows the list from 5 to just 2 documents. If I wanted to, I could further narrow the list, say, to show only the documents authored by me.

To clear the existing filters and go back to the original search, click the “Clear Filters” button.

Now let’s look at getting more data to search.


Improving search coverage Of course, the quality of search results depends on the quality of data added to the index. If the appropriate information isn’t in the index, even the most carefully constructed query isn’t going to find it.

In this section, we’ll show you how to index both local and remote content so that it can be found by your users.

Understanding Fields The first thing we’ll need to do before doing any indexing is understand just how LucidWorks Enterprise looks at the data we’re putting into it.

Each document is made up of one or more fields. You can see a list of existing fields by clicking the Index tab, and the Fields button in the administration user interface.


If you click a field, you’ll see the option to edit the properties of that field. When you’re just starting out, you’ll want to understand these properties:

• Name: This value is the name by which the field is known, both in the indexing process and in queries

• Field Type: This property determines how the field is handled. For example, in this case, we’re looking at English text (as opposed to German, etc.) rather than a date, a sortable number, and so on.


• Indexed: This property specifies whether the contents of this field are used to determine whether a document matches a particular query.

• Stored: This property determines whether the original value of the field is stored, potentially to be returned as part of a result.

• Multi-valued: This property determines whether a document can have multiple values for this field.

• Field Default: This value determines the value that will be used for the document if no value is given when it’s indexed.

• Search by Default: This property determines whether the field will be used in a search for which the user doesn’t specify a particular field.

• Include in Results: Make sure this option is checked if you want this field to show up in the search results for this document.

• Highlight: This property determines whether the given term will be shown in context for this field.


• Facet: This property determines whether this field shows up as an available filter on the search page. Note that documents without this field won’t show up in the counts for this facet.

• Use for Deduplication: In all likelihood, you will want to re-index your content as it changes. This setting enables you to determine how LucidWorks Enterprise knows this is the “same” document. For a product, it might be a product number. For HTTP data sources, it might be the URL.

You can also delete and add fields from this interface.

Indexing the Local Filesystem Even if you haven’t given any thought to what documents you’d like to search, you likely have a ready source of material right on your hard drive. To create a data source from local files, click the Index tab, then the Sources button, and the FileSystem button.

Enter a user-friendly name for your new data source and the full directory path in which the documents are stored. You have the option to drill down into subdirectories or not, as well as to follow symbolic links or not.


(Note that for security reasons, data indexed from the local filesystem will not automatically be available via a link from the search results. Search the product documentation for “linking” for information on how to configure LucidWorks Enterprise to activate those links.)

Click Create to create the data source.

Note that this process does not start the indexer; we’ll look at that under Scheduling in a moment.

Indexing HTTP Documents Another option is to index web documents. You might want to index the contents of your local intranet, or perhaps you have a repository of content that’s currently available via the browser. You can also use it to monitor external web sites. To set up a data source of web content, click the Index tab, then Sources, and Web.


The Name should be something you’ll recognize later, and the URL is the value at which you want the crawler to start.

The Allow Paths and Disallow Paths values enable you to control where the crawler goes. For example, if I were to index my own site, as I’m doing here, I might want only my own content, so I’ve specified only paths that start with my URL, using a regular expression to specify the rest of the path. Similarly, I might not want to index my administration pages.

You can use Disallow Paths to allow the crawler to follow links to external sites, but avoid, say, indexing tens of thousands of tweets.

The Crawl Depth specifies how far down the crawler will go. A depth of 0 crawls only the specified URL; to get only that page and the pages linked directly by it, specify a depth of 1.




Indexing Database Records Another fertile area for data indexing is the database. Here the effort required is a little greater, but so are the potential rewards. Before you index any database content, however, there are two tasks you must accomplish:

1) Determine the fields you’re going to be indexing from your database. It’s unlikely that the fields with which LucidWorks Enterprise is preconfigured will match your column names exactly, and unless you map those columns to existing fields, all of your data will wind up in the text_all field, making it difficult to search.

Fortunately, LucidWorks Enterprise gives you complete and easy control over field definitions. Use the Fields button to create any necessary fields.

2) Make sure the appropriate JDBC driver is available to LucidWorks Enterprise. To see the installed drivers, click the Index tab and the JDBC Drivers button. LucidWorks Enterprise ships with the sun.jdbc.odbc.JdbcOdbcDriver. To add a new driver, click Add Driver and use the form to upload the *.jar file.


Once you’ve accomplished these two steps, you’re ready to create the new data source. Click the Index tab, then Sources, and DB.

As usual, choose and enter a recognizable name for the data source, and enter the JDBC URL, minus any authentication information. For example:

jdbc:mysql://127.0.0.1/productDB

Select the appropriate JDBC driver. This is the actual class name you would use in a Java application, such as com.mysql.jdbc.Driver. Enter the username and password for the database.


Finally, enter the query used to extract the data from the database, mapping each column to a LucidWorks Enterprise field. For example:

select id, product_name, price, description, last_modified from products

In all likelihood, you will have information about a single item, or “document”, in several tables; to add it to your index, create the appropriate joins to add all of your data as a single query.

In some cases, you will have structures that can’t be added with a single query. In that case, you’ll want to add a Nested Query to define the relationship between entities. You may also want to define the criteria for Incremental Indexing. In other words, this query tells LucidWorks Enterprise how to know that data is “new”.



Indexing Solr Documents One final method of indexing data involves adding it directly using Solr’s native document format. To do that, you will need a Solr document, which is just an XML document, such as:

<add> <doc> <field name='id'>prod3_0</field> <field name='data_source'>Auxiliary Data</field> <field name='itemId_s'>3</field> <field name='itemType_s'>product</field> <field name='cat'>Printers</field> <field name='name_t'>Dokad SPE 3299 Printer</field> <field name='price_td'>99</field> <field name='blurb_t'>A great printer that doesn't use a lot of ink.</field> <field name='description_t'>Sed ut perspiciatis ...</field> <field name='text'>Dokad SPE 3299 Printer99A great printer that doesn't use a lot of ink.Sed ut ...</field> </doc> <doc> <field name='id'>prod4_0</field> <field name='data_source'>Auxiliary Data</field> <field name='itemId_s'>4</field> <field name='itemType_s'>product</field> <field name='cat'>Cameras</field>


<field name='cat'>Accessories</field> <field name='name_t'>Kinok UltraCam II</field> <field name='price_td'>550</field> <field name='blurb_t'>Boy, the Kinok UltraCam II is a great camera, and it hooks up to your printer terrifically.</field> <field name='description_t'>Lorem ipsum dolor sit amet, consectetur adipisicing elit, ...</field> <field name='text'>Kinok UltraCam II550Boy, the Kinok UltraCam II is a great camera, and it hooks up to your printer terrifically.Lorem ipsum dolor sit amet, ...</field> </doc>

</add>

To index this type of document, click the Index tab, then Sources, and Solr.

Enter a recognizable name, as well as the path to the actual document, or to a directory of Solr documents. Click Create to create the data source.

Note that this process does not start the indexer; we’ll look at that under Scheduling right now.

Scheduling Tasks We’re finally ready to look at scheduling the indexing of our data sources. Under the Index tab, click the Schedules button.


Click the icon next to DataSources to expand that option, and click the data source you want to index to highlight it.

You have the option to start indexing in a specific amount of time (such as 0 seconds to start immediately) or at a specific time on a specific day, or to tell LucidWorks Enterprise to start indexing all of your data sources immediately.

Also, if your data is likely to change, you can specify the frequency with which you want to reindex it.

You can also deactivate an index if you’d like to stop the indexing process. Deactivating an index won’t make the data unavailable, however. To do that, you’ll need to delete it altogether.

Deleting Data Sources Deleting a data source is a pretty straightforward process. Click the Index tab, and then the Sources button. Under the list of data source types, you’ll find a Delete button.


To delete a data source, highlight it and press the Delete button. Note that there is no confirmation dialog. Once you click Delete, it’s gone.

Sort of. While it will no longer be updated, the data indexed as part of that data source is still in the index, and queries will still return it. To get rid of it altogether, you will need to call the underlying engine directly through the ReST API. You can find information on how to do this in the documentation.

Organizing your data So far we’ve added just a few dozen documents, but in a real environment you may have thousands, or hundreds of thousands, or even millions of documents, some of which may have different structures or requirements or be used by different groups of people.

To solve this problem, LucidWorks Enterprise enables you to create “collections”, or groups of documents that can be managed and searched independently of each other.


Understanding collections At its heart, everything LucidWorks Enterprise does is done by collection; when you first install the software, it creates a default collection, collection1. When you add data, you’re adding it to that collection. When you search, you’re searching that collection.

But what if that’s not what you want? What if each one of your departments has its own data, or Administration has different requirements than Production, or the Compliance department needs all of their data searchable, but separate from the rest of the company? You might have documents in different languages for offices in different countries. These are just a few situations in which you might need to segregate your data into collections, each with its own list of fields, data sources, and searchable data.

To create a new collection, click the Collections tab. From here you can click Add Collection.

As you can see, for each collection, you have shortcuts to essential functions such as the Dashboard, where you can view statistics about how the server is running, Indexing, Data Sources, and so on. To add data for a particular collection, you can click Sources and add a new Data Source, just as we did before.

If you click the Search tab, you’ll see that you can now choose which collection to search:


So collections enable you to create separation between your actual data and how it’s treated. You also have the option to separate data based on users.

Who sees what? Using search filters One common requirement in building a large repository of information is the need to determine which users have access to what content. LucidWorks Enterprise includes the concept of search filters that enable you to limit your users’ access based on their identity.

To see existing filters or add a new one, click the Queries tab and the Search Filters button.

In this case, we’re adding a new filter called Statistics that tells LucidWorks Enterprise that Sam, Gina, and all of Accounting should only see documents that have a subject_matter value of statistics or predictions.


The DEFAULT search filter enables the admin user to see all documents.

To add a filter for a particular collection, click the Collections tab and the Filters shortcut for the appropriate collection.

(LucidWorks Enterprise integrates with your existing LDAP directory, or you can manage your users programmatically using the Users API.)

Now that we’ve looked at putting data in, let’s look at enhancing your users’ search experience when getting it back out again.

Improving the Search Experience LucidWorks Enterprise builds on the rich ecosystem that includes the Solr and Lucene, on which it is built. That means that you have access to all of the best bells and whistles available with Solr, plus even more, right at your fingertips. In this section, we’ll look at how to configure some of the most useful.

User Alerts When you’re dealing with huge amounts of data, one of the biggest challenges is keeping up with it as it grows. One way to do that is to use user alerts, which notify you when new content matching your queries has been added to the system.

To set up a user alert for a query, click the “Add this query as alert” link under the search box.

This button takes you to a page where you can specify the details of where you’d like to receive the alert.


The alerts arrive at the specified email address using the Name of the alert as the subject line, so you can specify it in a way that works with your mail filters.

You can also specify how often to check for new data, with the option to limit how often it actually sends you data.

Now, all that said, by default, email alerts are not enabled when LucidWorks Enterprise ships, because they require administrator configuration. To enable them, edit the file <LWE_HOME>/rails/config/alerts.yml

to include the appropriate SMTP information.


Even without additional configuration, however, you can still see the results of an alert. To do that, save the alert to add it to the list provided when you click the View saved alerts button under the search box.

By clicking the Preview link for the alert, you can see the latest results for this particular query.

Helping Users Create Their Queries Several of the functions available in LucidWorks Enterprise can help you help your users by providing guidance on what they should be searching for. These include auto-complete, spell-checking, and “find similar” links.

Auto-complete One of the best ways to make sure that users don’t wind up with “no results” is to guide them towards terms that actually exist within the index. And one of the best ways to do that is using auto-complete functionality.

Auto-complete looks at the characters the user has already entered and offers terms that start with those characters.


Spell-checking You can also check the spelling of the terms the user has entered against the existing index of terms, and offer suggestions. For example, if you were to search for “printe”, the system might suggest “printer” based on the content you have indexed.

Find Similar links The “find similar” functionality helps users by finding content they may not have known they were looking for. For example, a search for “sports” might find a document that contains the word “basketball”, even if the document doesn’t include the word “sports” at all.

If a similar result is available, the link appears under the existing result.


In order for these three functions to work, you’ll need to make sure of three things:

1) Make sure auto-complete and/or spell checking are enabled.

2) Make sure that at least one field is specified as a source for these terms.

3) For spell-checking and auto-complete, you’ll need to make sure that indexing for these terms has been performed.

Enabling These Functions To enable auto-complete, spell-checking, or find similar, click the Queries tab and the Settings button. Click the Search Settings item to highlight it.


Make sure that Enable auto-complete, Enable spell checking, and/or Show “find similar” links are checked, and click Save Settings.

Specifying Fields To specify one or more fields for these functions, click the Index tab and the Fields button. Highlight the relevant field.


Make sure that Index for Spell Checking , Index for Auto-complete, and/or Use in “Find Similar” are checked, and click Save Settings.

Indexing Finally, make sure that the spell-checking and/or auto-complete information has been indexed. To do that, click the Index tab and the Schedules button. Click the icon next to Activities to expand it and highlight spelling or auto-complete. Schedule these indexes just as you would schedule your data sources.

Now that we’ve got good queries, it’s time to make sure they return good results.

Improving Relevance One of the advantages of using a search platform such as LucidWorks Enterprise is that results can be ranked by relevance, with the results most likely to be what the user is looking for at the top of the list. Out of the box, LucidWorks Enterprise does a pretty good job, but there are ways that you can help LWE improve relevancy even more by including input from the best computer out there: the human brain.


Synonyms One way to provide better results is to provide LucidWorks Enterprise with groupings of words that have the same (or at least similar) meanings. For example, a search for “lawyer” should probably also find documents that only contain “attorney”. Most industries and subject areas have their own set of jargon and synonyms, and you can configure them directly from within the administration user interface.

Click the Queries tab, and then the Settings button. Highlight Synonyms and Stopwords, and then expand the Synonyms entry.

From here, you can add new entries or remove existing entries. Each line is considered a group; you can add as many comma-delimited terms as you like.

Click Save Settings when you’re finished.

Stopwords


In search parlance, a “stopword” is a word that’s so common that adding it to a query rarely increases the quality of results, and frequently decreases it. For example, if you did a search for “the City of Chicago”, “Chicago” would certainly provide good results. “City” might as well. But how many billions of documents that have nothing to do with Chicago contain the words “the” and “of”?

Fortunately, LucidWorks Enterprise understands the concepts of stopwords, and in most cases, will eliminate them from your query. It also understands how to handle stopwords on the back end so that they help improve relevance (for example, by judging the proximity of two words) rather than hinder it.

LucidWorks Enterprise starts with a list of several dozen stop words, such as “a”, “and”, “for”, and so on. You may find, however, that you need to add your own. To do that, click the Queries tab and the Settings button. Highlight Synonyms and Stopwords, and expand the Stopwords entry.

As with synonyms, you can use this interface to add, edit, or delete stopwords.


Click Scoring Perhaps the best way for LucidWorks Enterprise to know whether a result is really relevant for a particular search query is to keep track of whether a human thinks it is. Click scoring makes that happen.

When you enable click scoring, LucidWorks Enterprise tracks which results are most often clicked for a particular query, and “boosts” their relevance scores accordingly. It will then be more likely to present those results higher in the list for that query.

To enable Click Scoring, Click the Index tab and the Settings sub-tab, then highlight Click Scoring. Make sure that “User Click Scoring data in relevance ranking” is checked, then click Save Settings.

Once Click Scoring is enabled, you’ll need to make sure that the click scoring data is up to date. To do that, click the Schedules button and highlight click.


Just as you would for other features such as spell-check and autocomplete, make sure that the click scoring index is run periodically to ensure that your users get the best results.

Summary By providing a search development platform with a fast, flexible architecture built on open source, LucidWorks Enterprise harnesses the power of Solr/Lucene in a convenient, well-curated package, while sparing you the programming pain that would otherwise be required to get a basic system up and running.

In this document, we showed how to install a single-server instance of LucidWorks Enterprise, and how to index local, HTTP, and database content. We also looked at the basic concepts involved in performing search queries.

We then covered some of the bells and whistles that are available to make your life, and the lives of your users, easer, and how to configure them.

You should now have a fully-functioning search platform, ready for data and customization.


Next Steps

Next Steps For more information on how Lucid Imagination can help search application developers, employees, customers, and partners find the information they need, please visit www.lucidimagination.com to access blog posts, articles, and reviews of dozens of successful implementations.

LucidWorks Enterprise is the search solution development platform built on the power of Apache Solr/Lucene, developed by the enterprise search experts at Lucid Imagination. LucidWorks Enterprise leverages the disruptive innovation of the leading open source search technology, to deliver unmatched scalability to billions of documents, with subsecond query and faceting response time.

By building and extending the scalable power of Solr open source with vital new features, the search experts at Lucid Imagination have created an integrated platform that simplifies and empowers predictable, reliable search application development.

With LucidWorks Enterprise, you get the search results that matter – with a dynamic, tight between fit your business goals, processes, and the valuable information your customers and users need – today and as your search and business grow.

Or call: 1.650.353.4057

http://www.lucidimagination.com/�