haystack training

Getting The MostOut Of Haystack

Daniel LindsleyPragmatic Badger, LLC

Tuesday, December 28, 2010

Terminology

“Engine”• The actual search engine

• Here be interesting computer science problems

• Examples: Solr, Xapian, Whoosh

“Document”• A single record in the index

• Usually accompanied by 1+ fields of metadata

• Heavily processed

“Corpus”• The collection of indexed documents

• Latin for “body”

“Stemming”• Find the root of the word

• Part of the “magic” of search

• More on this later...

“Relevance”• A metric of how well a document matches

the query

• Search’s killer feature

• Hard to get 100% right

“Faceting”• Count of docs meeting certain criteria

within your result set

• Drill down!

• Think Amazon/eBay

• More on this later...

“Boost”• A way to artificially increase the relevance of

document

• Types: Document/Field/Term

Introduction toSearch

Search != RDBMS• The sooner you get over that, the easier

everything that follows will be.

• Think “document store”.

Stemming• Porter-Stemmer or Snowball

• The engine takes terms & hacks them down to the root word.

• Examples:

“testing” ! “test”

“searchers” ! “searcher” ! “search”

Inverted Index• The power of the engine starts here

• Basically a reverse mapping between the stemmed form of a term to a collection of documents containing the term

...“search”: [3, 104, 238],...

Inverted Index• Very fast lookups

• NOT a “contains” or “like” lookup unless you say so (slower)

Document Store• Flat structure

• Generally free-form/schema-less

• Easiest to think about each record as a dictionary

• No relations built-in

Why custom search?...or...

“Isn’t this what Google is for?”

Why custom search?• You control what is (and is not) indexed

• Better quality data goes into the index

• Information-specific handling

• Provide context-specific search

Introduction toHaystack

What is Haystack?At its simplest, Haystack is an abstraction layer for integrating Django with a search engine.

Why Haystack?• Familiar API

• Declarative

• “Looks” like Django

Why Haystack?• Pluggable Backends

• Support Solr & Whoosh out of the box, Xapian with a third-party backend (boo GPL!)

• Your code stays the same regardless of backend.

Why Haystack?• Advanced Features

• Faceting

• More Like This

• Highlighting

• Boost

Why Haystack?• Integration with third-party apps

• No need to fork their code

• Put the indexes in your code & register them

• Applies to django.contrib as well.

Why Haystack?• Real Live Documentation™!

• http://docs.haystacksearch.org/dev/

• Test Coverage!

• Decent coverage

• No new commits without tests

Enough shameless self-promotion

already!

UsingHaystack

Two Phase Approach• The “Data In” is SearchIndex

• The “Data Out” is SearchQuerySet

• Note: There’s a disconnect between your database & the search index

SearchIndex

SearchIndex• Provides the means to get data into the

• Something of a cross between a Form (the data preparation aspects) and Model (the persistence)

SearchIndexfrom haystack import indexes, sitefrom myapp.models import Entry

class EntrySearchIndex(indexes.SearchIndex):text = indexes.CharField(document=True, use_template=True)author = indexes.CharField(model_attr=‘user__username’)created = indexes.DateTimeField()

def get_queryset(self):return Entry.objects.published()

def prepare_created(self, obj):return obj.pub_date or datetime.datetime.now()

site.register(Entry, EntrySearchIndex)

`use_template=True`?• Use Django templates to prep the data

• Example:# search/indexes/myapp/entry_text.txt{{ obj.title }}{{ obj.author.get_full_name }}{{ obj.tease }}{{ obj.content }}

SearchQuerySet

SearchQuerySet• The reason to use Haystack

• Very powerful

• Forget views, forms, etc. They’re all thin wrappers around SearchQuerySet

SearchQuerySet• Fetches data from the index

• Very similar to QuerySet

• Intentional, to reduce conceptual overhead

• Lazily evaluated

• Chain methods

SearchQuerySet• By default, searches across all models

• Can limit using SearchQuerySet.models

• Caches where possible

SearchQuerySet>>> import datetime>>> from haystack.query import SearchQuerySet>>> sqs = SearchQuerySet().models(Entry)>>> sqs = sqs.filter(created__lte=datetime.datetime.now())>>> sqs = sqs.exclude(author=‘daniel’)

# Lazily performed the query when asked for results.>>> sqs[<SearchResult: myapp.entry (pk=u'5')>, <SearchResult: myapp.entry (pk=u'3')>, <SearchResult: myapp.entry (pk=u'2')>]

# Iterable interface.# Still hasn’t hit the DB.>>> [result.author for result in sqs][‘johndoe’, ‘sally1982’, ‘bob_the_third’]

SearchQuerySet# Hits the database once per result.>>> [result.object.user.first_name for result in sqs][‘John’, ‘Sally’, ‘Bob’]

# More efficient loading from database (one query total).>>> [result.object.user.first_name for result in sqs.load_all()][‘John’, ‘Sally’, ‘Bob’]

SearchView

SearchView• Class-based view

• Hit 80% of the regular usage

• A guideline to more advanced use

• Relies heavily on SearchForm

SearchForm

SearchForm• Outside of using SearchQuerySet, it’s a

standard Django form

• Defines a search method that does the necessary actions

SearchFormfrom django import formsfrom haystack.forms import SearchFormfrom myapp.models import Entry

class EntrySearchForm(SearchForm):# Additional fields go here.author = forms.CharField(max_length=255, required=False)

def search(self):sqs = super(EntrySearchForm, self).search()

if self.cleaned_data.get(‘author’):sqs = sqs.filter(author=self.cleaned_data[‘author’])

return sqs

SearchSite

SearchSite• Registry pattern

• Collects all registered SearchIndex classes

• Used by SearchQuerySet to limit results to only things Haystack knows about

• Think django.contrib.admin.site.

HaystackBest Practices

Common Fields• Try to find common fields as much as

possible

• Reuse where it makes sense

• But don’t shoehorn if it doesn’t work

It’s Just Python• When an out-of-box doesn't work for you,

use SearchQuerySet & write what you need.

• It's just Django & Python.

load_all• Appropriate use of

SearchQuerySet.load_all

• One hit to the DB per content type

• But do you need to hit the DB?

haystack training

Documents

haystack observatory - mit

the needle in the haystack (old)

haystack observatory

needle in an enterprise haystack

the haystack method

neroc haystack observatory

mike hecht mit haystack observatory

tow 2007: correlator theory kerry kingham - u.s. naval...

haystack international launch

the haystack product roadmap 2012

a magnet in a haystack

fall 2010 haystack gateway - haystack mountain school of

haystack -...

haystack ultra-wideband satellite imaging radar ...haystack...

haystack adminlaw (2003)

kraantjeswater in vlaanderen haystack international

needle in the haystack

haystack documentation - read the docs · 2019-04-02 ·...

about haystack

haystack sales