Fusion for data science

Grant Ingersoll CTO Lucidworks

gsingers

Scalable search and analytics in one

Get Started

httpsgithubcomLucidWorksfusion-examplestreemasterfusion-for-datascience-webinar

Get Started

httpsgithubcomLucidWorksfusion-examplestreemasterfusion-for-datascience-webinar

bull Best in breed search solution built on Apache Lucene and Solr

bull Easily capture signals like clicks shares ratings etc and make them actionable

bull Powerful data ingestion and analysis capabilities enabling machine learning recommendations and positive user feedback loops

bull Effortless scale leveraging proven frameworks and algorithms

bull Easy integration with big data tools like Hadoop

Fusion Foundations

Billions of Docs

Optional

REST

Security woven throughout

Prox

yLB

Recs

Worker

Pipes Metrics

NLP Sched

Blobs Msging

Connectors

Worker Cluster Mgr

Spark

Shards Shards

Solr

HD

FS

Shared Config Mgmt

Leader Election

Load Balancing

ZK 1

Zookeeper

ZK N

Signals

Fusion Architecture

Millions of Users

bull Data exploration and visualization

bull Easy Ingestion feature selection and data reduction

bull REST APIs for easy integration with commonly used tools

bull Quick and Dirty classification clustering

bull Powerful and scalable aggregations mathstats framework leveraging Apache Spark

bull Out of the box NLP tools for part of speech sentence detection named entity and more

bull OOTB recommenders plus Mahout extensions

Fusion Data Science Use Cases

Lucene Core search pluggable ranking advanced storage sparse matrix

Solr Faceting function queries basic stats scaling easy setup UIMA basic NLP search clustering

Fusion Pipelines ConnectorsCrawlers DashboardsUI Spark integration advanced stats large scale aggregations

Fusion Standing on the shoulders of giants

Data Exploration Demo

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

Billions of Docs

Optional

REST

Security woven throughout

Prox

yLB

Recs

Worker

Pipes Metrics

NLP Sched

Blobs Msging

Connectors

Worker Cluster Mgr

Spark

Shards Shards

Solr

HD

FS

Shared Config Mgmt

Leader Election

Load Balancing

ZK 1

Zookeeper

ZK N

Signals

Fusion Architecture

Millions of Users

bull Data exploration and visualization

bull Easy Ingestion feature selection and data reduction

bull REST APIs for easy integration with commonly used tools

bull Quick and Dirty classification clustering

bull Powerful and scalable aggregations mathstats framework leveraging Apache Spark

bull Out of the box NLP tools for part of speech sentence detection named entity and more

bull OOTB recommenders plus Mahout extensions

Fusion Data Science Use Cases

Lucene Core search pluggable ranking advanced storage sparse matrix

Solr Faceting function queries basic stats scaling easy setup UIMA basic NLP search clustering

Fusion Pipelines ConnectorsCrawlers DashboardsUI Spark integration advanced stats large scale aggregations

Fusion Standing on the shoulders of giants

Data Exploration Demo

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

bull Data exploration and visualization

bull Easy Ingestion feature selection and data reduction

bull REST APIs for easy integration with commonly used tools

bull Quick and Dirty classification clustering

bull Powerful and scalable aggregations mathstats framework leveraging Apache Spark

bull Out of the box NLP tools for part of speech sentence detection named entity and more

bull OOTB recommenders plus Mahout extensions

Fusion Data Science Use Cases

Lucene Core search pluggable ranking advanced storage sparse matrix

Solr Faceting function queries basic stats scaling easy setup UIMA basic NLP search clustering

Fusion Pipelines ConnectorsCrawlers DashboardsUI Spark integration advanced stats large scale aggregations

Fusion Standing on the shoulders of giants

Data Exploration Demo

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

Lucene Core search pluggable ranking advanced storage sparse matrix

Solr Faceting function queries basic stats scaling easy setup UIMA basic NLP search clustering

Fusion Pipelines ConnectorsCrawlers DashboardsUI Spark integration advanced stats large scale aggregations

Fusion Standing on the shoulders of giants

Data Exploration Demo

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

Data Exploration Demo

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

bull Ingestion

bull 60+ connectors plus easily push data in using REST APIs

bull Feature Selection

bull Analyzers for all types

bull Easily getcalculate weights for terms and attach payloads

bull Term VectorsTerm Dictionary

bull Data Reduction

bull Filters

bull Analyzers

bull Data quality tools

Ingestion Selection Reduction

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

bull Math

bull Search is essentially Vector Matrix

bull Aggregations

bull Enable advanced computation over both core content as well as Fusionrsquos signals

bull Make it easy to try out by leveraging Solr

bull Ship with prebuilt ldquonamedrdquo aggregations to cover common scenarios

Aggregations and Math

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

bull Effortless scale integrated with Fusion and Solr

bull Leverage existing libraries like

bull Mahout

bull Deep Learning 4J

bull GraphX MLLib

bull As easy as

bull binspark start

bull httpaggregatorjobstwitterhashtags_per_authorspark=true

Spark FTW

Aggregations Demo

bull Fusion powers recommendation use cases such as

bull People who bough this bought that

bull Related searches spellings and more

bull Session analysis

bull Fusion ships with several built in recommendation options - Graph and collaborative filtering based approaches

bull Easily enable multi-modal recommendations that combine - Content - Collaborative Filtering - Spatial - HistoricContext

Recommendations

bull Spark

bull APIs for running non-Lucid Spark jobs

bull Integration with 3rd party Spark instances (from major Hadoop distros)

bull Solr RDD extensions for term dictionary term vectors

bull UI for managing Aggregations

bull Full-fledged Graph API

bull More Math matrices functions etc

Whats Next

bull Lucidworks httpwwwlucidworkscom

bull Me grantlucidworkscom

bull Key Docs

bull httpsdocslucidworkscom

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregator+API

bull httpsdocslucidworkscomdisplayfusionAggregator+Functions

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregations+and+Recommendations

Resources

bull Fusion powers recommendation use cases such as

bull People who bough this bought that

bull Related searches spellings and more

bull Session analysis

bull Fusion ships with several built in recommendation options - Graph and collaborative filtering based approaches

bull Easily enable multi-modal recommendations that combine - Content - Collaborative Filtering - Spatial - HistoricContext

Recommendations

bull Spark

bull APIs for running non-Lucid Spark jobs

bull Integration with 3rd party Spark instances (from major Hadoop distros)

bull Solr RDD extensions for term dictionary term vectors

bull UI for managing Aggregations

bull Full-fledged Graph API

bull More Math matrices functions etc

Whats Next

bull Lucidworks httpwwwlucidworkscom

bull Me grantlucidworkscom

bull Key Docs

bull httpsdocslucidworkscom

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregator+API

bull httpsdocslucidworkscomdisplayfusionAggregator+Functions

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregations+and+Recommendations

Resources

bull Spark

bull APIs for running non-Lucid Spark jobs

bull Integration with 3rd party Spark instances (from major Hadoop distros)

bull Solr RDD extensions for term dictionary term vectors

bull UI for managing Aggregations

bull Full-fledged Graph API

bull More Math matrices functions etc

Whats Next

bull Lucidworks httpwwwlucidworkscom

bull Me grantlucidworkscom

bull Key Docs

bull httpsdocslucidworkscom

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregator+API

bull httpsdocslucidworkscomdisplayfusionAggregator+Functions

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregations+and+Recommendations

Resources

bull Lucidworks httpwwwlucidworkscom

bull Me grantlucidworkscom

bull Key Docs

bull httpsdocslucidworkscom

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregator+API

bull httpsdocslucidworkscomdisplayfusionAggregator+Functions

bull httpsdocslucidworkscomdisplayfusionSignals+Aggregations+and+Recommendations

Resources