building search engine in the social media era
DESCRIPTION
Rising of social networks like Twitter and Facebook has been flooding the web with user-generated contents in the amount that was never seen before. As we become more depending on the social media for daily information gathering and social interaction, we have started to demand more from search engines, such as real time conversations and social connections. Such new environment is forcing search engine technology to evolve. This presentation will first review the traditional search engine technologies and architecture. Then, we will talk about some of the new architectures for supporting real time search and social search. The last part of the presentation is devoted to discussion of the emerging real time web monitoring system, the cousin of search engine. Social media monitoring is predicted to become a mainstream tool for consumers and businesses because the inherent collective intelligece has the potential to further increase the efficiency of finding information and making decision as what search engine has done.TRANSCRIPT
![Page 1: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/1.jpg)
SDForum Event
Building Search Engine in the Social Media Era
twitter: @web2expresshttp://web2express.org
3/24/2010
![Page 2: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/2.jpg)
Agenda
1. The changing search environment 2. Traditional search engine architectures3. New architectures for real time search
and social search4. Semantic search 5. Emerging real time web monitoring6. Q/A
![Page 3: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/3.jpg)
The Changing Search Environment
![Page 4: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/4.jpg)
The Changing Search Environment
![Page 5: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/5.jpg)
The Changing Search Environment
![Page 6: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/6.jpg)
Agenda
1. The changing environment 2. Traditional search engine
technologies and architecture3. New architectures for supporting real
time search and social search4. Semantic search 5. Emerging real time web monitoring
system6. Q/A
![Page 7: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/7.jpg)
Open Source Lucene/Solr
from Grant Ingersoll
![Page 8: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/8.jpg)
Lucene/Solr Search for Web App
![Page 9: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/9.jpg)
Lucence Indexing/Searching
![Page 10: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/10.jpg)
Solr Server Architecture
from Yonik Seeley
![Page 11: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/11.jpg)
Scaling Lucene and Solr
from Mark Miller
![Page 12: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/12.jpg)
Agenda
1. The changing environment 2. Traditional search engine technologies
and architecture3. New architectures for supporting
real time search and social search4. Semantic search 5. Emerging real time web monitoring
system6. Q/A
![Page 13: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/13.jpg)
Near Real Time SearchLucene 2.9: IndexReader (old + new records) <-- IndexWriter Zoie (LinkedIn): Swap Indexes in memory
![Page 14: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/14.jpg)
Google Real-time Search
![Page 15: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/15.jpg)
Google Social Search
![Page 16: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/16.jpg)
Google Aardvark Social Q/A Engine
![Page 17: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/17.jpg)
Agenda
1. The changing environment 2. Traditional search engine technologies
and architecture3. New architectures for supporting real
time search and social search4. Semantic search 5. Emerging real time web monitoring
system6. Q/A
![Page 18: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/18.jpg)
Bing/Powerset NLP Search Engine
from Barney Pell
![Page 19: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/19.jpg)
Sindice: Retrieving semantic data
index API:sindice.com search engine:sig.ma
![Page 20: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/20.jpg)
Google Semantic Search
![Page 21: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/21.jpg)
Hybrid Search Engine
![Page 22: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/22.jpg)
Hybrid (Universal) Search
![Page 23: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/23.jpg)
Agenda
1. The changing environment 2. Traditional search engine technologies
and architecture3. New architectures for supporting real
time search and social search4. Semantic search 5. Emerging real time web
monitoring system6. Q/A
![Page 24: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/24.jpg)
Social Media MonitoringBenefits for businesses:
understand what people are talking about the company, brands, products, and competitions, etc.identify leads for marketing and sales;engage with customer and community conversations;support customers;cultivate product advocates;use customer feedback to improve products and brands;
Differences from search engine:
search engine for filtering real-time text analysis and semantic analysisauto-discovery of trends/patternscollective intelligencerule engine for routing right information to right person
![Page 25: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/25.jpg)
Basic Monitoring
![Page 26: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/26.jpg)
Scaling up Social CRMProblem: Today's social media program does not scale
Solution:Build a pipeline of social context flow and then apply advanced text analytics to automate routing of social contexts.
![Page 27: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/27.jpg)
Intelligent Monitoring Pipeline
![Page 28: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/28.jpg)
Social CRM Integration in the Clouds
![Page 29: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/29.jpg)
Turning CRM into Social CRM
![Page 30: Building Search Engine in the Social Media Era](https://reader035.vdocuments.us/reader035/viewer/2022070315/5550987eb4c90590208b4719/html5/thumbnails/30.jpg)
Agenda
1. The changing environment 2. Traditional search engine technologies
and architecture3. New architectures for supporting real
time search and social search4. Semantic search 5. Emerging real time web monitoring
system6. Q/A