using html textual and structural data for web image search cheng thao, ethan munson, jim dabrowski,...
DESCRIPTION
Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee. Which image is George Bush or has George Bush?. Which images are similar to this image?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/1.jpg)
Using HTML Textual and Structural Data Using HTML Textual and Structural Data for Web Image Searchfor Web Image Search
Cheng Thao, Ethan Munson,Cheng Thao, Ethan Munson,
Jim Dabrowski, Nikolas D. Bohne
University of Wisconsin-MilwaukeeUniversity of Wisconsin-Milwaukee
![Page 2: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/2.jpg)
Which image is George Bush or has George Bush?
![Page 3: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/3.jpg)
Which images are similar to this image?
![Page 4: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/4.jpg)
![Page 5: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/5.jpg)
<tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bill%20cosby.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Bill Cosby</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/betty%20white.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Betty White</center></FONT></em></td></tr><tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/tom%20brokaw.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Tom Brokaw</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/george%20bush.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Pres. George Bush</center></FONT></em></td></tr><tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/ed%20mcmahon.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Ed McMahon</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bob%20barker.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Bob Barker</center></FONT></em></td></tr>
Does the HTML source tell which images is George Bush?
![Page 6: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/6.jpg)
IntroductionIntroduction
-image search is difficult
- performance is slow
- image identification is a complex, inaccurate task
-most research on image search has emphasized analysis of image content
-few Web image search engines
- commercial: Alta Vista, Google
- research: WebSeek
-little research in textual image search
![Page 7: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/7.jpg)
HTML overview
-HTML document composed of:-head
-title-meta
-body-paragraph, -table,-text,-link,-image, …
![Page 8: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/8.jpg)
<html> <head> <title>Sample HTML</title> <meta keywords="html, html elements"> <meta description="showing a simple html and some html elements"> </head> <body>
<h1>HTML overiew</h1>
<p> first paragraph</p>
<table border="1"> <caption> Simple Table</caption> <tr><td>1<td>2<td>3 <tr><td>4<td>5<td>6 <tr><td>7<td>8<td>9 </table>
Here is a photo of George Bush. <br> <img src="g-bush.jpg">
</body></html>
![Page 9: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/9.jpg)
![Page 10: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/10.jpg)
Previous work - Previous work - Yelena Tsymbalenko
-studied HTML constructs and determine what can be used in image search.
-found the following to be effective- title of the page- image filename- image alt attribute
![Page 11: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/11.jpg)
Research GoalsResearch Goals
-What HTML features make good clues to the content of images?
- Structural features (document, table)
- File names or URLs
- Formatting of material (bold, heading)
- How can clues be combined into a single relevance rating?
![Page 12: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/12.jpg)
Image Search Study ProcessImage Search Study Process
-Downloading pages with matching text- Use existing search engine to identify matches- These pages provide a corpus of images- We download pages so that our corpus remains static
-Download acts as a snap shot
-Clue extraction-Analyze each page in corpus for all possible clues to image content
-Human relevance ratings-human rates if an image is relevant to the query-Statistical analysis to find clue-based relevance functions
![Page 13: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/13.jpg)
Downloading Software
query
URLs
Process: Downloading Web PagesProcess: Downloading Web Pages
queriesSearch Engine
Web Pages
<query>
images
Web pages and images are saved to local disk.
![Page 14: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/14.jpg)
<query><query>
<engine><engine> 11 </engine></engine>
<method><method> 22 </method></method>
<word><word> GeorgeGeorge </word></word>
<word><word> BushBush </word></word>
</query></query>
<query><query>
<engine><engine> 22 </engine></engine>
<method><method> 33 </method></method>
<word><word> BillBill </word></word>
<word><word> ClintonClinton </word></word>
</query></query>
Design: Queries in XMLDesign: Queries in XML
Multiple queries are stored in an XML fileMultiple queries are stored in an XML fileEngine:
1=Altavista, 2=Excite, 3=Hotbot, 4=Google
Method:
1 =or, 2 = and, 3 = expression
Search for George Bush using Alta Vista and must have all the words
Search for Bill Clinton using Hotbot and search for exact expression
![Page 15: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/15.jpg)
Process: Clue ExtractionProcess: Clue Extraction
Extraction Software
cluesqueries<query>
Clues Extraction Software clues
![Page 16: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/16.jpg)
Data to be analyzed
• For each page– Query used to find page– Source URL– For each image
• Source URL• Attributes• Position in document
– For each clue• Whether clue feature occurs in document at all• If feature occurs with text matching the query
– Position in document for each occurrence
![Page 17: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/17.jpg)
Relevant Rating Software
Query & image
Relevant/not
Process: Relevance RatingProcess: Relevance Rating
queriesHuman<query>
Presents images from each query to the user from the database, and record the human relevance rating back to the database.
![Page 18: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/18.jpg)
Clues : globalClues : global
Global Clues - clues that apply every image on the page
- filename of page
- path of page
- host of page
- title element of the web page
- keywords found in meta element
- description found in meta element
Why do we break the URL into three clues?
Different parts of the URL contributes different relevance factor to the overall relevance of the image in that page.
![Page 19: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/19.jpg)
Clues : globalClues : global
<HEAD><TITLE>Apple</TITLE>
<META NAME="keywords" CONTENT="Apple Computer, Power Macintosh, PowerBook, AppleWorks, WebObjects, iMovie, QuickTime, Desktop Movies, Software, Operating Systems, Mac OS, iMac, iBook">
<META NAME="Description" CONTENT="Visit www.apple.com for the latest news, the hottest products, and technical support resources from Apple Computer, Inc.">
<META HTTP-EQUIV="Expires" CONTENT="Fri, 26 Mar 1999 23:59:59 GMT"><META NAME="Date.Modified" CONTENT="19992109">
</HEAD>
![Page 20: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/20.jpg)
Clues : image fileClues : image file
Image file properties - external properties
- filename
- path
- host
An image can be from another host, and have different paths.
![Page 21: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/21.jpg)
Clues : common attributesClues : common attributes
Elements have common attributes
-title - describe what the element is
-id - used in identifying the element
-name - same as id, older HTML
Clues that use these attributes:
link, image, object, table, cell, row
![Page 22: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/22.jpg)
Clues: Image ContainerClues: Image Container
Link to an image <a>
- text enclosed within the link element
Embed image element <img>
-alt attribute (usually describes what the image is)
Object element <object>
-text that enclosed within the object element
![Page 23: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/23.jpg)
Clues: tableClues: table
Table (<table>):
- summary attribute - describes the table content
- caption - describes table content
- row heading
- row
- column heading
- column
- cell
- neighboring cells (above, below, right, left)
![Page 24: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/24.jpg)
Clues: tableClues: table
![Page 25: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/25.jpg)
Clues: tableClues: table
![Page 26: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/26.jpg)
Clues: tableClues: table
![Page 27: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/27.jpg)
Clues: tableClues: table
![Page 28: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/28.jpg)
Clues: tableClues: table
![Page 29: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/29.jpg)
Clues: tableClues: table
![Page 30: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/30.jpg)
Clues: tableClues: table
![Page 31: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/31.jpg)
Clues: headingsClues: headings
Heading elements(h1, h2, ..h6)
- headings above image
- headings below image
<h1>header above image</h1>
<img src=“sample.jpg”>
<h2>header below image</h2>
Heading can indicate a topic and images below the heading maybe relate to the heading. Some use headings as caption above images, and sometimes below images. Some headers are used where fonts should be used or bold should be used.
![Page 32: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/32.jpg)
In this photo, the heading comes after the image. Often if it is used as a topic, it usually comes before the image. But some images have heading as caption below the images.
![Page 33: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/33.jpg)
Clues : textClues : text
Emphasized text elements
- bold
- italic
- underline
- strong
- emphasis
- big
Body text - text that surrounds the image
- distance
![Page 34: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/34.jpg)
Current Project StatusCurrent Project Status
- Prototype download and clue extraction software nearly complete
- now testing implementation
- data (without human relevance ratings) in early November
- Recruiting students to build on-line relevance rating system
- hope to get students outside lab to help with ratings via Web interface
![Page 35: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/35.jpg)
Challenges for image search systemsChallenges for image search systems
- computing word distance from image
- Stylesheet used for presentation
- table pattern
- pattern of HTML elements usage
- CGI returned images
- structural boundaries
-patterns in Web page design
-HTML generators
![Page 36: Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee](https://reader035.vdocuments.us/reader035/viewer/2022062411/5681684c550346895dde4bb4/html5/thumbnails/36.jpg)
Cheng Thao, [email protected]