using yahoo pipes
DESCRIPTION
Slides from hands-on 'Bootcamp Sampler' session from THATCamp Canberra 2011. The aim of the session was to get data into web pages from data sources such as Trove's People and Organisations search using Yahoo PipesTRANSCRIPT
Using Yahoo Pipes to access data from web pages
THATCamp Canberra 2011
Why Yahoo Pipes?
• Browsers impose same-domain restriction on requesting XML/HTML/CSV content to include in web pages
• We can use server side technologies to retrieve data from external sources, but what about when we don’t have access to those technologies? (e.g. embedding content in a blog post)
• Yahoo Pipes convert data sources to JSON (JavaScript Object Notation) which can be included via JavaScript on web pages without restrictions
• http://pipes.yahoo.com
Example mashup
convictbook data sourcesTrove People & Organisations
Yahoo Pipes
Web page : hosted within a Content
Management System (HTML + CSS
+ JavaScript + jQuery)
Yahoo Pipes
SLQ convict data
Google Refine(Data clean up)
Google Fusion Tables
SRU Query Result (XML)
JSONJSON
CSV
CSV
Query (XML)
Visualisation links
Picture Australia data
CSV
Pipe : Trove People by namehttp://pipes.yahoo.com/pipes/pipe.info?_id=eed133e99c38a68d20d279e46cee66e6
Yahoo Pipes
JSON{ "count" : 1, "value" : { "title" : "ConvictIDGet", "description" : "Pipes Output", "link" : "http:\/\/pipes.yahoo.com\/pipes\/pipe.info?_id=c8b70fd9d9f6edcfbb7c0e351992e717", "pubDate" : "Thu, 06 Oct 2011 22:49:50 -0700", "generator" : "http:\/\/pipes.yahoo.com\/pipes\/", "callback" : "", "items" : [{ "convictid" : "200", "name" : "Frederick Beadle", "one_of" : "one of 296 convicts transported on the Pyrenees", "convicted" : "Central Criminal Court", "term_duration" : "10 years", "conviction_date" : "", "notes" : "Ticket of Leave.", "vessel" : "Pyrenees", "departure_date" : "31 January 1853", "destination" : "Western Australia", "microfilmref" : "Australian Joint Copying Project. Microfilm Roll 93, Class and Piece Number HO11\/18, Page Number 3 (3)", "surname" : "Beadle", "given_names" : "Frederick", "y:row" : "2", "description" : null, "title" : null }] }}
Tip: Use http://jsonlint.com/ to validate and format JSON
JSON for Trove People query{ "count" : 1, "value" : { "title" : "TrovePeople", "description" : "Pipes Output", "link" :
"http:\/\/pipes.yahoo.com\/pipes\/pipe.info?_id=c3412dfcaf40a54bdcd0273b293174e5",
"pubDate" : "Thu, 06 Oct 2011 22:50:47 -0700",
"generator" : "http:\/\/pipes.yahoo.com\/pipes\/",
"callback" : "", "items" : [{ "xmlns" :
"http:\/\/www.loc.gov\/zing\/srw\/",
"version" : "1.1", "numberOfRecords" : "265", "resultSetId" : "kc2umf", "resultSetIdleTime" : "300", "records”:
{"record" : [{"recordSchema" : "http:\/\/www.w3.org\/2005\/Atom","recordPacking" : "xml","recordData" : {"entry" : {"id" : "http:\/\/nla.gov.au\/nla.party-529154","link" : [{"href" : "http:\/\/nla.gov.au\/nla.party-529154”}, {"href" : "http:\/\/nla.gov.au\/anbd.aut-an35513577","rel" : "related”}, {"href" : "http:\/\/www.adb.online.anu.edu.au\/biogs\/A100015b.htm", "rel" : "related”}],
"author" : { "name" : "National Library of Australia Party Infrastructure”},"published" : "2008-12-10T15:20:01Z","title" : "Lawson, H. S. W.","content" : {"type" : "text\/plain","content" : "Alternate name\/s: Lawson, Harry Sutherland Wightman; Lawson, Harry Sutherland Wightman, Sir”}}}, "recordPosition" : "1”}, …} ]}}
Including JSON data using jQueryLive example:
http://jsfiddle.net/AnnaGerber/cbJrc/ <html><head><title>Demo</title></head><script type=“text/javascript” src=“jQuery.js”></script><script type="text/javascript"> function findPeople(pName) { var url = "http://pipes.yahoo.com/pipes/pipe.run?_id=eed133e99c38a68d20d279e46cee66e6&_render=json&name=" + pName; $.getJSON( url, function( data ) { var outputHTML = "”; $.each(data.value.items[0].records.record, function( index,item ) { var record=item.recordData.entry; outputHTML += "<li><a target='_blank' href='” + record.link[0].href +"'>" + record.title + "</a></li>"; }); $('#output').html(outputHTML); });});$(document).ready(function(){ findPeople(‘Lawson’); });</script></head><body><h1>People</h1><ul id=“output"></ul></body></html>
Tip: use browser inspector +console.log() to inspect data