Download - Using Yahoo Pipes
Using Yahoo Pipes to access data from web pages
THATCamp Canberra 2011
Why Yahoo Pipes?
• Browsers impose same-domain restriction on requesting XML/HTML/CSV content to include in web pages
• We can use server side technologies to retrieve data from external sources, but what about when we don’t have access to those technologies? (e.g. embedding content in a blog post)
• Yahoo Pipes convert data sources to JSON (JavaScript Object Notation) which can be included via JavaScript on web pages without restrictions
• http://pipes.yahoo.com
Example mashup
convictbook data sourcesTrove People & Organisations
Yahoo Pipes
Web page : hosted within a Content
Management System (HTML + CSS
+ JavaScript + jQuery)
Yahoo Pipes
SLQ convict data
Google Refine(Data clean up)
Google Fusion Tables
SRU Query Result (XML)
JSONJSON
CSV
CSV
Query (XML)
Visualisation links
Picture Australia data
CSV
Pipe : Trove People by namehttp://pipes.yahoo.com/pipes/pipe.info?_id=eed133e99c38a68d20d279e46cee66e6
Yahoo Pipes
JSON{ "count" : 1, "value" : { "title" : "ConvictIDGet", "description" : "Pipes Output", "link" : "http:\/\/pipes.yahoo.com\/pipes\/pipe.info?_id=c8b70fd9d9f6edcfbb7c0e351992e717", "pubDate" : "Thu, 06 Oct 2011 22:49:50 -0700", "generator" : "http:\/\/pipes.yahoo.com\/pipes\/", "callback" : "", "items" : [{ "convictid" : "200", "name" : "Frederick Beadle", "one_of" : "one of 296 convicts transported on the Pyrenees", "convicted" : "Central Criminal Court", "term_duration" : "10 years", "conviction_date" : "", "notes" : "Ticket of Leave.", "vessel" : "Pyrenees", "departure_date" : "31 January 1853", "destination" : "Western Australia", "microfilmref" : "Australian Joint Copying Project. Microfilm Roll 93, Class and Piece Number HO11\/18, Page Number 3 (3)", "surname" : "Beadle", "given_names" : "Frederick", "y:row" : "2", "description" : null, "title" : null }] }}
Tip: Use http://jsonlint.com/ to validate and format JSON
JSON for Trove People query{ "count" : 1, "value" : { "title" : "TrovePeople", "description" : "Pipes Output", "link" :
"http:\/\/pipes.yahoo.com\/pipes\/pipe.info?_id=c3412dfcaf40a54bdcd0273b293174e5",
"pubDate" : "Thu, 06 Oct 2011 22:50:47 -0700",
"generator" : "http:\/\/pipes.yahoo.com\/pipes\/",
"callback" : "", "items" : [{ "xmlns" :
"http:\/\/www.loc.gov\/zing\/srw\/",
"version" : "1.1", "numberOfRecords" : "265", "resultSetId" : "kc2umf", "resultSetIdleTime" : "300", "records”:
{"record" : [{"recordSchema" : "http:\/\/www.w3.org\/2005\/Atom","recordPacking" : "xml","recordData" : {"entry" : {"id" : "http:\/\/nla.gov.au\/nla.party-529154","link" : [{"href" : "http:\/\/nla.gov.au\/nla.party-529154”}, {"href" : "http:\/\/nla.gov.au\/anbd.aut-an35513577","rel" : "related”}, {"href" : "http:\/\/www.adb.online.anu.edu.au\/biogs\/A100015b.htm", "rel" : "related”}],
"author" : { "name" : "National Library of Australia Party Infrastructure”},"published" : "2008-12-10T15:20:01Z","title" : "Lawson, H. S. W.","content" : {"type" : "text\/plain","content" : "Alternate name\/s: Lawson, Harry Sutherland Wightman; Lawson, Harry Sutherland Wightman, Sir”}}}, "recordPosition" : "1”}, …} ]}}
Including JSON data using jQueryLive example:
http://jsfiddle.net/AnnaGerber/cbJrc/ <html><head><title>Demo</title></head><script type=“text/javascript” src=“jQuery.js”></script><script type="text/javascript"> function findPeople(pName) { var url = "http://pipes.yahoo.com/pipes/pipe.run?_id=eed133e99c38a68d20d279e46cee66e6&_render=json&name=" + pName; $.getJSON( url, function( data ) { var outputHTML = "”; $.each(data.value.items[0].records.record, function( index,item ) { var record=item.recordData.entry; outputHTML += "<li><a target='_blank' href='” + record.link[0].href +"'>" + record.title + "</a></li>"; }); $('#output').html(outputHTML); });});$(document).ready(function(){ findPeople(‘Lawson’); });</script></head><body><h1>People</h1><ul id=“output"></ul></body></html>
Tip: use browser inspector +console.log() to inspect data
