Download - WORKSHOP: REAL LIVE DATA FOR CS COURSEScs.berry.edu/sinbad/ccscne18-workshop-slides.pdf · Earthquake feeds US Geological Survey JSON variable *Fuel economy data US EPA XML 35,430

WORKSHOP: REAL LIVE DATAFOR CS COURSESCCSC:NE 2018 @ UNIVERSITY OF NEW HAMPSHIRE

NADEEM ABDUL HAMID

REAL LIVE DATA FOR CS COURSES

WARM-UP

http://services.faa.gov/docs/services/airport/

http://services.faa.gov/docs/services/airport/


“HELLO WORLD” / SINBAD — JAVA

import core.data.*;

public class Warmup { public static void main(String[] args) {

DataSource ds = DataSource.connect(“http://services.faa.gov” + “/airport/status/MHT?format=application/xml"); ds.load();

System.out.println(ds.fetchString(“Name")); System.out.println(ds.fetchString("Status/Reason")); System.out.println(ds.fetchString("Weather/Temp")); } }


“HELLO WORLD” / SINBAD — PYTHON

from sinbad import *

ds = Data_Source.connect(“http://services.faa.gov/airport/" + “status/MHT?format=application/xml")ds.load()print(ds.fetch("Name", "Status/Reason", "Weather/Temp"))


“HELLO WORLD” / SINBAD — RACKET

(require sinbad)

(define ds (sail-to "http://services.faa.gov/airport/status/MHT? format=application/xml" (manifest) (load)))(fetch ds "Name" "Status/Reason" "Weather/Temp")


TUTORIAL PLAN

▸ Welcome; Intros [5 mins.]

▸ SINBAD: Overview [15 mins.]

▸ Q&A

▸ Install & Setup [10 mins.]

▸ Quick Start Tutorials [40 mins.]

▸ (regroup/break - 10 mins.)

▸ Data Source Exploration [40 mins.] Course Activity Generation

▸ (regroup/break - 10 mins.)

▸ Discussion • Feedback [30 mins.]

▸ Wrap up [15 mins.]


MOTIVATION

▸ The “Age of Big Data”

▸ Incorporate the use of online data sets in introductory programming courses (CS1/2)

▸ Provide a simple interface

▸ Hide I/O connection, parsing, extracting, data binding


REAL LIVE DATA FOR CS COURSES▸ Motivate & engage: Make it really “real” ▸ Direct access via URL, e.g. “https://data.baltimorecity.gov/Culture-Arts/

Restaurants/k5ry-ef3g/” ▸ Live data (e.g. current weather conditions, airport status, currency

exchange rates) ▸ Support open-ended assignments/projects (students can choose data

sources) ▸ Target learning outcomes ▸ Easily access atomic data/collections of atomic data ▸ Binding happens automagically for structured data ▸ Students decide/control data representation ▸ Grade functions, not data access

▸ Don’t use up course time: Make it smooth ▸ Syntax <= Scanner (e.g. in Java) ▸ Transparently parse, cache, convert data ▸ Seamlessly integrate with course assignments


APPROACH▸ Infer necessary information about the data source

automatically ▸ Data format, schema, field names, etc.

▸ Provide transparent local caching

▸ Sampling facilities for testing & development

▸ Provide runtime binding to data structures/classes in the user code

▸ Deal as gracefully as possible with messy data (e.g. missing fields)


USAGE▸ 3-step approach: • Connect • Load • Fetch

▸ Infer data format if possible — XML, CSV, JSON

▸ Display inferred structure of data

▸ Fetching atomic values

▸ provide a path into the data

▸ Structured data:

▸ provide multiple paths of datasupplied to the class/struct constructor (Java/Racket)

▸ Collections: lists / arrays (Java)


OTHER FUNCTIONALITY

▸ Data source specifications

▸ Query parameters

▸ Cache control

▸ Processing support


SOME EXPERIENCE▸ CSC 103: Creative Computing (@Berry College)

▸ Processing-based; Tutorial-style labs

▸ Sample data sets used/discovered by students:

▸ Students get engaged and excited!

Name Source Type Records(Asterisk indicates data set discovered by students)*1000 songs to hear before you die opendata.socrata.com XML 1,000Abalone data set UCI Machine Learning Repository CSV 4,177*Airport Weather Mashup NWS + FAA XML fixed*Chicago life expectancy by community data.cityofchicago.org XML ˜80Earthquake feeds US Geological Survey JSON variable*Fuel economy data US EPA XML 35,430*Jeopardy! question archive reddit JSON 216,930Live auction data Ebay XML 100/pageMagic the Gathering card data mtgjson.com JSON variableMicrofinance loan data Kiva XML variable*SEC Rushing Leaders 2014 ESPN CSV (manual) variable

Figure 5. Examples of data sources used in lecture examples and student final projects.

One issue that arose in some instances was finding a link tothe raw data provided by a service. The primary URL for manyweb services often provides an interactive interface or visualizationof the data. It takes some digging around to find a direct URL tothe underlying XML data or a JSON rendering. Related to this,some students had initial difficulty understanding the distinctionbetween data rendered in a human-friendly form (such as is visiblewhen a web page is viewed in a browser), and the raw data in aformat amenable to machine-processing by their program. Whenthey found a web page that displayed a set of data in tabular form,they would attempt to use that page’s URL to connect to the datasource.

Another concern that did not seem to be a major problem, butmay have limited the choice of data sources, was that some sitesrequire “developer” registration and an API key to be obtained inorder to access data. Some students were hesitant to register forsuch. Also, this required various query parameters to be set on theDataSource object before it is loaded. Determining the necessaryparameters involved reading into the developer documentation of aparticular web service, an experience which varies in pleasantnessfrom site to site.

Overall, insofar as operation of the library itself was concerned,things seemed to go fairly smoothly. As noted earlier, introducingthe library earlier on in the course, and integrating it more thor-oughly with the course content would make the students more com-fortable with it and allow them a better sense of how to access datafor the purpose of analysis or visualization. And, of course, moreextensive field studies, using the library in other courses and insti-tutions, are necessary to gauge its true usefulness.

5.2 Technical Challenges

Based on the limited testing and use so far, our current implementa-tion appears to meet many of the goals laid out in Section 2. How-ever, there are a few areas where improvement is clearly needed.One is to provide a more robust approach to error handling and re-porting. Currently, exceptions are raised when things do not matchup as expected at various stages of loading and instantiating databut the error messages can be improved. This could be achievedpartly by providing a high-level description of the source of a prob-lem, rather than just a line number in a file, to help a novice pro-grammer understand what happened.

As alluded to in the previous section, one complication in ac-cessing data sources is providing a user-specific API key or accesstoken. While this can be achieved using the existing set methods,it can be frustrating for students to figure out how to specify thenecessary parameters. Also, rather than having user API tokens ex-posed directly in source code of a program, which might be sharedwith others, it would be better to have private user data stored in

a separate file. We need to investigate a mechanism for doing thisin a simple manner, combined with a curated set of data sourcespecifications for common APIs (Ebay, Twitter, etc.) that requirespecial parameters, to make it easier for students to access these.Also, a GUI tool for managing data source-related parameters, asmentioned in the next section might be very helpful.

In terms of performance, the library handles many small tomedium-size data sets well. However, the current implementationloads the entire data set into memory. Along with the adoption ofXML as a common intermediate format, this results in significantdelays when loading large data sets (thousands of records). Evenwith caching, there is at least a two-fold delay loading a CSV file,for example. First the data must be loaded and converted to XML;then the entire converted data is scanned again to infer its schema.Depending on the fetch operation that is then executed, there maybe further delay in processing. Some ways to alleviate this wouldbe to cache the data after it has been converted to XML, and tocache the inferred schema as well as the data itself. Even withmore intelligent caching, however, XML is not the most efficientrepresentation of data and in many cases it would be much moreefficient (time and space) to access CSV data, for example, using adata structure other than an XML object.

Another approach to improve performance for large data setsis to abandon XML as the common intermediate format and adoptmore abstract mechanisms to encapsulate arbitrary data format ob-jects and allow them to be traversed using a uniform interface. Thisis an interesting software architecture challenge that is currentlyunder investigation and would require a major reorganization of theinternals of the library. A complementary feature would be to alsoprovide methods for streaming or paginating data. This remains tobe explored.

5.3 Additional Future Directions

In addition to addressing the more immediate technical challengesof the previous section, there are some other ideas for future de-velopment that would improve the library’s usefulness. One is theability to invoke a GUI interface to configure various settings forthe library and parameters for a particular data source. Upon load-ing a data source, it could also allow the user to view usage info(as an alternative to printUsageInfo), preview the available data,provide help for invoking methods of the library, and perhaps evengenerate code snippets.

As mentioned in Section 4, the fetch() methods currentlyonly support the specification of flat signatures. It is possible todirectly construct and provide a type signature object to the librarythat involves, for example, lists of structures with nested lists orstructures. Providing a simplified way to supply such signatures via

5 2015/6/29


MORE EXPERIENCE

▸ “Designing Programs: Problem-solving & Abstraction” ▸ CS1 based on How to Design Programs (htdp.org) ▸ Functional, using Racket-lite ▸ Sinbad integrated in labs/assignments throughout semester (first time

in progress) ▸ Actual coding exercises the same, just tie in to data

“I really enjoy the variety and creativity of the different problems that we do, both in class and on the homework.”

“...I really enjoy the assignments we have because we’re applying what we’re learning to real things, like weather, …”

“...I really like the type of data we’re analyzing in the labs and assignments because it’s realistic…”

Experience - CSC120

● “Designing Programs: Problem-solving & Abstraction”○ CS1 based on How to Design Programs (htdp.org) ○ Functional, using Racket-lite ○ Sinbad integrated in labs/assignments throughout semester (first time in progress)○ Actual coding exercises the same, just tie in to data

Rea

lly li

ke

Nei

ther

like

no

r dis

like“I really enjoy the variety and creativity of the different problems that

we do, both in class and on the homework.”

“...I really enjoy the assignments we have because we’re applying what we’re learning to real things, like weather, …”

“...I really like the type of data we’re analyzing in the labs and assignments because it’s realistic…”W

hat f

eatu

res

of th

e co

urse

do

you

like

or d

islik

e?


Sample Activities/Data Sources● Current weather conditions (NWS)

○ Generate an image from data○ Load and compare two locations○ Interactive statewide weather conditions app

● Bike Share Data (NYC, Chicago)○ Arithmetic, conditionals

■ Compute usage fees based on type of user■ Latitude/longitude computations

○ Representing structured data, building strings■ Generate Google map URL from a trip record

○ Simple list algorithms■ What age is the youngest rider in the data? The oldest?

○ Filtering, aggregation■ If we arrange riders into 5-year age ranges which range do the most riders fall into?


Q & A

?

QUICK START TUTORIALACTIVITY

Go to: http://cs.berry.edu/sinbad

Follow Installation instructions for your language.

Follow Quick Start link for your language

end at ~10:10am

http://cs.berry.edu/sinbad


DATA SOURCE EXPLORATION • COURSE ACTIVITY GENERATION

▸ Data Bazaar @ cs.berry.edu/sinbad: Curated data sources + sample code

▸ Let’s break into groups (2/3/4?)

▸ Pick a data source (Kiva, Divvy, …) & a language

▸ Explore the data source website

▸ Look at sample Sinbad code for your language

▸ Pick one or several programming competencies (handout)

▸ Develop a course assignment - i.e. write solution code if possible

▸ Stop @ ~11:00am to share & discuss.


DISCUSSION / FEEDBACK


CHALLENGES

▸ Students understanding data formats/representations (HTML vs XML)

▸ Data is messy

▸ Maintaining balance - avoid students getting lost in data

▸ Data exploration/browsing

▸ Error handling/debugging

▸ Developer/API registration

▸ Repository of assignments, labs, examples

▸ Evaluation

�19


RELATED WORK

▸ CORGIS Dataset Project - http://think.cs.vt.edu/corgis/

▸ Great tips from experience:https://think.cs.vt.edu/pragmatics/

▸ BRIDGES Project - http://bridgesuncc.github.io/

�20

http://think.cs.vt.edu/corgis/

https://think.cs.vt.edu/pragmatics/

http://bridgesuncc.github.io/


CONCLUSION

▸ Facilitate incorporation of online data sources into programming assignments

▸ Painlessly

▸ Seamlessly

�21

cs.berry.edu/sinbad

Use a data set in your next assignment!



DATA SOURCE SPECIFICATION FILE‣ Data source URL and format. ‣ Human-friendly name and description, along

with URL to a project or informational page about the data source.

‣ A specification of pre-supplied and user-supplied (required and optional) query parameters or path parameters. The latter are user-provided strings that are substituted in for placeholders in the URL path.

‣ Programmatic options specific to the particular data source object (such as a header for CSV files).

‣ Cache settings, such as cache directory path or timeout.

‣ A data schema defining the exposed data structures and fields from the source with various helpful annotations such as textual descriptions of fields that can be displayed by printUsageString().

{ "name": "Geographical Data - Peru", "format": "TSV", "path": "http://download.geonames.org/export/dump/PE.zip", "infourl": "http://www.geonames.org/",

"options": [ { "name": "fileentry", "value": "PE.txt" }, { "name": "header", "value": "geoid,name,asciiname,altnames,lat,long,feature-class,feature-code,cc,cc2,admin1,admin2,admin3,admin4,pop,elev,dem,tz,mod" }], }

DataSource.connectUsing("geospec-pe.spec");

TEACHING CS COURSES WITH REAL LIVE DATA

FUTURE

▸ GUI tools

▸ Additional data formats

▸ HTML tables, web scrapers (regexps)

▸ Customized for popular APIs (ebay, twitter, etc.)

▸ Streaming, pagination, sampling…

▸ More curriculum resources (CS1/2)

▸ Evaluation of effectiveness

BART, ET AL. FIGURE 2Figure 2: A simple program demonstrating the Java Earthquake library

1 import java . u t i l . L i s t ;2 import java . u t i l . HashSet ;3 import realt imeweb . ea r thquake s e rv i c e . main . EarthquakeService ;4 import realt imeweb . ea r thquake s e rv i c e . domain . Earthquake ;56 public class EarthquakeDemo {78 public stat ic void main ( S t r ing [ ] a rgs ) throws EarthquakeException {9 // Use the EarthquakeServ ice l i b r a r y

10 EarthquakeService es = EarthquakeService . g e t In s tance ( ) ;1112 es . connect ( ) ; // Remove to use the l o c a l cache

1314 // 5 minute de lay , but i f we use the cache no de lay i s needed !

15 int DELAY = 5 ⇤ 60 ⇤ 1000 ;1617 HashSet<Earthquake> seenQuakes = new HashSet<Earthquake >() ;1819 // Po l l s e r v i c e r e g u l a r l y

20 while ( true ) {21 // Get a l l ear thquakes in the pas t hour

22 Lis t<Earthquake> l a t e s t = es . getEarthquakes ( His tory .ALL) ;23 // Check i f t h i s i s a new ear thquake

24 for ( Earthquake e : l a t e s t ) {25 i f ( ! seenQuakes . conta in s ( e ) ) {26 // Report new ear thquakes

27 System . out . p r i n t l n ( ”New quake ! ” ) ;28 seenQuakes . add ( e ) ;29 }30 }31 // Delay to avoid spamming the weather s e r v i c e

32 Thread . s l e e p (DELAY) ;33 }34 }35 }

Figure 3: A simple program demonstrating theRacket Weather library

1 ( require ”weathe r�o f f l i n e . rkt ”)23 ; s t r i n g �> s t r i n g

4 ; Consumes a c i t y and re turns whether i t s

5 ; curren t temperature i s ”hot ” or ”co ld ”

6 (define ( report�weather c i t y )7 (cond [(< 70 ( get�temperature c i t y ) )8 ”hot ” ]9 [ else ”co ld ” ] ) )

1011 ; We use the o f f l i n e ve r s i on o f the l i b r a r y

12 ; f o r c on s i s t e n t check�expects

13 (check�expect ( report�weather ”Nome, AK”)14 ”co ld ”)15 (check�expect ( report�weather ”Miami , FL”)16 ”hot ”)

• API documentation and student-oriented user guidesfor each language and library.

• Alternative datasets for the internalized data cache(e.g., instead of business reviews from around “Blacks-burg, VA”, there might be another dataset for “Indi-anapolis, IN”).

• Reduced variants of the libraries for targetted assign-ments.

• Example assignments that use the library.

All resources are open-source and fully supported. Theyare being continuously refined and extended.

3.3 Prototyping ToolAn important byproduct of our project is the creation of

an online tool for rapidly prototyping new libraries. Most ofthe code used in our libraries follows the same pattern forany given language. First, requests are made to a web ser-vice and raw data is returned (typically as XML or JSON).Next, the data is parsed into some intermediary, semi-struct-ured form using dictionary and list types that are nativeto the language. Finally, the data is encoded into a read-only class or struct, depending on the disposition of the lan-

EQUIVALENTimport core.data.*;import java.util.Date;import java.util.HashSet;import java.util.List;

public class EarthquakeDemo { public static void main(String[] args) { int DELAY = 5; // 5 minute cache delay DataSource ds = DataSource.connectAs("JSON", "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson"); ds.setCacheTimeout(DELAY); ds.load(); ds.printUsageString(); HashSet<Earthquake> quakes = new HashSet<Earthquake>(); while (true) { ds.load(); // this only actually reloads data when the cache times out List<Earthquake> latest = ds.fetchList("Earthquake", "features/properties/title", "features/properties/time", "features/properties/mag", "features/properties/url"); for (Earthquake e : latest) { if (!quakes.contains(e)) { System.out.println("New quake!... " + e.description + " (" + e.date() + ") info at: " + e.url); quakes.add(e); } } } }}

PLUS…// this class may be instructor-provided, or left to students to define as an exerciseclass Earthquake { String description; long timestamp; float magnitude; String url; public Earthquake(String description, long timestamp, float magnitude, String url) { this.description = description; this.timestamp = timestamp; this.magnitude = magnitude; this.url = url; } public Date date() { return new Date(timestamp); } public boolean equals(Object o) { // introductory CS students would probably implement a simpler version of this if (o.getClass() != this.getClass()) return false; Earthquake that = (Earthquake) o; return that.description.equals(this.description) && that.timestamp == this.timestamp && that.magnitude == this.magnitude; } public int hashCode() { // technically, hashCode() should be overridden if equals() is return (int) (31 * (31 * this.description.hashCode() + this.timestamp) + this.magnitude); }}

OUTPUT (1)The following data is available: a structure with fields: { type : * metadata : a structure with fields: { api : * count : * generated : * status : * title : * url : * } features : A list of: structures with fields: { id : * type : * geometry : a structure with fields: { type : * coordinates : A list of: * } properties : a structure with fields: { alert : * cdi : * code : *

... type : * types : * tz : * updated : * url : * } } }

OUTPUT (2)New quake!... M 0.8 - 3km WSW of Tahoe Vista, California (Wed Jul 06 21:50:36 EDT 2016) info at: http://earthquake.usgs.gov/earthquakes/eventpage/nn00550830

New quake!... M 1.2 - 9km WNW of Cobb, California (Wed Jul 06 21:44:22 EDT 2016) info at: http://earthquake.usgs.gov/earthquakes/eventpage/nc72659116

New quake!... M 1.2 - 47km E of Cape Yakataga, Alaska (Wed Jul 06 21:37:45 EDT 2016) info at: http://earthquake.usgs.gov/earthquakes/eventpage/ak13742485

New quake!... M 1.5 - 3km WSW of Tahoe Vista, California (Wed Jul 06 21:30:39 EDT 2016) info at: http://earthquake.usgs.gov/earthquakes/eventpage/nc72659111

New quake!... M 2.0 - 90km N of Redoubt Volcano, Alaska (Wed Jul 06 21:16:04 EDT 2016) info at: http://earthquake.usgs.gov/earthquakes/eventpage/ak13742480

Download - WORKSHOP: REAL LIVE DATA FOR CS COURSEScs.berry.edu/sinbad/ccscne18-workshop-slides.pdf · Earthquake feeds US Geological Survey JSON variable *Fuel economy data US EPA XML 35,430

Top Related