Webscraping for journalistsCAJ May 13, 2011

“A little Wget magic”

Webscraping

Using software that simulates a web browser to download large quantities of information from a web site.

Why webscrape?

• Assemble your own copy of online data• Save time pointing-and-clicking

Why webscrape?

• Data publishers (governments) want you to access data on their terms

Is it legal?

Yes. But.

Do it ethically.Watch for robots.txt

Tools for scraping

• DownThemAll (2)

• APIs• Wget• Custom scripts