1. 1. From Archives to Climate Science: a story of pirates, old weather, serendipity, data rescue, lovelorn sailors, perseverance, community values, and clearing the fog Paula Goodale Research Associate, Information School The Secret Life of a Weather Datum @PaulaGoodale, @lifeofdata with thanks to Jo Bates (iSchool), Joan Arthur (Old Weather)
2. 2. The Secret Life of a Weather Datum www.lifeofdata.org.uk
3. 3. Romilly Close Pi weather station designer (Aerospace Engineering UG student) Fred Sonnenwald - Pi weather station project co-supervisor (Civil Engineering) Sophie Rutter Usability testing (Information School) Jo Bates, Information School, University of Sheffield Paula Goodale, Information School, University of Sheffield YuWei Lin, University for the Creative Arts Dave Mee Tandot/The Garden #lifeofdata
4. 4. The Secret Life of Data? Big data, open data, managing data, data = oil .Lots of talk about data! Critical Data Studies Dalton and Thatcher (2014): in both its production and interpretation, all data big included is always the result of contingent and contested social practices that afford and obfuscate specific understandings of the world How to capture the complex socio-cultural shaping of (big) data infrastructures? Where to begin? Where to end? How to talk to people about it?
5. 5. The Secret Life of a Weather Datum Research Questions 1. What is the journey that weather data produced by the UKs Met Office takes from its production through to its collation and re-use as big weather data in different contexts? 2. What socio-cultural values and practices are articulated in the transformation of this data on its journey from production through to various contexts of collation, distribution and re-use, and how do these socio-cultural values and practices themselves transform as they interact with the data over the course of its journey? 3. What institutional policies and practices, and government policies and legislation, shape the distribution and licensing of weather data for re-use in different contexts? 4. How can the complexity of the socio-cultural dynamics shaping the production, collation, distribution and re-use of big weather data be communicated to a wider audience?
6. 6. Methods Case studies and participants Semi-structured interviews and Observations Documentary evidence and Digital ethnography Thematic analysis Data production Weston Park weather station Met Office Amateur observers Old Weather Policy maker Climate Science Met Office Hadley Centre Climatic Research Unit, UEA Intergovernmental Panel on Climate Change (IPCC) Met Office projects ACRE and Old Weather Financial Markets Weather Market Data supplier [Financial markets] Policy maker Citizen Science Amateur observers Old Weather / Archives Met Office projects - WOW and Old Weather
7. 7. Methods Case studies and participants Semi-structured interviews and Observations Documentary evidence and Digital ethnography Thematic analysis Towards ethnography Site visits Multiple interviews Observation Field notes Photographs Video Interviews - core themes Own role, data practices Motivations and challenges Relationships with other people and organisations External environment e.g. economic and policy issues Tailored to individuals Observations working environment Communal and personal space Branding and artefacts Personal interactions Technology
8. 8. Methods Case studies and participants Semi-structured interviews and Observations Documentary evidence and Digital ethnography Thematic analysis Extending coverage and filling in gaps Official records Access restrictions Discourse Community Documentary evidence Reports Policy documents Research papers Web sites Corporate publications Digital ethnography Twitter IPCC conference Online forums Citizen scientists
9. 9. Methods Case studies and participants Semi-structured interviews and Observations Documentary evidence and Digital ethnography Thematic analysis Coding Manual 2-3 coders Coding themes Attitudes, values and beliefs Valuable activities and practices Social relationships The data journey Working papers One per case Analysis by coding themes By organisation Cross-case analysis Web site Research papers
10. 10. lifeofdata.org.uk #lifeofdata
11. 11. Old Weather Case Study: Data collection 1x interview ACRE project 1x interview Met Office climate scientist 1x interview Climate historian 2x interviews Old Weather members Content / discourse analysis of Old Weather forum Content analysis of documents and web pages
12. 12. Old Weather: Climate Science Challenges To improve climate models you need more data, over longer time periods, from more locations I usually say to people if youve got a weather observation, if its south of the equator, well take it. Right. If its north of the equator before about 1950 well take it. If its north of the equator, outside 1950, and its not in North America, Europe Japan, or the North Atlantic were probably still interested. [MO_03]
13. 13. Pirates ACRE project Atmospheric Reconstructions of the Earth A loose network of like-minded climate scientists engaged in data recovery projects I often think about you know, the organisations work a bit like the Royal Navy, we work like Pirates of the Caribbean. [MO_06] Its very difficult to get funding for data recovery because people want scientific results. A lot of the funding bodies or people who make these decisions dont realise that unless youve got the data, youre not going to do any science, but they dont want to fund the recovery. As soon as you say data rescue, data recovery, they switch off. [CRU03]
14. 14. A Mission Challenge: knowing whats available and where its located I compiled an inventory of every major Royal Navy vessel from 1800 to 1950, where it was, what its movements were, where its logbook is, and I also compiled reports for them, which are available, concerning logbooks generally, where they are in the archives, what they consist of, what some of the problems are in using them, this sort of thing. [CRU_03]
15. 15. Serendipity My favourite research method is serendipity [CRU_03]
16. 16. When it says miscellaneous in my view its because its got lots of stuff with numbers on and they didnt know what to do with it. [CRU_03]
17. 17. Data Rescue and Recovery Challenge knowing the data Well, my expertise is finding the data, knowing how it was collected, knowing some of the problems of the, what Ill do is Ill look through the logbooks, and Ive got a lot of experience, Ive looked at thousands of them, and I anticipate the problems theyre going to have with the data. And sometimes I say, Youre going to have this problem. This is how you overcome it. [CRU_03]
18. 18. Data Rescue and Recovery Challenge - records may not be kept in ideal conditions it shows the stuff in archive is just piled up and its all mouldering. I didnt actually go there but it was--, its horrendous. Things are just falling to pieces and its just piled up on the shelves and its all Its irreplaceable. [CRU03]
19. 19. Digitisation and Preservation and the archivist looks at them and says, How good condition is this record? You know, Can we just photograph it, or do we need to repair it first? Is it going to fall to pieces on us? Does it need curating? So they do the curating as necessary, and then we try and photograph it, and that requires a bit of human expertise because the quality of the ink after 150 years is not always great. Okay. But usually we get fairly legible photographs out of it. [MO_03]
20. 20. Old Weather: Transcribing the logs Process Select a ship Transcribe the log book Weather data Notes Each one transcribed at least 3 times Cross-check for errors Data sent to Met Office for processing Challenges Hand-writing! Place names Can be dull / repetitive
21. 21. There are some things computers cannot do well From 4 to 6 light breezes from the S x W and cloudy - latter part light rain Air 80. Water 82
22. 22. TWYS Type What You See Typed records are a rare treat!
23. 23. Lovelorn Sailors Not just numbers inventories, events, stories Attachments formed to historical people A rich source of community engagement Edited notes are captured for future reference Sharing with historians www.naval-history.net
24. 24. Our interest in our subject, our own humanity for our scribes and for each other. Produces a powerful community, a family. [Joan Arthur, Old Weather]
25. 25. Community I think the sense of contributing to something that I care about, but also definitely the forum. The forum is massively important. [OW_02] One of the other threads is Signs of Old Weather Addiction [OW1] Helping with transcriptions Sharing stories, fun and games Personal interactions, keeping up morale Recognition and information from the scientists is also valued
26. 26. Motivations I think it is, yeah, you know it's the weather, it's the history, and it's the forum I think for me are the sort of the three key important things that have sort of kept me interested in it really. [OW_02] I think initially it was because of the whole thing to do with climate change. Because obviously, I've got a concern for the planet, you know, and the way that we treat it and so on. And so actually doing something that was in some way going to contribute to that kind of research attracted me. [OW_01]
27. 27. Reasons for Transcribing Eveleigh et al, 2012
28. 28. Serious Leisure: Volunteers http://www.seriousleisure.net/slp-diagrams.html
29. 29. Data Processing Transcribed data is sent to the Met Office Transformed into a standard IMMA data format Checked for errors Uploaded to ICOADS data centre for access by climate scientists across the world Primarily used in reanalysis projects Reanalysis is a scientific method for developing a comprehensive record of how weather and climate are changing over time. In it, observations and a numerical model that simulates one or more aspects of the Earth system are combined objectively to generate a synthesized estimate of the state of the system. A reanalysis typically extends over several decades or longer, and covers the entire globe from the Earths surface to well above the stratosphere.
30. 30. Old Weather Data in Climate Science Video: National Maritime Museum, www.oldweather.org http://player.vimeo.com/video/15388983
31. 31. My job is fog removal But the yellow region that's where because we did our citizen science Old Weather Project, and we got new weather observations, there now isn't as much fog as there used to be. So the yellow area, I call that the glow of discovery. That's where there isn't any fog any longer. [MO3] Video: https://vimeo.com/99917882
32. 32. Other Archival Data Climatic weather stations - Weston Park, Museums Sheffield - More than 130 years of weather data - Collected and preserved by generations of museum curators - Archived by the Met Office and by the Museum - Used in climate science - Available for research - Public good data and weather station belongs to the people of Sheffield
33. 33. Other Archival Crowdsourcing Projects Transcribe Bentham BBC World Service archive TNA Operation War Diary http://helpinghistory.com/ Crowd-sourcing 101
34. 34. References Dalton, C. & Thatcher, J. (2014). What Does a Critical Data Studies Look Like and Why Do We Care? Seven Points for a Critical Approach to Big Data. Society & Space [Online]. Available: http://societyandspace.com/material/commentaries/craig-dalton-and-jim- thatcher-what-does-a-critical-data-studies-look-like-and-why-do-we-care-seven-points- for-a-critical-approach-to-big-data/ Eveleigh, A. M. M., Jennett, C., Blandford, A., Brohan, P., Cox, A. L. (2014). Designing for dabblers and deterring drop-outs in citizen science. CHI'14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ( pp.2985-2994). New York, NY, USA: ACM Eveleigh, A. (2014). Crowding out the archivist? Locating crowdsourcingwithin the broader landscape of participatory archives. In Ridge, M. (Ed.).Crowdsourcing our cultural heritage ( ). Ashgate. Eveleigh, A., Jennett, C., Lynn, S., Cox, A. L. (2013). "I want to be a captain! I want to be a captain!": Gamification in the Old Weather citizen science project.Gamification'13: Proceedings of the First International Conference on Gameful Design, Research, and Applications. ( pp.79-82). New York, NY, USA: ACM Press Stebbins, R. A. (1996). Volunteering: A serious leisure perspective. Nonprofit and Voluntary Sector Quarterly, 25(2), 211-224.
35. 35. Web sites Life of Data: http://www.lifeofdata.org.uk/ Life of Data research data archive: https://archive.org/search.php?query=secret%20life%20of%20 a%20weather%20datum Old Weather: http://www.oldweather.org/ Old Weather Forum: http://forum.oldweather.org/ 130 years of Weston Park Weather Station: http://www.museums-sheffield.org.uk/blog/2012/9/130-years- of-weston-park-weather-station
36. 36. Thanks! Paula Goodale @PaulaGoodale, p.goodale@sheffield.ac.uk The Secret Life of a Weather Datum @lifeofdata, www.lifeofdata.org.uk