Content Working Group

2015 NDSA Web Archiving Survey Report HighlightsNicholas Taylor (@nullhandle)Web Archiving Service ManagerStanford University Libraries

Archives 2016: Web Archiving RoundtableAugust 3, 2016

Content Working Group

NDSA Web Archiving Survey Working Group

Jefferson BaileyInternet Archive / Archive-It

Edward McCainUniversity of Missouri

Abbey PotterLibrary of Congress

Abbie GrotkeLibrary of Congress

Christie MoffattNational Library of Medicine

Nicholas TaylorStanford University Libraries

Content Working Group

NDSA Web Archiving survey background2011• 78 respondents• program info• tools/services• access• policies

2013• 92 respondents• program info

• staff, metrics, skills, content concerns

• tools/services• access/discovery

• new options

• policies• embargo, social

media, robots.txt, resources

2015• 106 respondents• program info

• areas of progress, collaboration interests + barriers

• tools/services• replication targets

• access/discovery• researchers

• policies

Content Working Group

Respondent Characteristics

“Lego People” by Scoobay under CC BY-NC-SA 2.0

Content Working Group

increasing proportion of universities

Archive

Universi

ty

Commercial

Consortium

Fed Gov

Loca

l Gov

State Gov

Hist. S

ociety

K12 School

Museum

Public Li

braryOther

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

2011 2013 2015

Content Working Group

SAA WebArch RT tops group affiliations

group 2011 2013 2015

8% 7% 8%

31% 33% 39%

45% 50%

Content Working Group

growth in staffing at the middle

.25 FTE56%

.5 FTE25%

1 FTE

6%

1 to 3 FTE

7%

3+ FTE7%

2013

.25 FTE58%

.5 FTE13%

.75 FTE5%

1 FTE5%

1 to 3 FTE13%

3+ FTE6%

2015

Content Working Group

Maturity and Progress

“Apple Mouse Evolution” by raneko under CC BY 2.0

Content Working Group

shift from pilot to production (all responses)

Planning Pilot Production Discontinued0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

2011 2013 2015

Content Working Group

shift from pilot to production (longitudinal responses)

2011 2013 20150

2

4

6

8

10

12

14

16

18

Planning Pilot Production

Content Working Group

increased perceptions of progress

Significant progress

29%

Some progress43%

About the

same15%

Slightly worse

off2%

Much worse

off2%

Other10%

2013

Significant progress

49%

Some progress22%

About the same19%

Other10%

2015

Content Working Group

perceived progress on data capture, appraisal, vision

Vision + O

bjective

sPolic

y

Resources +

Workfl

ow

Risk M

anagement

Appraisal +

Selecti

on

Scoping

Data Capture

QA + Analysis

Storage + O

rganization

Preservation

Metadata/Descr

iption

Access/

Use/Reuse

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

40.00%

17.65%

34.12%

5.88%

45.88%

37.65%

58.82%

22.35%

10.59% 12.94%

24.71%

16.47%

Content Working Group

low perceived progress on access, metadata, QA

Vision + O

bjective

sPolic

y

Resources +

Workfl

ow

Risk M

anagement

Appraisal +

Selecti

on

Scoping

Data Capture

QA + Analysis

Storage + O

rganization

Preservation

Metadata/Descr

iption

Access/

Use/Reuse

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

15.85%

28.05%24.39% 24.39%

12.20%9.76%

12.20%

36.59%

21.95%

29.27%

37.80%

52.44%

Content Working Group

Archiving Focus

“Ant Farm Media Van v.08 (Time Capsule) in Bellewether at Southern Exposure” by Steve Rhodes under CC BY-NC-SA 2.0

Content Working Group

more archiving of own content

Own content 3rd-party content Both Other0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

2011 2013 2015

Content Working Group

access, cost, quality are top valued metrics

Access

+ use Cost

Data volume

Institutional b

uy-in

Loss

Quality

Risk M

anagementOther

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

2013 2015

Content Working Group

tools, appraisal, QA are top valued skills

Appraisal/s

election

Archivi

ng tools

Collaboration/co

mmunication

Domain experti

se

Metadata

Quality assu

rance

Software deve

lopment

Web tech

nologiesOther

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

2013 2015

Content Working Group

collaboration interest in QA, capture, metadata

Risk m

anagement

Capture optimization

Collecti

on development

APIs

Metadata standards QA

Tool deve

lopmentOther

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

40.91%

52.27%

31.82%27.27%

51.14%

64.77%

46.59%

3.41%

Content Working Group

“Photocopier” by Joriel "Joz" Jimenez under CC BY-NC-ND 2.0

Tools and Services

Content Working Group

more using multiple archiving solutions

Local External Both0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

25.40%

60.32%

14.29%

19.51%

63.41%

15.85%

4.81%

63.29%

30.38%

2011 2013 2015

Content Working Group

flat data transfer from service provider

Transfer data Do not transfer data0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

19.15%

80.85%

20.29%

79.71%

20.27%

79.73%

2011 2013 2015

Content Working Group

transferring data locally + to other services

Local External Both0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

58.82%

47.06%

5.88%

Content Working Group

trust provider + building locally for data transfer

Trust data ca

pture servi

ce provid

er

Building lo

cal in

frastr

ucture

No place to

store/m

aintain it

Not sure w

hat we’d do w

ith it

once w

e got itOther

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

2011 2013 2015

Content Working Group

Archiving Policies

“Handle With Care” by ServInt under CC BY-NC-ND 2.0

Content Working Group

more conditional handling of robots.txt

Always Never Sometimes/it depends Don't know0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

2011 2013 2015

Content Working Group

policies of other orgs key for policy-making

ARL Code of B

est Practi

ces

Oakland Arch

ive Polic

y

Secti

on 108 Study G

roup

Legal c

ounsel

Statutory

authority

Policies o

f other o

rgs

Previous N

DSA su

rveys

Other0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

2013 2015

Content Working Group

Landscape Summary

“Mt Baldy from Box Springs Mountain wi Theodolite” by signal mirror under CC BY 2.0

Content Working Group

overall picture• generally positive perceptions of progress• more moves to production• growing proportion of universities• growing proportion of focus on own content• staffing largely remains fractional w/ minor growth• majority use external service but hints of hybrid

approaches• more comfort w/ conditional policy approaches

Content Working Group

recurring themes: progress• data capture• highest perceived progress• 2nd highest collaboration interest

• appraisal• 2nd highest perceived progress• 2nd highest valued skill

Content Working Group

recurring themes: opportunities• quality assurance• highest collaboration interest• 3rd highest desired skill• 3rd lowest perceived progress

• access• lowest perceived progress• tied for highest valued metric

• metadata• 2nd lowest perceived progress• 3rd highest collaboration interest

Content Working Group

Nicholas Taylor@nullhandle

“Thank You” by vistamommy under CC BY 2.0