the digital public library of america (dpla) is a digital...

EVALUATING DIGITAL LIBRARIES 1

An Evaluation of the Digital Public Library of America and the Smithsonian’s History Explorer

as Effective Digital Libraries

Christine Skirka

Rutgers University


Abstract

The concept of a digital library is defined and reasons for evaluation are explored. Several

criteria are set forth as a framework for evaluation of two distinct digital libraries: the Digital

Public Library of America and the Smithsonian’s History Explorer. The six different criteria are

defined and the individual libraries are outlined and evaluated using the criteria. The analysis of

the digital libraries is very cursory and there is overall commentary about the two libraries and

their respective goals. Improvements are suggested and remarks are made for future research.


Introduction

Digital libraries are relatively new compared to traditional libraries, and they continue to

grow and develop in ways that are not easy to predict. The basis of these predictions is hard to

describe because digital libraries are severely lacking in a major way: evaluation. Evaluation of

digital libraries is something that can be time consuming and expensive, so it often is not a

priority for the institution. I can see the reasons behind this, but that does not mean efforts should

not be made to do an analysis on a somewhat regular basis, even if that means every five years.

An initial discussion will attempt to define what exactly a digital library is and what its

main attributes are. I also plan to evaluate two different digital libraries as an overall resource to

see if they are effective. I will do this using several different criteria which look at the overall

website, content, access, user needs, and organization, among other things. Ultimately, after a

review of the libraries, commentary will be made about if the digital libraries are effective in

terms of the goals they have described on their website.

The digital libraries that I have chosen are exemplary examples of resources that can be

used by many people for various purposes. I think they are a good representation of possible

digital libraries because they represent a public library and one which evolved from a museum. I

wanted to look at libraries from different institutional types to see if their goals might be the

same. Each digital library has a different goal, but all libraries share the ambition of making

information available to the users.

Evaluation of digital libraries is important because it gives the institution the ability to

monitor if the library is being used to the best of its ability. Digital libraries can be created and

not evaluated and function just fine. Evaluation allows for a more in-depth view of the overall

resources and their usage. Looking at the different facets of any digital library and the roles that


the various user types play permits the digital library to appeal to the governing body for

increased funds to further develop collections, add staff, or upgrade materials.

What is a digital library?

A clear-cut definition for “digital library” is something that is very hard to come by, let

alone define based on my own personal observations. I think that a major difficulty in writing a

definition comes from the fact that a digital library cannot be fit into one perfect little box.

Digital libraries are often extensions of physical libraries or collections. They have developed in

the digital age to allow access to people who would otherwise never have access to materials that

have long been kept on shelves or in drawers in institutional repositories. Each institution which

has created a digital library has made an attempt to showcase its most important and valued

materials so that people might find interest in them and share them with others. Because of this,

it is difficult to determine where a physical library’s purview ends and the digital library begins.

Several people have made great effort to attempt to define a digital library. In a report on

a workshop which investigated the social aspects of digital libraries conducted by Christine

Borgman et al. (1995), a rather lengthy, but clear definition is presented. This report defines a

digital library using two complementary concepts:

1. Digital libraries are a set of electronic resources and associated technical capabilities

for creating, searching and using information. In this sense they are an extension and

enhancement of information storage and retrieval systems that manipulate digital data

in any medium (text, images, sounds; static or dynamic images) and exist in

distributed networks. The content of digital libraries includes data, metadata that

describe various aspects of the data (e.g. representation, creator, owner, reproduction


rights) and metadata that consist of links or relationships to other data or metadata,

whether internal or external to the digital library.

2. Digital libraries are constructed – collected and organized – by a community of users,

and their functional capabilities support the information needs and uses of that

community. They are a component of communities in which individuals and groups

interact with each other, using data, information and knowledge resources and

systems. In this sense they are an extension, enhancement and integration of a variety

of information institutions as physical places where resources are selected, collected,

organized, preserved and accessed in support of a user community. These information

institutions include, among others, libraries, museums, archives and schools, but

digital libraries also extend and serve other community settings, including

classrooms, offices, laboratories, homes and public spaces. (p. 4)

While this definition is too lengthy to be included in most materials that might explore digital

libraries, it does highlight many important aspects of digital libraries that should be included in

any definition.

Borgman’s definition starts by pointing out the technical capabilities. Most people,

myself included, might not think of the technical aspect of a digital library initially though it is

important to note that a digital library cannot exist without technology. The definition then links

the technology to “digital data of any medium,” (Borgman, 1995). While the definition does go

on to identify specific document types, it also is ambiguous enough to allow for media type

extensions in the future. Allowing for unlimited extensibility has led to this definition still being

relevant 20 years after it was written. Borgman et. al. then go on to identify the importance of

data and metadata in a digital library. Discussing the content in these terms is something that I


would have considered earlier since that information makes up the entire library. The goal of the

entire first point seems to be that of the actual library. They discuss the technological

components that make up a digital library and what their needs and extensibility might be.

The second point goes on to look at the needs of the user and the surrounding

community. It is important to note that these two components are separate because they do, in

fact, complement one another. The library cannot exist without the technology, and will be

useless without a community of users. The definition states that digital library collections

“support the information needs and uses of that community,” (Borgman, 2995). This is

interesting to note because it implies that no two digital libraries should be the same. They

should be adapted and modified to meet the needs of the community which it is aimed to serve.

A digital public library in California has a completely different user community than that of a

small digital library from a historic commission in Ohio. The same user group might under some

circumstance use both libraries, but their research goals would likely vary for each library.

This definition also goes on to discuss the division between the digital and physical

library. “In this sense they are an extension, enhancement and integration of a variety of

information institutions as physical places where resources are selected, collected, organized,

preserved and accessed in support of a user community,” (Borgman, 1995). The use of the word

“enhancement” shows that digital libraries are important additions to the library as we have

known it for centuries. The library has always been a physical place, but a digital library breaks

out of the traditional confines to be something complementary. This also lends validity to an

institution which some might be skeptical of. Being that digital libraries are a rather new

concept, people might not trust that they are reliable or go-to sources of information. Many of us


have been brought up to go to the physical library for any host of reasons, but now we are able to

conduct much of the same research without ever leaving the living room.

Another shorted definition of digital libraries has been set forth by Donald J. Waters. In

his article “What are digital libraries?” (1998), he explores a working definition which was

created by the Digital Library Federation (DLF). The definition states that:

Digital libraries are organizations that provide the resources, including the specialized

staff, to select, structure, offer intellectual access to, interpret, distribute, preserve the

integrity of, and ensure the persistence over time of collections of digital works so that

they are readily and economically available for use by a defined community or set of

communities. (Waters, 1998)

Unlike Borgman’s definition, this one brings up the topic of the specialized staff that is necessary

to maintain and develop the collection, as well as provide assistance and guidance to those who

want to use it. This is an interesting, yet very important difference in the two definitions.

Borgman’s definition does address important issues, but fails in that it lacks any

acknowledgement that specialized staff is necessary for digital library creation, maintenance, and

customer support.

Another interesting difference between Borgman’s original definition and the one from

the DLF is that they analyze the need for digital libraries to maintain “persistence over time…

[and] …that they are readily and economically available,” (Waters, 1998). This notion that

digital libraries be maintained and updated over time is an important feature of this definition. A

digital library can be created today, but if efforts are not made to sustain it with most recent

technological advances, then it is useless. The issue of economics also lends some accountability

to those creating the digital library to produce something within certain fiscal constraints. Access


to information is always a good thing, but the requests for new materials versus the cost need to

be assessed and monitored. If the digital library is not being used to its utmost potential, then

there needs to be discussion about if developing a new collection would be helpful or harmful

overall. Ideally, an added collection would increase usage, but if usage is down, then it might not

be economically sound to use money or other resources in this way. The funds might be better

allotted to increased marketing or user outreach, for instance.

In an article which explores what digital libraries are, Christine Borgman (1999) states,

“The focus of the DLF definition is on the digital library as an organization who services include

the provision of information resources in digital forms,” (p. 236). While somewhat implied, this

statement reflects that the responsibility of the digital library lies in maintaining the digital files

in the collections. This does not necessarily include the maintenance of the original physical

items. Based on this, I would assume that the responsibility of maintaining the original items lies

with the owning institution, whether it be the physical library, or the digital library for born

digital resources.

Using a combination of the two suggested definitions of “digital library”, I think a happy

medium can be found. I think any digital library definition needs to discuss staff, content,

technology, users, and future growth. A better definition might be something like, “a digital

library is a set of digital resources which is intended to appeal to the designated community

through a variety of materials. This library is an enhancement of a physical library or may be a

standalone institution with plans to expand to further support the user community if necessary.

The technology that supports the library primarily supports information storage and retrieval

systems. Technologies should sustain any medium and contain metadata which creates links to

other data. Specialized staff is needed to control the digital library and support future expansion


and technological upgrades.” While not perfect, this definition covers all of the important ideals

that a digital library should strive to achieve.

Scope of evaluation

An evaluation of any digital library cannot be done without first identifying several

different criteria for analysis. This gives the researcher a definitive method by which to look at

digital libraries so that there is no variation in the assessment of the library being evaluated. I

believe that if enough digital libraries start using similar methods of evaluation criteria that a

standard could eventually be developed. In doing this, there is potential for more digital libraries

to allocate the resources needed to complete this process. I have found with various library

projects, the worst part is getting started, especially if there are no previously established

methods to guide my work. Implementation of a standard would reduce the overall difficulty of

completing the evaluation because the guesswork would be removed. Once one evaluation is

complete, the subsequent ones become easier because they can ideally be streamlined. Once staff

gets used to the process, evaluation can become second nature in some ways, and can possibly be

done as the library is being added to and maintained. The staff will know how to recognize

collections that are not being used or that are not fully developed. They will also be able to

identify and highlight the strengths of the digital library thus encouraging more people to look at

them thus increasing usage.

Evaluation of digital libraries can be done in a variety of different ways, some which may

be better than others. Some digital libraries might ask some of their users to assist through

surveys or focus groups. This approach would certainly show what collections are being used

most and give an impressions from the user about their satisfaction. Analytics can also be run on

the websites to get statistical data about overall usage. This will give you the hard and fast


information, but does nothing to get the human perspective. The statistics are very important in

situations like when a budget increase needs to be justified, but they should also be used with

some sort of human-generated information. Another method of evaluation can be done by library

professionals using the list of criteria that I have discussed previously. The criteria that I will be

using in my evaluation are: usability, effectiveness and efficiency, distribution of resources,

access, user needs & stakeholders, and content.

Criteria for evaluation

Usability

“Usability is a quality attribute that assesses how easy user interfaces are to use,” (Nielsen,

2012). This very basic definition opens the door for a wide variety of assessment and evaluation

to be completed. Usability, in this case, will be done from a professional standpoint, with the

perspective of a potential user in mind. Usability is an important criteria to be included in a

digital library evaluation because it is one of the basic fundamental units of any digital resource,

even beyond a digital library. If a user does not find a digital library easily accessible, they are

less likely to return in the future. If a digital library is clean cut and streamlined, the overall

usability reviews will be high.

Effectiveness and efficiency

Effectiveness and efficiency are two criteria that are symbiotic in nature; one fuels the other. The

digital library evaluation in terms of effectiveness will be done by looking at the overall

approach and feel of the main webpage. Does the initial web page send the user the right

message? Are the most important collections being highlighted? As far as efficiency, is the

digital library using its “space” well? I think of this in terms of a physical library’s use of shelves


—are the shelves being used properly or could the space be allotted for other types of books? I

want to see how this concept rolls over into digital libraries and determine if another approach

might be more efficient.

Distribution of resources

For this particular criterion, how are the resources laid out? Does their organization make sense

to a first-time or returning user? This evaluation will explore the different facets of the digital

library to see if they are logically laid out. A good layout shows that the digital library was well

planned and this is likely a good indicator of future development and institutional support. One

might think that layout makes little difference in future support, but I believe that if the

institution cares about small details like the page layout, then they care about the success of the

digital library into the future. If resources appear to be haphazardly organized, it indicates a lack

of accountability and responsibility on the part of the person/s in charge. Clear, well-defined

organization can make all the difference in increasing usage of a digital library.

Access

Being that there are many different types of digital libraries, there stands to reason that access be

an important issue among all of them. In order for a digital library to be successful, patrons must

have access to a wide variety of materials in several different formats. Are these digital libraries

easily accessible? Is the content built into the website, or do links lead to external sites? Are

materials available in full digitally, or in the case of some archives, is there simply a finding aid?

It is understood that not all materials can be available digitally at all times, but a great effort

should be made so that the resources that are available are readily accessible.


User needs & stakeholders

In this area, I will identify the various types of users and what their needs might be. The main

user groups that I plan on evaluating are: the public users, professionals in charge of the digital

library, and the decision-makers. What might each of these groups be looking for or what are

their goals when coming to the digital library website? Are those goals being met? Evaluation of

the different user perspectives can allow the digital library to see if their institutional goals are

being met.

Content

This last criterion will look at exactly what it says: the content. What is collected in the digital

library? Do the materials appeal to the intended demographic? How extensive is the metadata

and other technical data, if any? Without detailed and interesting content, the digital library is

useless. No matter how broad or narrow the scope of a digital library is, it needs to appeal to the

intended user group through its content. The layout and other issues discussed previously are key

factors as well, but without substantial content, there is little hope for a successful digital library.

The Digital Public Library of America

The Digital Public Library of America (DPLA) is a digital library that has been created as

a resource for a wide range of people. Its scope is national, and it covers topics on a great deal of

different topics. On the DPLA’s “About” page, it states:

The Digital Public Library of America brings together the riches of America’s libraries,

archives, and museums, and makes them freely available to the world. It strives to

contain the full breadth of human expression, from the written word, to works of art and


culture, to records of America’s heritage, to the efforts and data of science. DPLA aims to

expand this crucial realm of openly available materials, and make those riches more

easily discovered and more widely usable and used, through its three main elements: A

portal that delivers students, teachers, scholars, and the public to incredible resources,

wherever they may be in America, … A platform that enables new and transformative

uses of our digitized cultural heritage, …[and] An advocate for a strong public option in

the twenty-first century. (About, 2016)

This mission statement makes it very clear what the overall goals of the DPLA are and how they

intend to achieve them. One of the most important pieces of this mission statement states that “it

strives to contain the full breadth of human expression.” This is important to note because it

leaves the collection scope open ended and does not place limits on types of collected materials.

There may be some restrictions imposed because of copyright or other legal issues, but it is clear

that the Library will make every effort to express the culture and heritage of the people of

America.

In order for the DPLA to reach its extensive goals, they must have a great deal of support.

The DPLA has a governing board and several smaller committees which monitor the library, as

well as several contributing partners. The governing board is made up of public and research

librarians, technologists, authors, intellectual property scholars, media studies scholars, and

business experts. The smaller committees are broken into the Executive committee, Finance

committee, Governance committee, the DPLAfest committee, and the Partnerships and

Development committee (Board, 2016). Each of these committees has a clear definition and set

of responsibilities. The executive committee “will act for the Board, subject to the DPLA


bylaws, on matters requiring Board action between meetings, and other matters delegated by the

Board from time to time,” (Board, 2016). The finance committee’s

…[purpose] is to oversee the financial doings of DPLA; to review and evaluate DPLA’s

fiscal operation and its managers; to report to the Board and/or Executive Director on

DPLA’s finances, and/or any irregularities or issues; to provide advice and

recommendations to the Board of Directors, Executive Director, Director for Content,

and staff on how DPLA’s financial operations align with its mission, vision, and strategic

goals. An Audit subcommittee may be formed. … The Governance Committee has two

principal responsibilities: to promote the effective functioning of the Board and to

oversee the activities of all the operational committees. (Board, 2016)

The DPLAfest committee plans the annual DPLAfest conference. The partnerships and

development committee has quite a bit of responsibility with their main goal seeming to be

growing the DPLA in smart, decisive ways.

The Partnerships and Development Committee will make recommendations to the Board

on the ways in which DPLA can innovate and create strategic partnerships to further its

mission, such as DPLA’s efforts around eBooks. The Partnerships and Development

Committee will provide counsel on the development and implementation of DPLA’s

long-term sustainability plan(s), as well as identify and solicit funds from external

sources of support, in concert with the Business Development Director and Executive

Director. The committee will also provide advice and recommendations about potential

new markets/audiences and their needs, and how to meet those needs. (Board, 2016)

Meetings are held on a regular basis to discuss progress and any changes that might be

forthcoming. The partners submit various materials to be included in the collections. Some better


known libraries share materials like the National Archives and Records Administration and the

New York Public Library, as well as lesser known institutions like the Digital Commonwealth

and Indiana Memory. In total, there are thirty partners who contribute to the DPLA.

Usability

The DPLA has clearly taken the time to produce something with overall usability in mind. When

you go to the Library’s main webpage, the material is clearly and concisely presented (Digital,

2016). The different tabs and media panes present the main interest points of the DPLA. The

basic structure gives the user an easy site to navigate and explore. This structured looking design

indicates that other pages are organized and well-developed. As a researcher, simple design

makes using any web page much less stressful. In the case of a digital library, it will only help

people to navigate and learn the site in a more meaningful and effective way thus encouraging

them to use it on a regular basis.


The main webpage for the DPLA holds a great deal of information. As you can see in the images

below, the content is displayed in a way that makes navigation easy for the visitor. Along the top

of the page, there are clearly defined tabs with drop down menus for subtopics in the primary

navigation structure. A lot of this is background and technical information pertaining to the

library, but can be very helpful to first time and new users. In particular, the “About” and “Help”

tabs provide a great deal of information about the overall site and site navigation. I think that the

space on the main webpage is being used very effectively. The image on the top left changes

every few seconds which offers a glimpse into another featured collection. The rest of the data

panes have other featured content and correspond with the secondary navigation at the top of the

page.



The overall organization of the DPLA makes sense to any type of user who might utilize the site.

The right side of the page has three data panes with options to view all exhibitions, view by

place, or explore by date. The secondary navigation structure at the top of the page also has

options for exploration by map, timeline, bookshelf, and apps. It is appealing that the search box

states exactly how many items are currently in the holding for examination. The DPLA says that

they currently hold 13,253,889 items as of April 18, 2016. The main page also features


information about the various apps that have been produced by developers to use the Library’s

content in various ways. There are two main apps featured at this time, and I am assuming that

the displayed apps change as new ones are developed. There is also another pane along the

bottom right which features the DPLA Twitter feed and including real-time information from the

Library might attract a user who relies on social media. This social media connection could result

in relationships being developed with patrons. These relationships could be an asset when an

evaluation is done because their opinions will be honest and educated based on their usage of the

digital library.

Access

For the purposes of this research, I am looking at the collections by all exhibitions. I have

navigated through several of the collections, and they all seem to contain rather detailed

descriptions with sub-collections, or themes, displayed as well. Each individual collection has

information about credits and a citation along the left side of the page. The photos associated

with each collection have descriptions underneath. The screenshot below clearly displays the

various access points that the collections each have.


The DPLA has designed a layout which allows for increased access and usage. Once a patron

becomes familiar with the site, they will have no trouble navigating through other collections.

This standard view is cohesive, well-planned, and very user friendly.


As with most libraries, the public is the primary user group which they want to appeal to. I think

that most users will be using the DPLA for research or recreational purposes. The DPLA says in

their mission that they want to bring resources to students, teachers, scholars, and the public.

This distinct list of the intended users is interesting because the public is listed last. That further

supports that the main goal of the DPLA might not be for the general public. Many of the

collections are based on historical events and offer a wide variety of topics. This being said, the

users would likely use these collections for research. For the recreational user, they might be a

history fanatic who is looking for a large number of collections which have been centralized to

one website.

Another set of users that would have a vested interest in the DPLA are the board

members, small committees, and contributing partners. I have grouped them together because I


believe they all have a similar goal: help the DPLA succeed and become a reliable resource for

people around the world. They all work collaboratively to produce and share materials which

have interest among different demographics, age groups, and geographic locations.

I believe that the goals of the users are being met. The DPLA has made a concerted effort

to develop and display materials that are of a high quality and are from reputable institutions.

Their committees and staff seem to be dedicated to ensuring that the materials that the DPLA

hosts are of a high standard so that any users find the resources helpful.

Content

Based on cursory exploration of the content contained within the DPLA, it is very clear that time

has been taken to construct an impressive array of content. In the broad sense, the DPLA collects

materials from libraries, archives, and museums. Using a slightly narrower view, content can be

viewed by exhibition, map, timeline, bookshelf, or app. Overall, there are thirty main exhibitions

which cover topics like Activism in the US, Children in Progressive-era America, Indomitable

Spirits: Prohibition in the United States, and Race to the Moon. As stated earlier, each of the

individual exhibitions has themes which can be explored further. In the maps section of the site,

content can be viewed by format, contributing institution, partner, date, language, location, or

subject. Next to each navigation term, there is a list which outlines how many items are held

under that particular item. For instance, if you wanted to search under “moving image”, you

would be able to see that there are currently 22,385 moving images in the DPLA holdings. There

is also an interactive map if the user wants to navigate the maps in that manner. Each state has a

bubble over it with an estimated number of map holdings. When you click on a bubble, a box

pops up with a list of available resources from that state. The timeline option is laid out much

like the map one, and contains a graphic representation which is broken down by century with a


bar graph underneath is depicting how many items from that year are included in the DPLA. The

image below shows this.

As you can see, the older the material, the fewer items the DPLA has in their holdings. I find the

Bookshelf portion of the site navigation to be a bit cumbersome. It uses the same navigation

structure as the previous two, but I think because the library uses a large blue box for each book,

it is not the most user friendly display (seen below). Covers of books are not always available,

but a pictorial representation for some of the books may increase the usage in this area.

The final piece of the navigation structure is for the apps. I would be inclined to say this portion

of the site is not intended for the casual user. This portion of the site is meant to be used by


professionals who are linking the DPLA data to their own institution through the use of their in-

house developed app. Many of the apps have to do with metadata and content usage which is

most likely not what the casual user is looking for when coming to this site.

One of the most interesting parts of the DPLA site in terms of content is the fact that they

include their Metadata application profile under the “For developers” tab in the primary

navigation structure at the very top of the page. Once you navigate to this portion of the site, it

gives a description of the intended use of the Metadata application profile with a link to the

profile, as well as a link to the Introduction to the DPLA metadata model (Metadata, 2016). Both

documents provide a plethora of information about the content and description methods

employed by the DPLA. Many institutions do not make this material available, but I believe that

because so many other institutions are contributing material, it is in the best interest of the DPLA

to make the guidelines available so that their standard of excellence and quality is maintained

throughout.

Smithsonian’s History Explorer

The Smithsonian’s History Explorer is a site that might get increased usage simply based

on its name and assumed association with the Smithsonian museums in Washington, D.C. Based

on their “about” page,

[The] Smithsonian's History Explorer was developed by the National Museum of

American History in partnership with the Verizon Foundation to offer hundreds of free,

innovative online resources for teaching and learning American history. The site is

designed for use by K-12 teachers and students, afterschool program providers, families,

and individuals interested in lifelong learning. History Explorer's resources focus on

learning history by "reading" objects for the stories they hold about the nation and its


many peoples. Learning activities feature artifacts selected from over 3 million items in

the Museum's collections, and draw on the expertise of the Museum's renowned

curatorial staff. (About, 2014)

This mission statement is very clear that it is intended primarily for teachers and students to use

as an educational resource. Their approach is slightly different than many other libraries because

they state “resources focus on learning history by ‘reading’ objects for the stories they hold about

the nation and its many peoples,” (About, 2014). This idea that learning is supported by “reading

objects” is interesting because it approaches learning in a non-traditional manner. The History

Explorer is using their collection of physical objects to encourage growth and learning in a way

that is not often used. I think with the advent of libraries and the evolution of digital libraries,

this is becoming more common, but if far from second nature. As Anne R. Diekema et. al. state

in their article,

Digital resources on the web realize full potential only when they are found and used in a

meaningful way. Without access and use, they languish, a waste of potential and effort. In

short, unless digital resources find their way to users, they are not better than paper, print

and other non-digital formats that preceded them. (Diekama et. al., 2011).

It seems that one of History Explorer’s primary goals is to prevent exactly this scenario from

happening. By using their vast array of objects as teaching materials, they are bringing light to

items that otherwise would sit in storage somewhere until it was their turn to go on display.

Creating digital images of the physical object allows for greater usage and access, things that are

supremely important to any digital library.

The mission statement for History Explorer reminds me a bit of the definitions that

Borgman and Waters suggested. The History Explorer mission focuses on the user, the


community, content, specialized staff, and access. All of these main goals are outlined in one or

both of the digital library definitions. I think having a mission statement that so closely

resembles professional definitions of a digital library shows that great effort was put in during

the development stage of the Smithsonian’s History Explorer.

The History Explorer site does not have information about who contributes and monitors

the library other than noting that the Smithsonian National Museum of American History

Kenneth E. Bering Center and the Verizon Foundation are partners. Based on that information, I

would assume that the Smithsonian is responsible for supplying the information and

documentation while the Verizon Foundation supplies the financial backing to support the library

as an entity separate from the Smithsonian National Museum of American History. This seems to

be the best of both worlds. While the Verizon Foundation probably has some say in what content

is generated and displayed by the Smithsonian, they are most likely acting as the financial

support system. The Smithsonian is responsible for the creation and maintenance of all of the

content that is available on the History Explorer, including all of the teacher resources.

The Smithsonian’s History Explorer is a valuable site to be evaluated because it falls

outside of the normal view of what a digital library should be. A digital library is thought of as

an extension of a physical library in most cases, and this could be considered an extension of the

Smithsonian Museum, but it is also something entirely separate. The approach that the History

Explorer takes seems to be as an educational tool first with the support of a digital library acting

as a secondary influence. One cannot exist without the other, but the roles seem inverted.

“Instead of finding ways to integrate digital libraries material into existing and well used learning

environments, digital libraries act as an agent which offers novel tools for the educational

domain. This might be a very successful approach of those tools are appreciated and gain


popularity,” (Dobreva, 2015). This notion supports the History Explorer because the digital

library is simply the process by which teachers are teaching new concepts. This library offers a

vast number of ways for teachers to integrate the digital library materials into their lesson plans

without the teacher needing to do a lot of extra work or research. Bringing these resources all to

one place will allow increased usage and potentially bring people other than teachers to the site.

Once the students get used to using History Explorer in the classroom, they will eventually come

to use it at home as a dependable and preferred resource for research.

Usability

Being that the History Explorer is primarily intended for teachers, I would say that it is very easy

to navigate and learn. The primary navigation structure lays out the various resources available

to teachers and other users. There is clear information about navigation options with little extra

information. This allows the user to navigate more easily and to not waste time trying to find

what they are looking for.


The Smithsonian’s History Explorer is effective in its presentation of resources on the main page

(Smithsonian’s, 2016). The image below shows that there are two search options. The overall

search box is in the upper right corner while the explore search box is below that on the right.

The search box can be used for general searching. The explore search feature is designed for

much more specialized research. Users are able to explore and limit results by resource type,

grade level, historical era, and cross-curricular connections. Allowing teachers to limit searching

in this way allows them to find what they are looking for quickly and efficiently. There is also a

link at the top of the page for “how to use this site.” This is a wonderful option for first-time and

newer users. Sometimes this information is available on sites, but it is buried under other


information. Making the link so prevalent on the main page will allow users to become

comfortable with the site from the very beginning (Smithsonian’s, 2016).

The space on this main page is being used very well. Users are provided with a wide variety of

search methods. Having the Featured artifacts and resources so prevalent on the page also allows


teachers to view collections or items that might be new to the site or that are relevant for another

reason like an anniversary.


There are not an overwhelming number of resource access points on the main page. A user may

explore the content using the primary navigation structure at the top of the page, through either

of the two search features, or by the featured artifacts and resources. Compared to other digital

libraries, this structure is rather limited, but I feel that does not take away from the overall

effectiveness of the site. This organization makes sense for new users because it does not

overwhelm them with information and a great number of exploration options. Some might this

that this is a failure of the digital, but it should be seen as a great feature. More avenues to search

by could mean more opportunities to get lost or not be able to find your way to the intended

resource.

Access

Exploring the resources available through History Explorer is rather easy. The primary

navigation structure lets one search by lessons & activities, interactives & media, museum

artifacts, themes, books, teacher resources, and web links. When investigating each of the

options, you are able to scroll through collections or search using the search box at the top of the

page. There is secondary navigation at the right side of the page which allows further restriction

and/or manipulation of results. When delving further into a particular resource or collection, the

original documents are linked to external Smithsonian websites by clicking the “Get resource”

button along the top of the individual page. Many of these links are interactive in nature to

further promote interaction with the material. From what I can tell, all of the materials are

available in digital format through the various external Smithsonian websites. Some external


pages for the objects are more extensive than others, but they contain a great deal of information

regardless.


As previously stated, the primary users are teachers and educators. While the site states it

is really intended for anyone, most of the resources are designed for classroom adaptation and

integration. Teachers could use this site as a starting point when developing new lesson plans.

The public and other researchers should be considered to be the secondary users. For the most

part, they would use this site to support other research, not as a primary search tool. The

secondary users would most likely use one of the main Smithsonian websites for their research

and end up here for supporting information.

Content

The evaluation of the content contained within this site was done in a very basic way

through the primary navigation. While the resources are displayed in a meaningful way, their

contents can be a little more difficult to get to. Not that they cannot be found, but the process is a

little involved. All of the navigation tabs have the same display once selected. The interactives &

media screen is pictured below. The left side of the screen contains a list of available resources

with information about grade range, resource type, and date posted. The right side of the screen

contains several boxes that can be used to limit results by grade, resource type, historical era, or

cross-curricular connections. Each of these subdivisions contains other categories for further

limitation. By using this level of granularity, teachers are able to modify the results and focus on

finding resources that pertain to their intended lesson without having to search page by page.

Something as simple as allowing a teacher to limit the resources by grade level can make all the

difference in determining if the teacher uses History Explorer in the future.


When you click on a resource, in this case the “Miner’s cap and carbide lamp” under the

museum artifacts tab, you are brought to a page which displays information about the item. Each


item held by History Explorer has standard information about grade range, resource type, and

date posted with a brief description of the item. Here the right side of the screen does not contain

search limiters, but offers related resources and artifacts as well as keywords.


Having this content readily available makes overall site navigation much easier. Correlations can

be made and more substantial lessons can be developed. The keywords are important to note

because that indicates that there is most likely a standard metadata scheme that has been

implemented, though there is no reference made to any nor is there access to any sort of metadata

file.

The only search tab that really varies from the others is the Themes tab. When you select

it, you are brought to a page where collections of materials have already been organized by

relevant themes. These are featured collections and give teachers the capability to have major

lessons already arranged without needing to link them together. The Smithsonian cannot be

expected to do this for all materials, but for special collections it is definitely warranted and

helpful to the users.

Commentary

Used separately, or even in conjunction with one another, the Digital Public Library of

America and the Smithsonian’s History Explorer are both outstanding examples of digital


libraries. While their approaches are slightly different, even though both state their primary user

demographics are either teachers or students with the public being secondary. In terms of overall

approach, both digital libraries are effective in executing their mission. Their collections are

well-developed and easy to navigate. As someone who spends a lot of time navigating various

web pages, being able to use resources where planning is evident is refreshing.

It is difficult to see if there has been an effort to make these collections available over

time. While no one can predict future technologies, these digital libraries seem to be willing to

adapt even though they may not know what that means presently.

An important feature of many traditional libraries is that their collections are preserved

over time – they persist. An important feature of digital collections is a potential lack of

persistence. Libraries have no control whatsoever over persistence of digital journals,

indexes, and the like for which they have licensed access for a time. …Persistence may

become one of the most important criteria for digital libraries. (Saracevic, 2000)

The idea of persistence is something that many digital libraries are aware of. This may not be a

primary concern because the maintenance and updating of digital files can be time consuming

and costly. That does not mean that there should be no effort. Even if a journal ceases to be

published, libraries have a responsibility to try to maintain the published information in some

way. This might not always be possible, but library professionals should be prepared to make the

effort when and if it is ever necessary. Especially in the case of digital libraries, if no effort is

made to keep materials persistent and up-to-date technologically, there should be no effort to

begin with to develop them digitally.

Digital library evaluation: Are we too early or too late?


This question might seem next to impossible to answer. I think that we as a profession are

too early and too late in digital library evaluation. We are too early because we are unable to

predict what the future of digital libraries holds. With technology changing by the minute and

resource availability changing, digital libraries are left guessing what to do next and how exactly

to do it so that a quality product is created. I believe digital libraries are still in their infancy

stage. Many major libraries, archives, and museums have associated digital libraries that are

growing every day, and more people are using digital libraries as primary research tools. There is

still a lot of growing that digital libraries need to do yet in terms of standards and future

development.

We are too late in evaluating digital libraries because of the simple fact that there is little

in the way of evaluation methods or standards. There is minimal research out there, and libraries

have been struggling just to maintain their current collections let alone take the time to evaluate

them. It is of the utmost importance that some sort of evaluation standard be developed and set

forth as “best practice”. Many libraries use the Library of Congress subject headings or the

Dewey Decimal system to generate data. There are cataloging and metadata rules and standards

that libraries rely on for just about every aspect of their collections, whether physical or digital.

A major institution like the Library of Congress should develop evaluation guidelines so that

libraries can determine if their collections are being used to the best of their ability.

Suggestions for further research

This paper is far from a complete evaluation of the Digital Public Library of America and

the Smithsonian’s History Explorer. More research can definitely be done on the content and

metadata. This paper skimmed the surface of these areas, and there is definitely room for more

investigation to see what was documented, why, and how it was documented. Research can also


be done to see where the contents are lacking. Some collections are probably more developed

than others and research can be done to see if there is a reason behind this.

Research can also be done to truly gauge if digital library evaluation is a responsible

endeavor. Analysis of the resources needed in order to evaluate expenses, time, staff, and other

critical parts of any digital library evaluation can be done. Do libraries of different sizes need to

approach evaluations differently? Is completing a full-blown evaluation of a bigger digital library

more important for a bigger institution than a smaller one? Should certain materials be evaluated

more frequently than others?

The teacher resources are the parts of each digital library which should be explored the

most. Each site has a plethora of available teaching tools. Are they effective? Are they

appropriate for classroom use? Should they be adapted for different geographic regions,

demographics, age groups?

The role digital libraries play in education is growing. Memorizing information is not as

critical as essential skill in education anymore and students need to know how to find and

synthesize information. Digital libraries provide abundant quality resources for learners

to find information at any time and almost anywhere, allowing learners to develop new

skills rather than just acquire more information. (Diekama et. al., 2011)

The educational resources offered by these digital libraries may help teachers to reach students

who were struggling before. So much of our lives are centered on technology with kids being

more tech-savvy than adults in many cases. Bringing these technologies into the classroom will

help kids to interact with the materials in a way that they understand and are comfortable with.

Conclusion


Digital libraries might just be the way of the future for our field. They are allowing

access to materials that would otherwise remain in basements and filing cabinets until there was

reason to resurrect them. Digital libraries offer a user experience like never before. Digital

libraries can be used by any person with knowledge of computers for any purpose. They are

collecting materials of various formats and are broadcast to the world. Even though great strides

are being made, digital libraries are failing because there is no great effort being made to

evaluate them. Without proper evaluation, a true picture of usage is impossible. Analytics can be

run, but that only offers the hard data. Evaluation also needs to include humans. We are the

users, so we need to look at the collections and see if they work for us. Analytics and other

technical evaluation tools only prove that the digital libraries are effective for computers. That is

not the purpose of a digital library. The technical components are very important, but if the

digital material is not reaching the intended group of people in the intended manner, it is useless.

In order for digital libraries to effectively be evaluated, there needs to be a standard

procedure. First, a clear definition needs to be used to support all digital libraries and their

missions. While slight variation would be acceptable for specialized libraries, many would be

expected to honor the standard definition. Digital libraries also need to be evaluated using the

same criteria. Some of the most important evaluation criteria are usability, effectiveness and

efficiency, distribution of resources, access, user needs & stakeholders, and content. Mini-

evaluations can be done just using one or two of these criteria. By no means do these criteria

evaluate all aspects of a digital library. Evaluation needs to be done using a variety of human-

generated and technologically based tools. This can be time consuming and expensive, but will

be priceless to the institution. If a complete evaluation is done, they will have a clear picture of


how the library is being used. Suggestions for modification can be made and the library holding

can be strengthened.


References

About. (2014). Retrieved April 17, 2016, from https://historyexplorer.si.edu/about/

About. (n.d.). Retrieved April 17, 2016, from http://dp.la/info/

Board. (n.d.). Retrieved April 17, 2016, from http://dp.la/info/about/board/

Borgman, C. L. (1999). What are digital libraries? Competing visions. Inf. Process. Manage.,

35(3), 227-243.

Borgman, C. L., Bates, M. J., Bates, M. V., Efthimiadis, E. N., Gilliland-Swetland, A. J., Kafai,

Y. B., ... & Maddox, A. B. (1995). Social aspects of digital libraries. In Background

paper for UCLA-National Science Foundation Workshop.

Diekema, A. R., Leary, H., Haderlie, S., & Walters, C. D. (2011). Teaching Use of Digital

Primary Sources for K-12 Settings. D-Lib Magazine, 17(3), 4.

Digital Public Library of America. (n.d.). Retrieved April 18, 2016, from http://dp.la/

Dobreva, M., Angelova, G., Agre, G., Scicluna, R., Azzopardi, D., O'Dwyer, A., ... &

Polydoratou, P. (2015). Bridging the gap between digital libraries and e-learning.

CYBERNETICS AND INFORMATION TECHNOLOGIES, 15(4).

Metadata Application Profile. (n.d.). Retrieved April 19, 2016, from

http://dp.la/info/developers/map/

Nielsen, J. (2012, January 4). Usability 101: Introduction to Usability. Retrieved April 18, 2016,

from https://www.nngroup.com/articles/usability-101-introduction-to-usability/

Saracevic, T. (2000). Digital library evaluation: Toward evolution of concepts. Library trends,

49(2), 350-369.


Saracevic, T. (2016) Criteria and methods in evaluation of digital libraries: use & usability

[PowerPoint slides]. Retrieved from

http://comminfo.rutgers.edu/~tefko/Courses/e553/Lectures/

Smithsonian's History Explorer. (n.d.). Retrieved April 17, 2016, from

https://historyexplorer.si.edu/

Waters, D. J. (1998) What are digital libraries? CLIR (Council on Library and Information

Resources) Issues, No. 4. http://www.clir.org/pubs/issues/issues04.html#dlf

http://www.clir.org/pubs/issues/issues04.html#dlf

https://historyexplorer.si.edu/

http://comminfo.rutgers.edu/~tefko/Courses/e553/Lectures/

the digital public library of america (dpla) is a digital...

Documents