-
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
1/27
Overview
Process
Criteria
Resources
Evaluating, Selecting, and Implementing
Free/Libre/Open-Source Software
Marko Schtz
Department of Mathematical SciencesUniversity of Puerto Rico at Mayagez
Mayagez, PR
07 May 2009
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
2/27
Overview
Process
Criteria
Resources
Outline
1 Overview
2 Process
3 Criteria
4 Resources
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
3/27
Overview
Process
Criteria
Resources
Workshop Structure
presentation of background
independent work
review
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
4/27
Overview
Process
Criteria
Resources
Introduction
the problem
> FLOSS sought to fill some need
> wealth of FLOSS available: 100,000s of projects
stages of the process
identify candidate
review
compare
analyze
implement
requirements
criteria resources supporting the process
> existing evaluation frameworks
> project documentation templatesMarko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
5/27
Overview
Process
Criteria
Resources
Stages
involve user- and developer community
clarify and document requirements
identify candidates
review candidates
shortlist and prepare comparison
analyze results implement
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
O i
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
6/27
Overview
Process
Criteria
Resources
Clarify and Document Requirements
elicit
> user interviews> user surveys> brainstorming sessions
FLOSS development differs from proprietary development
> FLOSS: do one thing right and inter-operate> proprietary: cover many features, use lock-in (data formats)
to make users stay within application
> formulate neutral requirements
By e.g. requiring a single application for all identified needs
many possible FLOSS combinations will be ruled out
document use cases
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
O i
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
7/27
Overview
Process
Criteria
Resources
Identify Candidates
perform search on the Internet, including specialized sites:
http://sourceforge.net, http://directory.fsf.org,
http://freshmeat.net, http://debian.org,http://icewalkers.com, http://cpan.org.
research individual features/requirements to find
candidates
use name of well-known proprietary software with the termopen source on google
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://sourceforge.net/http://directory.fsf.org/http://freshmeat.net/http://debian.org/http://icewalkers.com/http://cpan.org/http://cpan.org/http://icewalkers.com/http://debian.org/http://freshmeat.net/http://directory.fsf.org/http://sourceforge.net/http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
8/27
Overview
Process
Criteria
Resources
Identify Candidates (cont.)
use existing lists
> A MITRE 2003 report lists 115 FLOSS generallyrecognized as safe
> The Generally Recognized As Mature (GRAM) contains39 FLOSS
> Les Trophes du Libre> Qualification and Selection of Open Source Software> sourceforge.net project of the month, community choice
awards> problem with all these lists: outdated
- can only be used as indicators
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://www.terrybollinger.com/dodfoss/dodfoss_html/index.htmlhttp://www.dwheeler.com/gram.htmlhttp://www.trophees-du-libre.org/http://www.qsos.org/http://sourceforge.net/community/index.php/potm/http://sourceforge.net/community/index.php/potm/http://www.qsos.org/http://www.trophees-du-libre.org/http://www.dwheeler.com/gram.htmlhttp://www.terrybollinger.com/dodfoss/dodfoss_html/index.htmlhttp://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
9/27
Overview
Process
Criteria
Resources
Review Candidates
use existing reviews and comparisons
mailing lists user and developer documentation
project statistics
code review
collect findings
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
10/27
Overview
Process
Criteria
Resources
Shortlist and Prepare Comparison
weed out using must-have, must-not-have requirements,
e.g. must have unencrypted, standards-based backup
format
feed back findings to user community
feed back findings to developer community
goal: less than a handful solutions (might consist of several
programs)
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
11/27
Overview
Process
Criteria
Resources
Analyze Results
document lessons learned in previous stages
weigh and evaluate criteria> absolute> relative ranking
obtain approval from intended users, administration,
management, customer
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
12/27
Overview
Process
Criteria
Resources
Implement
invite some early adopters
prototype alongside current solution collect and maintain automated black-box tests for most/all
use cases
allow for ample transition time
encourage constant feedback from users/stake-holders
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
13/27
Process
Criteria
Resources
Criteria
often criteria are competing
> example: code size vs number of features
kinds of criteria
> requirements
> project community
> project source code
> project evolution> deliverables> trials
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
14/27
Process
Criteria
Resources
Community
developers (number of, background of)
conferences
project activity (recent/sustained)
some projects have activity bursts and times where they
fall asleep
size, organization
leadership mentoring
GSoC, etc
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
15/27
Process
Criteria
Resources
Source Code
technologies
> portability hidden issues: Java may be less portable thane.g. C
> teams existing competence in those technologies
> teams desire, capability to learn these technologies> extends to e.g.
- programming language- database system- desktop environment- communications protocols
How well is the implementation documented?
code size
> is an expenditure!!! bigger is worse, not better> all that code has to be maintained
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
16/27
Process
Criteria
Resources
Project Evolution
How long does the project exist?
history
> How has it proceeded since then?> Have there been prolonged phases of inactivity?> How has developer community evolved?> Major obstacles/challenges faced?
maturity
> Is it based on a tried concept, architecture?
> What are the revisions it went through?> Will be different for different kinds of applications> Installations
- What are the showcase deployments/installations?- How many installations are there?
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
P
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
17/27
Process
Criteria
Resources
Deliverables
usability
quality
security
feature list: breadth/focus
documentation
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
P
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
18/27
Process
Criteria
Resources
Deliverables: Usability
often misused:
> ease of installation, initial startup: valuable for casual, newusers
> frequent/power users need growth path, e.g.:
- scripting- external control- plug-ins
setup prerequisites
> which are shared with other software that is likely to beused? e.g. requires KDE, requires PostgreSQL.Amortization.
> done once, updates should be rare. Amortize over usernumber, usage time.
> estimated time to update to future versions
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
Process
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
19/27
Process
Criteria
Resources
Deliverables: Quality
minor releases & point/patch releases
> criteria for version number increase> developer oriented might rely of code repository
open bugs
> bugs need to be reported> how many reporters?> what kind of bugs are reported?> assignment of appropriate priority> bug-fixing sprints held> avoidance of regressions> reactivity to security relevant bugs> security relevant bugs relative to software class> what is considered critical?> as with most criteria here: absolute numbers mean little
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
Overview
Process
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
20/27
Process
Criteria
Resources
Deliverables: Security
security vulnerabilities
> how many> how critical> over what period
out-of-box configuration secure?
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
21/27
Process
Criteria
Resources
Deliverables: Feature List
the project might be too broad then
> risk the team doea not develop sufficient expertise in onethe individual features
> one of the most successful and longlived approaches hasbeen the UNIX approach of small tools that do one thing,one thing only and that do that one thing right.
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
22/27
Process
Criteria
Resources
Trials
form an early adoption team: a group of users who start
using the software in real productive work: must be users
who want to use the new software
early adoption team needs to feed back as much as
possible on their experiences, issues, problems, questions,
surprises, etc
very carfully listen to early adoption team
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
23/27
Process
Criteria
Resources
Business Readiness Rating (BRR)
http://www.openbrr.org
evaluation methodology
provides
> template> some sample evaluations
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://www.openbrr.org/http://www.openbrr.org/http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
24/27
Criteria
Resources
Qualification and Selection of Open Source Software
(QSOS)
http://www.qsos.org
evaluation methodology
provides
> template> browser-based software to work with templates> sample evaluations
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://www.qsos.org/http://www.qsos.org/http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
25/27
Criteria
Resources
ReadySET
http://readyset.tigris.org
project documentation library
provides
> XHTML-based forms for software-engineering projects
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://readyset.tigris.org/http://readyset.tigris.org/http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
26/27
Criteria
Resources
Discussion
BRR and QSOS are often too rigid, e.g.:
> QSOS only allows three values per criterion: no, partial, full> BRR and QSOS strive for evaluation of criteria independent
deployment context- e.g. QSOSs books available- cannot be compensated with assignment of weights
> both can be applied with regard to deployment contextworst case: loss of comparability
ReadySET sometimes too software-development oriented
strip out parts too software-development oriented
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
OverviewProcess
http://goforward/http://find/http://goback/ -
8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software
27/27
Criteria
Resources
Independent Work
pick domain you know something about e.g.
> database systems>
photo management software> course management software> statistics software> data visualization> enterprise resource planning
develop a realistic scenario use the process described
Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/