![Page 1: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/1.jpg)
NMFS Use Case 1 review/ evaluation and next steps
April 19, 2012Woods Hole, MA
Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI)*Tetherless World Constellation, ** AOP&E
![Page 2: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/2.jpg)
Modern informatics enables a new scale-free framework approach
• Use cases• Stakeholders• Distributed
authority• Access control• Ontologies• Maintaining
Identity
![Page 3: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/3.jpg)
What’s happened since ..
• Implementation– Technology assessment– Leverage infrastructure– Rapid prototyping
• Now
• Evaluation• Open world, iteration
![Page 4: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/4.jpg)
Review and evaluation
![Page 5: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/5.jpg)
References
• Twidale, Randall and Bentley (1994) and references therein
Twidale, M., Randall, D. and Bentley, R. 1994, Situated evaluation for cooperative systems, Proceedings, Comp. Supp. Coop. Work 1994, Chapel Hill, NC, pp. 441-452.
5
![Page 6: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/6.jpg)
Metrics
• Things you can measure (numerical)• Things that are categorical
– Could not do before– Faster, more complete, less mistakes, etc.– Wider range of users
• Measure or estimate the baseline before you start
6
![Page 7: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/7.jpg)
Result/ outcome
• We will refer to the use case document• Outcome (and value of it) is a
combination of data gathering processes, that may include surveys, interviews, focus groups, document analysis and observations that will yield both qualitative and quantitative results. I.e. a discussion (today)
• Did we meet the goal?7
![Page 8: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/8.jpg)
Example: what we wanted to know about VSTO
• Evaluation questions are used to determine the degree to which the VSTO enhanced search, access, and use of data for scientific and educational needs and effectively utilized and implemented a template for user-centric utilization of the semantic web methodology.
• VO – appears to local and integrated and in the end-users language (this is one of the metrics)
8
![Page 9: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/9.jpg)
Evaluation (Twidale et al.)
• An assessment of the overall effectiveness of a piece of software, ideally yielding a numeric measure by which informed cost-benefit analysis of purchasing decisions can be made.
• An assessment of the degree to which the software fulfils its specification in terms of functionality, speed, size or whatever measures were pre-specified.
9
![Page 10: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/10.jpg)
Evaluation• An assessment of whether the software fulfils the
purpose for which it was intended.• An assessment of whether the ideas embodied in
the software have been proved to be superior to an alternative, where that alternative is frequently the traditional solution to the problem addressed.
• An assessment of whether the money allocated to a research project has been productively used, yielding useful generalizeable results.
10
![Page 11: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/11.jpg)
Evaluation
• An assessment of whether the software proves acceptable to the intended end-users.
• An assessment of whether end-users continue to use it in their normal work.
• An assessment of where the software fails to perform as desired or as is now seen to be desirable.
• An assessment of the relative importance of the inadequacies of the software.
11
![Page 12: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/12.jpg)
(Orthogonal) Dimensions of evaluations
Structured Less structuredQuantitative Qualitative
Summative Formative
Controlled experiments Ethnographic observations
Formal and rigorous Informal and opportunistic
12
![Page 13: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/13.jpg)
Formative and Summative
• evaluation carried out for two reasons:– grading translations = summative
evaluation (when the guests taste the soup)
– giving feedback = formative evaluation (when the cook tastes the soup)
13
![Page 14: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/14.jpg)
Iterating
• Evolve, iterate, re-design, re-deploy– Small fixes– Full team aware of the evaluation results and
implications– Decide what to do about the new use cases, or if
the goal is not met– Determine what knowledge engineering is
required and who will do it (participants in the evaluation may become domain experts)
– Determine what new knowledge representation– Assess need for an architectural re-design
14
![Page 15: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/15.jpg)
Summary
• Project evaluation has many attributes• Structured and less-structured• Really need to be open to all forms• A good way to start is to get members of your
team to do peer evaluation• This is a professional exercise, treat it that way
at all times• Other possible techniques for moving forward
on evolving the design, what to focus upon, priorities, etc.: SWOT, Porter’s 5 forces
15
![Page 16: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/16.jpg)
Summary
• By now, the reality of going into complete detail for the prototype implementation should be apparent
• Keeping it simple is also very important as you begin to implement
• Being prepared to iterate is essential• Now is the time to validate the models with
domain experts and the team
16
![Page 17: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/17.jpg)
Questions?
![Page 18: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/18.jpg)
Back shed
![Page 19: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/19.jpg)
Use case!
![Page 20: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/20.jpg)
Use Case• … is a collection of possible sequences of
interactions between the system under discussion and its actors, relating to a particular goal.
• The collection of Use Cases should define all system behavior relevant to the actors to assure them that their goals will be carried out properly.
• Any system behavior that is irrelevant to the actors should not be included in the use cases.– is a prose description of a system's behavior when
interacting with the outside world.– is a technique for capturing functional requirements of
business systems and, potentially, of an ICT system to support the business system.
– can also capture non-functional requirements
![Page 21: NMFS Use Case 1 review/ evaluation and next steps April 19, 2012 Woods Hole, MA Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI) *Tetherless World](https://reader033.vdocuments.us/reader033/viewer/2022052918/5a4d1ba07f8b9ab0599c7179/html5/thumbnails/21.jpg)
Developed for NASA TIWG
Use case format
• Use case name• Goal• Summary• Triggers• Basic flow• Alternate flow• Post conditions• Activity diagram• Preconditions in tabular form• Notes