presentation from 2007 dinner meeting
Post on 30-May-2018
216 Views
Preview:
TRANSCRIPT
-
8/9/2019 Presentation from 2007 Dinner Meeting
1/34
Metrics in Risk Determinationfor Large-Scale Distributed
Systems MaintenanceMaureen Ann Raley Advisor: Letha Hughes Etzkorn
University of Alabama in Huntsville Computer Science Department
Telephone: 703-325-3510 University of Alabama in Huntsville
Maraley@comcast.net Telephone: 256-842-6291
letzkorn@cs.uah.edu
-
8/9/2019 Presentation from 2007 Dinner Meeting
2/34
Nationwide Computer System
Upgrade
Upgrade networked computers, distributed throughoutthe continental US
Systems consisted of a mix ofmainframe computersand client/server systems used in a financialenvironment
Lessons-learned survey concentrated on the client
computers or workstations (no mainframes or servers) Data entry centers
Computer centers (in-house software developed and maintained)
Field offices (operations)
-
8/9/2019 Presentation from 2007 Dinner Meeting
3/34
Nationwide Computer System
UpgradeCondition: Firm deadline
Non-compliant systems at deadline would be either: Removed from operational use and transferred/disposed
Disconnected from the network and accorded strictlystand-alone status
No transfer of data other than hardcopy printout
Prohibited Sneaker-net transfer by floppy drive, modem, orother electronic means
-
8/9/2019 Presentation from 2007 Dinner Meeting
4/34
Nationwide Computer System
UpgradePurpose of the upgrade
Standardize the software and hardware throughout theagency
Modernize the computers and software components
Dispose of obsolete components
Introduce a layered operating system with administrativeand user privileges
-
8/9/2019 Presentation from 2007 Dinner Meeting
5/34
Nationwide Computer System
Upgrade
Expected benefits
More conducive to configuration management controls
Easier to maintain and upgrade
Barriers to prevent unauthorized software (games, favorite
programs, instant messaging, peer-peer networking) More secure and less susceptible to external or internal
hacking
-
8/9/2019 Presentation from 2007 Dinner Meeting
6/34
Nationwide Computer System
UpgradeSoftware
Predominately COTS Business automation application suites
Operating systems Network
Individual workstations
But also . . . programs developed in-house data entry and data analysis
And . . . in-house customized COTS-based systems
-
8/9/2019 Presentation from 2007 Dinner Meeting
7/34
Nationwide Computer System
Upgrade
Hardware
Commercial grade desktop computers
No consistency as to
Age
Manufacturer RAM, HD, I/O, capabilities
-
8/9/2019 Presentation from 2007 Dinner Meeting
8/34
Nationwide Computer System
Upgrade
System Priorities
Availability
Data integrity
Performance
Security
-
8/9/2019 Presentation from 2007 Dinner Meeting
9/34
Nationwide Computer System
UpgradeLessons Learned
94 personnel interviewed at 12 different locations
Data entry clerks
Inventory control specialists
Information technology personnel
Middle and upper management
Secretaries and administrative assistants.
-
8/9/2019 Presentation from 2007 Dinner Meeting
10/34
Nationwide Computer System
UpgradeLessons Learned Questionnaire
1. What were the biggest obstacles you found in
achieving nationwide compliance?2. Based on what you have learned during the
nationwide compliance effort, what would youwant to see done differently if we were starting
the effort now?3. How is training being handled in your
organization? Are there adequate resources? Istime allocated for training?
-
8/9/2019 Presentation from 2007 Dinner Meeting
11/34
Nationwide Computer System
UpgradeThe responses note:
1. Management change in direction
2. Lack of understanding in the side effects of managementdirectives
3. Need for bottoms-up input into management decisions
4. Status assessment based on an inaccurate inventory database
5. Understaffing
6. Mismatched arrival of hardware and software7. Unrealistic compatibility testing
8. Loss of functionality in the replacement software
9. Inadequate training
-
8/9/2019 Presentation from 2007 Dinner Meeting
12/34
Nationwide Computer System
Upgrade1. & 2.Change in management direction and
inadequate side effect analysis caused rework
Upper management would direct paths to the goals thatwere always not the most reasonable or efficient.
Management failed to identify all the side effects oftheir direction and the implications to the end users.
Management would also direct action, then recall thatdirective, and direct a different approach. This causedfrustration and rework.
-
8/9/2019 Presentation from 2007 Dinner Meeting
13/34
Nationwide Computer System
Upgrade3.Lack of bottoms-up input from users
Project office held weekly conference calls on nationwide
status and twice weekly conference calls on issues, as wellas proactive, engaged working groups consisting of thenational office and field coordinators, but . . . Some of these issues were due to management issuing directives
without field input or vetting.
Sometimes field-level management made the decisions for theirend-users without soliciting input.
Ensuring a bottoms up comment process would have helped to
mitigate some of the rework.
-
8/9/2019 Presentation from 2007 Dinner Meeting
14/34
Nationwide Computer System
Upgrade1., 2. & 3. A process to vet upper management
decisions could help to reduce the amount of
rework by the end users.Choose a set of end users at diverse locales to beta
test the management directives.
Distribute proposed directives for comment and ensure
both headquarters personnel and field users, reviewedthese proposals.
Direct field management to allot time to end users toensure a meaningful review.
-
8/9/2019 Presentation from 2007 Dinner Meeting
15/34
Nationwide Computer System
Upgrade4. Inaccurate inventory database
The IS inventory database was queried weekly to provide reports on
the HW and SW compliance status. Inaccuracies largely due to the small, mobile nature of the
computers and the large numbers of them to track.
Effort to correct IS database was being worked by another office,but did not coincide with the project office milestones.
As the inventory database became more accurate, a realisticestimation of the work done and yet to be done became clearer asthe project deadline approached.
Modern inventory control methods, such as radio frequencyID(RFID) tagging could be used for more accurate tracking of small,easily movable components.
-
8/9/2019 Presentation from 2007 Dinner Meeting
16/34
Nationwide Computer System
Upgrade5. Excessive workload, staffing shortages
F
ield coordinators worked the upgrade effort in addition totheir normal workload. Because of this, some field coordinators were not as dedicated to
the upgrade effort as others
Additional staff (skilled support) was used at HQ
Management should also augment staffing levels in thefield.
-
8/9/2019 Presentation from 2007 Dinner Meeting
17/34
Nationwide Computer System
Upgrade6. Mismatched arrival of replacementHW/SW
Delays were as long as several months, due to vendor supply shortages
Inadequate storage space for new HW platforms arriving first
Old HW platforms could not be removed without completereplacements
Incompatible or upgrade violation: new SW and old HW; new HW andold SW
Installation of new SW on old incompatible platforms caused systemsfailures and rework for reformatting and reinstallation of old SW.
-
8/9/2019 Presentation from 2007 Dinner Meeting
18/34
Nationwide Computer System
Upgrade7.Lack of realistic compatibility testing
Compatibility testing was not independent and had no oversight.
The division overseeing the upgrade did the compatibility testing.
Testing was done on idealized machines with the new COTSsoftware suite at HQ, not in the field, not by field end-users, notwith in-house field application, and not under field workloadconditions.
Most of the software was compatible, BUT some criticalincompatibilities existed with the field applications not used at HQ.
Testing should have followed software engineering bestpractices independent tests, oversight, and under operationalconditions.
-
8/9/2019 Presentation from 2007 Dinner Meeting
19/34
Nationwide Computer System
Upgrade8.Loss of functionality in the new software
Transition from one office automation software suite (i.e. Word
Perfect, Lotus, Oracle, email) to another (MS Word, Excel, Access,Outlook).
Data-entry transition: line-entry to graphical, mouse-driven system.
In-house developed replacement programs incurred the "loss of
functionality" complaint less often than COTS software.
In-house programming team came closer to maintaining the basicfunctions of the old in-house software.
"Not all the right features" and "too many unneeded features" issues arecommon to COTS based systems (not tailored to a specific task, butinstead are developed for mass market sales).
A more rigorous COTS selection process could mitigate theinadequacies.
-
8/9/2019 Presentation from 2007 Dinner Meeting
20/34
Nationwide Computer System
Upgrade9. Inadequate training
Usually did not address the new skill sets needed. Not always tailored for the different skill levels.
Frequently expected to be self-taught.
Expected to occur in addition to the normal workload.
-
8/9/2019 Presentation from 2007 Dinner Meeting
21/34
Nationwide Computer System
Upgrade9. Inadequate training
The most successful training: combination of introductory overview,
followed by hands-on classroom training with a knowledgeable andmotivated instructor.
Unsuccessful training:
Sometimes none at all
Unmotivated, poorly trained instructors
Self-study -- a CD ROM at one's desktop, (ok if computer literate,overwhelming if not)
Information technology and computer specialists adapted with ease.
Non-computer oriented people, such as secretaries, administrativeassistants, and data entry clerks, had much more trouble.
-
8/9/2019 Presentation from 2007 Dinner Meeting
22/34
Nationwide Computer System
UpgradeBenefits
Uniform nationwide software and hardware
More effective configuration management Easier maintenance and upgrades installation
Cyclical replacement or upgrades hardware andsoftware easier to plan and effect
Layered operating systems (administrative user
privileges) inhibit installation of ad hoc end usersoftware
End Game Assessment has applications for recoveryfrom hostile information system attacks
-
8/9/2019 Presentation from 2007 Dinner Meeting
23/34
Observations Not all risks are known and can be planned for in
advance
This project mitigated many of the problemsencountered during the transition by:
Consistent monitoring of the hardware and softwarecomponents compliance status using inventory data
An issues database, addressed weekly A risks database for issues with a probability of
occurrence with negative consequences, addressed everytwo weeks
-
8/9/2019 Presentation from 2007 Dinner Meeting
24/34
Possibilities for
Future Research
Quantification of the loss due to rework and theeffect of double or treble responsibilities on the
lower level staff (data not available)
Further investigation on the consequences ofeffective and ineffective management decisions
Development of metrics for risk analysisDevelopment of metrics for risk analysis
-
8/9/2019 Presentation from 2007 Dinner Meeting
25/34
Nationwide Computer System
UpgradeActual Compliance Growth from Inventory Data
Compliance Growth -- Field Division
0
20
40
60
80
100
120
1 3 5 7 9 11 1 3 1 5 17 1 9 21 23 2 5 27 2 9 3 1 33 3 5 37 39 4 1 4 3 45 4 7 4 9 51 53 5 5 57 5 9 6 1
Week
Compliance Growth -- IS Division
0.00
20.00
40.00
60.00
80.00
100.00
120.00
1 3 5 7 9 11 1 3 15 1 7 1 9 21 2 3 25 2 7 2 9 31 3 3 35 3 7 3 9 41 4 3 4 5 47 4 9 51 5 3 5 5 57 5 9 61
Week
Fi gur e 1 . P e r c e nt a ge Fi e ld Co m p li ance Gro w th Fi gu r e 2. P e r c e nt a ge IS C o m pl ia nce G row th
-
8/9/2019 Presentation from 2007 Dinner Meeting
26/34
-
8/9/2019 Presentation from 2007 Dinner Meeting
27/34
Metrics Considerations
The determination of an appropriate set of metrics to analyze risk during themaintenance phase of a distributed system upgrade
Standard (actual data), as well as normalized metrics Normalizing would deal with the varying number of devices (with this sample data, the total number of units in
the system changes as new units are added, old ones are disposed of, and as the inventory accuracy grows). Bynormalizing the metric suite, we can compare distributions of different size.
An adaptive sizing model that deals not with the total number of systemdevices (units), but only with the units modified during a certain period
A period of time, i.e., a week
A time-independent period, or threshold, determined by number of devices, i.e. 100. Using this model, insteadof dividing by the actual number of devices in the system, as in the standardized model, would be divided by thenumber of recently modified devices.
A sliding window might used as well.
A history complexity metric for each location (component) to assess the effectof the complexity of the period.
This could help determine if risk increases during bursty or chaotic periods and during periods of high activity.
Validation: Statistical analysis of the results of the actual compliance growth(from the inventory data) vs. the results of the alternative metrics.
-
8/9/2019 Presentation from 2007 Dinner Meeting
28/34
Ta le 1. Soft are etrics Sets Com ariso#
o
t areT& Panel(
T P)
o
t are ngineeringInstitute(
I)
overnment
-P-70-14
-9-70-14
b
ective
1 ost ---------- ----- ost eviations
Trackdevelopmentexpenditures
2
chedule
cheduleProgress
chedule
eviationsTrack
progress vs.schedule
eadiness to
proceed tonext phase
3
omputer
esource
tilization
omputer
esource
tilization
omputer
esource
tilization
Trackplanned v s.actualmemoryutilization,I/
channels,throughput
4
o
t are
ngineering nvironment(
level)
---------------
o
t ar e
evelopmentTools
ssessmento
contractorsdevelopmentenvironment
5
equirementsTraceability
--------------- ---------- ----- Tracesrequirementsto design,code, andtest
6
equirements
tability
o
t areVolatility, nit evelopmentProgress
equirements e
inition &
tability
easureschanges inrequirementsand thee
ect ondevelopment
e
ort7 esign
tability
o
t areVolatility, esign omplexity
esign
tructureTrack designchanges andtheir e
ecto
developmentandcon
igura tion8 omplexity
o
t are
ize esign omplexity
--------------- Provides
indication o
potentialproblemareas heretest e
ort
should be
ocused.
9 Breadth o
Testing
TestingProgress
Test
overage
easures theextent andsuccess o
testcoverage.Providesindication o
su
iciencyo
testing.
10
epth o
Testing
TestingProgress
Test u
iciency
easures thedegree to
hich therequired
unctionalityhas beensuccess
ullydemonstrated
11 FaultPro
iles---------------
e
ect (Fault)
ensity
easures thenumber o
aults vs.time. Tracksopen, closedtroublereports by
priority
12
eliability --------------- ---------------
ttempts to
predictsystemdo ntime bytrackingtroublereports usingmath models
13
anpo er(optional)
Personnel
evelopment
anpo erTracks
personnelloading andturnover
14
evelopmentprogress(optional)
--
esignProgress
!
nitdevelopment
progress
TestingProgress
evelopmentProgress(
ompleteness)
Providespercentageo
developmentcompletion
across allphases and
orkelements
15 Incremental
elease
ontent
--------------- Tracksschedule andunits perrelease totrack
unctionalpreservation
16upportability Indicates
ho easy/di
icultthe so
t ar e
ill be tomaintain
-
8/9/2019 Presentation from 2007 Dinner Meeting
29/34
OneP
ossibility - Entropy Metrics Essentials of mathematical theory of information - A
Mathematical Theory of Communication, Claude Shannon(1949)
Established fundamental bounds on the performance ofcommunication systems in the presence of noise
Exerted an enormous influence on many disciplines --communications, biology, mathematics, physics.
Entropy has been defined in terms of the information content of
software and used to measure code complexity Also has been used effectively as an indicator for reusability
-
8/9/2019 Presentation from 2007 Dinner Meeting
30/34
Entropy Metrics
Have been used successfully to:
Measure SW quality during SW development
Used directed graphs to model a SW system and measure coupling, cohesion,size, length, complexity at module level (Allen, Khoshgoftaar, et al, 1996 -present)
Measured complexity in object-oriented design (Davis, Etzkorn, Bansiya,Gholston, et al, 1999 - present)
Measure SW complexity during SW maintenance
Based on message flowbetween modules, extended to COTS (Chapin, 1988)
Measure Cost Growth during large-scale system development
Queried experts to develop a cost model validated to 3% accuracy vice 300%predicted (Martin, Lenz, Glover, et al, 1981)
Measure SW complexity using process entropy (not code)
Theorized the number of times a module was modified adversely affected codecomplexity, validated 13%-45% improvement (Hassan Holt, 2003)
-
8/9/2019 Presentation from 2007 Dinner Meeting
31/34
-
8/9/2019 Presentation from 2007 Dinner Meeting
32/34
B
ackup Slides
-
8/9/2019 Presentation from 2007 Dinner Meeting
33/34
-
8/9/2019 Presentation from 2007 Dinner Meeting
34/34
Shannons EquationC.E. Shannon, in A Mathematical Theory ofCommunication, (1948) proposed to measure the amount
of uncertainty, or entropy, in a distribution by the followingequation:
n
Hn(P) = -7 (pk* log2pk)
k=1
n
where pku 0, k 1, 2, . . . n and 7pk= 1
k=1
top related