towards a new paradigm to resolve the software crisis
Post on 18-Oct-2014
603 views
DESCRIPTION
February 5, 2003: “Towards a New Paradigm to Resolve the Software Crisis”. Talk given at University of North Carolina, Chapel Hill.TRANSCRIPT
Cover Page
Uploaded June 27, 2011
Towards a New
Paradigm to Resolve
the Software Crisis Author: Jeffrey G. Long ([email protected]
Date: February 5, 20033
Forum: Talk presented at the University of North Carolina, Chapel Hill.
Contents
Page 1: Proposal
Pages 2‐28: Slides (but no text) for presentation
License
This work is licensed under the Creative Commons Attribution‐NonCommercial
3.0 Unported License. To view a copy of this license, visit
http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
Towards A New Paradigm to Resolve the Software Crisis Various kinds of metrics have been used to indicate the scope and magnitude of the software crisis. Some of these pertain to the high percentage of uninstalled systems; others to the increasing length of the applications backlog in most shops; others to the high lifetime cost of developing and maintaining software, whether custom-made or packaged; and others to the opportunity costs. All of these metrics indicate that the software business continues to be in a state of crisis which persists in spite of technical advances such as database management systems; structured code; improved and more formal development practices; the use of packaged software, code generators, computer-aided software engineering tools; component-based, object-oriented, multi-tier software architectures; and colossal budgets. The problem becomes worse as society expects more from computers and becomes more dependent upon them. User and organizational frustrations with this problem have caused CIOs to be replaced regularly, and, increasingly, have resulted in a strong trend to simply outsource all MIS computer and software operations -- and blame. This talk presents a new paradigm that has been developed and tested since 1985. It is based on an analysis of the nature of rules, their underlying common structure, and how rules are and ought to be represented. The approach is called “Ultra-Structure”. Rules are typically hard-coded into software applications, and the maintenance of these rules as they change is the primary cause of subsequent software changes. One unfortunate consequence of this standard approach is that subject experts must somehow communicate the rules they wish to see automated to programmers who typically are not experts in the subject matter of the application; much is lost in the translation. Another unfortunate consequence is that the software becomes large, often millions of lines of code, such that no one involved in the project can really comprehend or manage it. There have been of course numerous initiatives directed at solving these problems, but the solutions have been ineffectual and the problems they address are actually secondary and symptomatic rather than primary. The basic premise of this new approach is that we can resolve all of the major difficulties with software development, and generate significant new kinds of benefits for users, by removing most rules and all knowledge of the world from software and instead representing them the same way we represent data, i.e. as tables in a relational database. This approach combines key features of the normally disparate areas of management information systems, expert systems, and simulations, borrowing the strengths of each and largely eliminating the known problems of each. The approach can be applied to any kind of rule-based system, whether the application is intended to support business, scientific, or other application areas. It has been applied thus far to business and natural language applications, and an early prototype of its applications to biology has been developed by Dr. Gidding’s laboratory.
Towards A NewTowards A New Paradigm to Resolve the
S f C i iSoftware Crisis
Jeff Long
Feb 5, 2003
P d A dProposed Agenda
C t di dCurrent paradigm and consequences thereof (15 minutes)
Proposed alternative paradigm and consequences thereof (15 minutes)
Technical analysis of rules and their implementation (15 minutes)p ( )
General Q&A (15 minutes)
2/5/03 Copyright 2003 Jeff Long 2
P diParadigm
A f l ki t bj tA way of looking at a subject
An example, pattern, archetype, or model
A set of unconscious assumptions we have about a subjecthave about a subject
2/5/03 Copyright 2003 Jeff Long 3
C t S ft P diCurrent Software Paradigm
All li ti d fi d i t fAll applications are defined in terms of algorithms and data
Algorithms are the rules which are used to manipulate datato manipulate data
The model for this is the abacus
2/5/03 Copyright 2003 Jeff Long 4
Algorithms are implemented as softwareAlgorithms are implemented as software Everything else, such as inventory quantities or employee names isquantities or employee names, is implemented as data
Thus most rules (e.g. work processes, decision trees) are implemented asdecision trees) are implemented as software
2/5/03 Copyright 2003 Jeff Long 5
I di t CImmediate Consequences
E li ti i l t fEvery application requires a lot of software, often >1 million lines
No one can really understand or manage this level of complexity
Th b i ftThere are numerous bugs in any software system
Th ft h t b h d bThe software has to be changed as bugs are found and eliminated
2/5/03 Copyright 2003 Jeff Long 6
Th i l t f “ i t ” f thThere is a lot of “maintenance” of the software
The software has to be changed as the rules change
R l h bRules change as users become more sophisticated
R l h d t t lRules change due to external pressures as well
2/5/03 Copyright 2003 Jeff Long 7
Th h t ft th lThe changes to software themselves introduce new bugs
Extensive testing is required in order to have any comfort level
E t i d t ti i b tExtensive documentation is necessary, but often skipped; maintenance programmers don’t know why system was so designeddon t know why system was so designed
2/5/03 Copyright 2003 Jeff Long 8
Additi l A tiAdditional Assumptions
Users know what they want and canUsers know what they want and can communicate that to programmers
Programmers who have experience in anProgrammers who have experience in an industry can understand user requirementsUser requirements can be documented in aUser requirements can be documented in a form that is less complex than the actual working system
Software can be designed the same way as any other complex technology
2/5/03 Copyright 2003 Jeff Long 9
R lt t D tResults to Date
Increasing application backlog (maybeIncreasing application backlog (maybe use open source software)2/6 development projects cancelled;2/6 development projects cancelled; additional 3/6 considered failuresTrue value of very good designers andTrue value of very good designers and programmers increasingly recognizedBugs have greater and greaterBugs have greater and greater consequences; viruses don’t help
2/5/03 Copyright 2003 Jeff Long 10
Hi h t f Chi f I f tiHigh turnover of Chief Information Officers (average tenure: 18 months)
Increasing use of packagesThis typically forces changes in work processes, or else requires expensive customization (>2x package price)
Increasing use of outsourcing upon surrender (35-45% by 2005)
2/5/03 Copyright 2003 Jeff Long 11
L L d?Lessons Learned?Subject experts cannot communicate allSubject experts cannot communicate all requirements to programmers
their expertise took many years to acquirep y y qtheir own understanding will evolve
Subject experts must see working prototypes, t t tinot paper representations
Subject experts must be able to directly and continuously update rules as neededcontinuously update rules as needed“Corporate” knowledge must be externalized
2/5/03 Copyright 2003 Jeff Long 12
A Alt ti P diAn Alternative Paradigm
Remove most rules from softwareRemove most rules from software
Represent rules in canonical form, as fdata in a small set of tables
Externalize “corporate” knowledge in l i l k l d brelational knowledgebase
Let subject experts manage data
2/5/03 Copyright 2003 Jeff Long 13
I di t CImmediate ConsequencesSoftware size is reduced by 2+ orders ofSoftware size is reduced by 2+ orders of magnitude
simpler to create, manage, understand, test, p , g , , ,document, and teachremaining software has no knowledge of the world; it provides basic control logic that knowsworld; it provides basic control logic that knows what tables to check in what order, how to resolve conflicts, etc.
S ft d l t t i ll dSoftware development team is very small and manageable
2/5/03 Copyright 2003 Jeff Long 14
“Corporate” knowledge is externalized and isCorporate knowledge is externalized and is in a form anyone can see and understand
Knowledge is actionable by the computer: i h ki d i ireasoning, error checking, decision support
Subject experts can enter, change, and otherwise manage rules directly withoutotherwise manage rules directly, without programmer assistance
Data no longer merely defines certain details of the overall system; it defines the identity of the system
2/5/03 Copyright 2003 Jeff Long 15
Programmers do not need to know orProgrammers do not need to know or understand all rules, just enough to determine the classes of rules and thedetermine the classes of rules and the proper animation proceduresSerious prototyping becomes feasible;Serious prototyping becomes feasible; communications with users improvesTesting & QA can be far more rigorousTesting & QA can be far more rigorousDocumentation can be more complete
2/5/03 Copyright 2003 Jeff Long 16
R lt t D tResults to DateCommercial system: 7x growth withCommercial system: 7x growth with $1K/month software expense
Other prototypes in language, biology, legal, p yp g g , gy, g ,games, and artificial life seem to confirm basic claims and expectations
Cl SEI Vi i “Th i h fCloser to SEI Vision: “The right software, delivered defect free, on time and on cost, every time ”
2/5/03 Copyright 2003 Jeff Long 17
every time.
A l i f R lAnalysis of Rules
St t t f l d d i fStatement of rules and device for executing them can be different; need
t b ft f b thnot be software for both
Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z”
2/5/03 Copyright 2003 Jeff Long 18
“If” l if diti d“If” values specify conditions under which each rule is examined; these are
ll d “f t ” i Ult St t Thcalled “factors” in Ultra-Structure Theory (UST)
“Then consider” values specify additional criteria that must be considered; these are called “considerations” in UST
2/5/03 Copyright 2003 Jeff Long 19
F t b i k i thi dFactors become primary keys in third-normal form RDBMS
Alternate keys can be specified if useful
Control logic (“animation procedures”) g ( p )reads relevant rules, including rules about selecting rules, and carries out g ,specified actions
2/5/03 Copyright 2003 Jeff Long 20
O i f l “ l l ” l ( thOne informal, “molecular” rule (e.g. three strikes and you’re out in baseball) may be translated as many atomic rulestranslated as many atomic rules
Basic process is todefine what exists (existential rules)define what exists (existential rules)
define relations between these (network & authorization rules)
define processes (protocol & meta-protocol rules)
2/5/03 Copyright 2003 Jeff Long 21
Tens of thousands of rules can beTens of thousands of rules can be grouped into a small number of classes based on their syntax and semanticsbased on their syntax and semanticsThese classes can be managed easily by the tools of a RDBMSby the tools of a RDBMSDesign can proceed by iterative prototype; small prototypes can easilyprototype; small prototypes can easily evolve to necessary level of complexity
2/5/03 Copyright 2003 Jeff Long 22
I l t tiImplementation
S ti f l i t l d fiSeparation of rules into classes defines tables in the RDBMS; there is no practical limit on number of rows in a tablelimit on number of rows in a table
Referential integrity and field edits ensure that rules maintain integritythat rules maintain integrity
Queries, report writers and forms make access to rules easy for authorized usersy
2/5/03 Copyright 2003 Jeff Long 23
Th R l f H th iThe Ruleform Hypothesis
CComplex system structures are created by not-necessarily complex processes; and these processes are created by the animation of p ycompetency rules. Competency rules can be grouped into a small number of classes whose form is prescribed by "ruleforms". While the p ycompetency rules of a system change over time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically p g ypossible competency rules that might apply to the system, and constitutes the deep structure of the system.
2/5/03 Copyright 2003 Jeff Long 24
y
Th C RE H th iThe CoRE Hypothesis
WWe can create “Competency Rule Engines”, or CoREs, consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing p g y gbroad family resemblances, e.g. all corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, g g, y,whose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation procedures for p y peach engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language.
2/5/03 Copyright 2003 Jeff Long 25
g g g
Ultra-Structure is an Example of Notational Engineering
M bl i i t thMany problems in science, government, the arts, and engineering are caused by the way we represent themwe represent them
These problems cannot be resolved with more computing power or more moneymore computing power or more money
They require a new abstraction which can be the basis of a notational revolution
2/5/03 Copyright 2003 Jeff Long 26
R fReferencesLong J and Denning D “Ultra-Structure: A design theory forLong, J., and Denning, D., Ultra-Structure: A design theory for complex systems and processes.” In Communications of the ACM (January 1995)Long, J., “A new notation for representing business and other g, , p grules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)Long, J., “How could the notation be the limitation?” In Long, J. ( t dit ) S i ti S i l I N t ti l(guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure" In Proceedings of the 20thDocuments Using Ultra-Structure . In Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (October 1999)
2/5/03 Copyright 2003 Jeff Long 27