introduction to agile architecture - …...2016/01/07 · chapter 2 - challenges with agile 2.2...

Introduction to AgileArchitecture

Web Age Solutions Inc.USA: 1-877-517-6540Canada: 1-866-206-4644Web: http://www.webagesolutions.com

The following terms are trademarks of other companies:

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

IBM, WebSphere, DB2 and Tivoli are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

For customizations of this book or other sales inquiries, please contact us at:

USA: 1-877-517-6540, email: [email protected]: 1-866-206-4644 toll free, email: [email protected]

Copyright © 2016 Web Age Solutions Inc.

This publication is protected by the copyright laws of Canada, United States and any other country where this book is sold. Unauthorized use of this material, including but not limited to, reproduction of the whole or part of the content, re-sale or transmission through fax, photocopy or e-mail is prohibited. To obtain authorization for any such activities, please write to:

Web Age Solutions Inc.439 University AveSuite 820TorontoOntario, M5G 1Y8

Table of ContentsChapter 1 - Challenges with Traditional Architecture.........................................................5

1.1 The Need for Senior Management Support..............................................................51.2 Business People Underestimate the Complexity.......................................................61.3 Role Clarity...............................................................................................................61.4 Governance...............................................................................................................71.5 Variables Outside Architecture's Control..................................................................81.6 Analysis Paralysis.....................................................................................................81.7 Architecture Dismissed by Agile Teams...................................................................81.8 Scarce Resources.......................................................................................................91.9 The Challenge of Alignment.....................................................................................91.10 Rogue IT..................................................................................................................91.11 There's No ROI in Infrastructure...........................................................................101.12 Summary...............................................................................................................10

Chapter 2 - Challenges With Agile....................................................................................112.1 Business' Need for Predictability............................................................................112.2 Shortsightedness......................................................................................................122.3 Forest and Trees......................................................................................................122.4 Difficulties with Securing a Team Member from the Business..............................122.5 The Cycle of Formalization....................................................................................132.6 Integration Challenges............................................................................................132.7 Operations Team Seen as a Blocker........................................................................142.8 Reinventing Standards............................................................................................142.9 Possible Security Issues..........................................................................................142.10 Bias to Build over Buy..........................................................................................152.11 Team Member Criticality......................................................................................152.12 Agile Team Member Qualifications......................................................................152.13 Difficulties Acquiring Proper Support..................................................................152.14 Summary...............................................................................................................16

Chapter 3 - How Architecture Fits in with Agile...............................................................173.1 Common Ground.....................................................................................................173.2 Building High Quality Systems..............................................................................183.3 Solutions of Value...................................................................................................183.4 Rapid Delivery........................................................................................................183.5 Maintaining Systems Reliability.............................................................................193.6 Design.....................................................................................................................193.7 Reuse.......................................................................................................................193.8 Discipline................................................................................................................193.9 Measurement...........................................................................................................203.10 The Core Differences............................................................................................203.11 How is Agile Even Possible?................................................................................213.12 Open Source..........................................................................................................213.13 Code Control and Versioning................................................................................223.14 Test Driven Development (TDD).........................................................................223.15 Continuous Delivery Automation.........................................................................22

3.16 Stack Overflow......................................................................................................233.17 Twitter and GitHub...............................................................................................233.18 What Does Architecture Produce That an Agile Team Needs?............................24 1.19 What Should Architecture Provide to an Agile Team?........................................25 1.20 Summary.............................................................................................................25

Chapter 1 - Challenges with Traditional Architecture

ObjectivesThis chapter will enable participants to:

Understand the challenges Architects face;

Explore the issues that make Architecture challenging to implement;

Understand why friction often develops between Agile teams and Architects.

1.1 The Need for Senior Management Support

CIO's generally recognize the need for Architecture (but not always)

Senior business people understand there should be a lead designer or strategist (but may think that's the CIO's job)

Beyond that, the value of EA is abstract and may be seen as somewhat academic

Doing the work of Architecture requires access to senior people across the enterprise

Working with Senior Management

Senior management tends to think of IT as “plumbing”: something to be installed. Their primary focus is on developing new business and evolving the corporation to keep up with changes in market demand. Systems are often seen as a necessary evil—much harder to change than organization and so are source of frustration.

Vendors bombard them with messages about how easy it is to innovate by adopting their products and they usually don't have the background to sort out what is or isn't difficult


1.2 Business People Underestimate the Complexity

Business people often underestimate the complexity of systems

They don't understand why estimating is difficult

Comparisons to construction estimates are usually false analogies—one-of-a-kind structures severely overrun their estimates all the time

IT specialists often underestimate the work as well because no one really knows what it is until it's done

It's more a problem of not knowing what will ultimately turn out to be required than underestimating the work we know about now

Better estimating can lead to the perception that EA is a blocker because the numbers are higher

1.3 Role Clarity

Many organizations use the title “Architect” to refer to senior developers who aren't really doing Architectural work

There is often a lack of clarity around the dividing line and hand-offs between Enterprise and Solution Architecture

Also between Solution Architecture and Development

IT Architecture Associations don't provide much help: there is a lack of clear guidelines on the difference between EA and Software Architecture

6


1.4 Governance

Source: [CISR]

IT Governance may be underdeveloped leading to a lack of clarity around the scope and authority of the Architecture group

The Center for Information Systems Research describes IT Governance as being primarily concerned with decision rights

CISR identifies 5 key area of decision rights

◊ IT Principles

◊ IT Architecture

◊ IT Infrastructure

◊ Business Application Needs

◊ IT Investment and Prioritization

It may not be clear who gets to decide in each of these areas

The Architecture function may not have the power to enforce decisions

7


1.5 Variables Outside Architecture's Control

Packaged software often predetermines architecture e. g. ERP's come with an implied architecture

Mergers and acquisitions bring unplanned technologies into the mix and upset carefully planned architectures

It may be difficult to enforce standards when applications are built by outside contractors

Needing to be ready to divest a subsidiary may (intentionally) turn it into a technology island

Outsourcing may put control of those parts of the architecture in the hands of the outsourcer

1.6 Analysis Paralysis

Architects can fall into “Architecture for Architecture's sake”

Risk of spending too much time on Current State Analysis

Technologies are continually changing making architectural decisions extraordinarily difficult—today's best choice is probably not tomorrow's

Business users are often unclear about their priorities—they're struggling to react to changes in their markets and can't predict the future either

Modern businesses are complex entities

At some point the analysis must be cut off and decisions made

1.7 Architecture Dismissed by Agile Teams

Agile zealots can see Architecture as the antithesis of Agile

Architecture is seen as a relic of a Waterfall past

Architects may be seen as blockers and their arguments discounted

8


before even heard

Black-and-white thinking is always a risk—some people see everything as a dichotomy: it's either Agile or it's wrong

1.8 Scarce Resources

High quality architects are hard to find

They are usually high-salary “staff” employees which puts them under continual scrutiny

It's easy for architects to get caught up with delivery on projects they're assigned to making them less available for architectural work

As a result Architecture departments are often understaffed

1.9 The Challenge of Alignment

Architecture must always be aligned with stakeholder needs and requirements—it's a Principle!

Business units are rarely fully aligned from a business perspective

Every business unit believes its requirements are unique

Attempting to achieve consensus of any kind across the entire enterprise is the ultimate “cat herding” exercise

1.10 Rogue IT

In any large enterprise there will be smaller IT organizations that crop up

9


within business units

Sometimes they are legitimate: special-purpose systems may be best managed by a small, decentralized, specialized group

Occasionally it is the result of business units attempting to exert local control

Such groups typically try to stay “under the radar” and as a result do not communicate with central IT and may not even be aware of corporate standards

1.11 There's No ROI in Infrastructure

Architecture is often about building the platform for applications

Complete applications can generate positive ROI but infrastructure without applications is just “cost”

Many organizations make the mistake of attempting to calculate ROI at the project rather than program level

As a result they look at infrastructure-building as a poor investment and so don't invest sufficiently in platforms

1.12 Summary

Doing architecture has been likened to attempting to “nail Jello to a tree”

It's attempting to bring order to a continually shifting landscape of business needs and technologies

It requires highly skilled, senior practitioners with broad but deep experience, judgment and business/social skills

Local groups in business units like autonomy and will resist central control

Platform-building can be seen as a waste of money

Agile teams can see Architecture as “the enemy” even though in the end an Architecture must exist by design or by default

10

Chapter 2 - Challenges With Agile


Understand why Agile can be difficult in practice

Identify some of the pitfalls of Agile

Understand some of the broader needs of the business that Agile does not address

2.1 Business' Need for Predictability

Businesses need to balance investment and operational costs with expected revenues to remain viable

Most businesses have a well-defined annual planning and budgeting cycle

They need to be able to understand what they are getting for how much money to judge whether it is a worthwhile investment

To do this they will ask for an upfront estimate and ask for “order of magnitude” but expect it to be within 10%


2.2 Shortsightedness

Agile addresses the issue of connecting the team to the business by insisting a businessperson be on the team

That person will usually be looked to by the rest of the team to make all the requirements/features decisions

They may not have the background or experience to make good calls, either from a business or technology perspective

In the worst case the team can be whipsawed as the surrogate user and the team react to decisions that prove to be poor ones as the bigger picture unfolds

Because the team is focused on a single application or department they may not have visibility to important strategic changes happening at the corporate level and so may be focused on things that are not that important in the relative scheme of things

2.3 Forest and Trees

It's easy to get so focused on short-term delivery in sprints that there is never a good time to step back and assess the bigger picture

Reviews and retrospectives can be the first to get cut if the sprints are too ambitious

If rigorous test driven development isn't used the team can get into a viscous cycle of meeting sprint deadlines by deploying buggy code then losing more and more time each week to operational problems which causes more shortcuts to be taken

2.4 Difficulties with Securing a Team Member from the Business

Often Agile teams face delays in securing a team member from the business

12


Often the person assigned is not an 'A' player

If the person is not a strong decision-maker both in timeliness and quality it can have a big impact on the project quality and results

2.5 The Cycle of Formalization

Any new method starts out lean and based on a few key principles

Over time it has a tendency to adopt formalities and trappings

Scrums can become longer and waste time unless carefully managed

Well-meaning attempts to avoid problems recurring in the future get institutionalized into procedures

People who don't understand how development works “in the real world” will continue to assume that specification and documentation of formal process will improve a method

Auditors will continue to ask for evidence of conformance to formal process

Agile (like all) evangelists need to add to the methodology over time to have something new to say

Any method runs the risk of becoming “a religion” where people go through the motions and fail to question whether prescribed activities or deliverables are appropriate in the circumstances

2.6 Integration Challenges

Agile teams will tend to focus on rapid delivery for their project sponsor and downplay or ignore corporate standards

When integration becomes important the code base may be poorly positioned for it

In the worst case the team may become an isolated island with poor relations with other teams

13


2.7 Operations Team Seen as a Blocker

The operations team which will naturally focus on reliable and stable systems operations often is seen as a blocker by an Agile team

The operations group will ask for controls and manageability features that the team didn't plan for

The operations team may not understand the need for identical testing and production environments so code that works in test has problems in production

The operations team will ask for formal documentation of procedures that an Agile team will avoid doing

The operational team will want formal sign-offs before promotion to production that will stand in the way of meeting sprint objectives

Agile teams will often attempt to set up their own operational environment to avoid dealing with shared operational services, potentially adding cost

2.8 Reinventing Standards

Agile teams like to be self-sufficient and choose their own standards or work without them

Teams often begin coding almost immediately without agreeing in advance on coding standards or design patterns (or the team leader simply adopts what they used on their previous project, perhaps at another company)

The team will often choose its own tools in a similar manner

2.9 Possible Security Issues

Agile teams don't usually have a security expert

Corporate standards for writing secure code may be ignored

Risk of security vulnerabilities slipping into production

14


2.10 Bias to Build over Buy

Agile teams are strongly development-oriented and will naturally prefer custom business solutions over packaged ones

The team may not be aware of licensed solutions that already exist elsewhere in the enterprise

They may ignore packages that would be more cost-effective than custom code resulting in a more expensive final solution

2.11 Team Member Criticality

Because Agile teams tend to use their own tools and standards, it is difficult to replace a lost team member from within the company

Loss of resources has an immediate and lasting impact to productivity, often requiring the team to go outside to the marketplace with the attendant costs and overhead of advertising, interviewing, training, etc.

2.12 Agile Team Member Qualifications

Agile teams assume a high level of professionalism, discipline and knowledge of the team members

The team lead must be particularly strong and able to make good decisions across many technologies and package alternatives

The team must be able to work at all levels, from business analysis and architectural to coding and testing, possibly also implementation and operations

Requires people who know continuous integration: test-driven development, integration server technology, automated deployment, etc.

2.13 Difficulties Acquiring Proper Support

Agile works best in an open environment similar to a startup with lots of whitespace for brain-storming and conducive to teamwork

Project teams often need to work in less-than-ideal accommodations,

15


often temporary spaces with poor amenities

Sometimes the location is a corporate facility with enclosed offices

Bandwidth for access to shared repositories, integration servers, etc. may be limited

2.14 Summary

The business must decide what systems-creating activities it should invest in and how much they will cost in order to effectively manage value-for-money

Agile side-steps the issue by offering a constant burn rate in lieu of a TCO estimate

For custom software Agilists are probably right: the industry has never been able to figure out how to estimate software costs with any precision

Doing Agile well is not simple

But modern solutions need not be all custom code—some components can be costed, planned and procured

16

Chapter 3 - How Architecture Fits in with Agile


Define the common ground between Architecture and Agile

Begin to describe the features of an integrated approach

3.1 Common Ground

Building high quality systems

Providing solutions of value

Rapid delivery

Maintaining systems reliability and stability

Design

Reuse

Discipline

Measurement


3.2 Building High Quality Systems

Architects and Agilists would agree that the core objective is to deliver high quality systems

Architects might focus more on platforms and packaged software whereas Agilists think more in terms of the business logic and custom code

They both want the best possible result for the business

They both believe in integration of systems and information across the enterprise

They both know the enemy is Complexity

3.3 Solutions of Value

They both understand that:

◊ Time is money

◊ Resources are scarce

◊ They're there to create value for the business

3.4 Rapid Delivery

They would both agree that:

◊ Time is of the essence

◊ A system that does not yet exist has no value

◊ Early wins are important and build success

Enterprise Architecture Always Exists

It is often said that the architecture of an enterprise exists, whether it is described explicitly or not. This makes sense if you regard the architecture as existing in the system itself, rather than in a description of it. Certainly, the business practice of Enterprise Architecture has emerged to make the system structures explicit in abstract architecture descriptions. Practitioners are called "enterprise architects.”

18


3.5 Maintaining Systems Reliability

They'd also agree:

◊ The systems infrastructure must remain robust

◊ Bugs must be kept out of production

3.6 Design

They both:

◊ Consider themselves designers and craftspeople

◊ Use design patterns extensively

◊ Know that early mistakes are very expensive to correct later

◊ Know systems should be modular, separate concerns, and show high cohesion / low coupling

◊ Know designs should be subjected to many eyeballs and creative criticism

◊ Know good design requires a variety of skills and knowledge that almost never all resides in a single person—teams are important

3.7 Reuse

Agilists and Architects would agree:

It's always preferable to use a previously-existing solution than to create a custom one

Shared solutions are better

3.8 Discipline

Both take a highly disciplined approach

19


To achieve high reliability Agilists must be very disciplined in their deployments and use automation heavily

3.9 Measurement

Both would agree that measurement is fundamental to improvement

Group Discussion

What else would they agree on?

Given so many areas of agreement, what's the issue?

3.10 The Core Differences

The role and value of formal documentation

The extent to which there is such a thing as “Requirements”

How much can be planned vs. needs to be discovered

How releases need to be managed: continuous vs. target architectures

Local vs. global optimization

◊ Agile teams see the minute detail that isn't visible at the architectural level (and the devil lives there...)

◊ Enterprise Architects see the needs of the whole enterprise and want common if not ideal solutions

Decisions by team vs. committee

20


3.11 How is Agile Even Possible?

Open source and GitHub: reduced the cost of custom development of sophisticated systems by making pre-written code libraries and examples easy to find and free to use

Code control: (especially Git) allowed many developers to work simultaneously but effectively on the same code base

Test Driven Development: enabled custom code to undergo continual short release cycles with confidence that the code would still work

Continuous Delivery automation: enabled testing and deployment to be done efficiently (and is becoming even more important with the need now to often test against thousands of different mobile devices)

Stack Overflow: made development more cost-effective by allowing developers to share solutions to obscure bugs

IRC and Twitter: made it easy to get answers to problems from the developer community and to broadcast important news (e.g. new releases, key bug fixes)

3.12 Open Source

Modern development achieves high reuse through leveraging open source libraries usually based on established patterns

Ideas quickly cross-pollinate

Popular libraries are usually re-implemented quickly in other programming languages than the original

A culture of sharing and support arose around open source where developers globally assist each other at no charge in the expectation that it is reciprocal

21


3.13 Code Control and Versioning

Code Versioning Systems (CVS's) allowed many developers to work on the same code base in a coordinated fashion

Previously developers were assigned areas of a code base to work on so they wouldn't make changes that would collide with other developers

This meant that developers could not be flexibly assigned to the areas of code most in need of immediate work

Integration testing was critical to confirm that the components build by the different developers or subteams would actually interoperate correctly

Git in particular allows very fast check-in and check-out of code versions, (which was a very slow process with large code bases with earlier CVS's)

Git made it possible for Agile team members to work on the highest priority tasks rather than only assigned ones

3.14 Test Driven Development (TDD)

Test Driven Development is used to different extents by different teams

The basic steps are:

◊ Before each change is made to the code, a test for a small increment of new behavior is written and observed to fail (otherwise the test is defective)

◊ The code is changed so the test passes

◊ Repeat until the new feature is fully implemented

◊ Check code into the repository main branch only when all tests accumulated to-date pass

TDD made it possible to implement changes to production on a regular basis with the confidence that there wouldn't be a major breaking change

3.15 Continuous Delivery Automation

Automation of the testing and deployment processes with tools such as Jenkins, Chef, Puppet, etc. made it possible to consistently rerun huge

22


batteries of accumulated tests quickly and have confidence that the deployment to potentially many servers would be done completely and accurately

More recently, container technology is making the predictability even greater by ensuring an entire working ecosystem of libraries is always deployed as a consistent image, and allowing different versions of the same library for different purposes to coexist in a deployment

3.16 Stack Overflow

The pace of change of popular libraries is staggering

New releases appear several times a year with hundreds of fixes and additional features in each

Library developers are much more willing to introduce breaking changes to correct previous mistakes than was the case prior to open source

Developers can't possibly keep up with all the changes in all the libraries they use, nor can they afford to stick to an old version when the support comes from the community rather than a vendor and is centered the most current version(s)

Questions are posted by developers to Stack Overflow and usually quickly answered; the Q and A remains online and is easily found by others using Google

Modern development is polyglot; knowing one programming language is not enough

Good developers know conceptually what they want to do but may not remember the exact syntax for the language and/or library they working with, so they Google as they go

Internet Relay Chat is also popular for connecting with other developers who might know the answer to a question

3.17 Twitter and GitHub

Twitter has become a primary news channel for developers

23


By following particular hashtags they know almost instantly when new libraries or versions become available

By following GitHub activity they know when library bug workarounds have been posted

Group Discussion

What could Architects learn and apply from this?

Why don't Agile teams seem to be interested in Architecture repositories?

3.18 What Does Architecture Produce That an Agile Team Needs?

Scope definition: Agile groups are usually assembled to do a project with some sense of a scope. Where does that scope come from?

Standards: They will choose their own standards if not given any

Funding: The project funding had to come from somewhere: is this a departmental project or a cross-divisional corporate initiative?

Package decisions: Again, the team will decide themselves if not prescribed

24


1.19 What Should Architecture Provide to an Agile Team?Group Discussion

What else should they be looking for or could they benefit from if provided by the Architecture group?

1.20 Summary There is common ground between Architecture and Agile

An integrated approach is needed to ensure the architecture is in place to support Agile applications

25

introduction to agile architecture - …...2016/01/07 · chapter 2 - challenges with agile 2.2...

Documents