19/11/2004 1

The Roller-coaster of Product Development.

Tony GibbsPresentation to BSc (Hons)

CRTS course @ UWE.22/11/2004

19/11/2004 2

Agenda War stories and successes of Product

Development. With thoughts from the University of Life!

Contexts Technical Environmental Financial Organisation Managerial

19/11/2004 3

My background BT apprenticeship before BSc (Hons)

degree. BSc (Hons) degree in MCI (now CRTS),

2(ii), from UWE in 1989 (I.e. here!) 15 years product development experience

post-degree so far. Embedded single board computer in small

team (me + 1 technician.) Complex system in large team (£25 million

dev’t spend.) Software team lead and Test team lead.

Now a part of a £2.7 billion UK-wide Public Safety radio project.

I’m a S/W Engineer mostly!

19/11/2004 4

Technical ups & downs One bug OFTEN covers another one.

My customer wanted some software from me urgently.

With a specific bug fixed in about 60K of C code. I fixed it on Christmas Eve, tested it, and sent it

out at 6pm. But the C code bug was covering a

Assembler bug which could not be tested for previously!

My customer found the 2nd bug first! Between Christmas and New Year!

19/11/2004 5

Technical ups and downs Other people & companies make mistakes

outside of your control or influence. A Prototype for a Digital TETRA radio was a PCB

that was A4-paper sized. The product’s PCB could only be shrunk to its required

size with arrival of a Digital ASIC. The ASICs all failed test because the Fab’s

subcontractor made a +ve mask from a –ve mask!

4 – 6 weeks elapsed slip in the product plan!

19/11/2004 6

Technical ups and downs Try to understand other people’s

pressures and needs, not just your own. I needed to host an RTOS (VxWorks) on some

new custom designed hardware, but using some existing available s/w device drivers.

H/w team lead needed to reduce component count and cost.

A £10 component was expensive. RTOS license cost £3 per board (run-time license.)

HOWEVER – SUCCESS!! USING AN RTOS SAVED DEV’T TIME & DEV’T MONEY on S/w!

Cost = Dev’t cost /n + Unit Cost. But I needed to listen to the H/w team as

well.

19/11/2004 7

Technical ups and downs Launch Timescales are short & getting shorter

(e.g. Motorola with 17 new phones in 2004.) Need to develop hardware & software in parallel. Need to REDUCE development RISK.

I needed to keep a 10 person software team working to design, code and unit test software WITHOUT any real h/w being available.

I needed to help h/w team be able to test prototype hardware. RTOS simulator on Sun Unix network allowed s/w & h/w

development to continue in parallel. Monitor prom helped BOTH h/w team & RTOS port.

Code was free from processor & support chip manufacturers.

SUCCESS!! Complex Real-time process ran 1st time at real speed (500s) !! A relief all-round!!

RTOS tools allowed parallel dev’t!!

19/11/2004 8

Technical Ups & Down Problems blamed on Software

might be Hardware related. RTOS being ported onto 2nd iteration

of custom hardware. Hardware chip count being reduced,

so a VHDL version of 16550 UART was put in an FPGA.

RTOS could not do RS232 serial comms with a bought in device driver.

Turned out to be a Hardware problem in VHDL UART.

19/11/2004 9

Technical Ups & Down If the Software has not been changed but it

does not run anymore, then it could be the Hardware. A complete batch of Processor boards came

through the production line, and 100% failed test. The software was being blamed, but it had not

been changed. The RTOS Software was not understood.

Source code control (e.g. using RCS) was & is important. Fault was intermittent. H/w team said they had not changed the

hardware. But it turned out to be a PCB change of the

0v ground tracking.

19/11/2004 10

Technical Ups & Down Software needs to be well

documented (in the code & on paper.) A student during a summer placement

had designed some hardware and assembler software for an alarm unit.

But the software was incorrect, and not documented well enough.

So…the hardware was changed with addition of NOT gates (inverters) to get product out on time.

19/11/2004 11

Technical Ups & Downs Learn from other people’s mistakes in

our industry, before you make them yourself! Design Software that avoids Memory

Leaks!!! Put wrappers around alloc() and free() so that you

can spot Memory Leaks, & use leak detection tools.

Make it easy for yourself in the Lab, because they are hard to spot in the field.

Upgrades in the field are slow & costly. E.g £75 per mobile to re-flash it, including

Logistics!! + Disruption/Down-time/Bad News.

19/11/2004 12

Technical Ups & Down Quality cannot be tested in, if it is poor at

the start. Design for Quality & for RE-USE.

Review the design on paper/screen before you code it.

Document it!! Use structured design methods. I used Yourdon RT-SA/SD because I knew it &

others knew it! Use automated code review tools, if available. Use Defensive programming

AT&T Telecomms outage in 1990s for Eastern USA seaboard for 24 hours.

CAUSE? No Default case in a C-language Switch statement.

COST? $50 million!!! For 1 line of code!!!

19/11/2004 13

Technical Ups & Downs. Design for Testability. Design in extra interfaces in to H/w,

PCBs, & Software, even if not populated with components in final products. Design in connectors for debugging equipment, e.g.

ICEs and Logic Analysers. Design in a (RS232) Serial Port or two.

Useful for output of debugging PRINTF statements. Design in LEDs or 7-segment displays you can use

for debugging or for attaching Oscilloscopes on to. I learnt this lesson in BSc (Hons) MCI year 1 in 1986,

and it is so useful, e.g. to watch an interrupt signal. Software is often on Critical Path to launch,

so make it easier to test & debug it.

19/11/2004 14

Environmental Not really an up or down, but something to

be aware of. RoHS and WEEE RoHS = “Removal of Hazardous

Substances” EU directive. WEEE = “Waste Electrical & Electronic

Equipment” EU directive. These will affect our industry a lot, and call

for more attention to recycling, resource efficiency and sustainable development.

Companies are already having to rework their equipment and processes to comply with these directives.

19/11/2004 15

Organisational Ups & Downs Marketing have requirements

which are sometimes imprecise, but which are as good as they can do at the time. Sometimes need to prototype feature /

look & feel. so it can be shown to Marketing and to

Potential customers. Need to prototype cheaply though.

To reduce the business RISK. Need to design to target costs.

19/11/2004 16

Organisational Ups & Downs Sales have to have something

to sell, even if it is being developed now! Need to be able to give Sales a clear

definition of what it will do, even when still in development.

Needs a clear up-front design. Design needs to be followed through on.

Because it will have been sold. Sales will need clear progress

information to pass on to customers. Sales brings in the money.

19/11/2004 17

Project Management Triangle

COST

QUALITY TIME

19/11/2004 18

Project Management Triangle The 3 aspects of the triangle have to be

balanced against each other with 1 or 2 taking priority over the others.

Not infinite time to launch products. However, how is Quality v Features

balanced? Difficult to upgrade an embedded real-time

system (e.g. mobile phone) if it has a bug in it. Mostly due to LOGISTICS of RECALL to UPGRADE.

If Time is tight, then may to sacrifice some Features in order to maintain Quality.

Shouldn’t just squeeze more Features in if Quality cannot be maintained.

19/11/2004 19

Financial Cash flow is the life blood of

organisations. So, Priorities have to be set and juggled. Budgets have to be set and met.

So spend on Capital items have to be justified. Quality improved? Timescale shortened? Both of the above together?

Tools might have to be rented or leased rather than bought (initially.) Can also send them back if not right for

the job!

19/11/2004 20

Managerial ups & downs The managers have to hold their

nerve when other ups & downs are going on.

But the managers need information to brief others. The good & best managers will try to

keep the rest of the company off the workers, so they can focus on the technical work.

But the workers need to supply the information, sometimes to the MD in person.

19/11/2004 21

Managerial Ups & Downs The Boss OFTEN knows better than you.

Through experience of doing it before. The Boss OFTEN has some more

information than you. Because higher in the organisation.

The Boss is SOMETIMES under more pressure than you. So it is SOMETIMES not productive to

challenge their requests. Support their requests. Not always easy to do this, and this is

the HARDEST lesson I have had to learn!!

19/11/2004 22

Summary (1) Timescales are getting shorter, whilst

Features Sets are becoming richer. Good Quality is expected, that meets

the product’s Cost & Requirements. Make sure it is safe for its market. Can’t sell a Rolls-Royce at Mini’s price. If got Quality S/w or H/w already,

then REUSE IT. PORT IT NOT RE-WRITE IT. It has already been tested &

debugged!!

19/11/2004 23

Summary (2) A quieter day is NOT coming, so

document it now!! Lab books, Comments in code, README

files. Someone will have to maintain it. Software is expensive, & it is OFTEN re-

used when it works already. RE-USE will happen more, because of

pressures on time & costs. Good documentation will help RE-USE.

Example – SRP1000 v SRP2000 radios. SRP1K = 4 years development. SRP2K = 12-15 months, with 90%+ SOFTWARE

RE-USE from SRP1K.

19/11/2004 24

Summary (3) Support the Boss, so they can support

you. Give them the information they need, and

listen to their advice & requests. Don’t give them unpleasant surprises!!

Learn from other people’s mistakes in our industry. Don’t design Memory Leaks in to

Embedded Real-time systems. Can’t just download a patch for it in the

field!! Try to understand other people’s

pressures & needs. Everyone is under pressure today!

19/11/2004 25

Good books. Roger Pressman, “The Art of

Software Engineering.” Explains lots of software engineering

basics for use in the real world. Steve McConnell, “Rapid

Development”, Microsoft Press. A Microsoft Project Manager who tries to

explain how to balance pressures & needs.

19/11/2004 26

Thank you for listening. Any Questions?