Nanjing University

International Partnership in Information Technology He (Jason) Zhang

4 June 2018 @ NTNU

Nanjing · China

Beijing

Shanghai

Hong Kong

❖ A cultural city with 2,600 years history

❖ the capital of China for ten dynasties, including the Republic of China

❖ A city of excellent specialists for scientific education

❖ 53 general higher universities with over 820,000 college students

❖ 540 various scientific research organisations with 530,000 research staffs

❖ No.1 among all Chinese cities in terms of the portion of population receiving higher education

❖

Nanjing University

❖ Data Technologies❖ Data Mining

❖ mining software process from repository

❖ mining software metrics for process modeling

❖ Natural Language Processing

❖ NLP for grey literature review

❖ NLP in support of qualitative analysis & synthesis

❖ The first Chinese modern university with the combination of education & research established in 1902

❖ 29 schools, institutes, or departments, 3 of which are the Department of Computer Science & Technology, Software Institute, and Artificial Intelligence Institute

❖ The top ranked State Key Research Laboratory of Novel Software Technology in all 32 state key research laboratories in information sciences in past 15 years

Curricula @ Software Institute

Nanjing University · DevOps Research Centre

❖ Software Institute

❖ Faculty Members

❖ Prof. He (Jason) Zhang

❖ Assoc. Prof. Dong Shao

❖ Dr. Guoping Rong

❖ Dr. Zheng Li

❖ Students❖ 8 Ph.D Students

❖ 20+ Master & Thesis Students

Traditional Research Interests

❖ Software Engineering Process

❖ Empirical Software Engineering

❖ Quality Assurance

❖ Software Architecture

❖ Service-Oriented Computing

Why DevOps?❖ “Through 2016, ~80% of outages impacting mission-critical services will be

caused by people and process issues, and more than 50% of those outages will be caused by change / configuration / release integration and hand-off issues.” – Gartner Report

❖ “The Knight Capital Group was an American global financial services firm […].[…] Knight was the largest trader in U.S. equities, with a market share of 17.3% on NYSE and 16.9% on NASDAQ. The company agreed to be acquired by Getco LLC in Dec 2012 after a trading error lost $460 million” – Knight Capital from Wikipedia

❖ This took 45 minutes and was an upgrade error.

DevOps is about…❖ “DevOps is a set of practices intended to reduce the time between committing a

change to a system and the change being placed into normal production, while ensuring high quality.”

❖ DevOps is about…

❖ Bringing “agile” methods to operation

❖ Encouraging collaboration between development and operations staff, get them talking

❖ Shared goals and teams of Devs and Ops

❖

Requirements

•Treat Operations personnel as first class stakeholders

•Get their input when developing requirements

Development

•Small teams•Limited coordination

•Unit tests

Build

•Build tools•Supports continuous integration

Testing

•Automated testing•Integration testing

Deployment

•Deployment tools•Supports continuous deployment

Execution

•Monitoring•Responding to error conditions

DevOps Influences…❖ DevOps practices will influence industry

❖ the way you organise teams

❖ the way you build systems

❖ even the structure of the systems that you build

❖ DevOps practices will influence research

❖ Software Architecture

❖ split application into small, well-scoped microservices

❖ Software Process

❖ Ops as first-class citizens throughout the lifecycle

Grand Questions of Interest

❖ What are the impacts of DevOps on software industry in general, or software practitioners in particular?

❖ How to support software organisations to adopt or migrate to DevOps?

❖ How to reengineer software technologies and advance the state-of-the-art of DevOps?

Data Tech. Empirical Methodology

Archit. Quality Security Deploy

Process

DevOps

Acad

emic

Ach

ieve

men

t

Data Mining

NLP

Industrial Impact

SLR

MLR

QRS

Doc Engineering

Mic

rose

rvic

es

DD

D

Logg

ing

Con

tain

er

Dev

SecO

ps

API

Gat

eway

Secu

re A

rch.

AgilityCI Lean KanbanCD Automated Process

Toolchain

DL

Qualitative Methods

Log

Ana

lysi

s

MobileInternet

Big Date

Cyber-PhysicalSystem

Smart autonomous system

。。。

Application Domains

Software Engineering

Fundamental Competency

Cyber-Security

Nanjing University · DevOps Research Centre

DFD

Organisation & Culture

Fundamental Competency ❖ Empirical Methodologies

❖ Evidence-Based Software Engineering (Systematic Reviews)

❖ methodological issues: data synthesis, quality assessment, validity …

❖ Qualitative Research in Software Engineering

❖ ethnography, field study, grounded theory …

❖ qualitative data analysis and synthesis

❖ Grey Literature in Software Engineering

❖ evidence extraction & assessment of grey literature

❖ multivocal literature review

❖ Data Technologies

❖ Data Mining

❖ mining software process from repository

❖ mining software metrics for process modelling

❖ Natural Language Processing

❖ NLP in support of qualitative analysis & synthesis

Qualitative Evidence SynthesisResearch Synthesis

aggregative

< method >

interpretive

< extracted information > < study design >

booleanlogic

frequency structuredquestions

< data >

findings raw data

meta-summary contentanalysis

comparativeanalysis

case survey

diversedesign

meta-ethnography

comparabledesign

< purpose >

theorygeneration

understanding

groundedtheory

narrativesynthesis

thematicsynthesis

DevOps Oriented Software Engineering ❖ Toolchain

❖ “DevOps Tooling Technical Report”

❖ Microservices Architecture❖ Impacts on Quality Attributes

❖ Microservice Granularity

❖ Migration from Monolith to Microservice System

❖ Decomposition for Microservices

❖ constraints

❖ domain-specific design

❖ data-flow diagram

❖ Benchmark Example for Performance Evaluation

❖ pet-store project

DevOps Tooling

MSBuildMSBuild

AzureAzure

Academic Literature: 144Academic Literature: 144

: 178: 178

: 76

: 76

: 117: 117

DevOps Tooling

Data-Flow Driven Service Decomposition

DevOps Oriented Software Engineering ❖ Quality

❖ Logging Practices

❖ *Security

❖ Secure Architecture

❖ DevSecOps

❖ *Deployment & Operation

❖ Process & Agility

❖ Automated Software Process

❖ Mining Software Process

❖ Software Process Simulation Modelling

❖ hybrid simulation modeling

❖ model verification & validation

❖ Agile / Lean / Kanban Development

❖ Interplay between Process and Product

❖ Organisation & Culture

Metrics for Process ModellingProcess

…

……

Product

Legend

Resource

……

Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentTime(270)

-HNKLLDL

HGM L

Project Duration

-HNKLLDL

HGM L

Test Duration

-HNKLL

Fix Duration

LTask Duration

-HNKLL

Review Duration

-HNKL-HNKL D

Work Time-HNKL

L

Interval between Releases

LLead time

LAnswer time

LDelivery time

HGM L

Domain Discovery Delay

HGM L

Design Filling Delay

HGM L

Analysis Discover Delay

HGM L

New Requirements Feedback Delay

L %

Requirement Selection Time L

Coding Duration

-HNKLLDL

Rework Duration

L

Interval between Requirements

-HNKL

Issue Assessment

Duration

Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentDefect

(246)

ML

# Injected Requirement

Defects

Injection Rate during Design

ML

# Design DefectsPercentage of

Fixed Requirement

DefectsDesign Defect

Density

MLML -HNKML HGMML LDML GCM

Injection Rate during Coding

ML

# Injected Code Defects

ML HGMFix Rate

Percentage Fixed DefectsML

# Fixed Defects

MLML LM L

Detected Defects in Test

Defects Detected in Reviews % ML -HNK

Defect Detection Rate

Overlap of Detection of Reviewers

ML

# Detected Defects

Rate of Defect Delivered

ML

# Detected Defects after

Test

ML

Released Defect Density

CF GLCHGE LLDefect Severity

ML -HNKML HGM

Defects Injection Rate

MLML GCM

Defect Density

CF GLCHGE LLDefect Type

ML

# Remaining Defects

Requirement Implementation Review Test Delivery & Maintenance Phase IndependentTask Size

(112)

A L

Task Size per Inspection

.GLI MCHGL# Inspections

GCML

Amount of ReworkLDL

L K MHKC L

Amount of Implementation

Work

MNK L CF

Demand for New Features

NCK F GMLTest Task Size

LDLTotal Tasks

KLHG DL LD-HNKL LD

Task Size

LHNK L LD

Task Required Resource

LDL L K MHKRequired Tasks

N LML

# Maintenance Requests

CF GLCHGE LLDelivery Times

Requirement Design ImplementationChange(48)

GA L# Requirement Changes

GA L CFRate of Requirement Changes Change request acceptance Rate Change Request Time Span

GA L# Design Changes

GA L# Code Changes

Design Implementation Review Test Phase IndependentEffort(143)

KLHG -HNKLKLHG LKLHG DLKLHG HGM LL

Total Effort

KLHG DLEffort per Item

Effort Allocation Percentage

KLHG -HNKLCoding Effort KLHG L

Test Effort

-HNKL GCMTest Effort RateKLHG DL

Effort per Test Case

-HNKLDesign Effort Review Effort

KLHG LRework Effort

Meeting EffortPreparation

EffortKLHG DLKLHG L M

Fixing Effort

Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentProcess

Factor(106)

CF GLCHGE LLTime Pressure

CF GLCHGE LLCoding Method

CF GLCHGE LLTask Priority

CF GLCHGE LLPolicy

CF GLCHGE LLRisk Impact

CF GLCHGE LLQuality Strategy

…

CF GLCHGE LLTest Method

OC PL

# Periodical Reviews

CF GLCHGE LL

Review Adjustment

Policy

CF GLCHGE LL

Domain Discovery Factor

CF GLCHGE LL

Analysis Discovery Factor

CF GLCHGE LL

Design Discovery Acceptance Factor

CF GLCHGE LL

New Requirements Feedback Factor

CF GLCHGE LL

New Requirements Scaling Policy

CF GLCHGE LLRequirements Volatility

CF GLCHGE LLRelease PolicyCF GLCHGE LL

Maintenance Adjustment

Factor

Design Implementation Review Test Delivery & Maintenance Phase IndependentDevelopment

Rate(75)

.

LLDL .M K MCHG

Development Rate

Discover RateH NF GMLH NF GML HGM

Design Rate Implementation Rate GCML CF

Delivery RateReview Rate Test Rate-HNKL MLDL

Rework Rate

Phase IndependentProcess Static Attributes(13)

CF GLCHGE LL# Increments

CF GLCHGE LL# Sprints

CF GLCHGE LL

Stage of Development

CF GLCHGE LL

Stage of Development

CF GLCHGE LL

Multisite Development

KLHGLTeam Size

FL# Team

KLHGL# PersonsKLHGL# Testers

KLHGL

# Experienced Staffs

KLHGLDatabase Size

HHFL

# Available Inspection Rooms

CECMC L # -HNKL DTest Facility Availability

Team Personal EnvironmentResource Size(194)

Team PersonalProductivity(107)

-HNKLLM L L

I C C MCHGL -HNK

Team Productivity / /

Individual Productivity

TeamPersonnel Continuity(34)

Turnover Rate Hiring RateKLHGL

Assimilation Rate …

Team Personal EnvironmentResource Factor(19)

CF GLCHGE LLTeam Cohesion

CF GLCHGE LLSelf Esteem

CF GLCHGE LLTeam Utility CF GLCHGE LL

Culture

CF GLCHGE LLAchievement Need

CF GLCHGE LLNative Language

CF GLCHGE LLPenalty Factor

Personal EnvironmentCapability(82)

CF GLCHGE LLCommunication Overhead

CF GLCHGE LLTraining Overhead

CF GLCHGE LLTask Switch Overhead

TeamOverhead(20)

CF GLCHGE LLSkill Level

CF GLCHGE LLExpertise

CF GLCHGE LLCapability

CF GLCHGE LLKnowledge

CF GLCHGE LLUse of Tools

Team PersonalExperience(27)

CF GLCHGE LLTeam Experience

CF GLCHGE LLIndividual Experience

Requirement Design Implementation Test Phase Independent

Product Size(185)

A LFPsTasksUnits

Size of Design Document

A LFPsStoriesUnits

Size of Requirement

DocumentH NE LKLHG$ LHGMKC NMCHGLGCML

Size of Product/System/ Project

CE LGCML

Size of Component

.L

Size of CodeLM L L# Test Cases

Size of Test Code

Test Case Pass Rate

Design Implementation Test Phase IndependentDetail Measure(29)

C AK FLHGMKHE EHPL

Detail Measure of Design Document CLCHG HNGM

H NEH E O KC E LEH E O KC E L

Detail Measure of Code

HGMKHE EHPLL L L

Detail Measure of Test Case HK L A

K AK I LGM G L

/ PHK L

Detail Measure of Artefact

Phase IndependentProduct Feature(30)

HK L AK AK I LGM G L

/ PHK L

Detail Measure of ArtefactCF GLCHGE LLSystem Type

CF GLCHGE LLUser Type

MNK L# Features

CF HG L# Time Zones

…

Phase Independent

Complexity(40)

CF GLCHGE LLComplexity

CF GLCHGE LLMaintainability

CF GLCHGE LLReliability

CF GLCHGE LLReusability

Phase IndependentEffectiveness(34)

CF GLCHGE LLTest Effectiveness

CF GLCHGE LLReview Effectiveness

Test Review

CF GLCHGE LLEffectiveness of the Process Phase IndependentCost(48) Test Phase Independent

Sub-Category (# Metrics) EntityInternal Category:

Sub-Category (# Metrics) EntityExternal Category:

GCMR%

Metric

…Rarely used and we can not list all of them:

The specific metric with its unit:

Phase Independent

Maintainability(2)

Phase Independent

Reliability(8)

Phase Independent

Reusability(4)

GCMR%

MetricThe Metric belongs to this Sub-Category and falls in this Type

-HNKTester Cost

Quality(28)

-HNKL MDefect Cost

Individual Cost

Project Cost

KLHG LCost per Effort …

MLML

LD OCL

Quality

MRisk Start/End Date …

Performance(8) Phase Independent

CF GLCHGE LLCost Performance Index

CF GLCHGE LLSchedule Performance Index

* HK -HNKDevelopment Performance …

Legend

Process Intenal

Process External

Resource Intenal

Product Intenal

Resource External

Product SizeDetail Measure

Product Features Complexity

Maintainability

Reliability

Quality

Time

Defect

Task Size

Change

Effort

ProcessFactor

Development Rate

Effectiveness

Cost

Resource Size

Productivity

Personnel Continuity

Capability

Experience

Generates& Detects& Fixes(69)

Product External

Overhead

Measures(5)

Impacts (15)

Impacts(3)Impacts(9)

Impacts(17)

Extends

Extends

Extends

Extends

Extends

Extends

Extends

Extends

Extends

Measures(9)

Changes(6)

Releases(9)

Extends Extends

Extends

Extends

ExtendsExtends

Extends

Extends

Extends

Measures����

Measures(1)

Measures(1)

Measures(1)

Measures(5)

Measures(7)

Impacts(22)

Measures(10)

Extends

Extends

Impacts (7)

Measures���Impacts(3)

Requires(1)

Impacts(2)

Impacts���

Extends Impacts(4)

Impacts (5)

Measures(9)

Measures(35)

Process Static Attributes

Resource Factor

Impacts(11)

Measures(12)

Mesures(16)

Measures (12)

Extends

Impacts(1)

Requires(33)

Produces(33)Measures(16)

Requires(25)Produces(25)Requires(18)

Produces(18)

Category

Sub-Category

Label(# evidence)

Extends

Requires(9)

Measures (4)

Impacts(7)

Impacts(8)

Metrics for Process Modelling

Industrial Impacts❖ Synergy by combining research, industry, and education

❖ 《DevOps: Theory, Method, and Practice》(1st DevOps textbook in China)

❖ DevOps curricular and teaching/training platform

❖ Industrial Collaborations on DevOps: Huawei, ZTE, Transwarp, DaoCloud …

❖ 《DevOps · China 2016 Report》

❖ 《DevOps · China 2018 Report》（available soon）

❖ 《DevOps Tooling Technical Report》

❖ 《Software Architecture for Practitioners》@ GeekTime

❖ ArchSummit 2017 / 2018 (Shenzhen)

❖ Keynotes & invited talks at CNUTCon 2017 / NASAC 2017 / NJSD 2018 / DOIS 2018 …

❖ DevOps tracks at IAS 2017 & NJSD 2018

❖ DevOps Special Group of CCF · Software Engineering & System Software Committees