toward an environment for comprehending distributed systems

Reverse Engineering Reverse Engineering (Dynamic Analysis)(Dynamic Analysis) © SERG

Toward an Environment for Comprehending Distributed

Systems


Outline

• Motivation

• Software comprehension environment

• Data gathering

• Data repository

• Preliminary results

• Conclusions and future work


Background

• Project– Comprehending distributed component-based

systems– Phase-I

• Development of a core toolset

– Phase-II (Iterative)• Modeling and visualization

• Evaluation


Characteristics of modern software systems

• Large and complex

• Distributed

• Written in multiple programming languages

• Developed using component-models


Software Comprehension Challenges

• Unavailability of source code– Vigder-2001, Weyuker-2001

• Physical distribution of components– Bruegge-1993, Mendonca-1999

• Use of pre-built component– Stroermer-2002, Vigder-2001, Weyuker-2001

• Multiple programming languages– Hassan-2002, Chen-1995,– Koschke-1998, Stroermer-2002– Demeyer-1999, CDIF-1994, XML/XMI-2002


Approach

• Software Comprehension Environment– Technical issues

• Distributed profilers

• Single data repository

• Data management

• Extensible architecture Data gathering and Repository subsystems

– Modeling and visualization subsystem

– Empirical evaluation• Survey/questionnaire

• Industrial-strength case studies – Cigna Corporation


Software Comprehension Environment

Program Data(Static &Dynamic)

Data ModelMeta-data

Model

Universal Schema

Meta-Data

Data Gathering Subsystem

Analysis and Visualization Subsystem

Dynamic Analyzers(Debugger/Profiler)

Static AnalyzersSoftware

Program(s) Visualization(Presentation)

ConceptualModels

NavigationHuman-Driven/Feature-Driven

Sourcecode

Execution Traces

Repository Subsystem


Data Gathering subsystem

DataCollectionAdapter

GDBProfiler


JVMPIJVMDIProfiler


Java StaticAnalyzer


Win32C/C++/VBProfiler

GNU C/C++Run-time

Java Run-time

Java Source-codeor Byte-code

Win32 C/C++ and VBRuntime

Data Collection Manager

Logical Time Server

Data Collection ServerTo DataRepository

XML Documents

Import

/Exp

ort



• Distributed Profiler– Local Profiler

– Logical Time Server

– Data collection manager

• Remote Interactions– Network interceptor

– Communication Endpoint

• Program Facts/Data– Static: Entities and Relations

– Dynamic: Entities, Relations and Events



• Endpoint entity– Local and Remote host– Local and Remote port numbers– Time-stamp

• Connects RelationLocal Host Remote Host Local Port Remote Port Time

Local Host Remote Host Local Port Remote Port Time+Delta

Delta < Threshold

Endpoint-A

Endpoint-B


Example: Sequence Diagram

Java Program C/C++ Win32 Program

UserClassjava.netSokect-

OutputStreamUserModule

(Winsock)Ws2_32.DLL

write()

recv()

send()

EP(A)

EP(B)

Logi

cal T

ime

EPTime(A)

EPTime(B)

Java.net.Sokect-

InputStream

read()



• Supported data collectors– WDBG Win32 Profiler– JVPROF Java Profiler– JSA Java static analyzer– Others, provided the conversion to XML

• Network-call interceptors – WDBG and JVPROF


Data Repository

• Logical Models– Language definitions– Program Data– Used-defined views

• Implementation– Relational database

• Query/Manipulation Language– SQL– SMQL


Language Definitions

• Model(General) = Graph(Eg , Rg)Eg: Set of entity types

Rg: Set of relation types

• Java, example:• Model(Java) = Graph(Ejava , Rjava)

Ejava Eg : Entity types supported by Java

Rjava Rg : Relation types supported by Java

Such that the source and destination entity Ejava

• C++, definingModel(C++) = Graph(Ec++, Rc++)

Ec++ = Ejava {template, struct, typedef, function}

- {interface, package}


Java Language Example

method

object interface

package

class

field

creates

reads

writes

contains

defines

declares

instanceof

declares

End-point

connects

implements

invokes


SMQL

• SMQL– Software Modeling Query Language

• Set-based:– Typed Set

• Entity Set

• Relation Set

• Event Set

– Generic Set

• Extensible using Java


SMQL

• Operations– Union, intersection, Difference

• Functions– Closure

Closure(Source, Target, RelationTypes)

Returns a Relation-Set

– Composition• Builds a hierarchical model from flat set of entities into a nested

set of entities

– Output: Saves a typed-set as an XML document– Output_dot: Saves a relation set as DOT graph


Example: Deployment Architecture

Environment

Windows 2000IP Address:192.92.200.3

LinuxIP Address:192.92.200.2

Jetty WebServer(Java JVM 1.4.2)

jvprofData Collection

Tear.exe

wdbgData collection

Data CollectionManager SCDR

SQL Server 2000

SQ

L

SM

QL Analysis

Jetty Source code

jsaData collection

Dyn

am

ic

Sta

tic


SMQL: Defining Entity Set

Entity Identifier

{

[caption = String ;]

type = StringSet ;

(EntityFirstOrderFilter)+

}

EntityFirstOrderFilter ::

include ( EntityAttrib ) = StringSet ;

| exclude ( EntityAttrib ) = StringSet ;


SMQL: Inheritance example

// Get Inheritance tree rooted at org.mortbay.http.HttpListener

// OR org.mortbay.http.HttpHandler

EntitySet Http

{

caption = "HttpListener" ;

type = {“interface”, “class”} ;

include (name) = { "org.mortbay.http.HttpListener", "org.mortbay.http.HttpHandler" } ;

include (project) = { "jetty" } ;

}

Inh_Http = closure(null, Http, { “implement”, “subclass” } ) ;

Output (“file.xml”, Inh_Http, Http) ;

Output_dot (“graph.dot”, Inh_Http) ;


SMQL: Inheritance example


SMQL: Events Example// Events from Tear & Jetty

import java:serg.sc.filter.CallGraph as CallGraph ;

EventSet Events{caption = "Event (Jetty & Tear) " ;type = { "method-entry",

"method-exit", "endpoint", "thread-start", "thread-end", "module-load","process-start","process-end"} ;

include (project) = {"jetty-rt", "Tear"} ;}

// Convert events trace into a CallGraph (Relation Set)

GC = CallGraph( events ) ;

// Save as resultsoutput ("tear-jetty.xml", events);

Output_dot (“GC.dot”, GC) ;


SMQL: Events Example


SMQL: User defined functions

• User-defined functions– IFilter interfaceImport java:Qualified-Java-Class-Name as FunctionName ;

• ExampleImport java:serg.sc.MyClass as MyFunction

EntitySet E

{

caption = "HttpListener" ;

type = {“interface”, “class”} ;

include (name) = { "org.mortbay.http.HttpListener” } ;

include (project) = { "jetty" } ;

}

X = MyFunction(E) ;


Modeling and visualization subsystem

SMQL Analysis Visualization

Vis

ua

l Sty

le

Na

vig

atio

na

lS

tyle

Software ViewModelRevisedModel

ProgramData


Progress

• Repository implementation– SQL Server & Hypersonic DB

• Data gathering– WDBG, JVPROF, and JSA

• Tools– SMQL– Dbmgr - Database and SMQL front-end– dbImport, dbexport


Conclusions and Future Work

• Software Comprehension Environment• Dynamic analysis of distributed systems• Multi-language systems• Common repository• Distributed profiler• SMQL

• Future work– Phase I

• Performance• Support for Microsoft.Net, VB and COM+

– Phase II (Early stage)• Construction and visualization of views• Empirical evaluation in an industrial setting (at Cigna)


ReferencesVigder-2001] M. Vigder. The evolution, maintenance, and management of

component-based systems. In G. Heineman and W. Councill, editors, Component-Based Software Engineering: Putting the Pieces Together, chapter 29, pages 527. 539. Addison-Wesley, 2001.

[Weyuker-2001] E. J. Weyuker. The trouble with testing components. In G. Heineman and W. Councill, editors, Component-Based Software Engineering: Putting the Pieces Together, chapter 28, pages 499.512. Addison-Wesley, 2001.

[Stroermer-2002] C. Stroermer, L. O'Brien, and C. Verhoef. Practice patterns for architecture reconstruction. In Proceedings of the 9th Working Conference on Reverse Engineering (WCRE), Richmond, VA, USA, October 2002.

[Hassan-2002] A. E. Hassan and R. C. Holt. Architecture recovery of web applications. In 24th International Conference on Software Engineering (ICSE), Orlando, Florida, USA, May 2002.

[Chen-1995] Y.-F. R. Chen, G. S. Fowler, E. Koutsoos, and R. S. Wallach. Ciao: A graphical navigator for software and document repositories. In Proceedings of International Conference on Software Maintenance (ICSM), Nice, France, October 1995. IEEE.


References[Demeyer-1999] S. Demeyer, S. Tichelaar, and P. Steyaert. Famix 2.0 – the

famoos information exchange model. Technical report, University of Berne, August 1999.

[CDIF-1994] Electronic Industries Alliance and International Standards Organization (EIA/ISO). CDIF - Framework for Modeling and Extensibility, 1994.

[XML/XMI-2002] Object Management Group. XML Metadata Interchange (XMI), version 1.2 edition, January 2002.

[Koschke-1998] R. Koschke, J.-F. Girard, and M.Wurthner. An intermediate representation for reverse engineering analysis. In Proceedings of the 5th Working Conference on Reverse Engineering (WCRE), Honolulu, HI, USA, October 1998.

[Bruegge-1993] B. Bruegge, T. Gottschalk, and B. Luo. A framework for dynamic program analysis. In Conference on Object Oriented Programming Systems, Languages, and Applications (OOSLA93), Washington, USA, September 1993.

[Mendonca-1999] N. C. Mendonca. Software Architecture Recovery for Distributed Systems. PhD thesis, Department of University of London, November 1999.