graph-based source code analysis of javascript repositories

Graph-Based Source Code Analysisof JavaScript Repositories

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Fault Tolerant Systems Research Group

Dániel SteinGábor Szárnyas

Content

1. Context

2. Tooling

3. Use Cases

4. Neo4j Observations

Continuous Integration (CI)

– Developers working together

– Prevent integration problems

– Examples

– Jenkins

– Hudson

– Travis CI

Verziókezelés

Fordítás

Fejlesztés

Egység- és integrációs teszt

DevelopmentVersion ControlSystem

CompilationUnit and

IntegrationTests

Apple,https://blog.codecentric.de/en/2014/02/curly-braces/

whoops

Apple,https://blog.codecentric.de/en/2014/02/curly-braces/

Static Analysis

– No need for compilation orexecution of the application

– Formatting, structural and semantic rule checking

– Can extend the workflow of continuous integration and improve it

– In this research we used codeanalysis utilizing patternmatching

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

CompilationUnit and

IntegrationTests

StaticAnalysis

Static Analysis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

CompilationUnit and

IntegrationTests

StaticAnalysis

Static Analysis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

– Java

– FindBugs

– PMD

– CheckStyle

CompilationUnit and

IntegrationTests

StaticAnalysis

Static Analysis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

– Java

– FindBugs

– PMD

– CheckStyle

– JavaScript

– ESLint

– Facebook Infer, Flow

– Tern

– TAJS

CompilationUnit and

IntegrationTests

StaticAnalysis

– Thorough code analysis is time-consuming and resource-intensive

– For large projects it can be too slow

Problems to Solve

unit tests

static analysis

☼ ☆☾☆

– Temporary solution: batching

Problems to Solve

unit tests

static analysis

☼ ☆☾☆

unit tests

static analyis

– Temporary solution: batching

Present results

as soon and as fast

as possible.

Problems to Solve

unit tests

static analysis

☼ ☆☾☆

unit tests

static analyis

Problems to Solve

– Memory limits appear when...

– Global rules are checked

– Storing the structure in-memory

– For large code repositories

– Not being incremental

– Batched execution simplydoes not cut it

– Small change inducescomplete recheck

Our Approach

– Incremental methodology– Instead of batched execution

– Update the prepared results with theeffects of the change

– Only store the required parts in thememory

analyzer

Δ2.-1.1.

VCS Workspace Abstact SyntaxTree

Abstract SemanticGraph

Well-formednessRules

Query Execution Database

Main.js | ++----

Dependency.js | +++++-

FIterator.js | ----

Parser.js | ++

AutomaticWell-formedness

Rule Evaluation

Manual Executionand Data Extraction

Querying and Transformation

discoverer

ChangeProcessor.js

CommandParser.js

FileIterator.js

iterators

DepCollector.js

FileDiscoverer.js

InitIterator.js

Main.js

whitepages

ConnectionMgr.js

DependencyMgr.js

neo4jValidation Report

<!><?>

Module

declaration

declarators

binding init

left right

Architecture overview

Main.js | ++----

FIterator.js | ----

Parser.js | ++

Rule Evaluation

discoverer

ChangeProcessor.js

CommandParser.js

FileIterator.js

iterators

DepCollector.js

FileDiscoverer.js

InitIterator.js

Main.js

whitepages

ConnectionMgr.js

DependencyMgr.js

<!><?>

Module

declaration

declarators

binding init

left right

Main.js | ++----

FIterator.js | ----

Parser.js | ++

Rule Evaluation

discoverer

ChangeProcessor.js

CommandParser.js

FileIterator.js

iterators

DepCollector.js

FileDiscoverer.js

InitIterator.js

Main.js

whitepages

ConnectionMgr.js

DependencyMgr.js

<!><?>

Module

declaration

declarators

binding init

left right

Main.js | ++----

FIterator.js | ----

Parser.js | ++

Rule Evaluation

discoverer

ChangeProcessor.js

CommandParser.js

FileIterator.js

iterators

DepCollector.js

FileDiscoverer.js

InitIterator.js

Main.js

whitepages

ConnectionMgr.js

DependencyMgr.js

<!><?>

Module

declaration

declarators

binding init

left right

Code Processing Steps

tokenizer

source code

tokens

parser

scope analyzer

tokenizer

source code

tokens

parser

scope analyzer

tokenizer

source code

tokens

parser

scope analyzer

tokenizer

source code

tokens

parser

scope analyzer

Sequence of statements

formalized in a given language

tokenizer

source code

tokens

parser

scope analyzer

Sequence of statements

formalized in a given language

tokenizer

source code

tokens

parser

scope analyzer

tokenizer

source code

tokens

parser

scope analyzer

token – the shortest character sequence still having meaning.

tokenizer

source code

tokens

parser

scope analyzer

tokenizer

source code

tokens

parser

scope analyzer

Token Token type

VAR (Keyword)

IDENTIFIER (Ident)

ASSIGN (Punctuator)

NUMBER (NumericLiteral)

DIV (Punctuator)

NUMBER (NumericLiteral)

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclarationStatement

VariableDeclaration

VariableDeclarator

BindingIdentifiername = `foo`

BinaryExpressionoperator = `Div`

LiteralNumericExpressionvalue = 1.0

declaration

declarators

binding init

left right

tokenizer

source code

tokens

parser

scope analyzer

Abstract Syntax Tree (AST)

– Tree representation of

– the grammar structure of

– the sequence of tokens.

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclaration

VariableDeclarator

BinaryExpressionoperator = `

declaration

declarators

binding init

left right

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

GlobalScope

Variablename = `foo`

Referenceaccessibility = `Write`

variables

references

children

Declarationkind = `Var`

declarations

astNode

tokenizer

source code

tokens

parser

scope analyzer

Code Processing StepsAbstract Semantic Graph(ASG)

– Graph, not necessarily tree.

– Semantic information besidesthe syntactic structure.

– Containscross-edges →

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

GlobalScope

variables

references

children

declarations

astNode

tokenizer

source code

tokens

parser

scope analyzer

Module

VariableDeclaration

VariableDeclarator

declaration

declarators

binding init

left right

GlobalScope

variables

references

children

declarations

astNode

AST vs ASG

20-40-50nodes

Overview of the Approach

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

Verziókezelés

Fordítás

Fejlesztés

Kódanalízis

CompilationUnit and

IntegrationTests

StaticAnalysis

VersionControlSystem

IntegratedDevelopmentEnvironment

Git, Visual StudioCode

tokenizer

source code

tokens

parser

scope analyzer

Git, Visual StudioCode ShapeSecurityShift

transformationIntegrated

DevelopmentEnvironment

tokenizer

source code

tokens

parser

scope analyzer

Git, Visual StudioCode ShapeSecurityShift Java, Cypher

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

Git, Visual StudioCode ShapeSecurityShift Java, Cypher Neo4j

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

transformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

transformationtransformation

graphdatabase

tokenizer

source code

tokens

parser

scope analyzer

resultprocessing

Graph Pattern Matching

VariableDeclarator

LNExpressionvalue = 1.0

– Graph pattern

– A declarative,

– graph-like formalism

– expressing constraints.

VariableDeclarator

– Graph pattern

– A declarative,

VariableDeclarator

binding be

– Graph pattern

– A declarative,

VariableDeclarator

binding be

– Graph pattern

– A declarative,

Graphpatternqueryexpressed in Cypherlookingforadivisionbyzero

binding

Resultsof thepatternmatching

Use Cases static analysis

– Searching for local badsmells (linter warnings)

– without a case

– value set more than once

– Not used variable

– Global rules– Unreachable code parts

– Framework

– Freely extendable

– User-defined rules

– Easier to use than visitorpattern solutions

Use Cases transformation

Control Flow Graph (CFG)

– Graph representation of

– every possiblestatement sequence

– during code execution.

statement

if condition

statement

statement statement

if condition

statement

statement statement

statement

condition

statement

statement statement

statement

condition

statement

statement statement

statement

condition

statement

statement statement

statement

condition

statement

statement statement

statement

return

statement

condition

Use Cases test generation

statement

statement statement

statement

return

condition

statement

– Inspecting control flows

– Is the given statement reachable

given the constraints on the

edges?

– Which one is the shortest route?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

statement

statement statement

statement

return

condition

statement

edges?

– Producing test input

for dynamic testing

statement

statement statement

statement

return

condition

statement

Use Cases type inference

– Supporting dynamically typed languages

– Python

– JavaScript / ECMAScript

Use Cases type inference

– Supporting dynamically typed languages

– Python

– JavaScript / ECMAScript

http://marijnhaverbeke.nl/blog/tern.html

Use Cases impact analysis

– Adapting to the continuous integration workflow

– Handling multiple branches

– Following the modifications in a branch

– File-level incremental granularity

– Giving differential reports to the developers

Why Neo4j?+++

– Quick prototyping

– Supporting transactions

– Great tooling

– Not scaling well

– Only disk-based

Remarks MERGE

– MATCH or CREATE

– Great for the lazy

– Can be expensive

– Possible solutions:

– Less MERGE

– Separating queries

– Create first if not present

– Use MATCH instead of MERGE

– Prevention

– Prepare the structure when

inserting the data

Remarks MERGE

Remarks if-then-else

– Not a language element in

Cypher

– Can be solved with a trick

– Verrrrrry sloww

– Solution:

– Two smaller, disjunct cases

– Not a language element in

Cypher

– Can be solved with a trick

– Verrrrrry sloww

– Solution:

– Two smaller, disjunct cases

28∞ vs 15 sec

These are not chickens.

Remarks reachability

– Transitive closure without

length constraints is slow.

– Transitive closure over

repeating node/edge pattern

is only possible using tricks.

Conclusions

– Source code analyzerframework

– Searching for global errorpatterns

– Close to real time feedback

– Type inference possible

– Test input generation possible

– Approach for both dynamicallyand statically typed languages

– Using Neo4j for

– Storing

– Pattern matching

– Transforming

– Version control

– Storing metadata

– Our work was supported by:

– ÚNKP*

– Microsoft Azure for Research

– MTA-BME Lendület Program

Project Details

– The frameworkprototype is open-source.

https://github.com/

ftsrg/codemodel-rifle

*Supported by the ÚNKP-16-2-I. New National Excellence Program of the Ministry of Human Capacities.

Project Details

– Supervisors

– Ádám Lippai

– Dávid Honfi

– Gábor Szárnyas

– Helped my research

– Tamás Soma Lucz

– Industrial case study

graph-based source code analysis of javascript repositories

Technology

javascript beginner's guide to javascript

eiw: javascript the language1 the javascript language

javascript specialist. lesson 1: introduction to javascript

polynomial graph-colorings - connecting repositories ·...

javascript session 01 - introduction to javascript

javascript for beginners - daniel...

javascript for · pdf...

klemen nagode - connecting repositories · ajax -...

measuring javascript performance, javascript games

javascript - valdosta state university javascript homework...

modern javascript - o’reilly...

javascript & jquery javascript and jquery introduction

“software theft detection for javascript programs based on...

javascript (js) 3.pdffull stack javascript (หร อ...

creating an open community platform - amazon s3€¦ ·...

graph-based analysis of javascript repositories · 2018. 2....

graph-based analysis of javascript source code...

javascript javascript introduction. q. what is javascript?...

javascript in javascript: inception (jsconf eu 2014)

analyzing javascript and the web with...