unbounded data model verification using smt solvers

Unbounded Data Model Verification Using SMT Solvers

Jaideep Nijjar Tevfik BultanUniversity of California, Santa Barbara

ASE 2012

Web Software Everywhere• Commerce, entertainment, social interaction

• We will rely on web apps more in the future

• Web apps + cloud will make desktop apps obsolete

Acknowledgement: NSF Support

It is not just NSF

Web Application Dependability

Web Application Dependability is a Problem• Web applications are hard to program

• Distributed behavior, interaction among many components and many languages

• Web applications are hard to test • Highly dynamic behavior and concurrency

• Web applications are easy targets for hackers• They are notorious for security vulnerabilities and unreliable

behavior

My research group’s goal: • Improving dependability of web applications using automated

verification techniques!

We have a hammer

Automated Verification

Unfortunately, life is complicated

Automated verification techniques

Web application dependability problems

Making it work

Separation of concerns + modularity+ abstraction/extraction

SMT-basedverification

Data modelproblems

Formal data model+ Properties

Three-Tier Architecture

Backend Database

Browser

Web Server

Three-Tier Arch. + MVC Pattern

Backend Database

Browser

Model

ViewsController

Web Server

• MVC pattern has become the standard way to structure web applications:

• Ruby on Rails• Zend for PHP• CakePHP• Struts for

Java• Django for

Python• …

Web Application

• Ruby on Rails

Data Model • ActiveRecords

Formal Model

• Sets and Relations

Unbounded

Verification• SMT

Solver

Our Approach

MVC DesignPattern

Automatic Extraction

AutomaticTranslation +AutomaticProjection+Properties

Outline

• Motivation• Overview of Our Approach• Rails Semantics• Translation to SMT-LIB• Experiments• Related Work• Conclusions

A Rails Data Model Exampleclass User < ActiveRecord::Base

has_and_belongs_to_many :roleshas_one :profile, :dependent => :destroyhas_many :photos, :through => :profile

endclass Role < ActiveRecord::Base

has_and_belongs_to_many :usersendclass Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroy has_many :videos, :dependent => :destroy, :conditions => "format='mp4'"endclass Tag < ActiveRecord::Base

belongs_to :taggable, :polymorphic => trueendclass Video < ActiveRecord::Base belongs_to :profile has_many :tags, :as => :taggableendclass Photo < ActiveRecord::Base ...

Role

*

0..1

1User

Profile

*

1

Video

*

1

Taggable

*Tag

1

* 1Photo

*

1

format=.‘mp4’

Rails Data Models

• Data model verification: Analyzing the relationships between data objects

• Specified in Rails using association declarations inside the ActiveRecord files• The basic relationships

• One-to-one• One-to-many• Many-to-many

• Extensions to the basic relationships using Options• :through, :conditions, :polymorphic, :dependent

The Three Basic Relationships in Rails

• One-to-One (One-to-ZeroOrOne)

.

• One-to-Many

class User < ActiveRecord::Base has_one :profileend.

class Profile < ActiveRecord::Base belongs_to :userend

class Profile < ActiveRecord::Base has_many :videosend.

class Video < ActiveRecord::Base belongs_to :profileend

User

Profile

0..1

Profile

Video

*

1

1

The Three Basic Relationships in Rails

• Many-to-Manyclass User < ActiveRecord::Base has_and_belongs_to_many :usersend

class Role < ActiveRecord::Base has_and_belongs_to_many :rolesend

User

Role

*

*

Options to Extend the Basic Relationships

• :through Option• To express transitive relations

• :conditions Option• To relate a subset of objects to another class

• :polymorphic Option• To express polymorphic relationships

• :dependent Option• On delete, this option expresses whether to delete the associated

objects or not

The :through Optionclass User < ActiveRecord::Base

has_one :profilehas_many :photos, :through => :profile

endclass Profile < ActiveRecord::Base belongs_to :user has_many :photosendclass Photo < ActiveRecord::Base belongs_to :profileend Profile

User Photo

**

0..1 1

1

1

The :dependent Option

• :delete directly delete the associated objects without looking at its dependencies

• :destroy first checks whether the associated objects themselves have associations with the :dependent option set

class User < ActiveRecord::Base has_one :profile, :dependent => :destroyend

class Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroyend

PhotoProfileUser *1 10..1

Formalizing Rails Semantics

• S: The sets and relations of the data model (data model schema)• e.g. { Photo, Profile, Role, Tag, Video, User} and the relations

between them

• C: Constraints on the relations• Cardinality constraints, transitive relations, conditional relations,

polymorphic relations

• D: Dependency constraints • Express conditions on two consecutive instances of a relation such

that deletion of an object from the first instance leads to the other instance

Formal data model: M = <S, C, D>

Formalizing Rails Semantics

• I = <O,R> is an instance of the data model M = <S,C,D>,denoted by I |= M, iff1. the sets in O and the relations in R follow the schema S, and 2. R |= C

• Given a pair of data model instances I = <O,R> and I’ = <O’,R’> (I, I’) is a behavior of the data model M = <S,C,D>,denoted by (I, I’) |= M, iff1. O and R and O’ and R’ follow the schema S2. R |= C and R’ |= C, and 3. (R,R’) |= D

Data Model Properties

Given a data model M = <S,C,D>, we define four types of properties: 1. state assertions (AS): properties that we expect to hold for each

instance of the data model2. behavior assertions (AB): properties that we expect to hold for

each pair of instances that form a behavior of the data model3. state predicates (PS): properties we expect to hold in some

instance of the data model4. behavior predicates (PB): properties we expect to hold in some

pair of instances that form a behavior of the data model

Outline


Translation to SMT-LIB

• Given a data model M = <S, C, D>we translate the constraints C and D to formulas in the theory of uninterpreted functions

• We use the SMT-LIB format

• We need quantification for some constraints


class Profile has_many :videosendclass Video belongs_to :profileend

(declare-sort Profile 0)(declare-sort Video 0)(declare-fun my_relation (Video) Profile).

SMT-LIB:

RAILS:

• One-to-Many Relation


class User has_one :profileendclass Profile belongs_to :userend

(declare-sort User 0)(declare-sort Profile 0)(declare-fun my_relation (Profile) User).(assert (forall ((x1 Profile)(x2 Profile)) (=> (not (= x1 x2)) (not (= (my_relation x1) (my_relation x2) ))) ))

SMT-LIB:

RAILS:

• One-to-One Relation


class User has_and_belongs_to_many :rolesendclass Role has_and_belongs_to_many :usersend

(declare-sort Role 0)(declare-sort User 0)(declare-fun my_relation (Role User) Bool)

SMT-LIB:

RAILS:

Many-to-Many Relation

Translating the :through Optionclass Profile < ActiveRecord::Base belongs_to :user has_many :photosendclass Photo < ActiveRecord::Base belongs_to :profileEndclass User < ActiveRecord::Base has_one :profile has_many :photos, :through => :profileend

(declare-sort Profile 0)(declare-sort Photo 0)(declare-sort User 0)(declare-fun profile_photo (Photo) Profile)(declare-fun user_profile (Profile) User)(declare-fun user_photo (Photo) User)(assert (forall ((u User)(ph Photo)) (iff (= u (user_photo ph)) (exists ((p Profile)) (and (= u (user_profile p)) (= p (profile_photo ph)) )) )))

Profile

User Photo

0..1

**

11

1

Translating the :dependent Option• The :dependent option specifies what behavior to take on

deletion of an object with regards to its associated objects• To incorporate this dynamism, the model must allow analysis of how

sets of objects and their relations change from one state to the next

class User < ActiveRecord::Base has_one :account,

:dependent => :destroyend

.

class Profile < ActiveRecord::Base belongs_to :userend

(declare-sort Profile 0)(declare-sort User 0)

(declare-fun Post_User (User) Bool)(declare-fun Post_Profile (Profile) Bool)

(declare-fun user_profile (Profile) User)(declare-fun Post_user_profile (Profile User) Bool)

Translating the :dependent Option

• Update sets relations of its associated object(s) based on the use of the :dependent option

• A relation is only updated if it is a :belongs_to or :has_and_belongs_to_many relationship• In the database, the foreign key is stored with the object that has

the :belongs_to relationship

(assert (not (forall ((x User)) (=> (and (forall ((a User)) (ite (= a x) (not (Post_User a)) (Post_User a))) (forall ((b Profile)) (ite (= x (user_profile b)) (not (Post_Profile b)) (Post_Profile b) )) (forall ((a Profile) (b User)) (ite (and (= b (user_profile a)) (Post_Profile a)) (Post_user_profile a b) (not (Post_user_profile a b)) )) ) ;Remaining property-specific constraints go here)))

Verification

• Once the data model is translated to SMT-LIB format we can state properties about the data model again in SMT-LIB and then use an SMT-Solver to check if the property holds in the data model

• However, when we do that, for some large models, SMT-Solver times out!

• Can we improve the efficiency of the verification process?

Property-Based Data Model Projection• Basic idea: Given a property to verify, reduce the size of the

generated SMT-LIB specification by removing declarations and constraints that do not depend on the property

• Formally, given a data model M = <S, C, D> and a property p, (M, p) = MP

where MP = S, C⟨ P, DP is the projected data model such that C⟩ P ⊆C and DP D⊆

• Key Property: For any property p, M |= p ⇔ (M, p) |= p• Implemented as part of our tool

• Algorithm Input: Active Record files, property p• Output: The projected SMT-LIB specification • Removes constraints on those classes and relations that are not

explicitly mentioned in the property nor related to them based on transitive relations, dependency constraints or polymorphic relations

Data Model Projection: ExampleProperty, p: A User’s Photos are the same as the User’s Profile’s Photos.

Role

*

0..1

1User

Profile

*

1

Video

*

1

Taggable

*Tag

1

* 1Photo

*

1

Data Model, M:

0..1

1User

Profile* 1Photo

*

1(M, p) =

Verification Overview

Translator

SMT Solver (Z3)

Verified

Counter-example Data Model

Instance

Unknown

Active Records

Formal Data Model

Data Model Properties

Projection

SMT-LIB Specification

Outline

• Motivation• Overview of Our Approach• Rails Semantics• Translation to SMT-LIB• Experiments• Related Work• Conclusion

Experiments

• We used five open-source Rails apps in our experiments:• LovdByLess: Social networking site• Tracks: An application to manage things-to-do lists• OpenSourceRails(OSR): Social project gallery application• Fat FreeCRM: Customer relations management software• Substruct: An e-commerce application

• We wrote 10 properties for each application

LovdByLess Tracks OSR Fat Free

CRM Substruct

LOC 3787 6062 4295 12069 15639

Data Model Classes 13 13 15 20 17

Types of Properties Checked

• Relationship Cardinality• Is an Opportunity always

assigned to some Campaign?

• Transitive Relations• Is a Note’s User the same as the

Note’s Project’s User?

• Deletion Does Not Cause Dangling References• Are there any dangling Todos after a User is deleted?

• Deletion Propagates to Associated Objects• Does the User related to a Lead still exist

after the Lead has been deleted?

Note

User Project

Experimental Results

• 50 properties checked, 16 failed, 11 were data model errors• For example in Tracks, a Note’s User can be different than Note’s

Project’s User• Currently being enforced by the controller• Since this could have been enforced using the :through option, we

consider this a data-modeling error• From OpenSourceRails: User deletion fails to propagate to

associated Bookmarks

• Leaves orphaned bookmarks in database• Could have been enforced in the data model by setting

the :dependent option on the relation between User and Bookmark

BookmarkUser*1

Performance

• To measure performance, we recorded • The amount of time it took for Z3 to run and check the properties• The number of variables produced in the SMT specfication

• The time and number of variables are averaged over the properties for each application

• To compare with bounded verification, we repeated these experiments using the tool from our previous work and Alloy Analyzer• The amount of time it took for Alloy to run• The number of variables generated in the boolean formula generated for

the SAT solver• Taken over an increasing bound, from at most 10 objects for each class to

at most 35 objects for each class

Performance: Verification Time

10 15 20 25 30 350

0.51

1.52

2.5LovdByLess

Alloy

Z3

Z3+proj

Scope10 15 20 25 30 35

02468

Substruct

Verifi

catio

n Ti

me

(s)

10 15 20 25 30 350

0.51

1.52

2.5OSR

10 15 20 25 30 3505

10152025

FatFreeCRM

10 15 20 25 30 3501234567

Tracks

Verifi

catio

n Ti

me

(s)

Performance: Formula Size (Variables)

Z3 Alloy

10 15 20 25 30 350

100200300400500600700800

LovdByLessTracksOSRSubstruct

Scope

No.

Var

iabl

es (t

hous

ands

)

Lovd

ByLess

Tracks OSR

Substr

uct

FatFr

eeCRM

0

40

80

120

160

200

non-proj proj

No.

Var

iabl

es

Unbounded vs Bounded Performance

• Why does unbounded verification out-perform bounded so drastically?

Possible reasons:• SMT solvers operate at a higher level of abstraction than SAT

solvers• Z3 uses many heuristics to eliminate quantifiers in formulas• Implementation languages are different

• Z3 implemented in C++• Alloy (as well as the SAT Solver it uses) is implemented in Java

Outline


Related Work• Previous work on Data Model Verification using Alloy

• [Cunha and Pacheco, SEFM 2009] maps relational database schemas to Alloy; not automated

• [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy, and uses the Analyzer to produces instances of the data model to show consistency

• These approaches bounded, not unbounded technique like ours• [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser

and business logic interactions• A different class of bugs than the data model related bugs we focus on

• Unbounded verification of Alloy specifications using SMT Solvers• [Ghazi et al, FM 2011], approach not implemented• More challenging domain since Alloy language contains constructs

such as transitive closures which do not appoar in the data models we extract

Related Work

• Specification and Analysis of Conceptual Data Models• [McGill et al ISSTA 2011, Smaragdakis et al ASE 2009, Halpin et al

IMRD 2009]• Model-driven (forward engineering) approaches, whereas we

performed model extraction (reverse engineering)• Formal modeling of Web Applications

• [Book et al ASE 2004, Hallé et al ASE 2010, Han et al MoDELS 2007]

• Focus on navigation aspects as opposed to data model• SMT Solvers More Efficient than SAT Solvers

• Observed in other verification domains [Cordeiro et al, ASE 2009]• The data model verification problem we investigate is different

from the problems studied in earlier works

Conclusions

• Goal: To automatically discover data model errors in Ruby on Rails web applications

• Approach: Automatically extract a formal data model, translate it to the theory of uninterpreted functions, and verify using an SMT-solver• Use property-based data model projection for efficiency

• Implementation: An automatic translator from Rails ActiveRecords to SMT-LIB• Handles three basic relationships and several options

(:through, :conditions, :polymorphic, :dependent)• Experiments: Found multiple data model errors on five open

source applications• Unbounded verification of data models is feasible and more efficient

than bounded verification!

Future Work

• Analyzing dynamic behavior• Model object creation in addition object deletion• Fuse the data model with the navigation model in order to

analyze dynamic data model behavior• Check temporal properties

• Automatic Property Inference• Current work requires manual property writing• Use the inherent graph structure in the the data model to

automatically infer properties about the data model• Automatic Repair

• When verifier concludes that a data model is violated, automatically generate a repair that establishes the violated property

Questions?

unbounded data model verification using smt solvers

Documents