Computers in Human Behavior 27 (2011) 730–740

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier .com/locate /comphumbeh

Developers’ perceptions of object-oriented frameworks – An investigation into theimpact of technological and individual characteristics

Gregor Polancic ⇑, Marjan Hericko, Luka PavlicFaculty of Electrical Engineering and Computer Science, University of Maribor, Slovenia

a r t i c l e i n f o

Article history:Available online 10 November 2010

Keywords:Object-oriented frameworksSystem characteristicsIndividual differencesEmpirical researchDeveloper’s perceptionsTechnology acceptance model

0747-5632/$ - see front matter � 2010 Elsevier Ltd. Adoi:10.1016/j.chb.2010.10.006

⇑ Corresponding author. Tel.: +386 2 220 7421.E-mail addresses: gregor.polancic@uni-mb.si (G. Po

mb.si (M. Hericko), luka.pavlic@uni-mb.si (L. Pavlic).

a b s t r a c t

Framework-based development is currently regarded as one of the most promising software develop-ment approaches, boasting increases in lead time, productivity and quality. However, many frameworksand framework-related projects still fail, which indicates that frameworks still have unsolved problems.In this article, we will identify and investigate the major framework characteristics and individual differ-ences that impact the most important users’ perceptions about frameworks. To test the causal relation-ships between these factors, we performed an online survey and analyzed the results using structuralequation modeling. The results support the technology acceptance model (TAM), which was used asan underlying theory. In addition, we found that framework characteristics and individual differenceshave a significant impact on users’ perceptions, especially in the case of voluntary framework use. BesideTAM constructs, the results also suggest an additional determinant for the acceptance of frameworks:‘‘confidence”. Despite the limits of our research, we foresee future research activities as well as theoret-ical and practical implications. Our results might be used to develop acceptable frameworks and for theevaluation of existing frameworks, their constituent parts and framework-related guidelines.

� 2010 Elsevier Ltd. All rights reserved.

1. Introduction a product family (also known as framework instances) (Srinivasan,

Software developers use different approaches to improve theproductivity and quality of their outcomes. According to Boehm(1999), three major approaches help to achieve these goals: (1) theuse of better development tools, (2) the use of better software devel-opment processes and (3) the reuse of existing software artifacts.Empirical results (Boehm, 1999) indicate that software reuse is themost promising approach, under the condition that common reusepitfalls can be identified and controlled. In accordance with this,Morisio, Ezran, and Tully (2002) investigated common reuse pitfallsand found that the two most critical software reuse success factorsare human factors and the product-type development approach.Fig. 1 presents these findings as a decision tree, where the numberson the arrows denote how many examples exercise some edge.

As seen in Fig. 1, the key to software reuse success is in consid-ering human factors and choosing a product family developmentstrategy. Product families are most commonly realized with ob-ject-oriented frameworks (Batory, Cardone, & Smaragdakis, 2000;Cunningham, Liu, & Zhang, 2006). Object-oriented frameworks(or simply ‘‘frameworks”) are incomplete systems that containcertain fixed aspects common to all applications in a productfamily, and certain variable aspects unique to each application in

ll rights reserved.

lancic), marjan.hericko@uni-

1999). Frameworks differ from other reuse techniques, such ascomponents, libraries or design patterns, because they aim to reuselarger grained components and higher-level designs. They also de-fine the flow of control and so act as the main program for instan-tiated applications.

Frameworks have a central role to play in the software develop-ment community (Manolescu, Noble, & Voelter, 2006), where theimpact of frameworks on software development have been inves-tigated by several researchers. They found that the proper use offrameworks increases software development productivity and thequality of software (Mamrak & Sinha, 1999; Moser & Nierstrasz,1996). The proper use of frameworks is mainly related to the ex-tent of framework use, where researchers found that the benefitsof framework use increase with the number of framework instan-tiations (Bockle, Clements, McGregor, Muthig, & Schmid, 2004).

However, developers have reported several problems related tothe development, instantiation and maintenance of frameworks(Bosch, Peter, Mattsson, & Bengtsson, 2000; Srinivasan, 1999; vanGurp & Bosch, 2001). In addition, researchers have reported thatthe product families’ engineering area is lacking in measurementsand experimentation (Frakes & Kang, 2005). All of these problemsdivert application developers from accepting new frameworks orcause them to abandon using frameworks before their investmentin frameworks actually provides a reward. Our research is relatedto this framework-acceptance problem and addresses issues,which are presented below.

Human factors

Product type

Reusefailure

Reusesuccess

No(7)

Yes(16)

Product

families(15)

Isolated

(1)

Fig. 1. Critical success factors of software reuse (Menzies, 2003, p. 475).

G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740 731

By searching through the available literature, we will investi-gate what the common framework-related technological charac-teristics and specialties of their users are. Secondly, by performinga survey, we will investigate if and how these factors are related tousers’ considerations about frameworks. In this step, we will baseour research on a proven theoretical basis, utilizing the technologyacceptance model (TAM). Third, considering our results and TAM,we will propose some solutions for improving the acceptanceand continuous use of frameworks. From the viewpoint of Fig. 1,we believe that investigating the above-presented frameworkissues will address the remaining critical success factor of softwarereuse: human factors.

The original outcome of this research is an empirically validatedcausal model that defines relationships between framework char-acteristics, user specialties and user’s perceptions. Based on themodel, we will present some of our current and planned activities,which are oriented towards improving the proposed model and topractical applications of the model.

2. Related work

2.1. Technology acceptance model

The technology acceptance model (TAM) (Fig. 2) as defined byDavis (1989) is one of the most cited and validated theoreticalmodels for examining the acceptance and use of information tech-nologies (IT) (King & He, 2006). A key assumption of TAM is thatexternal variables (EV) influence user attitudes, behavioral inten-tions (BI) and use, indirectly, via two user beliefs: perceived easeof use (PEOU) and perceived usefulness (PU). PU is defined as theextent to which a person believes that using the system will en-hance his or her job performance. PEOU is defined as the extentto which a person believes that using the system will be free of ef-fort (Davis, 1989).

TAM (Fig. 2) does not define which external variables influenceuser perceptions, because they are dependent on the type of theinvestigated system (Seddon, Staples, Patnayakuni, & Bowtell,1999). However, by manipulating proper external variables, sys-tem developers can gain better control over users’ beliefs of thesystem, and subsequently, their behavioral intentions and use of

External Variables

(EV)

Perceived Usefulness

(PU)

Perceived Ease of Use

(PEOU)

Behavioral Intention to

Use (BI)

Actual System Use

(U)

Fig. 2. Technology Acceptance Model (Davis, 1989). This version of TAM is knownas parsimonious TAM because it excludes attitudes.

the system (Hong, Thong, Wong, & Tam, 2001). Several researchershave found that two groups of external variables have a major im-pact on user beliefs: system characteristics and individual differ-ences (Chau, 1996; Davis, 1993; Hong et al., 2001; Venkatesh,2000).

Although TAM, and its successor TAM2 (Venkatesh & Davis,2000), have evolved into a more complex model known as theUnified Theory of Acceptance and Use of Technology (UTAUT)(Venkatesh, Morris, Davis, & Davis, 2003) there currently exist onlyfew studies confirming UTAUT’s validity and robustness. TAM, onthe other hand, is still tested by many researchers with differentpopulations of users and IT innovations (Sharp, 2007). Besides,(Hong, Thong, & Tam, 2006) concluded that TAM is the most parsi-monious and generic model that can be used to study both initialand continued IT adoption.

2.2. Framework characteristics

We investigated framework characteristics as a special case ofsoftware characteristics. Software characteristics are most oftenexpressed with international software product quality standards,where software quality is defined as ‘‘The totality of characteristicsof an entity that bear on its ability to satisfy stated or impliedneeds” (ISO 9126 2001, p. 20). These quality characteristics consti-tute a product quality model which provides the basis for specify-ing quality requirements and evaluating the quality of softwareproducts. One of the most general and commonly used softwareproduct quality models is ISO/IEC 9126-1 (ISO 9126 2001).

Frameworks are also a special case of reusable software, wherethe most important characteristic is reusability (Froehlich, Hoover,Liu, & Sorenson, 1998; Pree, Pomberger, Schappert, & Sommerlad,1998; van Gurp & Bosch, 2001). The REBOOT (Reuse Based onObject-Oriented Techniques) model defines reusability as acomposite of portability, adaptability, understandability and confi-dence (Sindre, Conradi, & Karlsson, 1995).

We considered the above findings when we searched forcommon framework characteristics within relevant literature(Table 1). The search space encompassed books, journals, confer-ence proceedings, theses and technical reports found in electroniclibraries (like Citeseer, Science Direct and IEEE).

As a result, we have identified four common framework charac-teristics. They were selected because they were frequently men-tioned in the search results (Table 1), either directly, included inan abstract factor or presented as a composite of more specific fac-tors. Three of these characteristics correspond to REBOOT (ReuseBased on Object-Oriented Techniques) reusability factors (Sindreet al., 1995) whereas one characteristic could be found in an ISO/IEC 9126 software product quality model (ISO 9126 2001). Thesecharacteristics are presented below.

2.2.1. AdaptabilityREBOOT defines adaptability (or ‘‘flexibility”) as the ‘‘ease with

which a component can be adapted to fulfill a requirement thatdiffers from that for which it was originally constructed” (Sindreet al., 1995, p. 135). REBOOT defines adaptability and portabilityas two independent factors where ISO/IEC 9126 models adapt-ability as a sub-characteristic of portability (ISO 9126 2001). Toreduce the number of investigated characteristics to a minimum,we decided to combine adaptability and portability into asingle factor.

Adaptability is the most common framework characteristic,because it increases the feasibility of framework instantiations(Bockle et al., 2004; Fayad & Hamu, 2000). On the other hand, sev-eral researchers have posted that adaptability usually comes withincreased complexity, which reduces the ease of using a frame-

Table 1Identified framework characteristics as a results of literature review.

Title Identified framework characteristics

Critical success factors of object-oriented frameworks (van Gurp & Bosch,2001)

Easy to learn, qualitative internal, good support, efficiency, appropriateness

Design, implementation and evolution of object-oriented frameworks:concepts and guidelines (Fontoura, 1999)

Flexibility, reusability, usability

Object-oriented application frameworks: the untold story (Fayad &Schmidt, 1997).

Handling complexity, insufficient support, adequate documentation, support formaintenance

Object-oriented application frameworks (Froehlich et al., 1998) Development effort, learning curve, integratability, maintainability, efficiency, lack ofstandards

Designing object-oriented frameworks (Shai, 1997) Ease of use, extensibility, flexibility, completeness, consistent codingPattern language for framework construction (Guido, Francesco, & Andrea,

1997)Framework user effectiveness, extensibility, simplicity, understandability, frameworkintegration

The evaluation of framework reusability (Sindre et al., 1995; Todd, 1997) Portability, adaptability, understandability, confidenceDevelopment of object-oriented frameworks (Freimut et al., 2002) Sufficient documentation, minimalistic architecture, utilization because of advantages,

easy to understand, easy to use, easy to extend

732 G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740

work (Froehlich et al., 1998; Mattsson, 1996). According to thesefindings, we declared two hypotheses:

H1: A user’s perception of the adaptability of a framework (FA)will have a positive effect on the user’s perception of the useful-ness of the framework (FU).

H2: A user’s perception of the adaptability of a framework (FA)

will have a negative effect on the user’s perception on the easeof using a framework (FEOU).

2.2.2. UnderstandabilityISO/IEC 9126-1 (ISO 9126 2001, p. 9) defines understandability

as ‘‘the capability of a software product to enable the user to under-stand whether the software is suitable and how it can be used forparticular tasks and conditions of use”. Because frameworks are gen-erally recognized as difficult to develop, learn and use, severalresearchers investigated framework understandability or the pro-posed guidelines for improving it (Cechticky, Chevalley, Pasetti, &Schaufelberger, 2003; Mathias, de Oliveira, & de Lucena, 2004;Oliveira, Alencar, Filho, de Lucena, & Cowan, 2004). Understand-ability as a system characteristic is often substituted with ease ofuse, which is a perceptional construct, commonly impacted byunderstandability. Thus, the following can be hypothesized:

H3: A user’s perception of the understandability of a framework(UD) will have a positive effect on the user’s perception on theease of using a framework (FEOU).

2.2.3. ConfidenceREBOOT (Sindre et al., 1995, p. 135) defines confidence as the

‘‘(subjective) probability that a module, program or system performsits defined purpose satisfactory (without failure) over a period oftime in another environment than for which it was originally con-structed.” Framework researchers usually investigate frameworkmaturity and fault tolerance (Fayad & Hamu, 2000; Froehlich etal., 1998; Mattsson & Bosch, 2006), which are according toREBOOT, criteria (sub-characteristics) which together constituteconfidence.

Confidence is important for two reasons. First, while the deci-sion to develop or use a framework is a long-term one, the poten-tial framework user needs to have confidence that the frameworkwill perform satisfactorily over a period of time. Second, whileframeworks constitute parts of end applications (framework in-stances) the potential framework defectiveness usually affects allof its instances. So, we can therefore hypothesize:

H4: A user’s perception of the confidence of a framework (CF)will have a positive effect on the user’s perception of the per-ceived usefulness of a framework (FU).

2.2.4. Efficiency

Efficiency is a product quality characteristic, defined as the

‘‘capability of the software product to provide appropriate perfor-mance, relative to the amount of resources used, under stated condi-tions” (Carva & Franch, 2006; ISO 9126 2001, p. 10). Whileframeworks enhance adaptability by employing additional levelsof indirection, this often reduces efficiency. Some papers have al-ready outlined efficiency as a potential drawback for using frame-works (Mattsson, 1996), especially in comparison with domainspecific languages (Deursen, 1997). Drawing on the above, wecan thus state the following hypothesis:

H5: A user’s perception of the efficiency of a framework (EF)will have a positive effect on the user’s perception of the useful-ness of a framework (FU).

2.3. Individual characteristics

In a comprehensive review of individual reactions to IT innova-tions, Nelson (1990) found that the success of IT innovations de-pends as much on technology as on individuals themselves.

2.3.1. Implementation gapAdopting a framework requires new software development and

maintenance processes, new assignments of job responsibilitiesand, for individual framework users, new skills and new knowl-edge. The wider the gap between ‘‘old” and ‘‘new” technology,the longer the time individuals need to learn the new skills and ac-quire knowledge and thus adapt it to the new work process. Theproduct line cost model includes a special cost factor related tothe above problems: organizational costs (Bockle et al., 2004).We conceptualized the above problem according to Chau (1996)as an ‘‘implementation gap” and hypothesized the following:

H6: A user’s perception of the implementation gap (IG) willhave a negative effect on the user’s perception on the ease ofusing a framework (FEOU).

2.3.2. Task-technology fitTask-Technology Fit (TTF) implies the matching of technological

capabilities with the demands of individuals. TTF posits that IT willbe used if, and only if, the functions available to the user supportthe activities of the user (Goodhue & Thompson, 1995). Severalresearchers have already integrated TTF constructs with TAM’sconstructs and found a significant impact of TTF constructs on per-ceived usefulness (Dishaw & Strong, 1999; Klopping & McKinney,2004). In the context of frameworks, researchers have outlined

G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740 733

the importance of adequacy or fit (Fayad & Hamu, 2000; Meier,1998). So we hypothesize:

H7: A user’s perception of the task-technology fit (TTF) is posi-tively related to the user’s perception of the usefulness of aframework (FU).

3. Research model

We derived the research model from TAM and considered thefollowing criteria: First, we believe that existing and experiencedframework users are more suitable for evaluating frameworkscompared to potential framework users. We therefore based theresearch model on the post-adoption version of TAM (Hong et al.,2006). Secondly, TAM studies have rarely investigated complextechnologies (Hong et al., 2001), whereas we found only onestudy investigating the acceptance of reusable software assets(Mellarkod, Appan, Jones, & Sherif, 2007). However, when consider-ing the effectiveness and simplicity of TAM, as well as its generalapplicability to different types of IT, we felt confident using TAM asthe theoretical basis of our research.

Based on the stated hypotheses, we defined a causal model aspresented in Fig. 3. According to previous TAM studies, we alsohypothesized a causal relationship between FEOU and FU (H8).

We did not investigate the impact of user perception constructson continuous framework use intention (dotted arrows on Fig. 3),because these relationships have already been demonstrated byother researchers (Mellarkod et al., 2007). In this way, we keptthe size of the questionnaire within reasonable boundaries.

The operationalization of the hypothesized research model wasbased mainly on existing sources. If possible, items, which reportedadequate, valid and reliable measures, were reused (see Table 2).

While we measured psychometric framework user responses,we based our operationalization on the Likert scale. The Likertscale is a multi-item scale, where typical items are statements.With the Likert scale, the respondent is asked to indicate their de-gree of agreement with a statement, in our case on a seven pointscale with the endpoints ‘‘Strongly disagree” and ‘‘Strongly agree”(Uebersax, 2006).

4. Research method

The hypothesized research model was validated with data thatwas obtained from a web-based survey. A structured questionnaire

Implementation gap(IG)

Task-technology fit(TTF)

Understandability (UD)

Adaptability (AD)

Confidence (CF)

Efficiency (EF)

Technologicalcharacteristics

Individual characteristics

Framew

H5

H1

H4

H6( - )

H7

H3

H2 (-)

Fig. 3. Hypothesized

was developed and administered to software professionals whohave reported having experience with using frameworks.

4.1. Instrumentation

We developed a survey instrument that contained instructionsasking respondents to identify the framework they most frequentlyuse during application development. Focusing a subject on a spe-cific framework is in line with Churchill’s recommendation ofdefining a unit of analysis for a more precise response and greatervalidity (Churchill, 1979). The development of the research instru-ment was based on the operationalization of the hypothesizedmodel (Table 2).To achieve the desired balance in the question-naire, some statements were worded with proper negations andall items in the questionnaire were randomly sequenced to reduceany potential ceiling effect. Beside Likert items, the instrument in-cluded demographic questions about the respondent’s gender, age,experience with frameworks and software development in general.

A pilot study of the survey instrument was conducted in orderto ensure that the subject could understand the items and mea-surement scales. Twenty-two students who have experience withframeworks participated in the pilot study. Completed question-naires, feedback from the subjects and observations by the authorsresulted in minor changes to the survey instructions, refinement inthe wording of several items and an additional explanation of sometechnical terms.

4.2. Sample

Surveys differ greatly in value, depending on how respondentsare sampled (McBurney & White, 2003) whereas identifying a sam-ple frame is an important step in ensuring that the population ofinterest has been correctly identified. The ideal candidate for ourstudy was an experienced framework user who has already used aspecific framework in several cases. For practical reasons, wesearched for candidates among members of open-source frameworkprojects. Sourceforge.net is the largest repository of open-sourceprojects with database snapshots available to researchers. In thesnapshot version ‘‘December 2007”, 5216 projects were classifiedas framework projects. The sample frame embraced 11,357 uniquemembers of these projects. We used an integer random generatorfor random sampling, where every member from the sample framehad an equal and independent chance of being selected in thesample.

Frameworkease of use

(FEOU)

Frameworkusefulness

(FU)

ork user perceptions

H8

Continuous framework use

intention

research model.

Table 2Question items used in the study.

Constructs and Likert items Source used

CF: Confidenc� CF1: I believe that the framework is mature� CF2: The framework fails frequently� CF3: The framework handles failures well if or when they occur

Sindre et al. (1995)

UD: Understandability� UD1: I believe the framework is self-descriptive Sindre et al. (1995)� UD2: The accessibility, level of detail and quality of framework documentation is good� UD3: The framework is easy to learn ISO 9126 (2001)

AD: Adaptability� AD1: The framework can be easily adapted or extended to fulfill application requirements Sindre et al. (1995)� AD2: The framework can be installed on different environments ISO 9126 (2001)� AD3: The framework can be easily transferred from one environment to another

EF: Efficiency� EF1: The framework requires too much of system resources ISO 9126 (2001)� EF2: The framework provides appropriate response and processing times

IG: Implementation gap� IG1: I felt that there existed a large gap between my existing skills and knowledge and those required by the framework Chau, (1996)� IG2: Learning the framework was not a matter of building on what I knew already� IG3: Major modification in our software development policies and procedures was necessary for the framework to truly fit in

TTF: task-technology fit� TTF1: The framework functions or services suited to application requirements in each individual case of its use New item� TTF1: The framework provides suitable set of functions for my tasks and user objectives in each individual case of its use ISO 9126 (2001)

FEOU: framework perceived ease of use� FEOU1: The framework is rigid and inflexible to interact with Moore & Benbasat (1991)� FEOU2: I find it is easy to get the framework to do what I want it to do� FEOU3: Overall, I believe that the framework is easy to use� FEOU4: Learning to operate the framework is easy for me� FEOU5: I find it takes a lot of effort to become skillful at using the framework

FU: framework perceived usefulness� FU1: I believe that using the framework will further increase my productivity� FU2: I believe that using the framework will further increase my job performance� FU3: I believe that using the framework will further enhance my job effectiveness� FU4: Overall, I believe the framework will be further useful in my job

Moore and Benbasat (1991)

734 G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740

The sample, which resulted from a systematic random samplingprocess, embraced 4000 different framework project members to-gether with their e-mail addresses. We sent personal invitations(e-mails) to participate in an online survey to the members ofthe sample. Sample members who did not respond within 48 hwere additionally reminded to participate in the survey. Of the4000 surveys mailed: 934 mails were opened, 710 surveys werestarted and 447 surveys completed. This puts the response rateat 11.2%. Fifty-six surveys had incomplete responses or did notpass controlled questions and were therefore not used. As a result,391 completed surveys were analyzed, meaning a response rate of9.8%.

5. Data analysis and results

We used descriptive statistics to analyze the properties of thesample. The means of different sub-samples were compared withT-tests for independent samples. In both cases, SPSS softwarewas used.

We used structural equation modeling (SEM) for testing the fitof the proposed theoretical model (Fig. 3) with empirical data. SEMwas applicable because it encourages confirmatory rather thanexploratory modeling. Thus, it is suited to theory testing ratherthan theory development.

SEM consists of two main parts (Anderson & Gerbing, 1988): themeasurement model showing the relations between the latentvariables and their indicators, and the structural model showingpotential causal dependencies between endogenous and exoge-nous variables. The measurement model was estimated using aconfirmatory factor analysis to test whether the proposed con-

structs possess sufficient validation and reliability. To ensure datavalidity and reliability: internal consistency, convergent validity,and discriminate validity were demonstrated. The structural modelwas estimated with path analysis.

We used AMOS as the SEM software for data analysis. AMOS is acovariance-based approach similar to LISREL, in which the covari-ance structure obtained from the observed data is used to simulta-neously fit measurement and structural equations specified in themodel. AMOS estimated both the measurement and structuralmodels using the full information maximum likelihood estimator.

5.1. Descriptive statistics

Table 3 shows the descriptive statistics of survey respondentsand their experiences with the framework that they evaluated inthe survey. The typical respondent of the survey was a 31.7-years-old male with a bachelor’s degree and 8–13 years of experiences insoftware development. On average, respondents reported 3–5 yearsof experience in using 3–5 different frameworks.

Only 33% of the respondents decided to evaluate the frameworkfrom the corresponding Sourceforge.net framework project, where67% of the respondents decided to evaluate another open source orproprietary framework. In most cases, respondents decided toevaluate a framework that they used on a voluntary and daily ba-sis. On average, they already instantiated this framework 3–5times.

5.2. Measurement model results

A measurement model is a confirmatory factor analysis modelwith unmeasured covariance between each possible pair of latent

Table 3Descriptive statistics of respondents.

Variable Values Freq. Valid percent(%)

Variable Values Freq. Valid percent(%)

Gender Female 9 2.3 Type of use Voluntary 302 78.0Male 378 97.7 Mandatory 85 22.0

Education High sch. 53 13.7 Number of completed projects with theevaluated framework

0 27 6.9Bachelor 153 39.5 1 53 13.6Master 125 32.3 2 71 18.3Doctorate 30 7.8 3–5 118 30.3Other 26 6.7 5–8 63 16.2

Software development experience(in years)

0–1 1 0.3 8–13 26 6.71–2 12 3.1 13–21 8 2.13–5 69 17.6 >21 23 5.95–8 94 24.0 Duration of framework use (in years) <1 66 16.98–13 102 26.1 1–2 102 26.213–21 69 17.6 2–3 100 25.6>21 44 11.3 3–5 81 20.8

Number of frameworks exploredso far

1 13 3.4 5–8 30 7.72 24 6.2 >8 11 2.83–5 118 30.5 Frequency of framework use Daily 198 51.05–8 83 21.4 Weekly 120 30.98–13 55 14.5 Monthly 31 8.0>13 years 93 24.0 Less than

monthly39 10.1

Experience in using frameworks(in years)

<1 12 3.1 Respondent type Early 220 56.31–2 31 7.9 Late 110 28.12–3 60 15.4 Not identified* 61 15.63–5 102 26.25–8 91 23.3 Respondent’s age (in years) Average 31.728–13 57 14.6 Std. dev 8.06>13 37 9.5

* Due to the technological restrictions of the online survey application.

G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740 735

variables. A measurement model consists of (1) straight arrows be-tween latent variables and their respective indicators, (2) straightarrows between the error and disturbance terms of their respectivevariables, and (3) no direct effects connecting the latent variables.According to this, we evaluated the measurement model usinggoodness-of-fit measures.

We included the following fit indices: the goodness-of-fit index(GFI), the adjusted goodness-of-fit index (AGFI), the comparativefit index (CFI), the normed fit index (NFI), the Tucker–Lewis coeffi-cient (TLI), the root mean squared residual (RMR), and the rootmean square error of approximation (RMSEA). Threshold valueswere taken from (Hong et al., 2006). We used same-fit indices foranalyzing the fit of the structural models.

Table 4 shows model fit indices for initial and revised mea-surement models, where the values that have exceeded theirrecommended criteria are bolded. Three of seven fit indicesexceeded their limits in the initial model. To improve the overallfit of the measurement model, we performed an iterative process,where only one item was changed per iteration. The modifica-tions were based on factor loadings and the modification indices,in cases where those modifications were theoretically justified.

Table 4Summary of fit indices (measurement models).

Rec. criteria Initial Revised

GFI P0.90 0.88 0.91AGFI P0.80 0.84 0.87NFI P0.90 0.88 0.92TLI (NNFI) P0.90 0.91 0.94CFI P0.90 0.92 0.95RMSR 60.10 0.11 0.10RMSEA 60.08 0.06 0.56X2 (df, p) – 737.53 (296, p < 0.001) 447.68 (202, p < 0.001)X2/df 63 2.48 2.21

Because we defined multiple items per construct, improper itemscould be excluded from the measurement model without causingany serious harm to the model. All fit indices of the revisedmeasurement model fell within the recommended criteria(Table 4).

We excluded one TTF item, one AD item and two FEOU itemsfrom the revised measurement model. In this way, we got a bettermeasurement model fit with the empirical data. Analyzing themeaning of the two remaining AD items (AD2 and AD3), wedecided to rename the adaptability construct to portability.According to this, we substituted the term ‘‘adaptability” withthe term ‘‘portability” in hypotheses H1 and H2.

For the resulting measurement items (Table 5) we also per-formed a T-test for independent groups, to compare if the itemmeans of early and late respondents differed significantly. Wefound no significant difference between the answers of early andlate respondents.

In the resulting set of measurement items, we examined inter-nal consistency, convergent validity and discriminant validity.Convergent validity was evaluated with Cronbach’s alpha, whichis a commonly used measure testing the extent to which multipleindicators for a latent variable belong together. Cronbach’s alpharanged from 0.74 to 0.95, meaning that all constructs exceed thethreshold of 0.70 for field research. We got similar results for Com-posite reliability (Cr), which was evaluated as (Bacon, Sauer, &Young, 1995):

Cr ¼P

kið Þ2P

kið Þ2 þP

ivarðeiÞð1Þ

where kiwas identified as the factor loading associated with item xi.var(ei) was identified as an error variance associated with item xi

and calculated as varðeiÞ ¼ 1� k2i . Discriminant validity was evalu-

ated with average variance extracted (AVE), which was defined as(Fornell & Larcker, 1981):

Table 5Descriptive statistics of remaining items.

N Mean Std.deviation

Factorloadings

Squared multiplecorrelations

AD2 391 2.00 1.39 0.89 0.82AD3 391 2.27 1.45 0.96 0.87CF1 391 2.45 1.45 0.70 0.48CF2 391 2.20 1.17 0.73 0.56CF3 391 2.59 1.27 0.70 0.48EF1 391 2.79 1.54 0.88 0.78EF2 391 2.27 1.09 0.65 0.42EF3 391 2.55 1.50 0.82 0.67IG1 391 5.22 1.58 0.77 0.61IG2 391 4.76 1.59 0.68 0.45IG3 391 5.01 1.65 0.69 0.46FEOU2 391 2.35 1.09 0.88 0.78FEOU3 391 2.46 1.21 0.82 0.67FEOU5 391 2.72 1.30 0.78 0.61FU1 391 1.87 1.07 0.89 0.80FU2 391 2.02 1.20 0.93 0.87FU3 391 1.99 1.18 0.95 0.90FU4 391 1.83 1.06 0.89 0.78TTF1 391 2.06 1.32 0.83 0.31TTF2 391 2.16 1.08 0.81 0.48UD1 391 3.07 1.48 0.78 0.61UD2 391 3.14 1.62 0.56 0.30UD3 391 2.79 1.38 0.77 0.61

Table 7Discriminant validity.

EF AD IG TTF UD CF FEOU FU

EF 0.62AD 0.12 0.85IG 0.08 0.09 0.51TTF 0.08 0.14 0.04 0.80UD 0.17 0.05 0.06 0.12 0.75CF 0.33 0.09 0.04 0.04 0.22 0.50FEOU 0.20 0.24 0.15 0.25 0.47 0.18 0.68FU 0.27 0.14 0.12 0.05 0.12 0.22 0.33 0.95

Note: Diagonals represent average variance extracted. Other entries represent theshared variance.

Table 8Overall fit and explanatory power of structural models.

Fit index Recommendedcriteria

Hypothesizedstructural model

Revised structuralmodel

GFI P0.90 0.90 0.91AGFI P0.80 0.87 0.88NFI P0.90 0.91 0.92TLI

(NNFI)P0.90 0.93 0.94

CFI P0.90 0.95 0.95RMSR 60.10 0.11 0.09RMSEA 60.08 0.06 0.06X2 (df, p) – 489.08 (207,

p < 0.001)377.45 (157,p < 0.001)

X2/df 63 2.41 2.15

R2FEOU

– 0.61 0.60

R2FU

– 0.42 0.41

R2CF

– – 0.41

736 G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740

AVE ¼P

k2iP

k2i þ

PivarðeiÞ

ð2Þ

The average variance extracted value for revised measurementitems lies between 0.5 and 0.85, reaching or exceeding the mini-mum value of 0.5 as proposed by Fornell and Larcker (1981), andalso indicating good internal consistency, as seen in Table 6.

Besides, Fornell and Larcker (1981) suggested that discriminantvalidity criteria are determined by the AVE value, namely whetherit exceeds the squared correlation between the constructs. Thefindings revealed good discriminant validity for all constructs(Table 7).

5.3. Structural model results

Using the revised measurement model, the structural modelwas analyzed. Two structural models were investigated: (1) thestructural model as hypothesized in Fig. 3; and (2) the revisedstructural model, based on the results of the hypothesized modeland modification indices. For each structural model, we examinedits goodness-of-fit using the same-fit indices as for the measure-ment model. The goodness-of-fit results for the hypothesized andrevised measurement model are presented in Table 8, in whichthe values outside the recommended thresholds have been bolded.In addition, we compared the results of mandatory and voluntaryframework users. The hypothesized structural model indicates thatthe RMSR fit index is outside the recommended values.

Table 6Constructs reliability.

Construct Numberof items

Mean Standarddeviation

Averagevarianceextracted

Compositereliability

Cronbach’salpha

AD 2 2.14 1.42 0.85 0.92 0.92CF 3 2.41 1.30 0.50 0.75 0.75EF 3 2.54 1.38 0.62 0.83 0.82IG 3 5.00 1.61 0.51 0.75 0.75PFEOU 3 2.20 1.19 0.68 0.87 0.86FU 4 1.93 1.13 0.84 0.95 0.95TTF 2 2.11 1.11 0.67 0.80 0.80UD 3 3.00 1.49 0.51 0.75 0.74

5.3.1. Hypothesized structural modelFig. 4 shows the hypothesized model, factor loadings (standard-

ized regression weights), significance of the relationships (themeaning of asterisks is explained in Table 9) and explained vari-ance in the dependent variables (R2).

All the hypothesized relationships, except EF ? FU were foundto be significant. The factor loadings (b coefficients) of significantrelationships ranged from 0.10 (AD ? FU) to 0.55 (UD ? FEOU).A deviation from the hypotheses was found for H2, where wehypothesized a negative impact but found a positive impact. Ashypothesized, we found a negative impact between IG and FEOU.In addition, our results confirmed the already demonstrated rela-tionship between FEOU and FU, with results accordant to TAMmeta-analysis (King & He, 2006). The amount of explained variancefor FEOU was 0.61 meaning that we embraced most of the FEOUantecedents. We found that the hypothesized model is more appli-cable to voluntary users. In the case of mandatory use, only tworelationships were found to be significant (Table 9).

5.3.2. Revised structural modelAfter considering the results and modification indices of the

hypothesized structural model, we created a revised structuralmodel (Fig. 5). The revised model indicated a better fit with thedata with all fit indexes within recommended values. The majormodification was done to the CF, which was defined as a depen-dent construct. TAM was already extended with similar constructwithin the context of Television–Commerce, where researchersdemonstrated a significant relationship between trust and userintentions (Yu, Ha, Choi, & Rho, 2005). EF was eliminated fromthe revised model because it did not demonstrate to have a signif-icant impact on FU.

Frameworkease of use

(FEOU)R2=0.61

Frameworkusefulness

(FU) R2=0.42

Implementation gap (IG)

Task-technology fit (TTF)

Understandability (UD)

Old: Adaptability (AD) New: Portability (PO)

Confidence (CF)

Efficiency (EF)

Technological characteristics

Individual characteristics

Framework user perceptions

Continuous framework use

intention

0.15*

0.08NS

0.10*

0.20***0.55***

- 0.29***

0.24**

0.32

***

Fig. 4. Results of the hypothesized model.

Table 9Moderating effect of voluntary and mandatory use in the hypothesized model.

Voluntary users Mandatory users All users

FEOU ? FU 0.27*** 0.44*** 0.32***

CF ? FU 0.26** �0.09NS 0.15*

EF ? FU 0.06NS 0.03NS 0.08NS

AD ? FU 0.09NS 0.09NS 0.10*

TTF ? FU 0.19* 0.37NS 0.24**

AD ? FEOU (�) 0.25*** 0.09NS 0.20***

UD ? FEOU 0.47*** 0.77*** 0.55***

IG ? FEOU (�) �0.31*** �0.17NS �0.29***

R2FU

0.39 0.47 0.42

R2FEOU

0.53 0.78 0.61

* p < 0.05.** p < 0.01.

*** p < 0.001.NS p P 0.05 (not significant).

G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740 737

The amount of explained variance was equal for CF and FU(0.41) where the highest variance explained FEOU (0.60). For vol-untary users and combined users all the relationships in the

Implementation gap (IG)

Task-technology fit(TTF)

Understandability (UD)

Portability (PO)

0.46***

0.37***

0.30***

0.55***

0.21***

-0.27***

Fig. 5. Results of the revis

revised structural model were found to be significant at p < 0.001(Table 10).

The results of the revised structural model (Fig. 5) show variousmeanings for framework-related stakeholders. It seems that frame-work users are aware of the fact that framework investments areonly rewarding in the long-term. So, they decide about adoptingor abandoning a specific framework, evaluating three acceptancedeterminants: FEOU, FU and CF. We demonstrated that confidence,which extends to the TAM model, increases with frameworkunderstandability and suitability (TTF) for a planned product fam-ily. As defined in the REBOOT model, confidence can be determinedwith maturity and stability of the framework.

FEOU was demonstrated to be the most prevalent acceptancedeterminant. This is consistent with common users’ opinions thatframeworks are mostly complex reusable software and thereforedifficult to use. According to the model, we believe that lightweightframeworks have better chances for user acceptance. We demon-strated that FEOU is determined by framework characteristics(UD and PO) and the implementation gap. As defined in theREBOOT model, portability can be improved with environmentalindependence and replaceability of the framework. We presume

Frameworkease of use

(FEOU)R2=0.60

Frameworkusefulness

(FU)R2=0.41

Continuous framework use

intention0.41

***

Confidence(CF)

R2=0.41

ed structural model.

Table 10Moderating effect of voluntary and mandatory use in TAM–CFU submodel.

Voluntary users Mandatory users All users

FEOU ? FU 0.39*** 0.47*** 0.41***

TTF ? FU 0.36*** 0.34** 0.37***

IG ? FEOU �0.29*** �0.14NS �0.27***

UD ? FEOU 0.48*** 0.77*** 0.55***

AD ? FEOU 0.25*** 0.12NS 0.21***

UD ? CF 0.32*** 0.25NS 0.30***

TTF ? CF 0.46*** 0.51*** 0.46***

R2CF

0.41 0.45 0.41

R2FU

0.38 0.47 0.41

R2FEOU

0.52 0.77 0.60

* p < 0.05.** p < 0.01.

*** p < 0.001.NS p P 0.05 (not significant).

IIToys tools

IRejected

tools

IIIPower user

tools

IVSuper tools

UsefulnessLow High

Low

High

Ease of use

Fig. 6. Usefulness/ease-of-use grid (Keil et al., 1995).

738 G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740

that IG can be reduced when developing frameworks using a com-mon and standardized technology.

The most important framework characteristic is understand-ability (UD). We demonstrated that UD impacts all three accep-tance determinants, whereas FU is impacted indirectly, throughFEOU. So, framework developers should focus their work onimproving framework understandability. Considering the REBOOTand ISO/IEC 9126 model, this can be done by managing frameworkcomplexity and improving the learn ability, self-descriptivenessand documentation quality of the framework.

6. Discussion and conclusions

Our study investigated the impact of major framework-relatedtechnological and user characteristics on a framework’s perceivedease of use and its perceived usefulness. We based our study ona well-proven theory: the technology acceptance model (TAM).In TAM-related studies, researchers have already demonstratedthe significance of relationships between user perceptions, theirintentions and behavior for different software types (Legris,Ingham, & Collerette, 2003). According to TAM related categoriesof modifications (King & He, 2006), the focus of our research hasbeen on identifying major external variables that impact users’perceptions within the context of frameworks. According toperformed literature review we did not find any similar researchin the investigated context.

To find answers to the stated hypotheses, we performed an on-line survey with a sample of 391 experienced framework users.The data, obtained from questionnaires, supports six of the eightstated hypotheses, whereas an additional hypothesis was foundsignificant but with the opposite impact as initially hypothesized.The TAM relationship between FEOU and FU was found to be con-sistent with previous studies (King & He, 2006).

We can summarize the results as follows: First, we demon-strated that perceived ease of use and perceived usefulness are va-lid within the context of frameworks. Second, both technologicalcharacteristics, task-technology-fit and individual differences areimportant determinants for the perceived ease of use and per-ceived usefulness of frameworks. Third, beside FEOU in FU, confi-

Identify framework type, corresponding to

FU-FEOU-CF grid

According to thdefine a strateg

moving the framto a more favou

position

Assess FU, FEOU and CF of the

framework

Fig. 7. Proposed framework

dence seems to be an important determinant for continuedframework use intention. Fourth, we confirmed that TAM is moreapplicable to voluntary users. With these theoretical results, weextended the existing knowledge in the field of framework’sresearch, related to the impacts of frameworks use (Bockle et al.,2004; Mamrak & Sinha, 1999; Moser & Nierstrasz, 1996).

6.1. Research limits

Readers should interpret our results considering the followinglimitations. First, there is no empirical evidence for the validityand completeness of the investigated technological and individualcharacteristics. We are aware that other search engines and otherelectronic libraries could lead to the uncovering of different frame-work characteristics.

Second, the sample frame used in our survey was taken frommembers of Sourceforge.net framework projects. Despite the factthat survey participants also evaluated proprietary frameworks,there is no evidence that the sample frame is a typical representa-tive of the population of interest – i.e. all frameworks users. Inaddition, respondents could be predisposed to be favorable aboutthe framework they have evaluated in the survey. So, future stud-ies need to be performed to investigate and compare different sam-ple frames.

The SEM approach was the most reasonable choice for our re-search. SEM offers great potential for theory development and con-struct validation. However, the readers of our study should beaware that the SEM approach has its own limitations.

We are also aware that we have only demonstrated that ourmodels have valid constructs, causal relationships and are a goodfit with the empirical data. However, since other unexamined mod-els may fit the data as well or even better, we are aware that an ac-cepted model is only a not-yet-disconfirmed model. In accordancewith these limitations, other framework-related external variables,which were not considered by us, might have a significant impacton user perceptions.

6.2. Practical implications

There are several practical implications of our research. First, tocontinuously improve frameworks, framework developers shouldpermanently survey (i.e. with a quick poll placed on the project’s

Identify guidelines and design patterns which support the defined

improvement strategy

e grid , y for ework rable

Apply identified guidelines and

design patterns to the framework

improvement process.

G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740 739

web page) framework users according to dependent factors of therevised structural model (CF, FU and FEOU), where the concretesurvey questions could be obtained from the research instrument.According to the TAM and results of our research, these factorshave a significant impact on framework acceptance. So, reactionsbased on such survey could increase both – the number of frame-work’s newcomers and long-term framework users.

Second, several researchers and practitioners already proposedcriteria and guidelines for framework selection and evaluation(Fayad & Hamu, 2000; Meier, 1998). When selecting a framework,potential framework users should also consider framework charac-teristics (UD and PO) and individual characteristics (TTF and IG),for which we demonstrated an impact on user believes.

Third, many framework researchers proposed guidelines forimproving frameworks as those proposed by Landin and Niklasson(1995), where effects of those guidelines on framework users re-main unknown. These effects could be analyzed comparing inves-tigated user perceptions (CF, FU and FEOU) before and after aguideline has been applied to a specific framework (i.e. performinga longitudinal experimental research). Observing those user per-ceptions as a function of time could indicate the impact of a spe-cific guideline on framework users (i.e. ‘‘after the guideline hasbeen applied to the framework, user perceptions of the frameworkare improving”).

6.3. Future work

We are continuing our work in several directions. Currently, weare performing research and development which will increase thepractical value of this research in the following ways: We areinvestigating the degree of difference in common elements, whichconstitute frameworks (for example: framework-related artifacts,design patterns and guidelines for framework improvements),influencing the constructs of investigated structural models. Forexample, we are investigating if different framework types, as de-fined by Robert and Johnson (1997), have an impact on how usersperceive frameworks. Besides this, Landin (1995) have defined aset of 71 framework-related guidelines and we plan to investigateif applications of those guidelines impacts the constructs. In thisway, we believe that it is possible to assess the effectiveness andappropriateness of different framework development approachesand elements which constitute frameworks.

Secondly, we are developing a framework improvement pro-cess, which is based on Keil’s et al. (1995) ‘‘usefulness/ease-of-use grid” (Fig. 6). The ‘‘usefulness/ease-of-use grid” provides a sim-ple way of representing the different combinations of PU and PEOUthat are possible. From the practitioner’s point of view, the gridprovides a means of mapping out the changes to move a systemto a more favorable position (see arrows on Fig. 6).

According to the results of the revised structural model (Fig. 5)we propose to extend the grid with an additional dimension (CF)and define typical frameworks, which correspond to different com-binations of FU, FEOU and CF. Combining a FEOU–FU–CF grid withpreviously presented framework assessments approach, we candefine a common framework improvement process as presentedbelow (Fig. 7).

The main idea behind the proposed framework improvementprocess lies in a user-oriented, iterative and evolutionary process,which improves frameworks stepwise, on the level of their commonconstituent elements. The first step in the process is the assessmentof FU, FEOU and CF (see Table 2), which can be performed by survey-ing framework users. According to the assessment results and theFU–FEOU–CF grid, the framework type can be identified. Afterwards,a strategy for moving the framework from a less favorable to a morefavorable position on the grid is defined. Finally, the framework isimproved with the implementation of guidelines and design

patterns, which support the strategy. The process complies well tothe evolutionary framework development model as proposed byRobert and Johnson (1997). In this way, we believe that the proposedframework improvement process could be incorporated into thedevelopment of open-source as well as proprietary frameworks.

Third, according to the revised structural model (Fig. 5), we planto perform additional empirical research, which will analyze theimpacts of FU, FEOU and CF on (continuous) user’s intentions.We are also planning to include new constructs into the revisedstructural model. For example, while adaptability has been unex-pectedly excluded from this research, we plan to investigate porta-bility and adaptability as two independent constructs – as definedin the REBOOT model.

Acknowledgments

The authors express their appreciation to the University of No-tre Dame for enabling access to the Sourceforge.net research data.The authors would also like to thank the survey participants fortheir answers and critical feedbacks.

References

Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice –A review and recommended 2-step approach. Psychological Bulletin, 103(3),411–423.

Bacon, D. R., Sauer, P. L., & Young, M. (1995). Composite reliability in structuralequations modeling. Educational and Psychological Measurement, 55(3),394–406.

Batory, D., Cardone, R., Smaragdakis, Y. (2000). Object-oriented frameworks andproduct lines. In: P. Donohoe (Ed.), First software product line conference (pp.227–247).

Bockle, G., Clements, P., McGregor, J. D., Muthig, D., & Schmid, K. (2004). CalculatingROI for software product lines. IEEE Software, 21(3), 23–31.

Boehm, B. (1999). Managing software productivity and reuse. Computer, 32(9),111–113.

Bosch, J., Peter, M., Mattsson, M., & Bengtsson, P. (2000). Object-orientedframework-based software development: Problems and experiences. ACMComputing Surveys, 32.

Carva, J. P., Franch, X. (2006). Extending the ISO/IEC 9126-1 quality model with non-technical factors for COTS components selection. In: International workshop onSoftware quality (pp. 9–14).

Cechticky, V., Chevalley, P., Pasetti, A., Schaufelberger, W. (2003). A generativeapproach to framework instantiation. In: Generative programming andcomponent engineering 2003, proceedings Vol. 2830 (pp. 267–286).

Chau, P. Y. K. (1996). An empirical investigation on factors affecting the acceptanceof CASE by systems developers. Information and Management, 30(6), 269–280.

Churchill, G. A. (1979). Paradigm for developing better measures of marketingconstructs. Journal of Marketing Research, 16(1), 64–73.

Cunningham, H. C., Liu, Y., & Zhang, C. H. (2006). Using classic problems to teachJava framework design. Science of Computer Programming, 59(1–2), 147–169.

Davis, F. D. (1993). User acceptance of information technology – Systemcharacteristics, user perceptions and behavioral impacts. International Journalof Man–Machine Studies, 38(3), 475–487.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptanceof information technology. MIS Quarterly, 13(3), 318–331.

Deursen, A. V. (1997). Domain – Specific languages versus object-orientedframeworks: A financial engineering case study. In: STJA’ 97 (pp. 35–39).Ilmenau Technical University.

Dishaw, M. T., & Strong, D. M. (1999). Extending the technology acceptance modelwith task-technology fit constructs. Information and Management, 36(1), 9–21.

Fayad, M., & Hamu, D. (2000). Enterprise frameworks: Guidelines for selection. ACMComputing Survey, 32(1), 4.

Fayad, M. E., & Schmidt, D. C. (1997). Object-oriented application frameworks.Communications of the Acm, 40(10), 32–38.

Fontoura, M. F. (1999). Object-oriented application frameworks: The untold story.In: Object-oriented programming, systems, languages, and applications(OOPSLA’99).

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models withunobservable variables and measurement error. Journal of Marketing Research,18(1), 39–50.

Frakes, W. B., & Kang, K. (2005). Software reuses research: Status and future. IEEETransactions on Software Engineering, 31(7), 529–536.

Freimut, B., Punter, T., Biffl, S., Ciolkowski, M. (2002). State-of-the-art in empiricalstudies, Virtuelles Software Engineering Kompetenzzentrum.

Froehlich, G., Hoover, J., Liu, L., Sorenson, P. (1998). Designing object-orientedframeworks. In: CRC handbook of object technology. CRC Press.

Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual-performance. MIS Quarterly, 19(2), 213–236.

740 G. Polancic et al. / Computers in Human Behavior 27 (2011) 730–740

Guido, C., Francesco, B., & Andrea, V. (1997). The evaluation of frameworkreusability. SIGAPP Application on Computer Review, 5(2), 21–27.

Hong, S. J., Thong, J. Y. L., & Tam, K. Y. (2006). Understanding continued informationtechnology usage behavior: A comparison of three models in the context ofmobile internet. Decision Support Systems, 42(3), 1819–1834.

Hong, W. Y., Thong, J. Y. L., Wong, W. M., & Tam, K. Y. (2001). Determinants of useracceptance of digital libraries: An empirical examination of individualdifferences and system characteristics. Journal of Management InformationSystems, 18(3), 97–124.

ISO 9126 (2001). ISO/IEC TR 9126-software engineering, product quality, quality model.International Organization for Standardization.

Keil, M., Beranek, P. M., & Konsynski, B. R. (1995). Usefulness and ease of use – Field-study evidence regarding task considerations. Decision Support Systems, 13(1),75–91.

King, W. R., & He, J. (2006). A meta-analysis of the technology acceptance model.Information and Management, 43(6), 740–755.

Klopping, I. M., & McKinney, E. (2004). Extending the technology acceptance modeland the task-technology fit model to consumer e-commerce. InformationTechnology, Learning and Performance Journal, 22(1).

Landin, N., Niklasson, A. (1995). Development of object-oriented frameworks. LundInstitute of Technology, Lund University, Department of CommunicationSystems.

Legris, P., Ingham, J., & Collerette, P. (2003). Why do people use informationtechnology? A critical review of the technology acceptance model. Informationand Management, 40(3), 191–204.

Mamrak, S. A., & Sinha, S. (1999). A case study: Productivity and quality gains usingan object-oriented framework. Software – Practice and Experience, 29(6),501–518.

Manolescu, D., Noble, J., Voelter, M. (2006). Patterns for successful object-orientedframework development. In: Pattern Languages of Program Design 5 (1st ed.) (pp.401–431). Addison Wesley Professional.

Mathias, I., de Oliveira, T. C., & de Lucena, C. J. P. (2004). A framework instantiationapproach based on the features model. Journal of Systems and Software, 73(2),333–349.

Mattsson, M. (1996). Object-oriented frameworks, a survey of methodological issues.Lund University, Department of Computer Science.

Mattsson, M., Bosch, J. (2006). Characterizing stability in evolving frameworks. In:Proceedings of TOOLS Europe (Vol. 99), (pp. 118–130).

McBurney, D. H., White, T. L. (2003). Research methods (6th ed.). WadsworthPublishing.

Meier, P. (1998). Critical success factors of object oriented frameworks. In: STJA 98.Mellarkod, V., Appan, R., Jones, D. R., & Sherif, K. (2007). A multi-level analysis of

factors affecting software developers’ intention to reuse software assets: Anempirical investigation. Information and Management, 44(7), 613–625.

Menzies, T. (2003). More success and failure factors in software reuse.

Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure theperceptions of adopting an information technology innovation. InformationSystems research, 2(3), 192–222.

Morisio, M., Ezran, M., & Tully, C. (2002). Success and failure factors in softwarereuse. IEEE Transactions on Software Engineering, 28(4), 340–357.

Moser, S., & Nierstrasz, O. (1996). The effect of object-oriented frameworks ondeveloper productivity. Computer, 29(9), 45.

Nelson, D. L. (1990). Individual adjustment to information-driven technologies – Acritical-review. MIS Quarterly, 14(1), 79–98.

Oliveira, T. C., Alencar, P. S. C., Filho, I. M., de Lucena, C. J. P., & Cowan, D. D. (2004).Software process representation and analysis for framework instantiation. IEEETransactions on Software Engineering, 30(3), 145–159.

Pree, W., Pomberger, G., Schappert, A., & Sommerlad, P. (1998). Active guidance offramework development. Software – Concepts and Tools, 3(16).

Roberts, D., Johnson, R. (1997). Patterns for evolving frameworks. in Patternlanguages of program design (Vol. 3), (pp. 471–486). Addison-Wesley LongmanPublishing Co. Inc.

Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions ofinformation systems success. Communications of AIS, 2(3).

Shai, B. -Y. (1997). Pattern language for framework construction. In: The 4th patternlanguage of programming conference.

Sharp, J. H. (2007). Development, extension, and application: A review of thetechnology acceptance model. Information Systems Education Journal, 5(9).

Sindre, G., Conradi, R., & Karlsson, E. A. (1995). The reboot approach to softwarereuse. Journal of Systems and Software, 30(3), 201–212.

Srinivasan, S. (1999). Design patterns in object-oriented frameworks. Computer,32(2), 24-+.

Todd, H. (1997). Development of successful object-oriented frameworks. ACM,115–119.

Uebersax, J. S. (2006). Likert scales: Dispelling the confusion. 8-7-2008. Ref type:electronic citation.

van Gurp, J., & Bosch, J. (2001). Design, implementation and evolution of objectoriented frameworks: Concepts and guidelines. Software – Practice andExperience, 31(3), 277–300.

Venkatesh, V. (2000). Determinants of perceived ease of use: Integrating control,intrinsic motivation, and emotion into the technology acceptance model.Information Systems Research, 11(4), 342–365.

Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technologyacceptance model: Four longitudinal field studies. Management Science, 46(2),186–204.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). Useracceptance of information technology: Toward a unified view. MISQuarterly, 27(3), 425–478.

Yu, J., Ha, I., Choi, M., & Rho, J. (2005). Extending the TAM for a t-commerce.Information and Management, 42(7), 965–976.