predicting programmers' personality via interaction ... · 80 such as mmpi-2 (butcher, 1989),...
Post on 15-Dec-2020
7 Views
Preview:
TRANSCRIPT
Predicting programmers' personality via interaction behaviourwith keyboard and mouseIftikhar A Khan, Osman Khalid, Waqas Jadoon, Rafi Us. Shan, Abdul Nasir
This exploratory research was conducted to study the relationship between Big Fivepersonality measurement scale and the interaction behaviour of the programmers withkeyboard and mouse to examine the possibility of creating a computer based objectivepersonality measurement scale. A field study with two analyses (N = 20), (N = 12) and alab study (N = 15) were conducted where N is the number of participants who participatedin the research. In the field study, interaction data were collected during normal PC useover several days. In the laboratory study, participants worked on a programming taskwhile their interaction with keyboard and mouse was being recorded in background. All ofthe participants rated their personality online and uploaded the data for analysis. Resultsfound inconsistent behaviour of all of the personality traits except ‘activity level’ in allstudies and thus suggested that a programmer’s ‘activity level’ can be predicted fromhis/her interaction behaviour with keyboard and mouse. This prediction will help indifferentiating good programmers from not so good programmers objectively.
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
2 Predicting Programmers' Personality via Interaction Behaviour with Keyboard & Mouse
3 Iftikhar Ahmed Khan, Osman Khalid, Waqas Jadoon, Rafi Us Shan, Abdul Nasir Khan
4 (iftikharahmed, osman, waqas_Jadoon, shan, anasir)@ciit.net.pk
5 COMSATS Institute of Information Technology, TOBE Camp Abbottabad, Pakistan
6 ABSTRACT
7 This exploratory research was conducted to study the relationship between Big Five personality
8 measurement scale and the interaction behaviour of the programmers with keyboard and mouse to
9 examine the possibility of creating a computer based objective personality measurement scale. A
10 field study with two analyses (N = 20), (N = 12) and a lab study (N = 15) were conducted where
11 N is the number of participants who participated in the research. In the field study, interaction data
12 were collected during normal PC use over several days. In the laboratory study, participants
13 worked on a programming task while their interaction with keyboard and mouse was being
14 recorded in background. All of the participants rated their personality online and uploaded the data
15 for analysis. Results found inconsistent behaviour of all of the personality traits except ‘activity
16 level’ in all studies and thus suggested that a programmer’s ‘activity level’ can be predicted from
17 his/her interaction behaviour with keyboard and mouse. This prediction might help in
18 differentiating good programmers from not so good programmers objectively.
19
20 Keywords: Programmers, Personality, Keyboard, Mouse, Interaction, Behaviour, Estimation,
21 Measurement
22 1. INTRODUCTION
23 The study of personality is one of the most active areas in the field of psychology (Burger, 2010).
24 One of the most agreed definition of personality is: “consistent behaviour patterns and
25 intrapersonal processes originating within the individual” (Burger, 2010). From definition it can
26 inferred that people have different personalities like some people make friends easily (extroverts)
27 others prefer to be alone (introverts). Similarly, some people are more business oriented while
28 others are service oriented. Therefore, studying personality is significant to identify the differences
29 among people behaviours. This implies that studying various human characteristics can help other
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
30 humans to decide how to deal with a particular personality characters (Burger, 2010). Moreover,
31 these characteristics can help in marketing products (Nunes et al., 2012), finding effects of
32 personality on peoples’ own economy (Phau et al., 2009) and finding the relation between
33 personality and health (McCool, 2008) etc. In order to improve the human-human or human-
34 machine interaction as well as to enhance the productivity, the knowledge about various
35 personality features has gained much significance.
36 Various personality measurement instruments like (Butcher, 1989), (McCrae and John, 1992),
37 (Cattell, 1989) etc. can be used to collect knowledge about the personality. Personality is a
38 combination of traits and states that can be compared with already formulated personality
39 categories (Cohen et al., 2012). There are various techniques employed by researchers to measure
40 personality, such as MMPI-2 (Butcher, 1989), Big Five IPIP (McCrae and John, 1992), 16 PF
41 (Cattell, 1989), etc. Readers can find a detailed overview about personality measurement in (Cohen
42 et al., 2012) and (Boyle, 2008). All of these measuring instruments require self-reporting. There
43 are various versions of self-reporting scales like the one which require respondents to rate more
44 items and some which require to rate less items. The more items a scale contain, more reliable will
45 be the results. However, several studies showed that shorter versions were highly correlated with
46 full versions (Donnellan et al., 2006). Shorter versions tend to take less time as compared to the
47 versions which have more items. However, even shorter versions require allocation of time on part
48 of the participants. In addition, self-reports pose various validity and reliability problems (Stone
49 et al., 1999).
50 An alternative solution is automated measurement of personality that automatically measures a
51 user’s personality while doing his/her routine tasks. There is very little research on this approach.
52 Some examples include Heckmann who designed General User Model Ontology (GUMO) to store
53 users’ profile and to represent their psychological information through different kind of
54 technologies like semantic web and userML (User Model Mark-up Language) (Heckmann, 2005).
55 Personality recognizer from linguistic and conversational cues were also introduced (Mairesse et
56 al., 2007). There is also research available that adopted the personality measurement scale on the
57 computers (offline as well as online) but participants still needed to self-report these instruments.
58 An example of such adopted instrument is a website by Goldberg and Saicoer1.
1 http://ipip.ori.org/
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
59 This research adopted the automated instrument to develop a behaviour based measurement of the
60 personality. This research supports the notion that personality can be expressed from behaviour.
61 For example, Brebner et al. (Brebner, 1983) showed that a change in visual stimulation caused a
62 higher level of motor activity in extroverts. Geen et al. (Geen, 1984) noted that extroverts prefer
63 more intense level of music as compared to introverts in a learning task. Extroverts interaction
64 time with the interfaces were quicker as compared to introverts (Saati et al., 2005). In this paper,
65 we explore the possibility that such interactive behaviours might also be visible through a keyboard
66 and mouse. As compared to other personality measurement techniques, a dispositional approach
67 is adopted to measure personality via usage of keyboard and mouse along with “Big Five” model
68 to describe personality characteristics.
69 Empirical studies and analysis performed on the gathered data indicated that it is possible to predict
70 some of participants’ personality traits from the interactive behaviour of computer programmers
71 with keyboard and mouse and thus forming the main scientific contribution of this paper. The rest
72 of the paper is organized as follows. In Section 2, we present the state of the art literature review.
73 A brief overview to Big Five Personality Measurement Model is provided in Section 3. Section 4
74 presents the field study we conducted in our work, whereas in Section 5, we present the conducted
75 laboratory study. Section 6 presents active windows analysis, and a discussion is provided about
76 programmer’s activity level as well as the proposed algorithm in Section 7. Finally, Section 8
77 presents the limitations of the proposed work along with conclusions and future work.
78 2. RELATED WORK
79 The task of personality measurement is complex. Various personality measurement scales exist,
80 such as MMPI-2 (Butcher, 1989), Big Five IPIP (McCrae and John, 1992), 16 PF (Cattell, 1989).
81 Such scales measure personality from a particular dimension. For example, in order to assess
82 biological-based traits, various electronic equipment like EEG or GSR (Heller, 1993) is used. For
83 subjective measurement of personality, subjective self-reporting scales like Big Five Factor
84 models (McCrae and John, 1992) are used. Alternatively, for objective measurement of
85 personality, objective scales like Objective Analytic Battery (OAB) (Cattell, 1955) is used.
86 Additionally, some scales, like multidimensional personality questionnaire is used to improve
87 multicultural effectiveness of the scales to measure personality (Van Der Zee and Van
88 Oudenhoven, 2000). All of these approaches are summarized under approaches like
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
89 psychoanalytic approach, the Freudian approach, Non Freudian approach, the humanistic
90 approach, the cognitive approach as well as Behavioural/Social approach and have been discussed
91 in detail by Burger (Burger, 2010).
92 Personality depicts the habits of living beings specifically humans (Burger, 2010). This means that
93 people learn to behave and respond in a certain way and their responses for most of the times is
94 predictable. As discussed before, psychologists have developed various instruments to measure
95 these behaviours. These behaviours are then matched with a standard set of agreed upon
96 behaviours and are termed as personality (Cohen, 1988). However, significant time and effort is
97 required towards administering and managing these scales (Boyle, 2008). In addition, according
98 to Boyle (Boyle, 2008), there are very limited actual tests of personality and most of the scales
99 according to him are “self-report scales or reports of others’ rating scales”.
100 To overcome difficulties with self-reporting (as discussed in introduction) and to reduce
101 management and administrative issues, researchers devised various methods. In the present era of
102 computing, various researchers diverted their efforts towards adoption of personality measurement
103 scale for computers. Some studies, such as (Nunes et al., 2012), attempted to standardize the
104 personality information across applications. The aforementioned studies took human
105 psychological aspects into consideration while making decisions by the use of XML (Extensible
106 Mark-up Language). In addition Zhang et al. (Liang et al., 2012) presented a framework to
107 synthesize 2D cartoons and to depict their personality. People now are also able to take paper less
108 personality tests via website2. Such methods of evaluation might have reduced the managerial and
109 administrative issues, however, these tests are still based on self-reporting. Another issue is of data
110 privacy and handling as data access and control is not in the hand of every researcher and is mostly
111 restricted to the organization who own these scales. The researchers using such scale need to
112 convert textual results into their own format in order to do any analysis on the data. To overcome
113 these problems a more autonomous approach to measure personality is required.
114 Various automating activities related to personality with the help of computers exist. For example,
115 an interactive story telling system devised by Su et al. (Su et al., 2005) utilized hierarchical fuzzy
116 rule based system to facilitate the personality and emotion control of the dynamic story character.
2 http://ipip.ori.org/
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
117 Similarly, Chittaro and Serra [23] presented a system for realistic behavioural programming of
118 virtual characters and based the characters’ programming on personality and probabilistic
119 automata (Chittaro and Serra, 2004). Various researchers such as [24], experimented on imparting
120 personality on virtual computer generated characters (Kshirsagar, 2002). One the same note,
121 Rousseau et al. [25] proposed a model to be used by intelligent automated actors. These actors in
122 turn were able to improvise their behaviour to interact with the users in a multimedia environment
123 (Rousseau, 1996). There is also research on personality and how it is related to online buying
124 behaviour (Bosnjak et al., 2007). For instance, the work presented in [26] imparts various
125 behaviours and personalities to computer characters by using different computational and
126 probabilistic models. However no research in authors’ knowledge addresses autonomous objective
127 prediction of personality of computer users in general and specifically programmers.
128 Programmers and computer users’ personality is being researched to improve their productivity
129 and to find right person for the right job. For example, Capretz (Capretz, 2003) reviewed empirical
130 studies on personality types of software engineers. Capretz concluded that software engineers are
131 significantly related to various personality types and are more likely to be sensing and thinkers
132 (STs), or Thinking and Judgmental (TJs), or Intuitive and thinkers (NTs) (Capretz, 2003). Software
133 professionals should be assigned roles from which it is evident that they have suitable personality
134 to carry out such tasks (Acuna et al., 2006), (Da Cunha and Greathead, 2007). Another example
135 of software professional personality should be considered while they work comes from (Salleh et
136 al., 2010) which found personality to be significantly related with pair programming students’
137 academic performance (Salleh et al., 2010).
138 In our proposed work, we analyse the behaviour of users on most commonly used devices, such
139 as keyboard and mouse. For computer users generally and for software programmers specifically
140 the keyboard and mouse are the most common devices for input and output. The keyboard and
141 mouse behaviour is already used by various researchers to predict moods of computer users. For
142 instance, in (Khan et al., 2013) and (Epp et al., 2011) the authors argued that it is possible to predict
143 some of the personality traits of programmers by recording their behaviour with keyboard and
144 mouse. In this paper, we utilized the Big Five Model of personality measurement. The next section
145 discussed in details the Big Five Model and the reasons why we selected this personality
146 measurement scale.
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
147 3. BIG FIVE PERSONALITY MEASUREMENT MODEL
148 Although there is a difference of opinion among researchers in how to measure and what to
149 measure in a person’s personality, but still it is agreed that factor analytic properties of personality
150 are consistent (Burger, 2010). Various investigations on dynamic data like: (Digman, 1990),
151 (Goldberg, 1992) and (McCrae and Costa Jr, 1997) found that five basic dimensions of personality
152 are more or less consistent. These five dimensions of personality are so extensively used by
153 researchers that now it is being referred to as Big Five Model of personality measurement (Burger,
154 2010), (Boyle, 2008). The most commonly used names of the five dimensions are Neuroticism,
155 Extraversion, Openness, Agreeableness, and Conscientiousness. As the authors Boyle et al.
156 (Boyle, 2008) stated that in the construction of the Big Five scale, researches have not only utilized
157 the theory but also used empirical factor analytic inputs.
158 Such approach of defining traits is also the most common approach implemented by computing
159 scientists to measure personality via computers (Nunes et al., 2012). According to Nunes et al.,
160 trait approach highlights clear psychological differences among individuals and is easier to
161 categorize, organise, and to implement. Because of its wider and universal use, we considered Big
162 Five personality model in our paper for measuring personality of the participants.
163 Big Five personality measure contains five main traits and each main trait is further divided into
164 six sub-trait thus forming a total of 35 traits including the main trait. This specific 35 trait
165 personality questionnaire is called as NEO PI-R, a revised version by Costa and McCrae (Costa
166 and MacCrae, 1992). The original scale contains 300 items and take on average 30-40 minutes to
167 complete the test. The short version of scale contains 120 items and could take 10-20 minutes to
168 complete. The original scale is more reliable than the shorter one, though, shorter version also
169 meets professional standards of reliability (Costa and MacCrae, 1992). The test can be taken at the
170 following website3. IPIP Big Five scale main traits, sub-traits, and few of the characteristics are
171 shown in the Table 1 below.
172 Table 1: Big Five personality traits, sub-traits, and some of the related characteristics adopted
173 from Burger (Burger, 2010)
3 http://www.personal.psu.edu/faculty/j/5/j5j/IPIP/
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
Main Trait Sub-trait Some Characteristics
FriendlinessGregariousness
Assertiveness
Activity Level
Excitement Seeking
Extraversion
Cheerfulness
Sociable versus retiring, Fun-loving versus sober
Affectionate versus reserved, Energetic versus lazy,
Hurried versus slow, Aggressive versus ambitionless,
Active versus inactive
Trust
Morality
Altruism
Cooperation
Modesty
Agreeableness
Sympathy
Soft-hearted versus ruthless
Trusting versus suspicious
Helpful versus uncooperative
Self-Efficacy
Orderliness
Dutifulness
Achievement Striving
Self-Discipline
Conscientiousness
Cautiousness
Well organized versus disorganized
Careful versus careless
Self-disciplined versus weak willed
Anxiety
Anger
Depression
Self-Consciousness
Immoderation
Neuroticism
Vulnerability
Worried versus calm
Insecure versus secure
Self-pitying versus self-satisfied
Imagination
Artistic Interests
Emotionality
Adventurousness
Intellect
Openness to
Experience
Liberalism
Imaginative versus down-to-earth
Preference for variety versus preference for routine
Independent versus conforming
174
175 These studies involve collection of the participants’ personality as well as their interactive
176 behaviour with keyboard and mouse. The next sections will present two studies and three analyses
177 performed in this paper. The results are presented in the corresponding study sections.
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
178 4. STUDY I – FIELD STUDY
179 4.1. Material
180 Most of the material collected and used in this study was also used in (Khan et al., 2013) to measure
181 mood of computer users from keyboard and mouse interaction. The basic idea of the experiment
182 was to record all keyboard and mouse related interaction of a user in log files. Therefore, we
183 developed a software application to record all the user interaction into log files. A mood rating
184 dialog was also designed to appear after every 20 minutes of logging to get participants’ mood
185 response. However, to reduce the concerns of participants over their privacy and security issues
186 various functionalities were developed in the application, such as:
187 Pause logging: Participants were able to pause application from logging for 5 and 10 minutes as
188 well as for custom specified period (in minutes) by clicking on the relevant menu. This option was
189 helpful for participants who wanted to do private tasks like bank transaction, etc.
190 Exit logging: Participants were also able to exit the application. Participants were instructed to
191 restart the application when they feel so. They were also told that application will restart itself
192 automatically on next computer boot.
193 Categorical Storage of keys: Some keys of the keyboard were divided into 4 categories namely:
194 ‘capital alphabets’, ‘lowercase alphabets’, ‘numeric keys’ and ‘special characters’. The keyboard
195 interaction were classified and recorded according to one of the categories instead of recording
196 original keys. In addition, mouse button events were recorded ‘Right Button’ or ‘Left Button’
197 depending on the right or left button click event. This mechanism further built the trust of
198 participants to continue using this application over several days.
199 4.2. Experimental setup
200 Departmental ethics committee namely CS-DARC (Computer Science Department Academic
201 Review Committee) reviewed and approved this experiment. A written consent for all studies was
202 obtained from the participants. The procedure of taking consent this way was approved by CS-
203 DARC. The developed application is run as a background application on the participants’
204 machines. The categorized keyboard interaction data described earlier were recorded under the
205 column ‘Category of the event’. In addition, the data for three more columns was recorded namely
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
206 as: ‘Window name’, ‘Keyboard and Mouse Event’, and ‘Date/Time of the event’. Further
207 description of these three columns is as follows:
208 Window name: As modern day operating systems are designed to run different windows
209 (applications) at a time, the window name field was used to record active window. Names of the
210 document under work were not recorded. Only general software names under the use were
211 recorded like ‘Microsoft Power point’, ‘Mozilla Firefox’, etc. The data was recorded based on the
212 assumption that a person’s behaviour with the usage of keyboard and mouse will be different for
213 different applications. For example, frequent events of keyboard are expected while using an
214 editing application, chatting application or Integrated Development Environment (IDE) used by a
215 programmer during process of coding. Similarly, frequent mouse clicks are expected on a mouse
216 driven games like age of empires and rise of nations, etc.
217 Keyboard and Mouse event: When a key is pressed it initiates a ‘key down’ event. When released,
218 a ‘key up’ event is initiated. Similarly, ‘mouse click’ event is initiated on mouse button click. Such
219 data was used to record the three events: ‘Key Up’, ‘Key Down’ as well as mouse clicks (‘Left’ or
220 ‘Right’) button.
221 Date/Time of event: This column was used to record the date and exact time up to milliseconds
222 on the initiation of an event using the system date and time functionality.
223 A mood dialogue (Khan et al., 2013) was embedded in the log files as a marker to measure an
224 interaction. Application also provided the choice to participants to uninstall the application and zip
225 the log files, or to continue the experiment. Short form of Big Five IPIP-NEO4 (Goldberg, 1999)
226 instrument containing 120 items were used in this research. Participants were instructed to
227 complete the personality questionnaire online on IPIP website and email the results to the
228 researcher.
229 4.3 Participants
230 A total of 26 computer users participated in the mood study but only 20 consented to take
231 personality test as well. Therefore this study considered response of only 20 participants in total.
232 All of the participants were invited via personal contact or emails. The mean age of the
233 participants’ was 28.5 years and a standard deviation of 2.9 years. Participants’ computer
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
234 experience mean was 5 years and standard deviation of 1.6 years. When asked from participants
235 how they rate themselves from the options as programmer, expert computer user, and a novice,
236 50% of them rated themselves as programmers. Expert computer users were 40% and 10% were
237 novices or beginners. There were only two females who participated in this study.
238 4.4. Results
239 4.4.1 Data preparation
240 Each participant continued logging for at least 5 days. Logging application was designed to create
241 a separate log file for each day with date and time stamp. All these file were merged together to
242 form a single file for a participant thus forming 20 different log files related to 20 participants. On
243 average, about a million lines of interaction of a participant with keyboard and mouse were
244 recorded. Therefore, a separate application was required to extract meaningful data from these
245 files. First step was to find a meaningful point around which data is analysed. This meaningful
246 point was identified as ‘mood rating’ embedded via mood rating dialog (Khan et al., 2013) in to
247 the log files. Average of behavioural data was calculated around every mood rating window4. An
248 interaction data of 3 minutes before mood rating and 3 minutes after mood rating were taken to
249 form a 6 minute window. The basic measures taken for each window were:
250 Self-reported mood: This was embedded in the log file. In the analysis part, this measure was
251 excluded because of no relevance to the study.
252 1. Total number of events (nEvents): Total events that took place within 6 minute window.
253 2. Average time between events (ATBE): Average of the time difference between occurrence of
254 two consecutive events
255 3. Standard deviation time between events (ATBE-STD): The standard deviation of ATBE
256 4. Total windows switched (WS): A switch by the participants from one window to another
257 window.
258 5. Number of backspace and delete key events (NBD): The events counter for only backspace
259 and delete keys. This is the counter of the possible errors committed by a user and the undo
260 action performed for those errors.
4 A window was formed around embedded mood rating in the log file and the interaction data before and after it
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
261 6. Numeric and alphabet keys (NAK): Counter for numeric and alphabet keys
262 7. Mouse clicks (MC): Counter for total number of clicks (including right and left button clicks)
263 8. Other keys (OK): All other keys that include special keys, arrow keys, semi colons, colons etc.
264 Each participant’s average of the above measures and that of his/her rated personality was put into
265 an excel file for further refinement. There were N rows of entries for each participant’s interaction
266 data, where N is the number of mood rating dialogs answered by participants. There was only 1
267 row of personality traits. Before averaging a participant’s interaction data over N rows, all the
268 events where interaction was less than 10 events in a window were filtered out. The reason was
269 that the participants might not be actively using keyboard and mouse but instead reading from
270 some website or busy in some other tasks. All events where difference of time between two events
271 was less than 50 milliseconds were also eliminated. The reason being that an experienced typist
272 on average cannot type a meaningful character of a sequence in less than 60 milliseconds (Card et
273 al., 1983). Considering an experienced typist, we reduced filter to 50 milliseconds just in case that
274 someone among the participants could be an experienced typist. In addition, all the events with a
275 difference of time more than 20 seconds were also eliminated on the same grounds of inactivity.
276 After required filtration of each participant data, all the interactions rows were averaged to a single
277 row. In addition to means, standard deviation of each interaction variables was also calculated.
278 4.4.2. Analysis
279 First step toward analysis of this data was to carry out correlations analysis. As aim of this study
280 is to find out the relation between keyboard and mouse interaction behaviour and personality traits.
281 Therefore, a correlation analysis was conducted between interaction behaviour variables and
282 personality trait variables. The results revealed that various traits and sub-traits are significantly
283 correlated with interaction behaviour of participants with keyboard and mouse. The results are
284 presented from Table 2–Table 7 below.
285 Table 2: Correlations between Extroversion and its sub-traits with that of keyboard and mouse
286 behaviour
Interaction Variables Gregariousness Activity Level Excitement
Seeking
Cheerfulness
ATBE-Mean -0.44*
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
ATBE-Standard Deviation -0.43* 0.44* -0.53*
NAK-Mean 0.40* 0.41*
NAK-Standard Deviation 0.41* 0.41*
OK–Mean 0.41*
OK- Standard Deviation 0.48*
287 Note: ** significant Correlation at 0.01 levels, * significant Correlation at 0.05 levels for N = 20.
288 Table 3: Correlations between Agreeableness and its sub-traits with that of keyboard and mouse
289 behaviour
Agreeableness Trust Morality Cooperation Sympathy
ATBE-Mean 0.41*
WS-Mean -0.56** -0.56** -0.52*
WS-Standard
Deviation-0.48*
NBD-Mean -0.51* -0.43*
NBD-Standard
Deviation-0.40* -0.40* -0.40*
NAK-Mean -0.45*
NAK-Standard
Deviation0.47*
290 Note: ** significant Correlation at 0.01 levels, * significant Correlation at 0.05 levels for N = 20.
291 Table 4: Correlations between Conscientiousness and its sub-traits with that of keyboard and
292 mouse behaviour
Conscientious
ness
Self-Efficacy Orderliness Dutifulne
ss
Cautiousn
ess
nEvents–Mean -.60** -.46*
WS-Mean -.42*
NAK-Mean .46*
MC–Mean .44*
MC–Standard Deviation .52*
OK-Mean -.40* -.43* -.55**
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
OK-Standard Deviation -.56**
293 Note: ** significant Correlation at 0.01 levels, * significant Correlation at 0.05 levels for N = 20.
294 Table 5: Correlations between Neuroticism and its sub-traits with that of keyboard and mouse
295 behaviour
Neuroticism Anxiety Self-Consciousness Vulnerability
ATBE-Mean .43*
WS-Mean .50*
NAK-M .51*
MC-M -.40* -.53*
MC-Standard Deviation -.40* -.40* -.51* -.41*
296 Note: ** significant Correlation at 0.01 levels, * significant Correlation at 0.05 levels for N = 20.Table
297
298 Table: 6: Correlations between Openness to experience and its sub-traits with that of keyboard
299 and mouse behaviour
Openness to
experience
Artistic
Interests
Emotionality Intellect Liberalism
nEvents-Mean .43*
ATBE-Standard Deviation .41*
WS-Mean -.40* -.54* -.60**
WS-Standard Deviation -.48* -.62** -.58**
300 Note: ** significant Correlation at 0.01 levels, * significant Correlation at 0.05 levels for N = 20.
301 Correlations are considered of small, medium, and large size with r = 0.1, r = 0.3 to 0.49, and r =
302 0.5 or above, respectively (Cohen, 1992). According to the aforementioned criterion, all
303 correlations revealed in Tables 2–7 either have medium or large effects.
304 5. STUDY II – LAB STUDY
305 Various personality traits were found to be significantly correlated with the keyboard and mouse
306 interaction variables mentioned in section 4.4.1. While in study 1, the interaction of the participants
307 with keyboard and mouse were logged in normal working conditions, in the second study
308 programming tasks of about an hour duration were given to participants as task to minimize
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
309 boredom and to reduce activity time. The idea was same to replicate the earlier findings with
310 different experimental setup and conditions. Research shows that personality of people mainly
311 remains constant over years and external conditions like time, environment etc. do not effect
312 personality of people (Shaffer, 2009). Therefore, it was hypothesized that interaction of
313 participants with keyboard may produce similar results as that of experiment 1. The material used,
314 preparation of the data, analysis and results are presented in the following sections
315 5.1. Material
316 The customized application used in study 1 with disabled mood dialogs were modified to log
317 keyboard and mouse interaction data. Original data was not recorded and instead categorical data
318 were recorded. Like previous study this study also measured participants’ personality by using
319 short form of IPIP-NEO Big Five personality instrument. In this study programming tasks
320 assigned to participants were to simulate a story (see section 5.2) in ALICE programming
321 language. ALICE is a free object oriented and simulation based language that is often utilized to
322 teach programming to children.
323 5.2. Experimental setup
324 Main differences from experiment 1 were in the setup of the experiment. A quiet room without an
325 external interference was reserved for the experiment. All of the participants were either
326 programmers or expert computer users. They were trained to program in ALICE. All participants
327 successfully completed four learning tutorials already present in ALICE documentation for
328 learning purposes. Two of the Aesop stories (‘the hare and the tortoise’ and ‘the cow and the frog’)
329 were assigned to participants to simulate. Music was played in background while participants were
330 performing tasks. The experiment was divided into two sessions with a gap of 10-20 minutes in
331 each session. The logging application was running in background to log keyboard and mouse
332 interaction during the entire duration of the experiment. Same variables as discussed in section 4.1
333 were also being logged in this study.
334 5.3. Participants
335 In this experiment, 15 participants were involved. All participants rated themselves as expert
336 computer users but with no experience of programming in ALICE. The participants mean age was
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
337 21 years with standard deviation of 3.1. The mean experience of participants of computer usage
338 was 7.3 years and standard deviation of 2.77. There were no female participants in this study.
339 5.4. Results
340 5.4.1 Data preparation
341 Average of eight behavioural measures such as: (1) number of events, (2) average time between
342 events, (3) standard deviation of the time between events, (4) total windows switched, (5) number
343 of backspace and delete keys, (6) number of alphabetic and numeric keys, (7) number of mouse
344 clicks, and (8) number all other keys were calculated over the duration of the experiment. Before
345 calculating average of behavioural variables over the sessions, the data was filtered with filters
346 like: ‘time difference between the two events’ should be greater than 50 milliseconds and less than
347 20 seconds The reason of using these filters is already discussed in section 4.4.1. This data was
348 merged with personality data in an excel file forming a single row with first eight columns of
349 interaction data and last columns containing personality data.
350 5.4.2 Analysis II
351 Pearson correlation was conducted between the interaction data and personality data. Various traits
352 and sub-traits of personality were again found to be significantly correlated with that of keyboard
353 and mouse interaction behaviour. Only one result appears to be significantly correlated in both
354 studies – the standard deviation of average time between events and activity level with r=0.59 and
355 p<=0.05. The results from this study and the earlier one indicates that behavioural measures have
356 no relation with personality traits except activity level and standard deviation of average time
357 between events. Activity levels fall within the sub-traits of extraversion. It is related to the level
358 of excitement and energy within individuals (Whalen and Gates, 2007). In both studies activity
359 level was found to be positively correlated with that of standard deviation time between events.
360 To confirm this effect, another analysis was planned by utilizing the data from the experiment 1.
361 6. ACTIVE WINDOWS ANALYSIS-STUDY I
362 As the first experiment was field study with data collected from participants working at their
363 offices/homes etc. therefore participants interacted with various windows during the experiment.
364 One of the columns ‘Active windows’ (on which the participants were working on) was being
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
365 recorded in the log files. Its purpose was to measure the keyboard and mouse interaction with
366 different applications. A skimming through these data revealed that participants interacted with
367 various different applications. There were various applications which do not need a lot of
368 interaction from participants like audio/video players, as well as Installation/Uninstallation
369 windows etc. The remaining applications were categorized as below:
370 1. Browsers
371 2. Word Editing
372 3. Messaging
373 4. Programming and Development
374 5. Games
375 Some of the windows/software were identifiable by their general name like Microsoft word and
376 Internet Explorer. However, some of the windows were not identifiable as they were tagged like
377 “unidentifiable window” or “False Window”. The reason was that these applications unlike various
378 other applications did not contain sub documents but one general name. For example a document
379 named ‘doc1’ will display a title like: ‘Microsoft Word – doc1’. Extracting the name of the
380 application without identifying the name of document becomes easy. However, some applications
381 like games were logged as ‘Unidentifiable Window’ to protect information related with
382 participants whereas applications without sub-documents with direct names were recorded as
383 ‘False Windows’
384 Keystroke analysis against ‘Unidentifiable windows’ and ‘false windows’ showed mouse clicks
385 and ‘other keys’ events dominated over all other events. As up, down, left, and right arrows were
386 included in ‘other keys’, therefore these windows were most likely to be games as usually games
387 use a lot of mouse clicks as well as direction keys. Similarly, use of alphabetic and numeric keys
388 was minimal compared to the aforementioned windows thus supporting the claim that the window
389 identifies some game being played. Only two of the entries after analysing the keys were not
390 clearly identifiable and therefore were left out of the analysis.
391 Different participants showed different frequency of use of the above mentioned five categories.
392 If total number of events by a participant for an application over five days were less than 1000
393 events, then this application was not included in the analysis. Browsers and Gaming applications
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
394 were used by all the 20 participants included in this analysis. However, word editing applications
395 with events greater than 1000 were only used by 16 participants. Chatting/Messaging was used by
396 19 participants. Programming and development related applications were used by only 12
397 participants. The filters like: time difference of 50 milliseconds between two events and less than
398 20 seconds were also applied on this data.
399 6.1 Results
400 Correlation analysis was conducted between ‘standard deviation of average time of events’ on five
401 categories of applications separately with that of ‘activity level’. Only the one sub-trait of
402 extroversion ‘activity level’ was considered as this variable showed significant correlations in
403 earlier two studies. Results revealed that activity level was only significantly correlated with the
404 standard deviation of the average time between events that occurred on the programming
405 applications (r = 0.646, p = 0.023). The outcomes support the notion that generally the participants
406 who tend to be programmers have positive correlation with their activity level.
407 DISCUSSIONS-PROGRAMMERS’ ACTIVITY LEVEL
408 In this study, three types of computer users participated: (a) programmers, (b) experts, and (c)
409 novices. These ranking were participants’ own perceptions. On analysis of the participants’
410 keyboard and mouse interaction data from experiment 1, it appeared that all the users either were
411 programmers or were users who used computer frequently. All of the participants used word
412 browsers and game applications. Majority participants (a total of 19) used chatting/email
413 applications as well as 16 used word editing applications. Similarly, of all the participants were
414 experienced computer users’ with mean experience of 2 years or more. In the second study, all the
415 participants’ were trained of programming task and eventually carried out that programming task.
416 Therefore, it is safe to presume that finding of this study largely dealt with programmers. Results
417 from all of the analysis showed that it is possible to predict programmers’ activity level from their
418 interaction behaviour with keyboard and mouse.
419 Activity level is a sub-trait of extraversion. According to (Goldberg, 1999) a person with positive
420 activity level is one who can manage multi-tasking, target goals and achieve them without wasting
421 his time. These are all attributes that belong to good programmers. Researches like (Stolee et al.,
422 2011) and (Staurt, 2010) are some of the examples that discussed on the positive characteristics of
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
423 the programmers. Therefore, the results of our research can provide a useful application to predict
424 good programmers vs. not so good programmers.
425 Programmers’ activity level might also be dependent on the extent of stress they face during
426 development. Nan and Harter et al. (Nan and Harter, 2009) demonstrated relationship between
427 schedule pressure and development activities in software development life cycle (SDLC). They
428 concluded in their paper that excessive pressure affect development performance negatively. An
429 indirect inference is that increased pressure may cause a programmer to rapidly press keyboard
430 keys, indicating high activity level on keyboard and mouse. Therefore, one of the findings of this
431 research is that the pressure to complete a task within a deadline may cause increase activity level
432 with the keyboard and mouse.
433 The variables having significant correlations were analysed to find what frequency of keyboard
434 interaction is related with which level of activity, where activity level is a participants’ personality
435 trait rating out of 100. Both analyses of experiment 1 revealed that in conditions where there is no
436 pressure on the programmers and when their activity level is equal to or above 60 then standard
437 deviation of difference between events time lies above 1600 milliseconds. In the pressure
438 development as in experiment 2, standard deviation time was reduced to 530 milliseconds for
439 programmers having activity level above 60. Considering these statistics, an algorithm to predict
440 programmers’ activity level is formulated as follows:
441 ALGORITHM
442 A general algorithm to measure a person’s activity level from his/her interaction behaviour with
443 keyboard and mouse is formulated. As evident from current research [Khan et. el, 2013] that this
444 algorithm will most likely to work on programming related application like I.D.Es’.
445 1: Begin Program
446 2: Let Ec a current keyboard or mouse interaction event
447 3: Let Ecp a previous keyboard and mouse interaction event. If this is the first event then Ecp = 0
448 4: Record programmers’ keyboard and mouse interaction over or at least for n minutes in a file
449 F where n >= 3
450 5: Set ‘previous event’ to null
451 6: For time t loop from t = 0.1 second to a finite time but greater than n minutes
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
452 7: Scan ‘current event’ from file F
453 8: If (’ previous event’ is not null) then
454 9: Let TD a variables that will store time difference between two events
455 10: TD = absolute (Ecp - Ec)
456 11: Comment: The line below shows filter applied on the interaction data
457 12: If (TD > 50 ms) and (TD < 20s)
458 13: Add the difference into an array Ai
459 14: End If (TD > 50 ms) and (TD < 20s)
460 15: End If (there is previous event)
461 16: Put ‘current event’ in ‘previous event’
462 17: Scan file for next event and put it into ‘current event’
463 18: End for
464 19: Let SD = Calculate Standard deviation of all the elements in Ai
465 20: If (Development Environment is a Pressure Environment) and
466 21: If (SD > 530 milliseconds)
467 22: Predication “Programmer has high activity level and might be a good
468 programmer”
469 23: End if (SD > 530 milliseconds)
470 24: Else (Development Environment is not a Pressure Environment)
471 25: If (SD > 1600 milliseconds)
472 26: Predication “Programmer has low activity level and might not be a good
473 programmer”
474 27: End if (SD > 1600 milliseconds)
475 28: End if ((Development Environment == Pressure Environment) and
476 29: End Program
477
478 7. CONCLUSIONS, LIMITATIONS AND FUTURE RESEARCH
479 Based on the participants as well as results of study 1, it is evident that most of the personality
480 traits and sub-traits correlate with keyboard and mouse interaction behaviour. However, only
481 activity level was found to be positively correlated in both the experiments and three analyses.
482 Moreover, as these participants had more than two years of experience with computers as well as
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
483 almost all participants showed significant usage of applications that need expertise, it can also be
484 safely concluded that these participants were either programmers or possessed programming skills.
485 The main conclusion of the research is that programmers’ personality specifically activity level
486 can be predicted from their interaction behaviour with keyboard and mouse events.
487 Like all empirical researches, this research also has some limitations. For example, sample
488 participants were mostly graduate and PhD students that limited the findings specific to only such
489 people. Another limitation was that these results are mostly representative of male population as
490 there were only two females in study 1 and no female in study 2. However, this limitation denotes
491 an overall trend of technology and engineering industry where female population is scarce. In
492 addition, 40% of the participants in both the studies were same. Therefore, there is a chance of
493 repetition in the observed correlations. However, as sampling was done in two different
494 environments, the aforementioned limitation might have negligible effect on overall results of this
495 research.
496 Various future possibilities exist from this research. This research laid a foundation for an
497 application to measure users’ personality (activity level) from their interaction behaviour with
498 keyboard and mouse. Further experiments to test each of five personality dimensions separately
499 with the keyboard and mouse interaction might enable researchers to detect personality with more
500 accuracy.
501 Acknowledgements
502 I am thankful to Dr. William-Paul Brinkman of Delft University of Technology, The Netherlands
503 under whose kind supervision, I have collected the data used in this paper.
504 8. REFERENCES
505 Acuna ST, Juristo N and Moreno AM. (2006) Emphasizing human capabilities in software development. 506 Software, IEEE 23: 94-101.507 Bosnjak M, Galesic M and Tuten T. (2007) Personality determinants of online shopping: Explaining online 508 purchase intentions using a hierarchical approach. Journal of Business Research 60: 597-605.509 Boyle GJM, G.; Saklofske, D. H.;. (2008) Personality measurement and testing: An Overview. In: Boyle 510 GJM, G.; Saklofske, D. H.; (ed) Personality and Social Sciences papers. Paper 294. London: SAGE, 511 1-29.512 Brebner J. (1983) A model of extraversion. Australian Journal of Psychology 35: 349-359.513 Burger JM. (2010) Personality: Wadsworth Cengage Learning.
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
514 Butcher JND, W. G.; Graham, J. R.; Tellegen, A; Kaemmer, B;. (1989) The Minnesota Multiphasic Personality 515 Inventory-2 (MMPI-2): Manual for administration and scoring, Minneapolis: University of 516 Minnesota Press.517 Capretz LF. (2003) Personality types in software engineering. International Journal of Human-Computer 518 Studies 58: 207-214.519 Card KC, Moran TP and Newell A. (1983) The Psychology of Human-Computer Interaction, Hillsdale, New 520 Jersey: Lawrence Erlbaum.521 Cattell HB. (1989) The 16PF: Personality in depth: Institute for Personality and Ability Testing Illinois.522 Cattell RB. (1955) Handbook for the Objective-Analytic Personality Test Batteries:(including Adult and Child 523 OA Batteries): Institute for Personality and Ability Testing.524 Chittaro L and Serra M. (2004) Behavioral programming of autonomous characters based on probabilistic 525 automata and personality. Computer Animation and Virtual Worlds 15: 319-326.526 Cohen, Swerdlik ME and Sturman ED. (2012) Psychological Testing and Assessment: An Introduction to 527 Tests and Measurement: McGraw-Hill Education.528 Cohen RJM, P.; Nathanson, L. S.; Swerdlik, M. E.;. (1988) Personality Assessment: An Overview. 529 Psychological Testing: An Introduction to
530 Tests and Measurements. Mountain View, CA: Mayfield Publishing Co.531 Cohen; J. (1992) Statistical Power Analysis. Current Directions in Psychological Science 1: 98-101.532 Costa PT and MacCrae RR. (1992) Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor 533 Inventory (NEO FFI): Professional Manual: Psychological Assessment Resources.534 Da Cunha AD and Greathead D. (2007) Does personality matter?: an analysis of code-review ability. 535 Communications of the ACM 50: 109-112.536 Digman JM. (1990) Personality structure: Emergence of the five-factor model. Annual review of 537 psychology 41: 417-440.538 Donnellan MB, Oswald FL, Baird BM, et al. (2006) The mini-IPIP scales: tiny-yet-effective measures of the 539 Big Five factors of personality. Psychological assessment 18: 192.540 Epp C, Lippold M and Mandryk RL. (2011) Identifying emotional states using keystroke dynamics. 541 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 715-724.542 Geen RG. (1984) Preferred stimulation levels in introverts and extroverts: Effects on arousal and 543 performance. Journal of Personality and Social Psychology 46: 1303.544 Goldberg LR. (1992) The development of markers for the Big-Five factor structure. Psychological 545 assessment 4: 26.546 Goldberg LR. (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level 547 facets of several five-factor models. Personality psychology in Europe 7: 7-28.548 Heckmann D. (2005) Ubiquitous user modeling: IOS Press.549 Heller W. (1993) Neuropsychological mechanisms of individual differences in emotion, personality, and 550 arousal. Neuropsychology 7: 476.551 Khan IA, Brinkman W-P and Hierons R. (2013) Towards estimating computer users’ mood from interaction 552 behaviour with keyboard and mouse. Frontiers of Computer Science: 1-12.553 Kshirsagar S. (2002) A multilayer personality model. Proceedings of the 2nd international symposium on 554 Smart graphics. ACM, 107-115.555 Liang Z, Xiao J and Zhuang Y-t. (2012) Synthesizing style-preserving cartoons via non-negative style 556 factorization. Journal of Zhejiang University SCIENCE C 13: 196-207.557 Mairesse F, Walker MA, Mehl MR, et al. (2007) Using Linguistic Cues for the Automatic Recognition of 558 Personality in Conversation and Text. J. Artif. Intell. Res.(JAIR) 30: 457-500.559 McCool LC. (2008) The Health Behaviours of Exercise and Dietary Intake: Links with Personality and 560 Coping, in a student sample.
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
561 McCrae RR and Costa Jr PT. (1997) Personality trait structure as a human universal. American psychologist 562 52: 509.563 McCrae RR and John OP. (1992) An introduction to the five-factor model and its applications. Journal of 564 personality 60: 175-215.565 Nan N and Harter DE. (2009) Impact of budget and schedule pressure on software development cycle time 566 and effort. Software Engineering, IEEE Transactions on 35: 624-637.567 Nunes MASN, Bezerra JS and de Oliveira AA. (2012) Personalityml: a markup language to standardize the 568 user personality in recommender systems. GEINTEC-Gestão, Inovação e Tecnologias 2: 255-273.569 Phau I, Sequeira M and Dix S. (2009) To buy or not to buy a.570 Rousseau D. (1996) Personality in computer characters. Proceedings of the 1996 AAAI Workshop on 571 Entertainment and AI/A-Life. 38-43.572 Saati B, Salem M and Brinkman W-P. (2005) Towards customized user interface skins: investigating user 573 personality and skin colour. Proceedings of HCI 2005 2: 89-93.574 Salleh N, Mendes E, Grundy J, et al. (2010) An empirical study of the effects of conscientiousness in pair 575 programming using the five-factor personality model. Proceedings of the 32nd ACM/IEEE 576 International Conference on Software Engineering-Volume 1. ACM, 577-586.577 Shaffer D. (2009) Social and personality development: CengageBrain. com.578 Staurt; A. (2010) 50 characteristics of a great software developer.579 Stolee KT, Elbaum S and Sarma A. (2011) End-user programmers and their communities: An artifact-based 580 analysis. Empirical Software Engineering and Measurement (ESEM), 2011 International 581 Symposium on. IEEE, 147-156.582 Stone AA, Bachrach CA, Jobe JB, et al. (1999) The science of self-report: Implications for research and 583 practice: Psychology Press.584 Su W-P, Pham B and Wardhani A. (2005) High-level control posture of story characters based on 585 personality and emotion. Proceedings of the second Australasian conference on Interactive 586 entertainment. Creativity & Cognition Studios Press, 179-186.587 Van Der Zee KI and Van Oudenhoven JP. (2000) The Multicultural Personality Questionnaire: A 588 multidimensional instrument of multicultural effectiveness. European journal of personality 14: 589 291-309.590 Whalen T and Gates C. (2007) A psychological profile of defender personality traits. Journal of Computers 591 2: 84-93.
592
593
PeerJ PrePrints | https://dx.doi.org/10.7287/peerj.preprints.1183v1 | CC-BY 4.0 Open Access | rec: 17 Jun 2015, publ: 17 Jun 2015
PrePrin
ts
top related