Proceedings of the First Workshop on Ethics in Natural Language Processing, pages 41–52,Valencia, Spain, April 4th, 2017. c© 2017 Association for Computational Linguistics

Building Better Open-Source Tools to Support Fairnessin Automated Scoring

Nitin Madnani1, Anastassia Loukina1, Alina von Davier2, Jill Burstein1 and Aoife Cahill1

1Educational Testing Service, Princeton, NJ2ACT, Iowa City, IA

1{nmadnani,aloukina,jburstein,acahill}@ets.org2Alina.vonDavier@act.org

Abstract

Automated scoring of written and spo-ken responses is an NLP application thatcan significantly impact lives especiallywhen deployed as part of high-stakes testssuch as the GRE® and the TOEFL®.Ethical considerations require that auto-mated scoring algorithms treat all test-takers fairly. The educational measure-ment community has done significant re-search on fairness in assessments and au-tomated scoring systems must incorporatetheir recommendations. The best way todo that is by making available automated,non-proprietary tools to NLP researchersthat directly incorporate these recommen-dations and generate the analyses neededto help identify and resolve biases in theirscoring systems. In this paper, we attemptto provide such a solution.

1 Introduction

Natural Language Processing (NLP) applicationsnow form a large part of our everyday lives. Asresearchers who build such applications, we have aresponsibility to ensure that we prioritize the ideasof fairness and transparency and not just blindlypursue better algorithmic performance.

In this paper, we discuss the ethical considera-tions pertaining to automated scoring of written orspoken test responses, referred to as “constructedresponses”. Automated scoring is an NLP appli-cation which aims to automatically predict a scorefor such responses. We focus on automated sys-tems designed to score open-ended constructedresponse questions. Such systems generally usetext and speech processing techniques to extracta set of features from responses which are thencombined into a scoring model to predict the fi-

nal score assigned by a human rater (Page, 1966;Burstein et al., 1998; Zechner et al., 2009; Bern-stein et al., 2010).

Test scores whether assigned by human raters orcomputers can have a significant effect on people’slives and, therefore, must be fair to all test takers.Automated scoring systems may offer some ad-vantages over human raters, e.g., higher score con-sistency (Williamson et al., 2012). Yet, like anyother machine learning algorithm, models usedfor score prediction may inadvertently encode dis-crimination into their decisions due to biases orother imperfections in the training data, spuriouscorrelations, and other factors (Romei and Rug-gieri, 2013b; von Davier, 2016).1

The paper has the following structure. We firstdraw awareness to the psychometric research andrecommendations on quantifying potential biasesin automated scoring and how it relates to theideas of fairness, accountability, and transparencyin machine learning (FATML). The second halfof the paper presents an open-source tool calledRSMTool2 for developers of automated scoringmodels which directly integrates these psychome-tric recommendations. Since such developers arelikely to be NLP or machine learning researchers,the tool provides an important bridge from the ed-ucational measurement side to the NLP side. Next,we discuss further challenges related to fairness inautomated scoring that are not currently addressedby RSMTool as well as methods for avoiding biasin automated scoring rather than just detecting it.The paper concludes with a discussion of howthese tools and methodologies may, in fact, be ap-

1Some of these problems were recently discussed at apanel focused on Fairness in Machine learning in Educa-tional Measurement that was held at the annual meeting ofNational Council for Educational Measurement (von Davierand Burstein, 2016).

2http://github.com/EducationalTestingService/rsmtool

41

plicable to other NLP applications beyond auto-mated scoring.

2 Ethics and Fairness in ConstructedResponse Scoring

At this point in the paper, we believe it is importantto define exactly what we refer to as fairness forthe field of scoring constructed responses, whetherit is done manually by humans or automatically byNLP systems.

A key concept here is the idea of a “construct”which is defined as a set of related knowledge,skills, and other abilities that a test is designed tomeasure. Examples of possible constructs includelogical reasoning, language proficiency, readingcomprehension etc. A fair test is one where dif-ferences in test scores between the test-takers aredue only to differences in skills which are part ofthe construct. Any consistent differences in scoresbetween different groups of test takers that resultfrom other factors not immediately related to theconstruct (i.e., “construct-irrelevant”) — e.g., test-taker gender — may indicate that the test is unfair.Specifically, for a test to be fair, the non-randomeffects of construct-irrelevant factors need to beminimized during the four major phases of a test:test development, test administration, test scoring,and score interpretation (Xi, 2010; Zieky, 2016):

1. Test development. All tests must be freeof bias, i.e., no questions on a test shouldinclude any content that may advantage ordisadvantage any specific subgroup of test-takers in ways that are unrelated to the con-struct the test is designed to assess. The sub-groups in this case are defined based on fac-tors that include test-taker personal informa-tion such as gender, race, or disability, butmay also go beyond the standard protectedproperties. For example, Xi (2010) discusseshow familiarity with the subject matter in anEnglish language proficiency test may impacttest performance and, thus, would require anexplicit analysis of fairness for a group de-fined by test-taker fields of study. Addition-ally, on the same test, test-takers whose na-tive languages use the Roman alphabet willhave an advantage over test-takers with nativelanguages based on other alphabets. How-ever, this advantage is allowable because it isrelevant to the construct of English compre-hension. To ensure bias-free questions, the

developers of the test conduct both qualita-tive and quantitative reviews of each question(Angoff, 2012; Duong and von Davier, 2013;Oliveri and von Davier, 2016; Zieky, 2016).

2. Test administration. All test-takers mustbe provided with comparable opportunities todemonstrate the abilities being measured bythe test. This includes considerations such asthe location and number of test centers acrossthe world, and whether the testing conditionsin each test center are standardized and se-cure. For example, Bridgeman et al. (2003)showed that, at least for some tests, exami-nee test scores may be affected by screen res-olution of the monitors used to administer thetest. This means that for such tests to be fair,it is necessary to ensure that all test-takers usemonitors with a similar configuration.

3. Test scoring. There should also be no biasin the test scores irrespective of whether theyare produced by human raters or by auto-mated scoring models. The unequal distribu-tion of social, economic, and educational re-sources means that some differences in per-formance across subgroups are to be ex-pected. However, differences large enough tohave practical consequences must be investi-gated to ensure that they are not caused byconstruct-irrelevant factors (AERA, 1999).

4. Score interpretation Finally, while mosttests tend to have a constant structure, theactual questions change regularly. Some-times several different versions of a test (“testforms”) exist in parallel. Even if two test-takers take different versions of a test, theirtest scores should still be comparable. Toachieve this, a separate statistical processprocess called “test equating” is often used toadjust for unintended differences in the diffi-culty of the test forms (Lee and von Davier,2013; Liu and Dorans, 2016). This processitself must also be investigated for fairness toensure that it does not introduce bias againstany group of test-takers.

In this paper, we focus on the third phase: thefairness of test scores as measured by the impactof construct-irrelevant factors. As Xi (2010) dis-cusses in detail, unfair decisions based on scoresassigned to test-takers from oft-disadvantaged

42

groups are likely to have profound consequences:they may be denied career opportunities and ac-cess to resources that they deserve. Therefore,it is important to ensure — among other things— that construct-irrelevant factors do not intro-duce systematic biases in test scores, irrespectiveof whether they are produced by human raters orby an automated scoring system.

Over the last few years, there has been a sig-nificant amount of work done on ensuring fair-ness, accountability, and transparency for machinelearned models from what is now referred to as theFATML community (Kamiran and Calders, 2009;Kamishima et al., 2012; Luong et al., 2011; Zemelet al., 2013). More recently, Friedler et al. (2016)proposed a formal framework for conceptualizingthe idea of fairness. Within that framework, theauthors define the idea of “structural bias”: theunequal treatment of subgroups when there is noclear mapping between the features that are easilyobservable for those subgroups (e.g., largely irrel-evant, culturally and historically defined charac-teristics) and the true features on which algorith-mic decisions should actually be based (the “con-struct”). Our conceptualization of fairness for au-tomated scoring models in this paper — avoidingsystematic biases in test scores across subgroupsdue to construct-irrelevant factors — fits perfectlyin this framework.

3 Detecting Biases in Automated Scoring

Human scoring of constructed responses is a sub-jective process. Among the factors that can im-pact the assigned scores are rater fatigue (Linget al., 2014), differences between novice and ex-perienced raters (Davis, 2015), and the effect ofraters’ linguistic background on their evaluationof the language skill being measured (Carey etal., 2011). Furthermore, the same response cansometimes receive different scores from differentraters. To guard against such rater inconsistencies,responses for high-stakes tests are often scored bymultiple raters (Wang and von Davier, 2014; Pen-field, 2016). Automated scoring of constructed re-sponses can overcome many of these issues inher-ent to human scoring: computers do not get tired,do not have personal biases, and can be config-ured to always assign the same score to a givenresponse.

However, recent studies in machine learninghave highlighted that algorithms often introduce

their own biases (Feldman et al., 2015) either dueto an existing bias in the training data or due toa minority group being inadequately representedin the training data. Automated scoring is cer-tainly not immune to such biases and, in fact,several studies have documented differing perfor-mance of automated scoring models for test-takerswith different native languages or with disabilities(Burstein and Chodorow, 1999; Bridgeman et al.,2012; Wang and von Davier, 2014; Wang et al.,2016; An et al., 2016; Loukina and Buzick, Inprint).

Biases can also arise because of techniques usedto develop new features for automated scoringmodels. The automated score may be based onfeatures which are construct-irrelevant despite be-ing highly correlated with the human scores in thetraining data. As an example, consider that moreproficient writers tend to write longer responses.Therefore, one almost always observes a consis-tent positive correlation between essay length andhuman proficiency score (Perelman, 2014; Sher-mis, 2014b). This is acceptable since verbal flu-ency — a correlate of response length — is consid-ered an important part of the writing proficiency.Yet, longer essays should not automatically re-ceive higher scores. Therefore, without propermodel validation to consider the relative impactof such features, decisions might be made that areunfair to test-takers.

On this basis, the psychometric guidelines re-quire that if automated scoring models are to beused for making high-stakes decisions for col-lege admissions or employment, the NLP re-searchers developing those models should performmodel validation to ensure that demographic andconstruct-irrelevant factors are not causing theirmodels to produce significant differences in scoresacross different subgroups of test-takers (Yang etal., 2002; Clauser et al., 2002; Williamson et al.,2012). This is exactly what fairness – as we defineit in this paper – purports to measure.

However, it is not easy for an NLP or ma-chine learning researcher to perform comprehen-sive model validation since they may be unfamil-iar with the required psychometric and statisticalchecks. The solution we propose is a tool thatincorporates both the standard machine learningpipeline necessary for building an automated scor-ing model and a set of psychometric and statis-tical analyses aimed at detecting possible bias in

43

engine performance. We believe that such a toolshould be open-source and non-proprietary so thatthe automated scoring community can not only au-dit the source code of the already available analy-ses to ensure their compliance with fairness stan-dards but also contribute new analyses.

We describe the design of such a tool in the restof the paper. Specifically, our tool provides the fol-lowing model validation functionality to NLP/MLresearchers working on automated scoring: (a)defining custom subgroups and examining differ-ences in the performance of the automated scor-ing model across these groups; (b) examining theeffect of construct-irrelevant factors on automatedscores; and (c) comparing the effects of such fac-tors in two different versions of the same scoringmodel, e.g., a version with a new feature added tothe model and a version without the same feature.

4 RSMTool

In this section, we present an open-source Pythontool called RSMTool developed by two of the au-thors for building and evaluating automated scor-ing models. The tool is intended for NLP re-searchers who have already extracted featuresfrom the responses and need to choose a learnerfunction and evaluate the performance as well asthe fairness of the entire scoring pipeline (thetraining data, the features, and the learner func-tion).

Once the responses have been represented as aset of features, automated scoring essentially be-comes a machine learning problem and NLP re-searchers are free to use any of the large numberof existing machine learning toolkits. However,most of those toolkits are general-purpose and donot provide the aforementioned fairness analyses.Instead, we leverage one such toolkit — scikit-learn (Pedregosa et al., 2011) — to build a toolthat integrates these fairness analyses directly intothe machine learning pipeline and researchers thenget them automatically in the form of a compre-hensive HTML report.

Note that the automated scoring pipeline builtinto the tool provides functionality for each stepof the process of building and evaluating auto-mated scoring models: (a) feature transformation,(b) manual and automatic feature selection, and(c) access to linear and non-linear learners fromscikit-learn as well as the custom linear learnerswe have implemented. In this paper, we will fo-

cus solely on the fairness-driven evaluation capa-bilities of the tool that are directly relevant to theissues we have discussed so far. Readers inter-ested in other parts of the RSMToolare referredto the comprehensive documentation available athttp://rsmtool.readthedocs.org.

Before we describe the fairness analyses im-plemented in the tool, we want to acknowledgethat there are many different ways in which re-searchers might approach building as well as eval-uating scoring models (Chen and He, 2013; Sher-mis, 2014a). The list of learners and fairness anal-yses the tool provides is not, and cannot be, ex-haustive. In fact, later in the paper, we discusssome analyses that could be implemented in futureversions of the tool since one of the core charac-teristics of the tool is its flexible architecture. See§4.4 for more details.

In the next section, we present in detail the anal-yses incorporated into RSMTool aimed at detectingthe various sources of biases we introduced earlier.As it is easier to show the analyses in the contextof an actual example, we use data from the HewlettFoundation Automated Student Assessment Prize(ASAP) competition on automated essay scoring(Shermis, 2014a).3 As our scoring model, weuse ordinary linear regression with features ex-tracted from the text of the essay; see Attali andBurstein (2006) for details of the features. Notethat since the original ASAP data does not con-tain any demographic information, we simulate anL1 attribute (the test-taker’s native language) forillustration purposes.4 The complete report auto-matically generated by RSMTool is available at:http://bit.ly/fair-tool. The report containslinks to the raw data used to generate it and toother input files needed to run RSMTool. We fo-cus on specific sections of the report below.

4.1 Differential Feature FunctioningIn order to evaluate the fairness of a machinelearning algorithm, Feldman et al. (2015) recom-mend preventive auditing of the training data todetermine if the resulting decisions will be fair, ir-respective of the machine learning model learnedfrom that training data. RSMTool incorporates sev-

3https:/www.kaggle.com/c/asap-aes/data/

4We believe it is more transparent to use a publicly avail-able dataset with simulated demographics, rather than a pro-prietary dataset with real demographics that cannot be sharedpublicly. The value added by the fairness analyses comesthrough in either case.

44

eral such auditing approaches borrowed from pre-vious research in both educational measurementand machine learning.

The first step in evaluating the fairness of an au-tomated scoring model is to ensure that the per-formance of each feature is not primarily deter-mined by construct-irrelevant factors. The tra-ditional way to approach this is to have an ex-pert review the features and ensure that their de-scription and method of computation are in linewith the definition of the specific set of skills thatthe given test purports to measure (Deane, 2013).However, features incorporated into a modern au-tomated scoring system often rely on multiple un-derlying NLP components such as part-of-speechtaggers and syntactic parsers as well as complexcomputational algorithms and, therefore, a quali-tative review may not be sufficient. Furthermore,some aspects of spoken or written text can only bemeasured indirectly given the current state of NLPtechnologies (Somasundaran et al., 2014).

RSMTool allows the user to explore the quan-titative effect of two types of construct-irrelevantfactors that may affect feature performance: cate-gorical and continuous.

4.1.1 Categorical FactorsThis group of factors generally includes variablesthat can take on one of a fixed number of possiblevalues, e.g., test-takers’ demographic characteris-tics, different versions of the same test question,or various testing conditions. We refer to thesefactors as “subgroups” though they are not alwayslimited to demographic subgroups.

When this information is available for all orsome of the responses, RSMTool allows the userto compare the feature distributions for differ-ent subgroups using box-plots and other distri-butional statistics such as mean and standard de-viations. However, feature distributions dependon the scores which may differ across subgroupsand, therefore, differences in a feature’s distri-bution across subgroups may not always indicatethat the feature is biased. To address this, RSM-Tool also includes Differential feature functioning(DFF) analysis (Penfield, 2016; Zhang et al., Inprint). This approach compares the mean values ofa given feature for test-takers with the same scorebut belonging to different subgroups. These dif-ferences can be described and reviewed directlyusing DFF line plots. Figure 1(a) shows a box-plot for the distribution of the GRAMMAR feature

by test-taker L1 subgroups in our sample dataset;Figure 1(b) shows a DFF line plot for the samefeature. These plots indicate that the values forthe GRAMMAR feature are consistently lower forone of the test-taker subgroups (L1=Hindi) acrossall score levels. If such a pattern were observed inreal data, it would warrant further investigation toestablish the reasons for such behavior.

4.1.2 Continuous FactorsThis type of construct-irrelevant factors includescontinuous covariates which despite being corre-lated with human scores are either not directly rel-evant to the construct measured by the test or, evenif they are, should not be the primary contributorto the model’s predictions. Response length, aspreviously discussed, is an example of such co-variates. Even though it provides an importantindication of verbal fluency, a model which pre-dominantly relies on length will not generate fairscores. To explore the impact of such factors,RSMTool computes two types of correlations: (a)the marginal correlation between each feature andthe covariate, and (b) the “partial” correlation be-tween each feature and the human score, with theeffects of the covariate removed (Cramér, 1947).This helps to clearly bring out the contributionof a feature above and beyond being a proxy forthe identified covariate. The marginal and par-tial correlation coefficients for our example areshown in Figure 1(c). It shows that although allfeatures in our simulated dataset contribute infor-mation beyond response length, for some features,length accounts for a substantial part of their per-formance.

4.2 Bias in Model Performance

Not all types of machine learning algorithms lendthemselves easily to the differential feature func-tioning analysis. Furthermore, the sheer numberof features in some models may make the resultsof such analyses difficult to interpret. Therefore,a second set of fairness analyses included intoRSMTool considers how well the automated scoresagree with the human scores (or another, user-specified gold standard criterion) and whether thisagreement is consistent across different groups oftest-takers.

RSMTool computes all the standard evaluationmetrics generally used for regression-based ma-chine learning models such as Pearson’s correla-tion coefficient (r), coefficient of determination

45

(a) Box-plots showing the distribution of standard-ized GRAMMAR feature values by test-taker na-tive language (L1). The dotted red lines representthe thresholds for outlier truncation computed asthe mean feature value ± 4 standard deviations.

(b) A differential feature functioning (DFF) plotfor the GRAMMAR feature. Each line representsan L1; each point shows the mean and 95% confi-dence intervals of the feature values computed fortest-takers with that L1 and that assigned score.

(c) Pearson’s correlation coefficients (r) between features and human scores: (a) Marginal: marginal correlationof each feature with human score (b) Partial − all: correlation of each feature with human score with the effectsof all other features removed, and (c) Partial − length: the correlation of each feature with human score withthe effect of response length removed. The two dotted lines represent correlations thresholds recommended byWilliamson et al. (2012).

Figure 1: Examples of RSMTool fairness analyses for categorical and continuous factors.

46

(R2), and root mean squared error (RMSE). Inaddition, it also computes other measures that arespecifically recommended in psychometric liter-ature for evaluating automated scoring models:quadratically-weighted kappa, percentage agree-ment with human scores, and the standardizedmean difference (SMD) between human and au-tomated scores (Williamson et al., 2012; Rami-neni and Williamson, 2013). These metrics arecomputed for the whole evaluation set as well asfor each subgroup separately in order to evaluatewhether the accuracy of automated scores is con-sistent across different groups of test-takers. Fig-ure 2 shows a plot illustrating how the model R2

computed on the evaluation set varies across thedifferent test-taker L1 subgroups.

Figure 2: The performance of our scoring model(R2) for different subgroups of test-takers as de-fined by their native language (L1). Before com-puting the R2, the predictions of the model aretrimmed and then re-scaled to match the humanscore distribution in the training data.

4.3 Model comparisonLike any other software, automated scoring sys-tems are updated on a regular basis as researchersdevelop new features or identify better machinelearning algorithms. Even in scenarios where newfeatures or algorithms are not needed, changes inexternal dependencies used by the scoring pipelinemight necessitate new releases. Automated scor-ing models may also be regularly re-trained to

avoid population drift which can occur when thetest-taker population used to train the model nolonger matches the population currently evaluatedby this model.

When updating an automated scoring systemfor one of the above reasons, one should not onlyconduct a fairness analysis for the new versionof the model, but also a comprehensive compar-ison of the old and the new version. For example,a change in the percentage of existing test-takerswho have passed a particular test resulting fromthe update would need to be explained not only tothe test-takers but also to the people making deci-sions based on test scores (von Davier, 2016).

RSMTool includes the functionality to conducta comprehensive comparison of two different ver-sions of a scoring system and produce a reportwhich includes fairness analyses for each of theversions as well as how these analyses differ be-tween the two versions. As an example, we com-pare two versions of our example scoring model— one that uses all features and another that doesnot include the GRAMMAR feature. The compar-ison report can be be seen here: http://bit.ly/fair-tool-compare.

4.4 Customizing RSMTool

The measurement guidelines currently imple-mented in RSMTool follow the psychometricframework suggested by Williamson et al. (2012).It was developed for the evaluation of e-rater, anautomated system designed to score English writ-ing proficiency (Attali and Burstein, 2006), but isgeneralizable to other applications of automatedscoring. This framework was chosen because itoffers a comprehensive set of criteria for both theaccuracy as well as the fairness of the predictedscores. Note that not all of these recommenda-tions are universally accepted by the automatedscoring community. For example, Yannakoudakisand Cummins (2015) recently proposed a differentset of metrics for evaluating the accuracy of auto-mated scoring models.

Furthermore, the machine learning communityhas recently developed various analyses aimed atdetecting bias in algorithm performance that couldbe applied in the context of automated scoring.For example, in addition to reviewing individualfeatures, one could also attempt to predict thesubgroup membership from the features used toscore the responses (Feldman et al., 2015). If this

47

prediction is generally accurate, then there is arisk that subgroup membership could be implic-itly used by the scoring model and lead to unfairscores. However, if the subgroup prediction hashigh error over all models generated from the fea-tures, then the scores assigned by a model trainedon this data are likely to be fair.

RSMTool has been designed to make it easy forthe user to add new evaluations and analyses ofthese types. The evaluation and report-generationcomponents of RSMTool (including the fairnessanalyses) can be run on predictions from any ex-ternal learner, not just the ones that are providedby the tool itself. Each section of its report isimplemented as a separate Jupyter/IPython note-book (Kluyver et al., 2016). The user can choosewhich sections should be included into the finalHTML report and in which order. Furthermore,NLP researchers who want to use different evalu-ation metrics or custom fairness analyses can pro-vide them in the form of new Jupyter notebooks;these analyses are dynamically executed and in-corporated into the final report along with the ex-isting analyses or even in their place, if so desired,without modifying a single line of code.

Finally, for those who want to make more sub-stantive changes, the tool is written entirely inPython, is open-source with an Apache 2.0 li-cense, and has extensive online documentation.We also provide a well-documented API whichallows users to integrate various components ofRSMTool into their own applications.

4.5 Model Transparency & Interpretability

The analyses produced by RSMTool only suggesta potential bias and flag individual subgroups orfeatures for further consideration. As we indi-cated earlier, the presence of differences acrosssubgroups does not automatically imply that themodel is unfair; further review is required to estab-lish the source of such differences. One of the firststeps in such a review usually involves examiningeach feature separately as well as the individualcontribution of each feature to the final score. It isimportant to note here that unfairness may also beintroduced by what is not in the model. An auto-mated scoring system may not cover a particularaspect of the construct which can be evaluated byhumans. If the performance across subgroups dif-fers on this aspect of the construct, the differencemay be due to “construct under-representation”

rather than due to construct-irrelevant factors.

The automated scoring models used in sys-tems such as e-rater for assessing writing profi-ciency in English (Attali and Burstein, 2006) orSpeechRater for spoken proficiency (Zechner etal., 2009) have traditionally been linear modelswith a small number of interpretable features be-cause such models lend themselves more easilyto a detailed fairness review and allow decision-makers to understand how, and to what extent, dif-ferent parts of the test-takers’ skill set are beingcovered by the features in the model (Loukina etal., 2015). For such linear models, RSMTool dis-plays a detailed model description including themodel fit (R2) computed on the training set as wellas the contribution of each feature to the final score(via raw, standardized, and relative coefficients).

At the same time, recent studies (Heilman andMadnani, 2015; Madnani et al., 2016) on scor-ing actual content rather than just language profi-ciency suggest that it is possible to achieve higherperformance, as measured by agreement with hu-man raters, by employing many low-level fea-tures and more sophisticated machine learning al-gorithms such as support vector machines or ran-dom forests. Generally, these models are built us-ing sparse feature types such as word n-grams, of-ten resulting in hundreds of thousands of predom-inantly binary features. Using models with sucha large feature space means that it is no longerclear how to map the individual features and theirweights to various parts of the test-takers’ skill set,and, therefore, difficult to identify whether anydifferences in the model performance stem fromthe effects of construct-irrelevant factors.

One way to increase the interpretability of suchmodels is to group multiple features by featuretype (e.g. “syntactic relationships”) and build astacked model (Wolpert, 1992) containing sim-pler models for each feature type. These stackedmodels can then be combined in a final linearmodel which can be examined in the usual mannerfor fairness considerations (Madnani and Cahill,2016). The idea of making complex machine-learned models more interpretable to users andstakeholders has been investigated more thor-oughly in recent years and several promising so-lutions have been proposed that could also be usedfor content-scoring models (Kim et al., 2016; Wil-son et al., 2016).

48

5 Mitigating Bias in Automated Scoring

So far we have primarily discussed techniquesfor detecting potential biases in automated scoringmodels. We showed that there are multiple sourcesof possible bias which makes it unlikely that therewould be a single “silver bullet” that can maketest scores completely bias-free. The approachcurrently favored in the educational measurementcommunity is to try and reduce susceptibility toconstruct-irrelevant factors by design. This in-cludes an expert review of each feature before it isadded to the model to ensure that it is theoreticallyand practically consistent with the skill set beingmeasured by the test. These features are then com-bined in an easily interpretable model (usually lin-ear regression) which is trained on a representativesample of test-taker population.

However, simpler scoring models may not al-ways be the right solution. For one, as we dis-cussed in §3, several studies have shown thateven such simple models may still exhibit bias.In addition, recent studies on scoring test-takers’knowledge of content rather than proficiency haveshown that using more sophisticated — and hence,less transparent — models yields non-trivial gainsin the accuracy of the predicted scores. There-fore, ensuring completely fair automated scoringat large requires more complex solutions.

The machine learning community has identifiedseveral broad approaches to deal with discrimina-tion that could, in theory, be used for automatedscoring models, especially those using more com-plex non-linear algorithms: the training data canbe modified (Feldman et al., 2015; Kamiran andCalders, 2012; Hajian and Domingo-Ferrer, 2013;Mancuhan and Clifton, 2014), the algorithm it-self can be changed so that it optimizes for fair-ness as well as the selection criteria (Kamishima etal., 2012; Zemel et al., 2013; Calders and Verwer,2010), and the output decisions can be changedafter-the-fact (Kamiran et al., 2012). A survey ofsuch approaches is provided by Romei and Rug-gieri (2013a). Future work in automated scoringcould explore whether these methods can addresssome of the known biases.

Of course, it is also important to note that suchbias-mitigating approaches often lead to a declinein the overall model performance and, therefore,one needs to balance model fairness and accuracywhich likely depends on the stakes for which themodel is going to be used.

6 Conclusion

In this paper, we discussed considerations thatgo into developing fairer automated scoring mod-els for constructed responses. We also presentedRSMTool, an open-source tool to help NLP re-searchers detect potential biases in their scoringmodels. We described the analyses currently in-corporated into the tool for evaluating the impactof construct-irrelevant categorical and continuousfactors. We also showed that the tool is designed ina flexible manner which allows users to easily addtheir own custom fairness analyses and showedsome examples of such analyses.

While RSMTool has been designed for auto-mated scoring research (some terminology in thetool and the report is specific to automated scor-ing), its flexible nature and well-documented APIallow it to be easily adapted for any machine learn-ing task in which the numeric prediction is gener-ated by regressing on a set of non-sparse, numericfeatures. Furthermore, the evaluation componentcan be used separately which allows users to eval-uate the performance and fairness of any modelthat generates numeric predictions.

Acknowledgments

We would like to thank the anonymous review-ers for their valuable comments and sugges-tions. We would also like to thank Lei Chen,Sorelle Friedler, Brent Bridgeman, Vikram Rama-narayanan, and Keelan Evanini for their contribu-tions and comments.

References

AERA. 1999. Standards for Educational and Psy-chological Testing. American Educational ResearchAssociation.

Ji An, Vincent Kieftenbeld, and Raghuveer Kan-neganti. 2016. Fairness in Automated Scoring:Screening Features for Subgroup Differences. Pre-sented at the Annual Meeting of the National Coun-cil on Measurement in Education, Washington DC.

William H. Angoff. 2012. Perspectives on Differen-tial Item Functioning Methodology. In P.W. Hollandand H. Wainer, editors, Differential Item Function-ing, pages 3–23. Taylor & Francis.

Yigal Attali and Jill Burstein. 2006. Automated EssayScoring with e-rater V. 2. The Journal of Technol-ogy, Learning and Assessment, 4(3):1–30.

49

Jared Bernstein, A. Van Moere, and Jian Cheng. 2010.Validating Automated Speaking Tests. LanguageTesting, 27(3):355–377.

Brent Bridgeman, Mary Lou Lennon, and AltameseJackenthal. 2003. Effects of Screen Size, ScreenResolution, and Display Rate on Computer-BasedTest Performance. Applied Measurement in Educa-tion, 16(3):191–205.

Brent Bridgeman, Catherine Trapani, and Yigal Attali.2012. Comparison of Human and Machine Scor-ing of Essays: Differences by Gender, Ethnicity,and Country. Applied Measurement in Education,25(1):27–40.

Jill Burstein and Martin Chodorow. 1999. Automatedessay scoring for nonnative english speakers. InProceedings of a Symposium on Computer Medi-ated Language Assessment and Evaluation in Nat-ural Language Processing, pages 68–75, Strouds-burg, PA, USA.

Jill Burstein, Karen Kukich, Susanne Wolff, ChiLu, Martin Chodorow, Lisa Braden-Harder, andMary Dee Harris. 1998. Automated Scoring Us-ing A Hybrid Feature Identification Technique. InProceedings of the 36th Annual Meeting of the As-sociation for Computational Linguistics and 17th In-ternational Conference on Computational Linguis-tics, Volume 1, pages 206–210, Montreal, Que-bec, Canada, August. Association for Computa-tional Linguistics.

Toon Calders and Sicco Verwer. 2010. Three NaiveBayes Approaches for Discrimination-Free Classi-fication. Data Mining journal; special issue withselected papers from ECML/PKDD.

M. D. Carey, R. H. Mannell, and P. K. Dunn. 2011.Does a Rater’s Familiarity with a Candidate’s Pro-nunciation Affect the Rating in Oral Proficiency In-terviews? Language Testing, 28(2):201–219.

Hongbo Chen and Ben He. 2013. Automated EssayScoring by Maximizing Human-Machine Agree-ment. In Proceedings of the 2013 Conference onEmpirical Methods in Natural Language Process-ing, pages 1741–1752, Seattle, Washington, USA,October. Association for Computational Linguistics.

Brian E. Clauser, Michael T. Kane, and David B. Swan-son. 2002. Validity Issues for Performance-BasedTests Scored With Computer-Automated ScoringSystems. Applied Measurement in Education,15(4):413–432.

Harald Cramér. 1947. Mathematical Methods ofStatistics. Princeton University Press.

Larry Davis. 2015. The Influence of Training and Ex-perience on Rater Performance in Scoring SpokenLanguage. Language Testing, 33:117–135.

Paul Deane. 2013. On the Relation between Auto-mated Essay Scoring and Modern Views of the Writ-ing Construct. Assessing Writing, 18(1):7–24.

Minh Q. Duong and Alina von Davier. 2013. Hetero-geneous Populations and Multistage Test Design. InRoger E. Millsap, L. Andries van der Ark, Daniel M.Bolt, and Carol M. Woods, editors, New Devel-opments in Quantitative Psychology: Presentationsfrom the 77th Annual Psychometric Society Meeting,pages 151–170. Springer New York.

Michael Feldman, Sorelle A. Friedler, John Moeller,Carlos Scheidegger, and Suresh Venkatasubrama-nian. 2015. Certifying and Removing Disparate Im-pact. Proceedings of the 21th ACM SIGKDD Inter-national Conference on Knowledge Discovery andData Mining, pages 259–268.

Sorelle A. Friedler, Carlos Scheidegger, and SureshVenkatasubramanian. 2016. On the (Im)possibilityof Fairness. CoRR, abs/1609.07236.

Sara Hajian and Josep Domingo-Ferrer. 2013. AMethodology for Direct and Indirect Discrimina-tion Prevention in Data Mining. IEEE Transactionson Knowledge and Data Engineering, 25(7):1445 –1459.

Michael Heilman and Nitin Madnani. 2015. TheImpact of Training Data on Automated Short An-swer Scoring Performance. In Proceedings of theTenth Workshop on Innovative Use of NLP for Build-ing Educational Applications, pages 81–85, Denver,Colorado, June. Association for Computational Lin-guistics.

Faisal Kamiran and Toon Calders. 2009. Classifyingwithout Discriminating. In Proceedings of the IEEEInternational Conference on Computer, Control andCommunication, pages 1–6.

Faisal Kamiran and Toon Calders. 2012. Data Prepro-cessing Techniques for Classification Without Dis-crimination. Knowledge and Information Systems,33(1):1 – 33.

Faisal Kamiran, Asad Karim, and Xiangliang Zhang.2012. Decision Theory for Discrimination-awareClassification. In International Conference on DataMining (ICDM).

Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh,and Jun Sakuma. 2012. Fairness-aware Classifierwith Prejudice Remover Regularizer. In Proceed-ings of the Joint European Conference on MachineLearning and Knowledge Discovery in Databases,pages 35–50.

Been Kim, Dmitry Malioutov, and Kush Varshney, ed-itors. 2016. Proceedings of the ICML Workshop onHuman Interpretability in Machine Learning.

Thomas Kluyver, Benjamin Ragan-Kelley, FernandoPrez, Brian Granger, Matthias Bussonnier, JonathanFrederic, Kyle Kelley, Jessica Hamrick, JasonGrout, Sylvain Corlay, Paul Ivanov, Damin Avila,

50

Safia Abdalla, Carol Willing, and Jupyter Develop-ment Team. 2016. Jupyter Notebooks — A Publish-ing Format for Reproducible Computational Work-flows. In Proceedings of the 20th International Con-ference on Electronic Publishing. IOS Press.

Yi-Hsuan Lee and Alina von Davier. 2013. Moni-toring Scale Scores over Time via Quality ControlCharts, Model-Based Approaches, and Time SeriesTechniques. Psychometrika, 78(3):557–575.

G. Ling, P. Mollaun, and X. Xi. 2014. A Study on theImpact of Fatigue on Human Raters when ScoringSpeaking Responses. Language Testing, 31:479–499.

Jinghua Liu and Neil J. Dorans. 2016. Fairness inScore Interpretation. In Neil J. Dorans and Linda L.Cook, editors, Fairness in educational assessmentand measurement, pages 77–96. Routledge.

Anastassia Loukina and Heather Buzick. In print. Au-tomated Scoring of Speakers with Speech Impair-ments. ETS Research Report Series, In print.

Anastassia Loukina, Klaus Zechner, Lei Chen, andMichael Heilman. 2015. Feature Selection forAutomated Speech Scoring. In Proceedings of theTenth Workshop on Innovative Use of NLP for Build-ing Educational Applications, pages 12–19, Denver,Colorado, June. Association for Computational Lin-guistics.

Binh Thanh Luong, Salvatore Ruggieri, and FrancoTurini. 2011. k-NN as an Implementation of Situ-ation Testing for Discrimination Discovery and Pre-vention. In Proceedings of the 17th ACM SIGKDDinternational conference on Knowledge discoveryand data mining, pages 502–510.

Nitin Madnani and Aoife Cahill. 2016. AutomatedScoring of Content. Presented at the panel on Fair-ness and Machine Learning for Educational Prac-tice, Annual Meeting of the National Council onMeasurement in Education, Washington DC.

Nitin Madnani, Aoife Cahill, and Brian Riordan. 2016.Automatically Scoring Tests of Proficiency in MusicInstruction. In Proceedings of the 11th Workshopon Innovative Use of NLP for Building EducationalApplications, pages 217–222, San Diego, CA, June.Association for Computational Linguistics.

Koray Mancuhan and Chris Clifton. 2014. Combat-ing Discrimination Using Bayesian Networks. Artif.Intell. Law, 22(2):211–238.

Mara Elena Oliveri and Alina von Davier. 2016. Psy-chometrics in Support of a Valid Assessment of Lin-guistic Minorities: Implications for the Test andSampling Designs. International Journal of Testing,16(3):220–239.

Ellis B. Page. 1966. The Imminence of ... Grad-ing Essays by Computer. The Phi Delta Kappan,47(5):238–243.

Fabian Pedregosa, Gal Varoquaux, Alexandre Gram-fort, Vincent Michel, Bertrand Thirion, OlivierGrisel, Mathieu Blondel, Peter Prettenhofer, RonWeiss, Vincent Dubourg, Jake Vanderplas, Alexan-dre Passos, David Cournapeau, Matthieu Brucher,Matthieu Perrot, and douard Duchesnay. 2011.Scikit-learn: Machine learning in Python. Journalof Machine Learning Research, 12:2825–2830.

Randall D. Penfield. 2016. Fairness in Test Scoring. InNeil J. Dorans and Linda L. Cook, editors, Fairnessin Educational Assessment and Measurement, pages55–76. Routledge.

Les Perelman. 2014. When the state of the art isCounting Words. Assessing Writing, 21:104–111.

Chaitanya Ramineni and David M. Williamson. 2013.Automated Essay Scoring: Psychometric Guidelinesand Practices. Assessing Writing, 18(1):25–39.

Andrea Romei and Salvatore Ruggieri. 2013a. AMultidisciplinary Survey on Discrimination Analy-sis. The Knowledge Engineering Review, pages 1–57.

Andrea Romei and Salvatore Ruggieri. 2013b. Dis-crimination Data Analysis: A Multi-disciplinaryBibliography. In Bart Custers, Toon Calders, BartSchermer, and Tal Zarsky, editors, Discriminationand Privacy in the Information Society: Data Min-ing and Profiling in Large Databases, pages 109–135. Springer Berlin Heidelberg.

Mark D. Shermis. 2014a. State-of-the-art AutomatedEssay Scoring: Competition, Results, and Future Di-rections from a United States demonstration. As-sessing Writing, 20:53–76.

Mark D. Shermis. 2014b. The Challenges of Emu-lating Human Behavior in Writing Assessment. As-sessing Writing, 22:91–99.

Swapna Somasundaran, Jill Burstein, and MartinChodorow. 2014. Lexical Chaining for MeasuringDiscourse Coherence Quality in Test-taker Essays.In Proceedings of COLING 2014, the 25th Inter-national Conference on Computational Linguistics:Technical Papers, pages 950–961, Dublin, Ireland,August. Dublin City University and Association forComputational Linguistics.

Alina von Davier and Jill Burstein. 2016. Fairnessand Machine Learning for Educational Practice. Co-ordinated Session, Annual meeting of the NationalCouncil on Measurement in Education, WashingtonDC.

Alina von Davier. 2016. Fairness Concerns in Com-putational Psychometrics. Presented at the panelon Fairness and Machine Learning for EducationalPractice, Annual Meeting of the National Councilon Measurement in Education, Washington DC.

51

Zhen Wang and Alina von Davier. 2014. Monitor-ing of scoring using the e-rater automated scoringsystem and human raters on a writing test. ETS Re-search Report Series, 2014(1):1–21.

Zhen Wang, Klaus Zechner, and Yu Sun. 2016. Mon-itoring the Performance of Human and AutomatedScores for Spoken Responses. Language Testing,pages 1–20.

David M. Williamson, Xiaoming Xi, and F. Jay Breyer.2012. A Framework for Evaluation and Use of Au-tomated Scoring. Educational Measurement: Issuesand Practice, 31(1):2–13.

Andrew Gordon Wilson, Been Kim, and William Her-lands, editors. 2016. Proceedings of the NIPS Work-shop on Interpretable Machine Learning for Com-plex Systems.

David H. Wolpert. 1992. Stacked Generalization.Neural Networks, 5:241–259.

Xiaoming Xi. 2010. How do we go about InvestigatingTest Fairness? Language Testing, 27(2):147–170.

Yongwei Yang, Chad W. Buckendahl, Piotr J.Juszkiewicz, and Dennison S. Bhola. 2002. AReview of Strategies for Validating Computer-Automated Scoring. Applied Measurement in Ed-ucation, 15(4):391–412, oct.

Helen Yannakoudakis and Ronan Cummins. 2015.Evaluating the Performance of Automated TextScoring systems. In Proceedings of the Tenth Work-shop on Innovative Use of NLP for Building Edu-cational Applications, pages 213–223, Denver, Col-orado, June. Association for Computational Linguis-tics.

Klaus Zechner, Derrick Higgins, Xiaoming Xi, andDavid M. Williamson. 2009. Automatic Scoring ofNon-native Spontaneous Speech in Tests of SpokenEnglish. Speech Communication, 51(10):883–895.

Richard S Zemel, Yu Wu, Kevin Swersky, ToniannPitassi, and Cynthia Dwork. 2013. Learning fairrepresentations. In Proceedings of ICML, pages325–333.

Mo Zhang, Neil J. Dorans, Chen Li, and Andre A.Rupp. In print. Differential feature functioning inautomated essay scoring. In H. Jiao and R.W. Lis-sitz, editors, Test fairness in the new generation oflarge-scale assessment.

Michael J. Zieky. 2016. Fairness in Test Designand Development. In Neil J. Dorans and Linda L.Cook, editors, Fairness in Educational Assessmentand Measurement, pages 9–32. Routledge.

52