extending the machine learning abstraction boundary: a … · 2020. 6. 18. · extending the...

Extending the Machine Learning Abstraction Boundary:A Complex Systems Approach to Incorporate Societal Context

Donald Martin, Jr.Google

Vinodkumar PrabhakaranGoogle

Jill KuhlbergSystem Stars

Andrew SmartGoogle

William S. IsaacDeepMind

ABSTRACTMachine learning (ML) fairness research tends to focus primarilyon mathematically-based interventions on often opaque algorithmsor models and/or their immediate inputs and outputs. Such oversim-plified mathematical models abstract away the underlying societalcontext where ML models are conceived, developed, and ultimatelydeployed. As fairness itself is a socially constructed concept thatoriginates from that societal context along with the model inputsand the models themselves, a lack of an in-depth understanding ofsocietal context can easily undermine the pursuit of ML fairness.In this paper, we outline three new tools to improve the compre-hension, identification and representation of societal context. First,we propose a complex adaptive systems (CAS) based model anddefinition of societal context that will help researchers and prod-uct developers to expand the abstraction boundary of ML fairnesswork to include societal context. Second, we introduce collaborativecausal theory formation (CCTF) as a key capability for establishinga sociotechnical frame that incorporates diverse mental models andassociated causal theories in modeling the problem and solutionspace for ML-based products. Finally, we identify community basedsystem dynamics (CBSD) as a powerful, transparent and rigorousapproach for practicing CCTF during all phases of the ML productdevelopment process. We conclude with a discussion of how thesesystems theoretic approaches to understand the societal contextwithin which sociotechnical systems are embedded can improvethe development of fair and inclusive ML-based products.

KEYWORDSComplex adaptive systems, ML system design, ML fairness, systemsthinking, system dynamics

1 INTRODUCTIONThe last decade has seen tremendous growth in the field of arti-ficial intelligence (AI), resulting in renowned scholars and worldleaders considering it a critical element of an ongoing fourth indus-trial/technological revolution [30, 94]. In large part this revolutionhas been driven by recent advancements, such as deep learning, inmachine learning model design and development. However, as thepace of adoption for these technologies accelerates, so too have con-cerns regarding the fairness, accountability and ethics of machinelearning (ML)models and algorithms both within the academic com-munity [13, 20, 38, 43, 61, 82] and among the general public [3].1,2 Agrowing body of research on machine learning fairness attempts to

1shorturl.at/ouJO42shorturl.at/bgxCK

build fairer machine learning systems, however it has been pointedout that these attempts primarily focus on the algorithms and mod-els, and their immediate inputs and outputs [96]. The limitations ofthis observational, statistical approach, when considering norma-tive, constitutive, process-oriented, socially-constructed conceptssuch as fairness, equity, and ethics [55], has been a recurring topicin recent fair-ML research [21, 29, 41, 87, 109].

The challenge of reconciling abstracted social and political con-siderations related to technological development is neither novel orlimited to machine learning. Human-Computer Interaction (HCI)and Science and Technology studies (STS) scholars have long high-lighted the struggle of technologists to identify and incorporatethese factors into their development processes [14, 49, 57, 58, 64].More recently, ML fairness scholars have argued the current MLsystem design processes exhibit a bias toward abstracting “awaythe social context in which systems will be deployed” [96] in pur-suit of manageable technical problems. This approach is fraughtwith ethical risks, as ignoring social factors could potentially leadto further exacerbating or introducing new harms in the socialcontext in which the systems are deployed. However, the fact thatresearchers interchangeably refer to the concept of social contextas the “sociotechnical puzzle” [96], “complex social reality” [16]and “the broader context” [60] illustrates a lack of clarity on whatsocial context is. This lack of clarity contributes to the tendency toabstract away social context during ML system design.

In order to combat the tendency to abstract away social context,this paper seeks to re-frame social context as a socio-cultural layer —which we will refer to as societal context — of the complex environ-ment in which all technical systems and the social actors that createand are affected by them, exist and interact. Specifically, we intro-duce and leverage the multidisciplinary complex adaptive system(CAS) theory to develop a taxonomy model of societal context.

Next, we leverage the taxonomy model and fundamentals ofproduct development processes to propose the concept of collabora-tive causal theory formation (CCTF), which we identify as a neededcapability for incorporating societal context into the ML systemdesign process in partnership with other (often excluded) stakehold-ers. We focus particularly on operationalizing CCTF through theuse of system dynamics (SD) [34, 101], which is a transparent andrigorous visual and analytical tool for facilitating recursive engage-ment among diverse stakeholders. In practice, SD and bottom-upvariants such as community based system dynamics (CBSD) areanalogous to other efforts in the ML fairness community [7, 111]seeking to aid developers and researchers who desire working aspartners with impacted stakeholders to develop greater perspectiveon the social and ethical dimensions of their research and products.

1

arX

iv:2

006.

0966

3v1

[cs

.CY

] 1

7 Ju

n 20

20

shorturl.at/ouJO4

shorturl.at/bgxCK

Martin et al. 2020

2 MODELING SOCIETAL CONTEXTIncorporating socially constructed concepts such as fairness andethics into a machine learning system design process requires adeep understanding of the societal context within which the systemwill operate. However, currently, within the ML system develop-ment ecosystem there is no concrete definition of societal context,nor a description of its key features and characteristics. We ar-gue that these elements are prerequisites for developing effectivestrategies and identifying useful frameworks that researchers andpractitioners can leverage to extend the ML system design abstrac-tion boundary to encompass societal context.

To make progress on defining societal context we must choose aperspective from which to think about what society is and what itskey elements and characteristics are. Models are an effective wayfor communicating, explaining and reasoning about the featuresand characteristics of complex concepts [74]. While the idea ofmodelling elements of the society has been actively researched fordecades [11, 69, 95], based on the groundwork laid by sociologistWalter Buckley, who asserted in 1968 that society was a complexadaptive system (CAS) [12], we leverage CAS theory to identify thesalient features and characteristics of societal context.

2.1 Complex Adaptive Systems (CAS)Complex Adaptive System theory has its origins in general systemstheory [108] — which emerged in the 1950s as a cohesive interdisci-plinary approach to study systems in all fields of science — and is aloosely organized field of study often bundled into the broader fieldof complexity science. Complex adaptive systems are complex inthe sense that they are comprised of components that are directlyor indirectly related in a causal network, and the behaviour of thesystem cannot be predicted based solely on the behaviour of itscomponents, and are adaptive in the sense that they adapt to thechanges in their environment by mutating or self-organizing theirinternal structures [5, 22, 23, 44, 68]. Examples cited in literature forCAS vary from small biological systems such as the cell, the embryo,the brain, and the immune system, to large social systems such asant colonies, social networks, organizations, and governments.

A detailed exposition of CAS and its various applications isoutside the scope of this paper.3 However, the key characteristicsof CAS include:• Complex: large number of active elements that continuouslyinteract through information and/or energy exchange.

• Distributed Control: individual elements of the CAS are nec-essarily unaware or oblivious to the system as a whole. Eachelement interacts with and reacts to its own local environmentand is governed by its own rules [22]. For example, an ant colonyis a CAS comprised of many individual ants (CAS in and of them-selves); no individual ant has the master plan for the complexnests the colony builds or has knowledge of each individual antsmotives or behaviors.

• Aggregation: individual elements of CAS combine to form aggre-gate elements. Aggregated elements at one level of organization3CAS is a large and deep discipline with many aspects we do not address in this

paper. For example, self-organization, chaotic behavior, fat tailed behavior and powerlaw scaling - key elements of CAS - are not necessary to detail for the purpose of thispaper, which is the introduction of our framework combining elements of CAS andcommunity based system dynamics

become building blocks for emergent aggregate properties ata higher level leading to hierarchical organization[44, 45]. Forexample amino acids combine to form proteins, proteins com-bine to form organelles and so on until ants and ant colonies areultimately formed.

• Adaptive: elements update their structures in order to adapt tothe constantly changing environment that results from elementinteractions.

• Non-linearity: interactions between the elements is often non-linear; small changes can have large effects in the system.

• Feedback loops: interactions are characterized by feedbackloops between elements that can be positive (reinforcing, ampli-fying) or negative (inhibiting, restraining).

• Time delays: interactions between elements may often involvetime delays; interventions and their impact on the system maynot be observed for months or years.

• History: system elements and the overall system have the abilityto store state and history so that the past helps to shape presentbehaviour[63].

• Stochastic: the system behavior may be inherently stochasticsince each element can have randomness in their inner struc-tures/processes.

• Emergence: The system elements interact in stochastic waysbut patterns emerge from these interactions in ways that arecounter-intuitive and hard to predict [100].

2.1.1 Key Element Types of CAS. Although CAS theorists agreethat adaptive agents are a primary element type of CAS there isno definitive, agreed upon taxonomy of CAS element types. Forthe purposes of this paper we have synthesized the various otherCAS element types proposed in literature [5, 44, 45] into two ad-ditional key element types to complement agents. Specifically weutilize the term precepts to encompass “internal models” [44, 45] and“strategies” [5], and the term artifacts to encompass “signals/tags”[44, 45] and “artifacts” [5]. Distinguishing these types helps us de-velop a richer representation for societal context, that for instance,separates objectives of agents, from mechanisms of precepts, andoutcomes manifested as artifacts. However, it is important to realizethat these distinctions are not always rigid, since some instances ofthese element types can incorporate properties of more than oneelement type. The following definitions for each element type willhighlight such instances.

Agents are the “bounded subsystems capable of interior process-ing” [45, 69]. Agents can be inorganic and inanimate (e.g. machinelearning system) or organic and living. They can be as simple as athermostat or bacteria, or as complex as an RNA molecule, a robot,or a human being. Aggregations of agents such as an organization,corporation or family are referred to as meta-agents.4 Some agents,such as human beings, are also CAS themselves.

Precepts are the internal rules and structures that constrain anddrive the behavior of agents and ultimately the overall system theagents comprise. Agents autonomously adapting to their environ-ments by changing these internal structures through processes suchas acclimatization, learning and self-organization is what makesCAS adaptive. For example in human immune systems, cells are

4Agents are equivalent to Actors and Meta-agents are equivalent to Actants inActor Network Theory [11]

2


agents whose internal structure consists of DNA that can change inresponse to its environment via mutations. Precepts are often mech-anisms for memory and persistence of state [54] as is the case withDNA which persists the instructions for generating the cells thatcomprise the human body. In general, precepts are highly complex,mostly invisible and extremely difficult to measure [25, 26].

Artifacts are the results or manifestations of agent behaviorin a CAS. Agent behavior generates, contributes to and changesthe environments in which they exist. In other words, artifactsreflect the underlying precepts of agents that are hard to directlyobserve or measure. For example an artifact of thermostat behav-ior would be the increased temperature of the room it is in. Anartifact of an RNA molecule would be a protein. Artifacts come inmany forms including ant hills, honey, odors, buildings, roads, laws,other agents such as offspring or organizations, cellphones, ML sys-tems,5 and economic systems. Some artifacts such as organizations,offspring or economic systems are also CAS and/or agents/meta-agents themselves. Similar to precepts, artifacts can also serve asmemory mechanisms — e.g. in the form of a hieroglyph, capacitor,book, or hard drive.

2.2 A Taxonomic Model of Societal ContextIt was Sociologist Walter Buckley who first introduced the ideathat Society can be thought of as a complex adaptive system [12].Since its introduction in the 1960s, CAS has been used to modelsocial systems of varying size and complexity such as supply-chainnetworks [19, 103] and individual organizations [10, 24, 93], tohealth care systems [8, 9, 79, 89] and economic systems [2]. Ourchoice of CAS as the framework to model societal context in MLFairness research stems from this rich lineage of its successfulapplications to model social systems.

In applying the CAS model to societal context, we consider soci-etal context to be primarily constructed by human agents [95] en-gaged in a continuous process of simultaneously satisfying their in-dividual complex needs [65] within their physical and socio-culturalenvironments and creating, changing and adapting (via their pre-cepts) to that same physical and socio-cultural environment. Inother words human agents simultaneously exist in, interact with andcreate (via artifacts) societal context. The precepts of human agentsinclude lower-level, deeply ingrained structures evolved over eonssuch as the fusiform gyrus which enables humans to recognize faces[66] and so called fixed action patterns theorized to drive instinctivebehavior [70]. For human agents higher-level precepts, central tosocialization and that impact decision making processes, includeemotions, learned concepts and patterns such as race and genderrole stereotypes, biases, beliefs, attitudes, assumptions, values andself-identity. Higher-level precepts also include the concept of ag-gregate models of the world — theorized to be central to reasoningand at the heart of human problem solving, decision making, causalinference and goal-directed counter-factual thinking [28, 110] —that various disciplines refer to as mental models [15, 33, 86].

Figure 1 depicts the taxonomic relationship between the keyelement types that comprise societal context from a CAS perspec-tive. Precepts drive and constrain the behavior of agents and are

5An ML system can also be classified as agent [81] in its own right.

Figure 1: TaxonomicModel of Societal Context using a Com-plex Adaptive System perspective.

reflected in the artifacts that result from that behavior. In turn, pre-cepts are influenced by the artifacts they are exposed to, resultingin the feedback loops that contribute to the dynamically complex[100] nature of societal context. In the following subsections wewill highlight key aspects of precepts (2.2.1) and artifacts (2.2.2) thatare relevant to ML Fairness and the extension of the ML systemdesign abstraction boundary.

2.2.1 Four Key Precepts. The primacy of human agents in creatingsocietal context makes their precepts — which shape what humanagents “see, think and do” [25] — the most influential feature ofsocietal context. As such any approach to extend the abstractionboundary of the ML system design process to include societal con-text should be centered around identifying and representing humanprecepts. In particular, the mental models at the foundation of thehuman decision processes [110] of the specific humans who fund,build, utilize, and are impacted by the products ML systems arecomponents of should all be considered endogenous to the systembeing designed. For instance, ignoring the role of human decisionprocesses (e.g. those of judges, defendants and their families) in thecase of risk assessment systems will lead to unfair outcomes [96].Below we enumerate four key precepts that are essential to uncoverthe societal context that surrounds ML systems/products. Althoughwe’ve enumerated these precepts separately, in reality they areinterdependent, overlapping and mutually influential. Additionally,they are supported and influenced by the values, emotions, biasesand stereotypes held by the agent.(1) perceived needs, ranging from fundamental physical needs

for food, water and warmth to socially-constructed needs suchas freedom, safety, fairness, justice and self-actualization [65].Satisfying some human agent’s (e.g. potential users and theorganizations that fund and build products for them) perceivedneed is often the motivating factor behind building a productan ML system might be a part of. For example pre-trial riskassessment products (e.g. COMPAS) are intended to address theperceived need to improve pre-trial risk decision making [18]held by criminal justice organizations. Perceived needs are veryrelated to and often interchangeable with perceived problems.

(2) perceived problems, or the perceived gap between a perceivedneed and the perceived state of satisfaction of that need. Per-ceived problems can range from an individual human agent’s

3

Martin et al. 2020

lack of food for their next meal to societal issues such as home-lessness, high rates of crime, immigration or poverty. Societalcontext (the system of agents, precepts and artifacts) producespatterns of behavior over time (artifacts) such as the growthin the number of homeless people. Some human agents canperceive those artifacts as problems to solve. Similar to per-ceived needs, the resolution of a perceived problem is oftenthe motivating factor that drives the development of products,6some of which employ ML systems.

(3) causal theories, are the mental models human agents holdabout the structure of cause-to-effect relationships betweenagents, precepts and artifacts that cause or lead to a specificproblem. As we’ve established earlier, no individual agent‘s per-ception of problem structure can be correct or complete as theyare oblivious to and incapable of perceiving and understandingsocietal context as a whole. In particular, human agents are cog-nitively incapable of constructing and managing internal causalstructures that can take into account the feedback loops andtime-delays that characterize problems produced by societalcontext [31]. For that reason we refer to these necessarily incom-plete perceptions of the causal structure of specific problems astheories[104]. Essentially, a human agent‘s causal theories aremicro-models of societal context. Each human agent’s collec-tion of causal-theories is a small patch in the overall fabric ofsocietal context. The causal theories of product managers, MLsystem designers, potential customers, and potentially impactedperipheral stakeholders are all critical aspects of the societalcontext that surrounds an ML system.

(4) goals and strategies for satisfying needs and/or solving prob-lems [5, 44]. Goals and strategies are closely linked to and oftenbased on causal theories. As a simple example, a job-seeker’scausal theory about why they cannot find a job may informa goal/strategy to move to another state, city or country orenroll in college or a training program. Once this goal/strategyis established, it can become a perceived need to be satisfied ora perceived problem to be solved.

2.2.2 ML Systems as Artifacts and Interventions. ML systems, aswell as their data inputs and outputs, are socially constructed arti-facts [78] of agent behaviors (driven by precepts) that once deployedwill become new elements of societal context. These ML systemsand output artifacts can be thought of as interventions on someaspect of the system of societal context to solve a perceived prob-lem (precept) or realize a goal (precept) that originated from thehuman precept aspect of that same societal context. ML systemsare increasingly being used as interventions on societal issues (akaperceived problems) within high-stakes domains such as healthand criminal justice. For example, risk assessment and predictivepolicing systems can be thought of as interventions on the criminaljustice system (artifact, meta-agent, and CAS) to solve some per-ceived problems (precepts) as perceived by a certain set of humanagents. As has been demonstrated extensively [27, 61], such inter-ventions can lead to fairness failures when the ML system design

6Understanding problems as perceived by peripheral stakeholders and socialgroups is a key analysis factor in the Social Construction of Technology (SCOT)method [78], and is identified as the key method to relate technical artifacts to their“wider context“.

abstraction boundary excludes the human precepts and feedbackloops that encompass other relevant regions of the societal context.

3 COLLABORATIVE CAUSAL THEORYFORMATION

Extending the abstraction boundary of the ML system design pro-cess to include societal context is a daunting task. The end-to-endProduct Development Process (PDP) provides the “local context”that shapes abstraction boundary decisions made duringML systemdesign. Hence, a practical first step is to focus on the PDP and thecausal theories of the human agents that own, participate and areimpacted by it. Here, a “product” can be a tool or system developedfor internal institutional use or for commercialization and externaluse by other institutions or individuals.

3.1 Causal Theories and the ProductDevelopment Process

The primary purpose of the PDP is to fulfill the goals and strategies(precepts) of 1) product funders (e.g. finance, sales & marketing,and product leaders, potential customers) and 2) product owners(e.g. product managers, ML system designers, user experience re-searchers) by enabling the design and delivery of products thatsolve perceived problems (precepts) and/or satisfy the perceivedneeds (precepts) of target stakeholders (agents). The causal theories(precepts) of product funders and owners reflect their understand-ing of the perceived problem they are working to solve and have anenormous impact on how the PDP operates and what it produces.As explained in section 2.2.1, these individual causal theories arenecessarily incomplete.

Throughout the PDP, product funders and product owners makehigh stakes design decisions based on their individual, incompleteimplicit causal theories about the problem the product is intendedto address. These high stakes decisions include deciding what therelevant factors (aka dependent and independent variables) of theproblem they have chosen to focus on are and how they are inter-related, who the target stakeholders of the problem solution orproduct are, who the peripherally impacted stakeholders are, whatproduct or sociotechnical system should be deployed to satisfy theneed/solve the problem, who should comprise the core productteam, whether or not it is appropriate to employ ML to solve theproblem and, if ML is chosen, what ML architecture is best suitedfor the problem at hand. Once the product is deployed it interactswith and influences the perceived needs and problems, goals andstrategies and causal theories of target and non-target stakeholders,resulting in feed-back loops.

When incomplete causal theories are the basis for understandingproblems and designing product-based solutions, the probabilityof unintended consequences, sub-optimal solutions and unfair out-comes that negatively impact the most vulnerable stakeholders islikely to increase. For example, incomplete causal theories can leadto excluding peripherally impacted stakeholders (e.g. the familyof someone arrested), their perception of problems to be solved,their causal theories about the factors that cause the problem aswell as their perspective of what a fair outcome is. In [78] the au-thors point out the important role of the perception of problems byperipheral stakeholders during the development lifecycle of new

4


technologies. Incomplete causal theories can also lead to excludingthe decision processes (driven by precepts) of targeted stakeholderssuch as the judges who use risk assessment frameworks to informbail decisions. Each of these exclusions lead to excluding criticalelements of societal context.

More complete causal theories would incorporate the causaltheories of target users and peripheral stakeholders and will likelylead to a more complete problem understanding, including whatinterrelated factors and feedback loops are most relevant for agiven problem, what the most effective interventions could be,what other problems are relevant and what the negative impacts ofan intervention could be.

3.2 The role of fair-ML researchers andpractitioners in the PDP

As our goal is to identify ways to extend the ML system designabstraction boundary to include societal context, we have chosento focus on the product development process that envelopes theML system design and produces ML systems. Fair-ML researchersare typically not owners or drivers of these processes, but ratheraudit and review the ML system design sub-process outputs — inputdatasets, training datasets, model/algorithms, and ML outcomes —for fairness failures, and develop techniques that ML system design-ers (a subset of product owners) can utilize for measuring/detectingand mitigating unfair results.

Due to the limits of purely observational and data analysis ap-proaches to achieving fairness in ML there has been an increasingnumber of fair-ML researchers delving into the topics of causality[17, 18, 51, 62], feedback loops [27] and time-delay [59]. A recur-ring method for representing the causal theories of researchers hasbeen via graphical models, called causal diagrams, of the presumedcausal inter-relationships between factors (aka variables) relevantto the problem to be solved or decision to be made. Often timesthese causal diagrams are constructed using directed acyclic graphs(DAGs). A number of these approaches leverage Structural CausalModels (SCMs) and Causal Bayesian Networks [18, 62] for measur-ing unfairness in datasets or building fair decision making modelsthat have biased data as inputs.7,8 Although these approaches in-corporate the concept of causality into ML fairness research, theytend to focus on leveraging those concepts to intervene on MLsystem inputs or outcomes, not for extending the ML system designabstraction boundary to include societal context.

3.3 Improving the PDP through CollaborativeCausal Theory Formation

As the causal theories of problem funders, product owners, tar-get stakeholders and peripheral stakeholders of the product beingdesigned comprise the core of its societal context, extending the ab-straction boundary of the ML System Design process to encompassthem requires updating the PDP to overcome four fundamentalweaknesses:

7SCMs are optimized for bridging the gap between qualitative and quantitativedescription and to explicitly deal with the fact that data alone cannot be utilized tounderstand the underlying causal structures that generated it [76]

8Critiques of counterfactual approaches caution that treating concepts such asrace and gender as variables vs complex systems in and of themselves limits theirreliability and effectiveness [53]

(1) there is often a lack of diversity on product funding and own-ership teams which decreases the richness and variety of thecausal theories they produce [16]. Solving for fairness and inclu-sion requires a deep understanding of unfairness and exclusion. Assuch the product funding and ownership teams, whose causaltheories and problem understanding drive the PDP, must beas diverse as possible (by race, gender, national origin, socio-economic status, etc.) and include people with deep understand-ing, through lived-experiences, of unfairness and exclusion.

(2) the PDP does not incorporate a systems approach to design thatacknowledges and contends with the fact that the products aredeveloped and deployed within a societal context that has thecharacteristics of a complex adaptive system.

(3) the PDP does not make the causal theories of product fundingand ownership teams explicit. This prevents them from beingtested for validity/completeness and from being improved upon.This often results in decisions being made based on availabledata, not on what might actually be relevant.

(4) peripheral stakeholders, including policy makers and thosebelonging to social groups that are most vulnerable to unfairoutcomes, are often excluded from meaningfully contributingto the product conception phase of the PDP.

Addressing weakness 1 is beyond the scope of this paper. How-ever, we believe that the value and primacy of diverse causal theo-ries will further validate the ongoing efforts to improve the diversityof product teams in the tech industry.

In order to address weaknesses 2-4, the typical PDP must be re-designed to elicit and leverage more comprehensive causal theoriesin all phases of ML product design, development and deployment,but particularly in the product conception and design phase. Prod-uct owners and the fair-ML researchers they partner with must a)adopt a systems-based approach to system design b) become keenlyaware of their own causal theories, including their limitations, aboutthe problem to be intervened on and c) develop the capability toexplicitly surface, share and compare their causal theories withthose of other key (obvious and non-obvious) stakeholders.

In other words, the PDP must incorporate the capability to col-laboratively discover, understand and synthesize the causaltheories of key stakeholders into new,more complete causaltheories that more accurately reflect the dynamic complex-ity of the societal context in which the ML-based product(intervention) will ultimately be deployed. We will genericallycall this capability collaborative causal theory formation (CCTF).Although there are a number of methods [72, 77, 80] across dis-ciplines such as anthropology, ethnography, economics and thesocial sciences, for discovering/eliciting the stories and perspec-tives of groups, these discipline-specific practices are not typicallyoptimized for collaboratively identifying causal theories or contend-ing with dynamic complexity. The outputs of CCTF are qualitativeand quantitative artifacts that contribute to comprehensive prob-lem understanding and causal theory improvement. For optimaleffectiveness CCTF should be performed in an open, explicit, multi-disciplinary manner that is optimized to incorporate perspectivesfrom people with lived experiences in the societal context beinginvestigated. To this end, we introduce system dynamics (SD) as apowerful tool to integrate CCTF capabilities in ML system design.

5

Martin et al. 2020

4 SYSTEM DYNAMICS (SD)System Dynamics was developed in the 1950s by Jay W. Forrester,who was challenged to see if his experience with control systemsand early aircraft simulators could be adapted to produce insightsinto supply chain dynamics that befuddled those working to man-age them [36]. The exploratory work was later developed into thecomputer simulation model detailed in Industrial Dynamics [35],which illustrated that the puzzling oscillations among inventoriesand orders were actually the result of the managers’ failure to ac-count for the feedback effects of their own decisions [31]. Afterapplying the modeling approach in the management field, Forrestermade the first effort to model social systems in his book Urban Dy-namics [32]. While this work generated insights about the feedbackloops connecting issues related to urban growth and decline, it alsoled to critiques [6] that the underlying model reflected and pro-moted the assumptions, worldviews and causal theories of Forresterand his collaborators while neglecting the same for people withlived experiences in urban settings and alternate political views.These critiques resulted in the evolution of participatory groupmodel building techniques that foster incorporation of diverse per-spectives [47]. In its now more than 60 years of practice, the SDfield has now broadened its application base to environmental sus-tainability [42, 92], urban planning [39], epidemiology [105], socialwelfare [48], education [106], and public policy [37].

SD is defined as the process of using both informal maps/diagramsand formal models with computer simulation to uncover and under-stand the dynamics of complex problems from a feedback perspective[85]. It is this emphasis on feedback — reinforcing and balancingprocesses that unfold over time — that distinguishes SD from othermodeling approaches, and makes it well-suited to incorporate thedynamically complex societal context into the ML system designprocess. To uncover and understand feedback processes, SD hasdeveloped a series of tools that vary in degree of formalism and aredesigned to provide insight into different aspects of the complexproblems they model [85]. Many of these tools are graphical innature, requiring modelers to make their causal theories explicit,thereby ensuring transparency [56].

Recognizing that effective action on a complex problem requiresthe perspectives, consensus, and coordination of multiple stake-holders, SD has evolved a rich framework to involve stakeholders inthe model building process to foster collaboration and learning [52].The reliance on visual tools allow insights and causal theories to beshared and understood with diverse stakeholder groups, enablinga history of participatory approaches [4, 71, 99, 107]. Communitybased system dynamics (CBSD) [46] is a particular SD practice ap-proach that engages stakeholders who are embedded in the systemof interest to conceptualize the problem, identify the related issuesand prioritize interventions based on model supported insights.These participatory aspects makes a compelling case for SD as away to perform CCTF in ML Fairness efforts and interventions.

4.1 Causal Loop DiagramsOne of the most commonly used visual tools in SD is the causalloop diagram (CLD). The main purpose of the CLD is to show thefeedback processes in a system (understood as the set of positedcausal structures related to the phenomenon of interest) using a

directed graph. Note that CLD is meant to communicate and elicithypothesized causal relations between variables in a problem space,and is understood to be informal, high-level, and incomplete.

An example of a CLD is shown in Figure 2a, which offers a sim-plified representation of a credit score based lending system. Thearrows in CLDs represent hypothesized causal links between vari-ables, with the arrowheads and polarity indicating the direction andthe nature of influence. Positive polarity represents relationshipswhere an increase (decrease) in one variable triggers an increase(decrease) in the other, all else equal. Negative polarity is used todepict relationships where an increase (decrease) in one variabletriggers a decrease (increase) in the other, all else equal. In theexample in Figure 2a, the relationship between Payments Made andAverage Credit Score is assumed to be of positive polarity since mak-ing payments towards debt generally builds credit, ceteris paribus,whereas the link between Loan Defaults and Average Credit Score isnegative, since defaulting generally results in score reductions. Anyincrease (decrease) in the average credit score of a group leads to acorresponding increase (decrease) in the number of loans receivedby that group, which in-turn increase (decrease) its borrower pool.

What distinguishes CLDs from other graphical modeling ap-proaches like DAGs [97] is that they are designed to capture feed-back loops. These loops can be of two types, based on their behaviorover time. Reinforcing feedback loops (labeled “R” in CLDs) arethose processes that amplify system behavior, and can create dy-namics of exponential growth or decline. These are often referredto as virtuous or vicious cycles. In turn, balancing feedback loopsare those that dampen or counteract change in a system (labeled“B” in CLDs). In Figure 2a, an example of a reinforcing feedbackloop is the one generated by the interactions of Payments Madeand Average Credit Score over time: as more borrowers repay theirloans, the better their credit scores become, which in turn increasesthe likelihood of receiving future loans (Loans Received). It also il-lustrates a balancing feedback loop, generated by the interaction ofLoan Defaults and Average Credit Score: as more borrowers defaulton their loans, the worse their credit becomes, limiting their abili-ties to qualify for loans (Loans Received), and thus their likelihoodto default again.

A further difference between the CLD and other graph basedapproaches (including those that include feedback like fuzzy cog-nitive maps [73]), is its ability to explicitly acknowledge wherethe relationship between two variables is mediated by the passageof time. These are typically denoted using the same symbol usedto represent capacitors in circuit schematics (||). In Figure 2a, thismeans that the impact of repayment on credit score is not only notinstantaneous but also that this delay has substantive impacts onthe behavior of the system [59].

While this deliberately simplified CLD includes only two loops,modelers are encouraged to incorporate as many variables andfactors as are required to explain the phenomenon of interest. Inparticular, CLDs and SD models more generally are not limitedto factors for which data is available, and are expected to insteadaim to include everything that is relevant (but nothing more) [100].Omitting variables on the basis of lack of quantitative data is ex-plicitly discouraged [31]; to assume their effect is zero, is probablythe wrongest assumption of them all. Despite the simplicity of a twoloop CLD like the one depicted in Figure 2a, the complexity of the

6


Figure 2: Simple examples of two visual approaches used in system dynamics (causal loop diagrams and stock and flow dia-grams) to reflect the hypothesized feedback structures relevant to a simplified Lending System scenario

lending system is still manifest: once the balancing loop is triggeredfor some populations, the reinforcing loop can turn from a virtuousto a vicious cycle, further limiting their abilities to build credit (aswe demonstrate in Section 4.3).

4.2 Stock and Flow DiagramsAmore formal treatment of the causal structures, including the con-cept of delays and their impact on the system is offered by stock andflow diagrams, perhaps the most commonly used tool in system dy-namics. In addition to representing relationships between variablesand feedback loops, stock and flow diagrams require explicit defini-tions of variables that accumulate, and the precise ways that theyaccumulate or are depleted over time. In these diagrams, variablesthat accumulate are called stocks and are drawn as rectangles, andthe processes that add to or drain them are called flows (inflows andoutflows) and are depicted as double-lined/thick arrows or “pipes”with valves. The “clouds” are the sources and sinks of the flows,and are assumed to have infinite capacity over the time horizonof the model. These clouds show the model’s assumed boundary —once information or material passes through the flows into a cloud,it ceases to impact the system.9

Figure 2b shows a stock and flow representation of the lendingsystem represented in the CLD (Figure 2a), in which Borrowersand Average Credit Score of the population are now represented as

9System dynamics practice encourages “challenging the clouds”[p. 132] [88]—inother words, critically examining the model’s boundary assumptions. Is it appropriateto exclude the stocks currently outside the model boundary? Do those excluded stockshave zero impact on the model? The visualization of stocks and flows supports thediscussion of what the the system boundary should include, and simulation modelingprovides ways to test the adequacy of assumed boundaries [100].

stocks, and are thus assumed to accumulate value over time. Thenumber of borrowers (units = people) accumulates the inflow ofpeople receiving loans per year and is depleted by the outflows ofpeople paying off the loan completely and defaulting on loans peryear. In this context the cloud before receiving loans indicates theassumption that there is an endless source of individuals who couldapply for loans. In turn, in this simplified model, those leaving thesystem by defaulting or paying off are assumed to not affect thesystem in any meaningful way,10 and are thus represented as cloudsat the ends of the outflows. It is important to note that in the processof converting the high-level CLD to a more formal stock and flowdiagram, a one-to-one correspondence for the system variables isnot enforced; rather the feedback loops are preserved and modeledin more detail. For instance, the causal path from Borrowers to theAverage Credit Score through the system variable Payments Madein the CLD is now represented through a flow (increasing) that isguided by the rate at which credit score increases per year as aresult of repayments made (avg increase per year of repayment), aswell as the maximum credit score possible (Max credit score).

The stock and flow diagram is a graphical representation of thesystem as differential equations that formalizes the behavior of thesystem over time. The behavior of a stock S can be determined bycalculating the integral of its flows over a given time horizon t ,such that:

S(t) = S0 +

∫ t

0(inflow(t) − outflow(t)) dt (1)

10the explicit and visual nature of this model boundary decision facilitates inputfrom and discussion by other modelers and stakeholders

7

Martin et al. 2020

For instance, in our example, lettingO be the number of borrowers,r be the rate of receiving loans, f be the rate of defaulting (failureto pay), p be the rate of paying the loan off (completely), and t isthe unit of time over which the system evolves, the value of O atany given time is defined as

O(t) = O0 +

∫ t

0(r (t) − p(t) − f (t)) dt (2)

Flows and rates are similarly formalized as functions of otherelements: stocks, other flows, and exogenous factors. For example,if parameters x and y refer to the probability of repayment andaverage loan term, respectively, repaying p is given by

p(t) = O(t) × x/y (3)

These parameters are determined by the modeler and their ex-plicit definition enhances the transparency of the modeler’s causaltheory. Together with a set of initial values for the stocks and pa-rameters, this system of equations complete an SDmodel, and allowmodelers to leverage the full power of the stock and flow representa-tion of a system through simulation of its evolution over time. In theabsence of empirical estimates, modelers can use values consistentwith opinions from relevant stakeholders and conduct sensitivityanalyses to understand the ranges of values for which the systempresents certain behavior modes — this is an important aspect ofSD that breaks the reliance on only the variables we have data for,and instead provides a way to incorporate qualitative insights intothe model building process.

4.3 Role of Simulation in SDDespite the usefulness of qualitatively mapping causal theoriesusing CLDs and stocks and flows, the cognitive load required totrack the state of the system over time for all but the simplestmodels is too high [31, 84, 98, 100]. Accordingly, numerical simula-tion approaches have typically been used to study the long-termimplications of the causal theory represented in the model. Morespecifically, simulation is used to test hypotheses about the rela-tionship between the system structure and the behavior it produces,and explore the impacts of parametric and structural changes (e.g.,adding/removing feedback loops or flows).

For instance, consider a scenario where a lending institutionhas a goal (a precept) to improve the profit margin while makingmore capital available to the markets they serve. This could beachieved by designing and implementing an ML-based system thatsearches the spaces of possible interventions and offers a policythat aligns with this goal. The lending institution would employproduct funders and owners whose precepts (e.g. goals, strategies,causal theories) drive the PDP that will produce the intervention inquestion. Suppose that such a predictiveML-based system identifiedtwo potential strategies that might both improve the profit marginand broaden access to capital. The first lowers the credit scorethreshold for granting loans to potential borrowers, and the second,provides longer loan terms for borrowers with lower credit scores.These interventions can be integrated into our simplified examplestock-flow model, as shown in Figure 3, where it becomes moreapparent that the former introduces a parametric change, and thelatter, a structural one (adding a balancing feedback loop to thesystem).

Figure 3: Stock and flow diagram of the lending systemhigh-lighting two proposed interventions — lowering the creditscore threshold for granting loans and adjusting the loanterm length based on credit score .

Figure 4: Simulation results comparing a lending institu-tion’s cumulative profits under different interventions inthe simplified lending system model, assuming an averagemonthly payment per borrower of $1000.

SD provides tools to simulate system behavior over time in re-sponse to these interventions. For the purpose of this discussion,we set the initial values for all the variables consistent with anaverage case, and monitor the evolution of the system for 20 yearswith the interventions implemented at year 10. Simulation of thetwo interventions reveals that they both achieve the goal of increas-ing profits over time for the lending institution. Figure 4 depictsthe lender’s profits under a scenario of no intervention (solid-linecurve), and the two aforementioned interventions (dotted-line anddashed-line curves). With respect to the lender’s profit margin,these interventions produce virtually indistinguishable results.

8


Figure 5: Simulation results comparing long-term trajectories of average credit scores for two groups of people (Group A withpayoff probability of 0.8 and Group B with payoff probability of 0.6) for different interventions in the lending system.

However, strategies that can produce seemingly equivalent re-sults with respect to the outcome of immediate interest, may havedisparate impacts on different sub-groups of the population — dif-ferences that would be difficult to trace without a more holisticview of the system. For instance, from an ML Fairness perspective,suppose that we have two sub-groups of people in the populationdistinguishable only by their payoff probabilities — 0.8 for GroupA and 0.6 for Group B.11 We set the initial values for all other vari-ables for both groups consistent with an average case (for instance,we chose an initial Average Credit Score of 550). As before, we mon-itor the evolution of the system for 20 years with interventionsimplemented at year 10.

This more holistic view of the system suggests that these seem-ingly equivalent strategies can have very different impacts on thetwo sub-groups that form our population. Figure 5 tracks the evolu-tion of average credit scores for members of those two groups. Theleft panel compares a scenario of no intervention to lowering theloan granting credit score threshold by 50% at year 10. The dottedblue line shows an increasing trajectory of the average credit scoresfor the group with a higher probability of payoff (Group A) underthe intervention. However, the same intervention appears to disad-vantage the group with the lower probability of payoff (Group B),widening the gap between the two over time. This illustrates theintuition derived from the CLD in Figure 2b, whereby the reinforc-ing loop can in fact operate very differently depending on initialconditions, generating undesirable consequences for some groupsin the population. The curves representing those same trajectoriesunder the no intervention scenario, depicted in grey, trace the sametrajectories so closely that they are barely noticeable.

Contrast that result to the simulation results of the loan termlength intervention, depicted on the right panel of Figure 5. Underthis intervention, the average credit scores for both groups improveover time, and particularly so for the group that initially had alower probability of pay off. While the gap remains, the systemreequilibrates to a more equitable position than that induced bythe alternative strategy. The credit score trajectories under this

11For simplicity, the payoff probability encapsulates all of the unobserved factorsthat can effect a borrowers ability to repay, such as average income, average educationlevels, etc.

new intervention also noticeably depart from the no interventionscenario (depicted in grey) — illustrating a well-known property ofsystems, whereby structural changes tend to have more powerfullong-term effects than those that simply adjust parameters [67].

We can use Figure 3 to guide the explanation for the trajectorieswe observe. The intervention of lowering the credit score thresholdfor loan granting increases the number of borrowers quickly, how-ever, with each group’s probabilities of payoff remaining constant,the number of defaults also increases. This disproportionately im-pacts the average credit scores of the group with a lower probabilityof payoff since each default invokes a large credit score penalty. Theintervention adjusting loan term length, however, slows the rateof defaulting, and at the same time allows borrowers more time toincrease their credit score as they make monthly payments. Whilethis balancing loop does not change the exogenous probabilities ofpayoff for each group, it does not allow the system’s reinforcingloop to privilege one group and harm another in the presence ofthe groups’ distinct probabilities of payoff.

Even though this is a simplified representation of the lending sys-tem (meant to illustrate the concepts we have introduced through-out the paper), its lessons, with respect to the benefits of incorporat-ing a holistic view of a system, hold in general. While a predictiveML model can reveal the relationships (as shown in Figure 5) be-tween lending thresholds, loan terms and overall profitability ofapproved loans, only a dynamic feedback perspective can uncoverthe long-term consequences of activating those relationships, andthe mechanisms through which they are likely to operate. As aresult it is paramount that these models have a boundary that isextensive enough to encompass the concerns and experiences ofthose embedded in the system.

More specifically, when considering proposed interventions,stakeholder groups impacted by the system may identify other(unanticipated) consequences, some of which may imply factors,feedback loops and problematic system behaviors not consideredduring the causal structure modeling process. Observing how feed-back loops can change the behavior of a system drastically (as itdid in our simple lending system model) highlights the need topartner with diverse stakeholders with unique and relevant causaltheories in an iterative modeling process that often begins with

9

Martin et al. 2020

establishing a “reference mode.” Establishing a “reference mode”makes the historical and desired future system behavior explicitand requires the participation of those who stand to be affected byits evolution.

4.4 Problem Definition with Reference ModesIn general, graphs like the ones in depicted in Figures 4 and 5 areindicative of what those who are modeling the system (typicallyproduct funders and owners) believe are important factors to focuson and what they consider to be desirable trajectories over time.Such preferences are not universal and are often not reflective of theprecepts (e.g. needs, goals, strategies) of non-target stakeholders.Should all credit scores increase over time? Are increases desirableeven if the number of borrowers declines? Should we be focusing oncredit scores or financial well-being? What about the profitabilityof lending institutions? In SD, the process of identifying what theproblematic and/or targeted behaviors are is known as defining thereference mode. The reference mode is comprised of three compo-nents: a clearly defined target/goal or desired-state, a clear idea ofthe current/problem state, and the clearly depicted gap between thetwo [83, 90]. It is often expressed as a graph over time, depicting thedesired and feared trends of an outcome variable of interest. Thesegraphs are highly influenced by the goals, strategies and implicitcausal theories of their creators.

In addition to enabling easy comparison to figures resulting fromsimulation like the ones in Figure 5, mapping the reference modevisually promotes dialogue, invites critique and revisions aboutwhat is considered the focal variable(s), what the goal pattern ofbehavior is, and what its associated time horizon is. Referencemodes also set the boundaries of the model, as the goal of SD is tomodel problems, not the whole system [101].

While reference modes can be informed by historical data [90],care must be taken to avoid choosing focal variables based solely ondata availability. To limit reference modes to indicators that alreadyexists runs the risk of conflating what is familiar, tangible, andpotentially biased, with what is actually important [102]. In a way,reference modes anchor the entire model building process. Not onlydo they define the dynamics of interest, but also—and perhaps morefundamentally—which variables are most important in the systemand which problematic dynamics (e.g. disparate outcome trends)ought to be addressed. Since these decisions determine in large partthe kind of insights that can be derived from the modeling exercise,it is of the utmost importance that those likely to be impacted bydecisions based on those insights, be partners in the model buildingprocess—particularly, in the definition of the reference mode. In MLfairness, this would imply collaborating with stakeholders to ensuretheir perspectives and causal theories inform the definition of thereference mode, and in so doing, identifying the most importantproblems and desired outcomes. In SD practice, this is typicallyachieved by engaging in participatory modeling, and specificallycommunity based system dynamics.

4.5 Community Based System DynamicsSD has a rich history of involving stakeholders in the model build-ing process to foster collaboration and learning [52]. Community

based system dynamics (CBSD) [46] is a particular SD practice ap-proach that engages stakeholders who are embedded in the systemof interest to conceptualize the problem, identify the related issuesand prioritize interventions based on model supported insights.More than just involving participants in the modeling process toelicit information, CBSD has the explicit goal of building capabil-ities within communities to use SD and systems thinking tools,distinguishing it from other participatory approaches in SD thatoften convene participants to gather information from them aboutan outsider-defined problem to inform an SD model and/or facili-tate activities for participants to interact with a simulation model[52]. Building capabilities enables stakeholders to more accuratelyrepresent their causal theories in the models, which is especiallycritical when the stakeholders are from marginalized communitiesthat are not represented in the modeling. In this view, individualand community perspectives on the structures that underlie every-day experiences are valued as valid and necessary sources of data,and community perspectives on the analysis and interpretation ofmodels are essential for realizing the value of the approach.

Best practices for engaging stakeholders in the process of estab-lishing the reference mode, hypothesizing the causal structure ofproblems using CLDs and stock and flow diagrams and refiningsimulation models are documented [46, 47]. These activities can beadapted for diverse contexts and support the development of capa-bilities for collaborative causal theory formation (CCTF). Overall,CBSD has been shown to be useful in a broad range of problemdomains such as maternal and child health [71], identifying foodsystem vulnerabilities [99], mental health interventions [107] andalcohol abuse [4], to name a few.

In the domain of ML (un)fairness, the practice of CBSD canhelp center the voices and lived experiences of those marginalizedcommunities that are typically negatively impacted by ML-basedproducts. If the goal is to design fairer ML-based tools and productsthat do not harm peripheral stakeholders, it is imperative to not onlypartner with those stakeholders to model the long-term dynamicscreated by those products when implemented in complex societalcontexts, but to also build the capabilities of stakeholders to defineand negotiate together what fairness means in those contexts.

5 DISCUSSION AND RECOMMENDATIONSIn this paper we add to the argument that in order to effectivelyevaluate and ensure fair outcomes for ML-based products, we needto expand the ML system design abstraction boundary to includethe broader societal context [59, 96]. Specifically, we identify andtackle three major weakness (3.3) in the typical PDP that impedesthe consideration of societal context at scale: 1) lack of systems-based approach to product development and design, 2) lack ofmethods to transparently articulate and improve the causal theoriesof product owners that drive the PDP, and 3) limited involvementof stakeholders and communities most negatively impacted by ML-based products. Towards addressing these gaps,

(1) we propose a CAS-based taxonomicmodel of the key interactingelements (agents, precepts and artifacts) of societal context thatML System designers and fair-ML researchers can use towardsextending the abstraction boundary.

10


(2) we introduce collaborative casual theory formation as an essen-tial capability to incorporate diverse stakeholder perspectivesinto designing fairer ML systems.

(3) we demonstrate system dynamics as a rigorous and scalableapproach to model the dynamic complexity that characterizesthe societal context in which systems will be deployed andpropose CBSD as a means to prioritize incorporating the causaltheories of potentially impacted peripheral stakeholders.

Our proposal to employ a complex systems approach to incor-porate and understand societal context leans on the rich historyof CAS theory and its successful application in a wide array ofdomains ranging from supply chain networks, health care systemsand economic systems. SD modeling tools can reflect precisely thecharacteristics that make societal context dynamically complex,namely feedbacks, accumulations, time delays, and the boundedrationality of agents. While efforts to incorporate causal theoriesinto ML-based products is not new [51], we reflect that the failureto incorporate these other characteristics that contribute to the dy-namic complexity of societal context into account can have harmfulresults. As Sterman writes, “Side effects are not a feature of reality,but a sign that the boundaries of our mental models are too narrow,our time horizons too short” [101].

Another advantage of our SD-based approach is that it drawsheavily on the visual diagramming conventions which emphasizetransparency and facilitate the engagement of diverse stakeholdersto add, revise and critique causal theories [46, 56]. A long lineageof participatory approaches within SD including CBSD and groupmodel building provide evidence of success in developing and usingsystem dynamics models in diverse contexts serve as resources forgroups interested in developing SD capabilities in their communi-ties/contexts [4, 71, 99, 107]. Moreover, a strength SD shares withother causal modeling approaches, including Bayesian networks[75, 76], is the correspondence between its visualizations and theirunderlying mathematical representations, which allows stakehold-ers to do more than visualize, but continue to develop deep insightsabout important data to collect, consider, and evaluate impact ofproducts and decisions through simulation as well [56, 101].

While the methods we have described here can be used to gaina deeper understanding of the societal context associated witha particular problem, we have not described how these methodscould be performed at scale by global product companies whohave customers in all parts of the world and operate in complexgeopolitical environments. Methods for scaling CCTF efforts, andfor managing and leveraging large quantities of qualitative societalcontext data to support industrial and global scale use cases areareas that require further research. Another set of open questionsconcern the representation of socially constructed but impactfulconceptions such as race and gender in CBSD models as well as inmachine learning [40, 50]. However, partnerships with communitiesto describe the causal structures of problems that impact them mayserve as fertile ground for answering these questions.

To begin extending the abstraction boundary, it is important firstto recognize that given the potentially expansive and authoritativenature of these systems approaches, careful ethical considerationsare needed to ensure that the design and deployment of thesemethods adhere to legal, ethical and moral guidelines.

In addition, we recommend that product owners (particularlyproduct managers and user experience researchers) and fair-ML re-searchers strive to augment product conception sprints and researchinitiation brainstorms with CBSD group model building sessionsto clarify the space of perceived problems (reference mode [91])relevant to the product or research effort being conceptualized. Theoutputs of these sprints will be the transparent, shared and morecomplete causal theories (aka dynamic hypothesis) about the causalstructures that cause the perceived problem. These sessions can bepurely qualitative and documented via CLDs and/or stock and flowdiagrams and will reveal other perceived problems and stakeholdergroups that should, respectively, be considered and fully participatein the next round of theory formation. These dynamic hypothesesare micro-models of the region of societal context most relevant tothe product in question and can serve as a critical component of theproduct requirements/specification document that typically servesas the primary input to the ML system design sub-process. Initiat-ing ML system design with a more comprehensive understandingof the relevant problem factors and their impacts essentially makesthem endogenous to the system and extends the abstraction bound-ary. These new inputs can also drive the criteria for choosing MLarchitectures and acquiring appropriate datasets vs. relying solelyon the relational inductive biases of individual ML system designersand starting with available datasets.

While the participation of all keys stakeholders in the ML-basedproduct development process is critical, transformative work in thisspace begins with centering on community stakeholders and theirperspectives on the problems at hand. SD practices in this spacemust especially protect the perspectives shared by marginalizedgroups, as models that very precisely and explicitly reflect theirvulnerabilities could be exploited. Preventing such exploitation re-quires working with community stakeholders as partners, not asmere informants whose perspectives (data) are mined to supple-ment a model based on the causal theories of product funders orowners. Moreover, investments in a CBSD approach to buildingCCTF capabilities can, over time, generate capabilities and interestin ML within communities currently underrepresented in ML-basedproduct development. Such capabilities and interests can foster anenvironment in which communities proactively model the problemsand social inequities [1] they care about and become full partnersin driving the development of holistic and fair solutions.

Product owners and fair-ML researchers motivated to build ca-pabilities and competence in applying CBSD to the task of CCTFshould prepare for a painstaking journey. Learning and integratinga systems approach into the existing PDP will require influenc-ing teams and stakeholders who are comfortable with existingapproaches.

Acknowledgments. We would like to thank Emily Denton, BenHutchinson, Sean Legassick, Silvia Chiappa, Matt Botvinick, ReenaJana, DierdreMulligan, and Deborah Raji for their valuable feedbackon this paper.

11

Martin et al. 2020

REFERENCES[1] Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and

David G Robinson. 2020. Roles for computing in social change. In Proceedings ofthe ACM Conference on Fairness, Accountability, and Transparency (FAT*). ACM,252–260.

[2] Philip W Anderson. 2018. The economy as an evolving complex system. CRCPress.

[3] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. MachineBias: there’s software used across the country to predict future criminals. Andit’s biased against blacks. ProPublica 2016.

[4] Yorghos Apostolopoulos, Michael K Lemke, Adam E Barry, and Kristen Has-smiller Lich. 2018. Moving alcohol prevention research forward-Part II: newdirections grounded in community-based system dynamics modeling. Addiction113, 2 (2018), 363–371.

[5] Robert Axelrod and Michael D Cohen. 2000. Harnessing complexity. BasicBooks.

[6] Kevin T Baker. 2019. Model Metropolis. https://logicmag.io/play/model-metropolis/

[7] Brhmie Balaram, Tony Greenham, and Jasmine Leonard. 2018. Artificial Intelli-gence: Real Public Engagement. Technical Report.

[8] James W Begun, Brenda Zimmerman, and Kevin Dooley. 2003. Health careorganizations as complex adaptive systems. Advances in health care organizationtheory 253 (2003), 288.

[9] Marge Benham-Hutchins and Thomas R Clancy. 2010. Social networks as em-bedded complex adaptive systems. JONA: The Journal of Nursing Administration40, 9 (2010), 352–356.

[10] Max Boisot and John Child. 1999. Organizations as adaptive systems in complexenvironments: The case of China. Organization Science 10, 3 (1999), 237–252.

[11] Attila Bruni and Maurizio Teli. 2007. Reassembling the social - An introductionto actor network theory. Management Learning 38, 1 (2007), 121–125.

[12] W.F. Buckley. 1968. Modern Systems Research for the Behavioral Scientist: ASourcebook. Aldine. https://books.google.com/books?id=KRYFAAAAMAAJ

[13] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu-racy disparities in commercial gender classification. In Conference on fairness,accountability and transparency. 77–91.

[14] Michel Callon. 1986. The sociology of an actor-network: The case of the electricvehicle. In Mapping the dynamics of science and technology. Springer, 19–34.

[15] L Jean Camp. 2009. Mental models of privacy and security. IEEE Technologyand society magazine 28, 3 (2009), 37–46.

[16] Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, and Kate Crawford.2017. AI now 2017 report. AI Now Institute at New York University (2017).

[17] Silvia Chiappa. 2019. Path-specific counterfactual fairness. In Proceedings of theAAAI Conference on Artificial Intelligence, Vol. 33. 7801–7808.

[18] Silvia Chiappa and William S Isaac. 2018. A Causal Bayesian Networks View-point on Fairness. In IFIP International Summer School on Privacy and IdentityManagement. Springer, 3–20.

[19] Thomas Y Choi, Kevin J Dooley, and Manus Rungtusanatham. 2001. Supplynetworks and complex adaptive systems: control versus emergence. Journal ofoperations management 19, 3 (2001), 351–366.

[20] Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A studyof bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163.

[21] Alexandra Chouldechova and Aaron Roth. 2018. The Frontiers of Fairnessin Machine Learning. CoRR abs/1810.08810 (2018). arXiv:1810.08810 http://arxiv.org/abs/1810.08810

[22] Paul Cilliers. 2002. Complexity and postmodernism: Understanding complexsystems. routledge.

[23] Rebecca Dodder and Robert Dare. 2000. Complex adaptive systems and com-plexity theory: inter-related knowledge domains.

[24] Kevin J Dooley. 1997. A complex adaptive systems model of organization change.Nonlinear dynamics, psychology, and life sciences 1, 1 (1997), 69–97.

[25] Jennifer L. Eberhardt. 2019. Biased: Uncovering the Hidden Prejudice That ShapesWhat We See, Think, and Do. Penguin Publishing Group. https://books.google.com/books?id=vpdeDwAAQBAJ

[26] Eileen Eckert and Alexandra Bell. 2005. Invisible force: Farmers’ mental modelsand how they influence learning and actions. Journal of Extension 43, 3 (2005),1–10.

[27] Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and SureshVenkatasubramanian. 2017. Runaway feedback loops in predictive policing.arXiv preprint arXiv:1706.09847 (2017).

[28] Kai Epstude and Neal J Roese. 2008. The functional theory of counterfactualthinking. Personality and Social Psychology Review 12, 2 (2008), 168–192.

[29] Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police,and punish the poor. St. Martin’s Press.

[30] Luciano Floridi. 2008. Artificial intelligence’s new frontier: Artificial companionsand the fourth revolution. Metaphilosophy 39, 4-5 (2008), 651–655.

[31] Jay W. Forrester. 1961. Industrial dynamics. 1961. Pegasus Communications,Waltham, MA (1961).

[32] Jay W Forrester. 1969. Urban dynamics. M.I.T. Press, Cambridge, Mass.[33] JayW. Forrester. 1971. Counterintuitive behavior of social systems. Technological

Forecasting and Social Change 3 (1971), 1–22.[34] Jay W. Forrester. 1994. System dynamics, systems thinking, and soft OR. System

Dynamics Review 10 (1994), 245–256. https://doi.org/10.1002/sdr.4260100211[35] Jay W. Forrester. 1997. Industrial dynamics. Journal of the Operational Research

Society 48, 10 (1997), 1037–1041.[36] Jay W. Forrester. 2007. System dynamics–a personal view of the first fifty years.

System Dynamics Review: The Journal of the System Dynamics Society 23, 2-3(2007), 345–358.

[37] Navid Ghaffarzadegan, John Lyneis, and George P Richardson. 2011. How smallsystem dynamics models can help the public policy process. System DynamicsReview 27, 1 (2011), 22–44.

[38] Ben Green. 2018. “Fair” Risk Assessments: A Precarious Approach for CriminalJustice Reform. In 5th Workshop on Fairness, Accountability, and Transparency inMachine Learning.

[39] Ji Han, Yoshitsugu Hayashi, Xin Cao, and Hidefumi Imura. 2009. Application ofan integrated system dynamics and cellular automata model for urban growthassessment: A case study of Shanghai, China. Landscape and urban planning 91,3 (2009), 133–141.

[40] Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2019. To-wards a Critical Race Methodology in Algorithmic Fairness. arXiv preprintarXiv:1912.03593 (2019).

[41] Moritz Hardt, Eric Price, Nati Srebro, et al. 2016. Equality of opportunity insupervised learning. In Advances in neural information processing systems. 3315–3323.

[42] Peder Hjorth and Ali Bagheri. 2006. Navigating towards sustainable develop-ment: A system dynamics approach. Futures 38, 1 (2006), 74–92.

[43] Anna Lauren Hoffmann. 2019. Where fairness fails: On data, algorithms, andthe limits of antidiscrimination discourse. Under review with Information, Com-munication, and Society (2019).

[44] John Henry Holland. 1995. Hidden order: how adaptation builds complexity.Number 003.7 H6.

[45] John H Holland. 2012. Signals and boundaries: Building blocks for complexadaptive systems. Mit Press.

[46] Peter S Hovmand. 2014. Community Based System Dynamics. Springer.[47] Peter S Hovmand. 2014. Group model building and community-based system

dynamics process. In Community Based System Dynamics. Springer, 17–30.[48] Peter S Hovmand and David N Ford. 2009. Sequence and timing of three

community interventions to domestic violence. American journal of communitypsychology 44, 3-4 (2009), 261.

[49] Ruogu Kang, Laura Dabbish, Nathaniel Fruchter, and Sara Kiesler. 2015. “MyData Just Goes Everywhere:” User Mental Models of the Internet and Implica-tions for Privacy and Security. In Eleventh Symposium On Usable Privacy andSecurity ({SOUPS} 2015). 39–52.

[50] Os Keyes. 2018. The misgendering machines: Trans/HCI implications of auto-matic gender recognition. Proceedings of the ACM on Human-Computer Interac-tion 2, CSCW (2018), 88.

[51] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt,Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discriminationthrough causal reasoning. In Advances in Neural Information Processing Systems.656–666.

[52] Gábor Király and Péter Miskolczi. 2019. Dynamics of participation: System dy-namics and participation–An empirical review. Systems Research and BehavioralScience 36, 2 (2019), 199–210.

[53] Issa Kohler-Hausmann. 2018. Eddie Murphy and the Dangers of CounterfactualCausal Thinking About Detecting Racial Discrimination. Available at SSRN3050650 (2018).

[54] James Ladyman, James Lambert, and Karoline Wiesner. 2013. What is a complexsystem? European Journal for Philosophy of Science 3, 1 (2013), 33–67.

[55] Kai Lamertz. 2002. The social construction of fairness: Social influence andsense making in organizations. Journal of Organizational Behavior 23, 1 (2002),19–37.

[56] David C Lane. 2008. The emergence and use of diagramming in system dynamics:a critical account. Systems Research and Behavioral Science: The Official Journalof the International Federation for Systems Research 25, 1 (2008), 3–23.

[57] John Law et al. 1987. Technology and heterogeneous engineering: The caseof Portuguese expansion. The social construction of technological systems: Newdirections in the sociology and history of technology 1 (1987), 1–134.

[58] Jialiu Lin, Shahriyar Amini, Jason I Hong, Norman Sadeh, Janne Lindqvist, andJoy Zhang. 2012. Expectation and purpose: understanding users’ mental modelsof mobile app privacy through crowdsourcing. In Proceedings of the 2012 ACMconference on ubiquitous computing. ACM, 501–510.

[59] Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018.Delayed impact of fair machine learning. arXiv preprint arXiv:1803.04383 (2018).

[60] Kristian Lum, Elizabeth Bender, and Wilkerson. [n.d.]. FAT* 2018 TranslationTutorial: Understanding the Context and Consequences of Pre-trial Detention.Association for Computing Machinery. https://www.youtube.com/watch?v=

12

https://logicmag.io/play/model-metropolis/

https://logicmag.io/play/model-metropolis/

https://books.google.com/books?id=KRYFAAAAMAAJ

https://arxiv.org/abs/1810.08810

http://arxiv.org/abs/1810.08810

http://arxiv.org/abs/1810.08810

https://books.google.com/books?id=vpdeDwAAQBAJ

https://books.google.com/books?id=vpdeDwAAQBAJ

https://doi.org/10.1002/sdr.4260100211

https://www.youtube.com/watch?v=hEThGT-_5ho



hEThGT-_5ho[61] Kristian Lum and William Isaac. 2016. To predict and serve? Significance 13, 5

(2016), 14–19.[62] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2019. Fair-

ness through Causal Awareness: Learning Causal Latent-Variable Models forBiased Data. In Proceedings of the Conference on Fairness, Accountability, andTransparency. ACM, 349–358.

[63] Antonio Majdandzic, Boris Podobnik, Sergey V Buldyrev, Dror Y Kenett, ShlomoHavlin, and H Eugene Stanley. 2014. Spontaneous recovery in dynamical net-works. Nature Physics 10, 1 (2014), 34–38.

[64] Giuseppe Mantovani. 1996. Social context in HCl: A new framework for mentalmodels, cooperation, and communication. Cognitive Science 20, 2 (1996), 237–269.

[65] Abraham H Maslow. 1943. A theory of human motivation. Psychological review50, 4 (1943), 370.

[66] Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. 1997. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuro-science 9, 5 (1997), 605–610.

[67] Donella Meadows. 1999. Leverage points: Places to intervene in a system. TheSustainability Institute Hartland, VT.

[68] John H Miller and Scott E Page. 2009. Complex adaptive systems: An introductionto computational models of social life. Vol. 17. Princeton university press.

[69] Marvin Minsky. 1988. Society of mind. Simon and Schuster.[70] Howard Moltz. 1965. Contemporary instinct theory and the fixed action pattern.

Psychological Review 72, 1 (1965), 27.[71] Wolfgang Munar, Peter S Hovmand, Carrie Fleming, and Gary L Darmstadt.

2015. Scaling-up impact in perinatology through systems science: Bridging thecollaboration and translational divides in cross-disciplinary research and publicpolicy. In Seminars in perinatology, Vol. 39. Elsevier, 416–423.

[72] Douglas Noble and Horst WJ Rittel. 1988. Issue-based information systems fordesign. (1988).

[73] Osonde Osoba and Bart Kosko. 2019. Beyond DAGs: Modeling Causal Feedbackwith Fuzzy Cognitive Maps. arXiv preprint arXiv:1906.11247 (2019).

[74] Scott E Page. 2018. The Model Thinker: What You Need to Know to Make DataWork for You. Hachette UK.

[75] Judea Pearl. 2009. Causality. Cambridge university press.[76] Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause

and effect. Basic Books.[77] S Peck. 1998. Group model building: facilitating team learning using system

dynamics. Journal of the Operational Research Society 49, 7 (1998), 766–767.[78] Trevor J Pinch and Wiebe E Bijker. 1984. The social construction of facts and

artefacts: Or how the sociology of science and the sociology of technologymight benefit each other. Social studies of science 14, 3 (1984), 399–441.

[79] Paul E Plsek and Trisha Greenhalgh. 2001. The challenge of complexity inhealth care. Bmj 323, 7313 (2001), 625–628.

[80] Fatemeh Rabiee. 2004. Focus-group interview and data analysis. Proceedings ofthe nutrition society 63, 4 (2004), 655–660.

[81] Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-FranÃğoisBonnefon, Cynthia Breazeal, Jacob W. Crandall, Nicholas A. Christakis, Iain D.Couzin, Matthew O. Jackson, Nicholas R. Jennings, Ece Kamar, Isabel M.Kloumann, Hugo Larochelle, David Lazer, Richard McElreath, Alan Mislove,David C. Parkes, Alex ‘Sandy’ Pentland, Margaret E. Roberts, Azim Shariff,Joshua B. Tenenbaum, and Michael Wellman. [n.d.]. Machine behaviour. 568,7753 ([n. d.]), 477–486.

[82] Filippo A Raso, Hannah Hilligoss, Vivek Krishnamurthy, Christopher Bavitz,and Levin Kim. 2018. Artificial Intelligence & Human Rights: Opportunities &Risks. Berkman Klein Center Research Publication 2018-6 (2018).

[83] Nelson Peter Repenning, Don Kieffer, and Todd Astor. 2017. The most underratedskill in management. MIT Sloan Management Review.

[84] George P Richardson. 1991. Feedback thought in social science and systems theory.University of Pennsylvania.

[85] George P Richardson. 2011. Reflections on the foundations of system dynamics.System Dynamics Review 27, 3 (2011), 219–243.

[86] George P Richardson, David F Andersen, Terrence A Maxwell, and Thomas RStewart. 1994. Foundations of mental model research. In Proceedings of the 1994International System Dynamics Conference. EF Wolstenholme, 181–192.

[87] Rashida Richardson, Jason Schultz, and Kate Crawford. 2019. Dirty Data, BadPredictions: How Civil Rights Violations Impact Police Data, Predictive PolicingSystems, and Justice. New York University Law Review Online, Forthcoming(2019).

[88] Barry Richmond. 1993. Systems thinking: critical thinking skills for the 1990sand beyond. System dynamics review 9, 2 (1993), 113–133.

[89] William B Rouse. 2008. Health care as a complex adaptive system: implica-tions for design and management. Bridge-Washington-National Academy ofEngineering- 38, 1 (2008), 17.

[90] Khalid Saeed. 1992. Slicing a complex problem for system dynamics modeling.System Dynamics Review 8, 3 (1992), 251–261.

[91] Khalid Saeed. 1998. Defining a problem or constructing a reference mode. Depart-ment of Social Science and Policy Studies, Worcester Polytechnic Institute.

[92] Ali Kerem Saysel, Yaman Barlas, and Orhan Yenigün. 2002. Environmental sus-tainability in an agricultural development project: a system dynamics approach.Journal of environmental management 64, 3 (2002), 247–260.

[93] Marguerite Schneider and Mark Somers. 2006. Organizations as complex adap-tive systems: Implications of complexity theory for leadership research. TheLeadership Quarterly 17, 4 (2006), 351–365.

[94] Klaus Schwab. 2017. The fourth industrial revolution. Crown Business.[95] John R Searle, S Willis, et al. 1995. The construction of social reality. Simon and

Schuster.[96] Andrew D Selbst, Sorelle Friedler, Suresh Venkatasubramanian, Janet Vertesi,

et al. 2018. Fairness and Abstraction in Sociotechnical Systems. In ACM Confer-ence on Fairness, Accountability, and Transparency (FAT*).

[97] Ian Shrier and Robert W Platt. 2008. Reducing bias through directed acyclicgraphs. BMC medical research methodology 8, 1 (2008), 70.

[98] Herbert Alexander Simon. 1982. Models of bounded rationality: Empiricallygrounded economic reason. Vol. 3. Massachussetts Institute of Technology Press.

[99] Krystyna A Stave and Birgit Kopainsky. 2015. A system dynamics approach forexamining mechanisms and pathways of food supply vulnerability. Journal ofEnvironmental Studies and Sciences 5, 3 (2015), 321–336.

[100] John D. Sterman. 2000. Business dynamics: Systems thinking and modeling for acomplex world. McGraw-Hill.

[101] John D. Sterman. 2001. System dynamics modeling: tools for learning in acomplex world. California management review 43, 4 (2001), 8–25.

[102] John D. Sterman. 2006. Learning from evidence in a complex world. Americanjournal of public health 96, 3 (2006), 505–514.

[103] Amit Surana, Soundar Kumara*, Mark Greaves, and Usha Nandini Raghavan.2005. Supply-chain networks: a complex adaptive systems perspective. Interna-tional Journal of Production Research 43, 20 (2005), 4235–4265.

[104] Joshua B Tenenbaum and Thomas L Griffiths. 2003. Theory-based causal infer-ence. In Advances in neural information processing systems. 43–50.

[105] Kimberly M Thompsonab and Radboud J Duintjer Tebbensc. 2008. Using systemdynamics to develop policies that matter: global management of poliomyelitisand beyond. System Dynamics Review 24, 4 (2008), 433–449.

[106] Jean-François Trani, Parul Bakhshi, Alan Mozaffari, Munib Sohail, HashimRawab, Ian Kaplan, Ellis Ballard, and Peter Hovmand. 2019. Strengtheningchild inclusion in the classroom in rural schools of Pakistan and Afghanistan:What did we learn by testing the system dynamics protocol for communityengagement? Research in Comparative and International Education 14, 1 (2019),158–181.

[107] Jean-Francois Trani, Ellis Ballard, Parul Bakhshi, and Peter Hovmand. 2016.Community based system dynamic as an approach for understanding and act-ing on messy problems: a case study for global mental health intervention inAfghanistan. Conflict and health 10, 1 (2016), 25.

[108] Ludwig Von Bertalanffy. 1950. The theory of open systems in physics andbiology. Science 111, 2872 (1950), 23–29.

[109] MeredithWhittaker, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaz-iunas, Varoon Mathur, Sarah Mysers West, Rashida Richardson, Jason Schultz,and Oscar Schwartz. 2018. AI now report 2018. AI Now Institute at New YorkUniversity.

[110] Sai Wing Yeung. 2011. An investigation of human inductive biases in causalityand probability judgments. Ph.D. Dissertation. UC Berkeley.

[111] Meg Young, Lassana Magassa, and Batya Friedman. 2019. Toward inclusive techpolicy design: a method for underrepresented voices to strengthen tech policydocuments. Ethics and Information Technology 21, 2 (2019), 89–103.

13


Martin et al. 2020

APPENDIXModel Variables and Underlying EquationsWe have placed all models used in this paper at this anonymous online drive (https://figshare.com/s/1d341bf4e24815d7db99) which can beinspected (i.e., simulating using different settings) using the free Stella Player software (https://www.iseesystems.com/softwares/player/iseeplayer.aspx). The descriptions of different variables, and their associated types and units are shown in Table 1. Then, for j ∈ {A,B},where A and B are two population groups, we define the various terms, their initial values and their relationships as follows:

Aj (t) = Aj0 +

t∫0

(n(t) − d(t)) dt

O j (t) = O j0 +

t∫0

(r (t) − p(t) − f (t)) dt

r j (t) = α × τ × дj (t)p j (t) = O j (t) × x j

f j (t) = O j (t) × (1 − x j )

nj (t) = O j × ι × σ

τ× σ − S(t)

σ

d j (t) = Aj (t) × f (t) × δ

τ

υ j (t) ={10 if (t) < 10 Under adjusting loan term intervention10 × x otherwise

λj (t) ={400 if (t) < 10 Under lowering credit score threshold intervention200 otherwise

x j (t) =(

11 + exp(−(47.89 − 0.083 ×Aj (t)))

− 1)× 4 + 5

дj (t) = 11 + exp(−(3.57 + 3.43 ∗Aj (t)/λ))

λ = 400α = .5τ = 10000υ = 10

π j =

{0.8 if j = A

0.6 if j = B

σ = 850ι = .04δ = .25

Simulation Settings:• Simulation start time = 0• Simulation end time = 20• Time units = years• DT = 1/12

14

https://figshare.com/s/1d341bf4e24815d7db99

https://www.iseesystems.com/softwares/player/iseeplayer.aspx

https://www.iseesystems.com/softwares/player/iseeplayer.aspx


Variable Type Units Symbol

average credit score stock points/people Sborrowers stock people Oreceiving loans flow people/year rpaying off flow people/year pdefaulting flow people/year fincreasing credit score flow points/year ndecreasing credit score flow points/year dloan granting threshold parameter points λapplication rate parameter dimensionless/year αtotal population parameter people τaverage loan term parameter year υ

probability of payoff parameter dimensionless πA, πBmax credit score parameter points σaverage increase per year of repayment parameter dimensionless × people /year ιaverage decrease per default parameter dimensionless × people/year δfraction of loans granted auxiliary dimensionless дinterest rate auxiliary dimensionless ieffect of credit score on loan term length auxiliary dimensionless x

Table 1: Description of system variables used for the model described in Figure 2.

15

extending the machine learning abstraction boundary: a … · 2020. 6. 18. · extending the...

Documents