FLUENT Fast Learning from Unlabeled Episodes of Next-Generation Tailoring

TLA Meta-Adaptation Data

Contract: W911QY-16-C-0019

Prepared for:

Dr. Sae Schatz, Director Advanced Distributed Learning (ADL) Initiative

Prepared by: SoarTech

3600 Green Ct., Ste. 600, Ann Arbor, MI48105 Phone:734-327-8000•Fax:734-913-8537•Web:www.soartech.com

2

Contents Figures............................................................................................................................................. 5Tables .............................................................................................................................................. 6RevisionHistory ............................................................................................................................... 6Introduction ..................................................................................................................................... 7Year1 Outcomes as Starting Points ................................................................................................. 7Meta-Adaptation Data Recommendations ...................................................................................... 8Science of Learning Data .............................................................................................................. 10

Assumptions .............................................................................................................................. 11Naming, Exemplars and Inclusivity ...................................................................................... 11Computational Format .......................................................................................................... 11Additional Frameworks ........................................................................................................ 11References ............................................................................................................................. 12Ordering ................................................................................................................................ 12Alignments ............................................................................................................................ 12

Frameworks Overview .............................................................................................................. 12Performance Measurement Frameworks .................................................................................. 13

What data concepts should be included? .............................................................................. 13Competency Frameworks ......................................................................................................... 14

What data concepts should be included? .............................................................................. 14What data alignments are needed? ........................................................................................ 15

Goal & Learning Objective Frameworks .................................................................................. 15What data concepts should be included? .............................................................................. 15What data alignments are needed? ........................................................................................ 15

Credential and Learning Outcome Frameworks ....................................................................... 16What data concepts should be included? .............................................................................. 16

Transcript Framework ............................................................................................................... 16What data concepts should be included? .............................................................................. 16

Context Frameworks ................................................................................................................. 17What data concepts should be included? .............................................................................. 17

Activity Trait Frameworks ........................................................................................................ 17What data concepts should be included? .............................................................................. 17What data alignments are needed? ........................................................................................ 18

Learner Trait Framework .......................................................................................................... 20

3

What data concepts should be included? .............................................................................. 20What data alignments are needed? ........................................................................................ 23

Experience Frameworks ............................................................................................................ 24What data concepts should be included? .............................................................................. 24What data alignments are needed? ........................................................................................ 24

Pedagogic Decisions Frameworks ............................................................................................ 25What data concepts should be included? .............................................................................. 25

Description Languages .............................................................................................................. 28Overview .................................................................................................................................. 28Alignment Description Language ......................................................................................... 29

Simple Text Alignment ......................................................................................................... 29SimpleLinkedAlignment ....................................................................................................... 31DiscreteMeasurementAlignment .......................................................................................... 32ContinousMeasurementAlignment ....................................................................................... 34AlignmentToAlignment ........................................................................................................ 34Alignments and Time ............................................................................................................ 35Summary ............................................................................................................................... 36

Needs Description Language .................................................................................................... 36Asset Data ..................................................................................................................................... 38

Activity State ............................................................................................................................ 38Learner Data .................................................................................................................................. 38

Learner State ............................................................................................................................. 38Inference & Prediction Algorithm Use Cases Supporting Meta-Adaptation ................................ 39

Inference and Prediction Use Case Overview ...................................................................... 39Manually Generate New Data and Alignments ........................................................................ 40Infer Individual Activity Alignments from Activity Source Material ...................................... 41Infer Individual Activity Alignments from Existing Activity Alignments ............................... 42Infer Individual Activity Alignments from Similar Activities ................................................. 42Infer Activity Population Needs Framework Alignments from All Existing Activity Alignments ................................................................................................................................ 42Generate New Learner Data ...................................................................................................... 43Infer Individual Learner Alignments from Experience Framework Alignments ..................... 43Infer Individual Learner Alignments from Existing Learner Alignments ................................ 44Predict Individual Learner Needs from Existing Learner Alignments ..................................... 44

4

Predict Individual Learner Needs from Alignments of Similar Learners ................................. 45Infer Individual Activity Alignments from Learner Population Alignments ........................... 45Predict Individual Activity Needs from Learner Population Alignments ................................ 46Predict Activity Population Needs from Learner Population Alignments ................................ 46Utilize Activity and Learner Alignments for Meta-adaptation ................................................. 47

Appendix A: Data Centric Hard Problems ................................................................................... 47Cross Cutting ............................................................................................................................ 47Science of Learning .................................................................................................................. 48Activity Data ............................................................................................................................. 48Learner Data .............................................................................................................................. 49

References ..................................................................................................................................... 50

5

FiguresFigure 1 TLA Conceptual Diagram ................................................................................... 8Figure 2 SimpleTextAlignment Activity Example ............................................................ 30Figure 3 SimpleTextAlignment Learner Example ........................................................... 30Figure 4 LRMI Alignment Object (Barker, 2014) ............................................................ 31Figure 5 SimpleLinkedAlignment Activity Example ........................................................ 32Figure 6 DiscreteMeasurementAlignment Learner Example .......................................... 33Figure 7 ContinuousMeasurementAlignment Activity Example ...................................... 34Figure 8 AlignmentToAlignment Activity Example .......................................................... 35Figure 9 Desired Future Alignment ................................................................................ 36Figure 10 Inference & Prediction Use Case Summary ................................................... 40

6

TablesTable 1 Science of Learning Frameworks Overview ...................................................... 12Table 2 Competency Framework Alignments ................................................................ 15Table 3 - Goal & Learning Objective Framework Alignments ........................................ 16Table 4 Data Alignments: Activity Trait Framework ....................................................... 18Table 5 Data Alignments: Learner Trait Framework ...................................................... 23Table 6 Experience Framework Alignments ................................................................... 24Table 7 Alignments and Time ......................................................................................... 35Table 8 Need Definition .................................................................................................. 36Table 9 Conditional Patterns Identifying Need ............................................................... 37Table 10 Need Resolution Patterns ............................................................................... 38Table 11 Manual Generation of Data And Alignments ................................................... 41Table 12 Inferences from Activity Sources ..................................................................... 41Table 13 Inferences from Activity Alignments ................................................................ 42Table 14 Inferences from Similar Activities .................................................................... 42Table 15 Inferences from all Existing Activity Alignments .............................................. 42Table 16 Learner Source Data ....................................................................................... 43Table 17 Individual Learner Alignments ......................................................................... 44Table 18 Inferences about Learner Alignments from Existing Learner Alignments ....... 44Table 19 Inferences about Learner Alignments with Needs Frameworks from Existing Learner Alignments ........................................................................................................ 44Table 20 Inferences about Learner Alignments with Needs Frameworks from Similar Learner Alignments ........................................................................................................ 45Table 21 Inferences about Activity Alignments from Learner Population Alignments .... 45Table 22 Predict Activity Alignments with Needs Frameworks ...................................... 46Table 23 Predict Activity Population Needs from Learner Population Alignments ......... 46Table 24 Activity and Learner Alignments for Meta-adaption ......................................... 47 RevisionHistory

Name Date Reason for Change Version SoarTech 3/29/2018 Initial Draft 0.1.0

7

IntroductionThis document presents the data related requirements for supporting meta-adaptation in the long term. Requirements for the TLA demonstration in 2018 will be a subset of the requirements presented here, and will be covered in separate 2018 design documents. This document’s purpose is to identify the types of data and data interactions that are necessary for algorithmically producing recommendations that result in meta-adaptation. The analysis included identifying data centric hard problems elicited in Year1 and evaluating potential solutions, considering both human and software use cases. The data concepts included in our analysis are grounded in learning theories that identify instructional variables and their optimal relationships. Over the past century, educational psychologists and researchers have posited many theories to explain how individuals acquire, organize, and deploy skills and knowledge. We considered the three major classes of learning theories (a) behaviorist learning theories, (b) cognitive-information processing learning theories, and (c) cognitive-constructivist learning theories (Driscoll, 1994), in addition to (d) emerging neurobiological learning theories, which represent a physiological approach to correlating learning to electrical and chemical activities taking place in the brain and central nervous system (Hirumi, 2016). From these theories about the relationships among instructional variables and the analysis of TLA use cases, we defined the categories of data necessary to enable meta-adaptation, as well as computationally accessible description languages needed for a machine to reason over the data categories.

Year1OutcomesasStartingPointsAnalysis of the Year1 prototype and Fort Bragg exercise revealed that while software services are an important part of a TLA instantiation, the TLA is fundamentally about data. As a result of analyzing Year1’s results we re-organized the high level conceptual diagram for the TLA to reflect its data centric nature, as shown in Figure 1.

8

Figure 1 TLA Conceptual Diagram

Analysis of why this re-organization is important is presented in the TLA Design Document 0.1.5. The analysis of Year1’s work also identified some data centric hard problems that impact meta-adaption including:

• Lack of agreement on terminology • Currently granularity and breadth of standards is not sufficient to support adaptation • Organization of Science of Learning Concepts is challenging because many concepts

have not been formalized yet into software compatible formats • Theories represented in the data are not static, they change over time • Alignments are a central hard problem that cannot be ignored as an anomaly • Long term acceptance requires achieving a pleasing balance between machine

automation and human control • Good recommendations require specific and detailed information about the situation,

which imposes a data sharing requirement that all participants may not be comfortable with

• Currently manual curation burden is prohibitive For more information on these challenges see Appendix A: Data Centric Hard Problems. The analysis presented in this document detailing how to curate the TLA data sharing necessary for enabling meta-adaptation was guided by seeking means to overcome these challenges.

Meta-AdaptationDataRecommendationsProducing personalized recommendations requires sufficient computationally accessible input for recommendation algorithms to reason over. Like human instructors, the recommendation

9

algorithms need information about learners and activities in order to suggest the best learning experiences in the best sequence for each individual or group of learners. Learners have different strategies, approaches, and capabilities for learning that are a function of prior experience and heredity (Hirumi, 2016). The more information available for analysis, the better the match that can be made among variables (e.g., learner, activity, learning goal). Information about content to be learned (i.e., the learning objectives of each activity) and how those objectives relate to over-arching learning goals (e.g., to pass an exam, to learn a new procedure) is also necessary, so that the most appropriate activities to help a particular learner achieve those learning goals are recommended. For example, if a user has a ninth-grade reading level and an activity presents text at a twelfth-grade reading level, that activity should not be recommended for that particular learner. It is crucial that the algorithms producing meta-adaptation recommendations are grounded in learning science. Additionally, to meet the needs of educational researchers, the TLA must be flexible enough to allow recommendation algorithms to be adapted and changed over time as educational research advances. To do this, the supporting data must be broad enough to provide input needed not just for one learning science theory driving a specific recommender, but for many theories driving many recommenders. The categories of data needed include:

• Learner Traits and States • Activity Traits • Learning Objectives, Learning Outcomes, Goals and Credentials • Finer Grain Experience Tracking and Context • Pedagogic Decisions • Relationships Between Concepts

To have broad appeal to educational researchers, the TLA needs to be approachable to more than just engineers, using vocabulary educational experts they can identify. To be operationalizable, the TLA must find a way to standardize that vocabulary to serve as a bridge between instructional experts, educational researchers and engineers. Therefore, our analysis has focused on organizing the approach to data to achieve the following goals:

• Enhanced Community Involvement: Place definitions of the vocabulary relevant to learning in the TLA Science of Learning Data Model to provide a natural place for science of learning experts (scientists who are not necessarily engineers) to contribute to the theory that is operationalized in a TLA instantiation.

• Enhanced Standards Organization: Organize the data definitions so standards can be modular, each focused on a concept category that that expresses the intrinsic properties of the concept and its expected interactions with other concepts.

• Reusable Object Descriptions: Define basic concepts such as characteristics and interactions relevant to learning in a modular way so they can be utilized for describing both learners and activities.

• Enhanced Sharing Across Software Systems: Express the common vocabulary for discussing Activity or Learner characteristics a location that is available to everyone to ensure that the information is not encapsulated inside proprietary components or services.

To achieve these goals, we recommend: • Richer concept definitions in Science of Learning Data Model.

10

• Standardize frameworks that organize the vocabulary of concepts into modular, logical groupings in the Science of Learning Data model.

• Standardize description languages that can be used to express relationships between concepts in different frameworks.

• Define standardized languages of primitives that can be used by frameworks to express many specific theories.

• Use standards profiles to express specific theories of interest in specific TLA instantiations.

• Use the vocabulary defined in the Science of Learning Data model to describe Learner State and Activity State.

One of the key outcomes of our analysis is that meta-adaptation requires a much richer set of concept definitions in the Science of Learning data model. In Year1, only Competency was defined in the Science of Learning Data. This is valuable and gave us a vocabulary for discussing KSAs. But to enable meta-adaptation algorithms we also need the vocabulary to discuss goals & learning objectives, learner traits, learning context and pedagogic decisions. To meet our flexibility objectives, the concepts defined in Science of Learning frameworks should be standardized using common ground primitives. Instead of encoding a single research viewpoint as a standard, instead, we should identify primitives that can be used to express many research viewpoints. For example, a KSA object is a common ground primitive that allows the definition of many specific competencies. Similarly, we should identify primitives for discussing the other Science of Learning data such as pedagogic decisions and learner traits that are flexible enough to allow many instructional research perspectives to be expressed. The remainder of this document will describe the Science of Learning frameworks needed to provide the concept vocabulary and the description languages needed to express relationships between concepts. It will then describe how the frameworks and description languages are utilized in the description of Learner State and Activity State and how the data contained in the Science of Learning Data Model, Learner Data Model and Asset Data Model is utilized by the inferences and predictions necessary for realizing meta-adaptation. ScienceofLearningDataFrameworks should provide the vocabulary for discussing ontologies of objects, their relationships, and the specific theories they represent. Frameworks should consist of primitives that can be used to express concepts (e.g., “CompetencyNode”) and that can be used to create tools that enable Learning Experts to express many specific instances of the concept (e.g., “Conduct Pen Test”). The primitives should be slow changing and general enough to be encoded into a standard that is applicable to all instances of the TLA. Profiles (like an xAPI profile) should be used to express specifics that are applicable to a single instantiation of the TLA. The frameworks provide the definitions of “what exists” that influences learning. Some of the frameworks described here already exist in the science of learning community, and others are new frameworks that the community should develop to support future-looking learning across systems using the TLA. It is anticipated that, as the TLA is developed, new frameworks and primitives will be discovered to be necessary. It is recommended that the frameworks be organized around logical groupings of concepts. It is further recommended that standards be organized around frameworks, one standard per framework. This is the strategy that Medbiquitous (Medbiquitous, 2017) is using very successfully in their medically oriented standards.

11

AssumptionsIn this section, we discuss the underlying assumptions of our analysis. Naming,ExemplarsandInclusivityThe TLA aims to be inclusive, encouraging learning experts to integrate their models and theories with the TLA. To do so, the Science of Learning frameworks provide a place for existing and future concepts relevant to learning to be expressed. The descriptions of the frameworks and concepts are intended to be read as functional requirements; the intent is to describe the concept that would be useful to include. The specific words used for the framework name or any concept within the framework that are ultimately encoded into a standard should not be dictated by any one party. They should be determined by the community effort led by ADL as part of the standardization process. In addition, we want the language of primitives to be capable of expressing multiple theories. To convey our meaning regarding the types of learning theories that should be represented, we will list well-known examples by name when it is applicable and cite the appropriate source. The specific examples referenced are not intended as an exhaustive list of all known theories and their elements. Nor are they intended to be a list of specific theories to endorse. Instead they are illustrative of the concepts that researchers in the field have studied, or are studying currently, that we can use to understand the breadth of information that researchers in the future may wish to study. They should be considered representative exemplars. The concepts described here are intended as a starting point. First, they serve as functional requirements that enable identification of existing matching specifications and gap analysis to identify where the existing specifications do not cover data we anticipate needing. As specifications are matched to the functional requirements, we expect that the naming of individual primitive constructs will change to reflect good naming choices that have already been encoded in standards by the community. Second, we encourage new concepts to be included as they are discovered to add value. In particular, if existing standards identified for use have additional constructs that add to the set of concepts expressed here, the intent is to include them, not to omit them because they are not already listed here. The goal of what is listed here is to present a realistic description of the breadth of data necessary for supporting good meta-adaptation. ComputationalFormatTo be useful in real-time software instantiations, the frameworks’ concepts must be computationally accessible. That is, the data must be encoded in a machine-understandable way to allow complex algorithms to reason over them. It is assumed that all data concepts discussed in this document should be encoded in a machine-accessible format. Unique identifiers are part of enabling machine accessibility. We will assume that all framework and object instantiations will include unique identifiers to make referencing them from another framework possible. We will not list this exhaustively in each functional description, to allow the functional description to focus on concepts, not implementation detail. AdditionalFrameworksThe set of frameworks described here represents the logical grouping of instructional variables already known to be important. It is not yet clear what the optimal level of detail for any one framework will be. We anticipate that as complexity increases, elements that are currently included in Learner Trait or Activity Trait frameworks may need to be broken out into individual frameworks to better facilitate modularity. The standards development done by the community

12

and guided by ADL will guide division of the functional requirement concepts into the specific frameworks that should be standardized. We recommend introducing a new framework any time it can enhance modularity. The frameworks that are recommended here should be considered as a starting point for introducing the necessary modularity. The word “frameworks” has been used instead of “framework” because it is envisioned that more than one valid conceptualization of the primitive constructs is needed to express the idea is possible. Therefore, more than one framework may exist in a functional category. ReferencesTo facilitate modularity, when defining the concepts that describe the objects and relationships in one framework, the definition may reference an object or relation defined by another framework. For example, describing a goal such as “Obtain a formal certification in Cyber Security” should not redefine the concept of competency and the concept of a certification, but should instead reference those concepts in a shared framework where they are defined. OrderingThe order in which the frameworks are presented, and the order in which concepts in each framework are presented, is not intended to indicate relative importance. The ordering represents logical groupings only and is intended to facilitate understanding of the concepts. AlignmentsIn regard to alignments, it is our assumption that the number of alignments among instructional variables (e.g., learner, activity, assessment, etc.) and frameworks will need to grow as additional relationships are discovered to be relevant (and necessary) to producing good recommendations. FrameworksOverviewAn overview of the frameworks known to be needed is shown in Table 1 Science of Learning Frameworks Overview. Table 1 Science of Learning Frameworks Overview

Framework Concept Examples Competency Frameworks What are the Job Tasks and KSAs that can be

learned? Hierarchy of skills, knowledge, and abilities (e.g. Perform Pen Testing) that job tasks (e.g. Cyber Security).

Goal & Learning Objective Frameworks

What are the objectives for learning and relationships between them?

Become an expert in Cyber Security, obtain a formal certification, become a more confident public speaker

Credential & Learning Outcome Frameworks

What learning outcomes can achieved? What organizations govern achievement certifications?

Diploma issued by university, Certificate issued by skill sustainment organization

Activity Trait Frameworks What are the characteristics and state of activities that can influence learning?

Activity X has human readable description Y, Activity X uses baseball metaphors

Learner Trait Frameworks What personal characteristics and state can influence learning?

12th grade reading level, Motivated by social-learning, High self-regulatory ability, Language processing disorder

13

Experience Frameworks What moment-to-moment actions can impact learning directly or indirectly?

Learner answered quiz X’s question Y on competency Z correctly, Learner added event X to calendar Y

Context Frameworks What external situations influence learning directly or indirectly?

Home vs. work, Available devices, Solo vs. group, Online vs. face-to-face, Weather, Traffic, Crew rest conditions and schedule, time of day (likelihood of fatigued state)

Pedagogic Decisions Frameworks

What learning theories, instructional methods, and pedagogic sequencing strategies are relevant?

Learner-Centered vs. Teacher-Directed, Whole-task vs. Part-task, Hierarchical vs. Holistic (e.g., Case-based), Deliberate Practice Theory (Ericcson et al., 1993) vs. Cognitive Transformation Theory (Klein & Baxter, 2006)

Performance Measurement Frameworks

What are the specific rulers (scales) that can be used for measurement, and what are the concepts appropriate to measure with each ruler?

Ruler Examples: Percent correct, Response time, Pass-fail, Rubric-based score, Rubric-based assignment to proficiency category Concepts Measured: Skill proficiency Knowledge (depth and breadth) Transfer to novel context KSA acquisition

Transcript Frameworks What are academic, military and employment history is relevant?

University Degree, Employment Job Titles, Military Rank and Deployment History

Detailed descriptions of these frameworks are provided in the subsequent subsections. PerformanceMeasurementFrameworksPerformance measurement frameworks should provide the vocabulary for describing rulers or scales for measuring learner performance and measuring characteristics of activities that influence prediction of their likely impact on learner performance. Whatdataconceptsshouldbeincluded?Models for Measuring Learner Performance

• Job Task Proficiency Levels o Novice/Proficient/Expert (Jonassen, 2011) o Naval Aviators scored U/F/G/E o USMC rifle qualifications, numeric scores from 0 to 250 with thresholds to

achieve Marksman, Sharpshooter, or Expert • Competency Mastery Estimate Levels

o held/not held o Four Stage

§ 0 -> the learner does not know about the skill or does not care about the skill

§ 1 -> the learner cannot accomplish the skill but does want to § 2 -> the learner can accomplish the skill but only with effort § 3 -> the learner can accomplish the skill without effort

• Test Scoring

14

o percentage 0-100% o pass/fail

• Intelligence Level § standardized test score such as IQ defined to have mean value 100 and

standard deviation 15 relative to a given population • Reading Level

§ K12 Grade Level, College Level (Common Core State Standards Initiative, 2018)

• Knowledge check Level § Pass § Fail

Models for Measuring Activities • Challenge level (based on Parsons (2008))

§ 1 - Not at all challenging (easy) § 2 - Presents some challenge § 3 - Presents significant challenge § 4 - Presents insurmountable challenge

• Interactivity level (IMI level) § passive § limited participation § complex participation § real-time participation

• User rating (e.g., 5 stars) • Quality level (e.g., low, medium, high) • Replayability score (e.g., low, medium, high) • Level of fun (e.g., low medium high) • Sophistication of instruction (e.g., low, medium, high) • Learner independence level (amount of scaffolding) • Learner control level (e.g., self-paced vs timed, skip allowed etc.) • Level of repetition (e.g., low, medium, high)

Models for Measuring Goals

• Level of progress (e.g. percent complete) CompetencyFrameworksCompetency frameworks should provide the vocabulary for describing the Job Tasks and KSAs that can be learned and the relationships between them. The word competency is used to denote a single item that can be learned. Whatdataconceptsshouldbeincluded?Objects

• Job Tasks with a human-readable name and description • Terminal Learning Objectives (TLOs) and Enabling Learning Objectives (ELOs) with a

human-readable name and description • Competencies with a human-readable names and description, and a type (e.g.,

knowledge, skill, or ability).

15

Relationships • KSAs that support each Job Task. Each KSA supports only one Job Task • TLOs/ELOs that support each Job Task. • Prerequisite and hierarchical relationships between Job Tasks • Prerequisite and hierarchical relationships between TLOs and ELOs • Prerequisite and hierarchical relationships between KSAs

Whatdataalignmentsareneeded?The types of alignments, the frameworks to which they are aligned, and the purposes they serve are detailed below in Table 2. Table 2 Competency Framework Alignments

Alignment Frameworks Aligned To Type of measurement appropriate for measuring Job Task proficiency. May be specified once in a framework instantiation rather than per Job Task.

performance measurement framework

Type of measurement appropriate for measuring mastery of TLOs and ELOs. May be specified once in a framework instantiation rather than per TLO or ELO.

performance measurement framework

Type of measurement appropriate for measuring mastery of KSAs. May be specified once in a framework instantiation rather than per KSA.

performance measurement framework

Goal&LearningObjectiveFrameworksGoal and learning objective frameworks should provide the vocabulary for describing reasons for learning and the outcomes sought. Whatdataconceptsshouldbeincluded?Goal & Learning Objective Classification Theories

• Cognitive learning theories (e.g., Anderson et al., 1992; Bloom, 1956; Gagne´, 1977; Merrill, 2002)

• Affective learning theories (e.g., Krathwohl Bloom & Masla, 1964) • Psychomotor learning theories (e.g., Simpson, 1966)

Goal & Learning Objective Description Theories

• A.B.C.D. method (Heinich, et al., 1996) Goal

• Affiliated classification theory • Description of the goal using an appropriate Goal & Learning Objective Description

Theory • Goal pursuit motivation/rationale (e.g. personal, academic, job requirement) • Goal priority • Desired goal achievement timeframe • Goal Status (e.g. actively pursuing, abandoned, completed) • Origin (e.g. learner-produced or instructor-mandated)

Whatdataalignmentsareneeded?The types of alignments, the frameworks to which they are aligned, and the purposes they serve are detailed below in Table 2.

16

Table 3 - Goal & Learning Objective Framework Alignments

Alignment Frameworks Aligned To Job Task, KSA, TLO or ELO to be pursued by the goal competency framework Learning outcome expected as a result of goal completion credential and learning

outcome framework Context for goal pursuit context framework Measurement or threshold for determining goal success performance measurement

framework Level of progress toward goal completion

performance measurement framework

CredentialandLearningOutcomeFrameworksCredential and Learning Outcome Frameworks should provide the vocabulary for describing what learning outcomes can achieved and what organizations are authorized to issue the outcomes. Whatdataconceptsshouldbeincluded?Learning Outcomes

• Micro Credentials (e.g. mastered a specific KSA, TLO or ELO) • Certifications • Diplomas • Awards • Licenses • Accomplishments • Skill Sustainment

Models of Types of Learning Outcomes

• Verbal information, intellectual skills, discriminations, defined concepts, cognitive strategies, attitudes, motor skills (Gagne´, 1977)

Organizations

• Accredited to issue credentials • Accredited to store credentials

TranscriptFrameworkThe Transcript Framework should provide the vocabulary for describing academic, employment and military records. Whatdataconceptsshouldbeincluded?Employment Transcript

• Job Title • Organization • Start/End Date • Location • Team • Supervisor

Academic Transcript • School Attended

17

• Start/End Date • Location • Classes taken • Advisors/Instructors

Military Transcript • Branch • Rank • Deployment Locations • Start/End Date • Medals or Distinctions

ContextFrameworksA context framework should provide the vocabulary for external factors that influence learning directly or indirectly. Note: It is not entirely clear if the concepts described here should be part of a single framework, or if they should be broken out into multiple frameworks (such as device types). The concepts described here have been limited to external context (outside a person). Internal context is included in the description of Learner Traits. It is clear that describing external conditions in a common location is valuable for describing both Activity and Learner state. For example, a learner has a set of currently available devices, and an activity is compatible with a set of specific devices. It is preferable to have a common vocabulary for devices that can be used in both descriptions. Whatdataconceptsshouldbeincluded?Devices

• Category (e.g., PC, Laptop, Tablet, Phone, Smart Watch etc.) • Platform (e.g., operating system, web) • Sensors (e.g. webcam, fingerprint scanner, physio sensor) • Input Modality (e.g. mouse, keyboard, joystick, touchscreen, microphone) • Output Modality (e.g. monitor vs HMD, screen dimensions, speakers)

Environment & Location

• Public/private • Home/work/business • GPS coordinate • Weather conditions & Environmentals affecting physical performance (e.g. Ambient

temperature) • Traffic conditions

ActivityTraitFrameworksActivity trait frameworks should provide the vocabulary for describing characteristics of an Activity that can influence learning directly or indirectly. Whatdataconceptsshouldbeincluded?Basic Properties

• Human Readable Name • Human Readable Description • Vendor, if applicable (e.g., Moodle, CyberScorpion, Sero!, etc.)

18

• Publisher/Author • Rights: copyright, licenses, etc. • Version number (e.g., version 1.2) • Icon/Preview

Interoperability Properties

• Launch URI • Configuration options - Note: In Year1 there was a single parameter. However, we

anticipate future exploration of launch to reveal a better way to define the vocabulary for describing configuration options, and a runtime API for invoking them. In future, the framework should only provide the vocabulary building blocks for discussing configuration options.

• Protocols supported (e.g., REST, pub/sub) • Required Services depended upon for minimum functionality • Authorization Level describing what the asset is authorized to access • Metadata interrogation endpoint - If the activity supports responding dynamically to

programmatic requests for metadata, this endpoint is where the requests should be directed

Learning Centric Properties Note: The majority of learning centric properties are alignments rather than simple properties.

• Expected time needed for completion • Minimum suggested time for completion • Category (e.g., text, simulation, video, audio, static content, interactive game, flash

cards, quizzes, etc.) • Intended audience • Associated keywords • Scope (e.g. course, unit, lesson, event, element) • Instructional Type (e.g. presentation, practice, assessment, problem, demonstration,

discussion, support) Population Characteristics In addition to describing the traits and characteristics of individuals, we may also need properties specific to describing populations of activities in future. Examples include

• Refresh Needs • Content Gaps

Whatdataalignmentsareneeded?The types of alignments, the frameworks to which they are aligned, and the purposes they serve are detailed below in Table 4. Table 4 Data Alignments: Activity Trait Framework

Alignment Frameworks Aligned To Relationship per MOS, Job or Job Task (e.g. Prerequisite, Teaches, Assesses)

competency framework

Relationship per KSA, TLO, or ELO (e.g. Prerequisite, Teaches, Assesses) competency framework Challenge level per KSA, TLO, ELO, or Job Task competency framework,

performance measurement framework

19

Interactivity level per KSA, TLO, ELO, or Job Task competency framework, performance measurement framework

Level of fun performance measurement framework

Average user rating performance measurement framework

Average learner success rate performance measurement framework

Average learner attempt rate performance measurement framework

Replayability score performance measurement framework

Quality level performance measurement framework

Sophistication of instruction performance measurement framework

Learner independence level (amount of scaffolding) performance measurement framework

Learner control level performance measurement framework

Level of repetition performance measurement framework

Expected Learner gains in KSA mastery or Job Task proficiency upon completion

competency framework, performance measurement framework, credential & learning outcome framework

Goals and learning objectives that this Activity supports goal & learning objective framework

Credentials and learning outcomes that this Activity supports credential & learning outcome framework

Type of measurements this Activity produces if it performs assessment or scoring

performance measurement framework

Locations that are appropriate for performing this Activity context framework Category of device this Activity can run on or requires to run context framework Platform or operating system this Activity can run on or requires to run context framework Sensors this Activity can use or requires for operation context framework Input modalities required or compatible with the Activity context framework Output modalities are required or compatible with the Activity context framework Physical traits & capabilities of a learner that are compatible with this activity

learner traits framework

Physical fitness levels of a learner that are compatible with this activity learner traits framework Internal cognitive states of a learner that are compatible with this activity learner traits framework Thinking styles of a learner that are compatible with this activity learner traits framework Intelligence types of a learner that are compatible with this activity learner traits framework Personality traits of a learner that are compatible with this activity learner traits framework Preferences of a learner that are compatible with this activity learner traits framework Interests of a learner that are compatible with this activity learner traits framework Transcript qualities of a learner that are prerequisites for this activity transcript framework Pedagogic sequencing dimensions this activity is appropriate for pedagogic decisions

framework Locus of control this activity is appropriate for pedagogic decisions

framework

20

Instructional strategies this activity is appropriate for pedagogic decisions framework

Learning theories this activity is compatible with pedagogic decisions framework

Instructional philosophies this activity is compatible with pedagogic decisions framework

Instructional design models this activity is compatible with pedagogic decisions framework

Support mechanisms this activity employs pedagogic decisions framework

Task types this activity is compatible with pedagogic decisions framework

LearnerTraitFrameworkA learner trait framework should provide the vocabulary for describing facts about a Learner that can influence learning and provide a basis for personalization including their basic information, internal mental models, physical traits and preferences. Note: Privacy concerns intersect with data that enables personalization. Finding a balance where personalization is meaningful without being intrusive is an ongoing hot topic in the tech industry. ADL has conducted a study on privacy concerns for the TLA that is outside the scope of this document. This document does not attempt to reconcile privacy concerns. The data this section describes, if available, would be useful for personalization algorithms to take advantage of. We have considered the data that might be useful for a recommendation system from the perspective of what would might a human instructor know about a learner when personalizing their learning? If we wish for algorithms to produce recommendations that are personalized similar to the way an instructor can personalize learning for an individual learner, then it stands to reason that the same types of information available to an instructor may need to be available to a recommendation algorithm. Which specific data is actually utilized in a TLA instantiation will require careful consideration of the tradeoffs between privacy concerns and functionality that is outside the scope of this document. Whatdataconceptsshouldbeincluded? Basic Learner Information that Supports Record Management

• User Name • Real Name • Email Address • Social Media IDs (user names) • Phone number • Physical Mailing Address

Personal Information that May be Used for Research with Participant Permission

• Gender (M/F/L/G/T/B/+) • Age/Birthday • Ethnicity • Relationship Status • Number of children • Academic performance measures

21

o Grade point average o Standardized test scores (GRE, SAT, ACT, ASVAB, etc.)

Physical Traits & Capabilities

• Height • Weight • Fitness Level

o Muscular endurance o Muscular strength o Cardiovascular endurance o Flexibility o Body composition

• Maximum Exertion level o Muscular endurance o Muscular strength o Cardiovascular endurance o Flexibility o Body composition

• Physical Challenges o Sight (Yes/no/Impaired/RGBlind/ColorBlind) o Hearing(Yes/no/Impaired) o Hands(Yes/No/Impaired) o Full Body(yes/no/impaired)

Measures of Physical Fitness (from the Army Physical Fitness Test)

• Number of Timed Sit Ups • Number of Timed Push Ups • 2-Mile Run Time

o Alternative: 800-yard swim time o Alternative: 6.2-mile bike time o Alternative: 2.5-mile walk time

Internal Cognitive States that might be studied with respect to implications for learning support

• Intrinsic motivation; models include: o ARCS model of motivational design (Keller, 1983; Hirumi, 1993)

§ Attention § Relevance § Confidence § Satisfaction

o Herzberg’s (1966, Herzberg, Mausner, & Snyderman, 2011) theory of job motivation

o Locke and Latham’s (1990) goal-setting theory of motivation o Bandura’s (1977) self-efficacy theory o McClelland’s (McClelland, Atkinson, Clark, & Lowell, 1953) theory of

intrinsically driven achievement motivation • Arousal

o Inverted-U hypothesis o Hancock’s (1989) model of stress and sustained attention

• Frustration

22

• Boredom • Engagement • Cognitive workload • Fatigued vs. Alert • Self-Regulation during learning (see review by Zimmerman, 2013) • Affect, Emotion and Mood States; examples include:

o Anger, aversion, courage, dejection, desire, despair, fear, hate, hope, love, sadness (Arnold, 1960)

o Rage, terror, anxiety, joy (Gray, 1978) o Anger, interest, contempt, disgust, distress, fear, joy, shame (Tomkins, 1984) o Anger, disgust, anxiety, happiness, sadness (Oatley & Johnson-Laird, 1987)

Learning Styles; examples include:

• Visual, aural, read/write, and kinesthetic (VARK; Fleming, 1995) • Visual, aural, and kinesthetic (VAK, e.g., Barbe, Swassing, & Milone, 1979) • Dunn and Dunn’s (1992) model of learning styles • Honey and Mumford’s (1992) Learning Styles Questionnaire • Kolb’s (1999) Learning Style Inventory • Vermunt’s (1994) Inventory of Learning Styles (ILS) • Surface, deep, and achieving learning styles (Biggs, 1987) • Other categories:

o Connector learners (try to make connections to existing knowledge) o Self-aware learners (awareness of learning strategy used and its

effectiveness) o Experienced vs Novice (at learning) learners

Thinking Styles, such as:

• Allinson and Hayes’ (1988) Cognitive Styles Index • Gregorc’s (1979) Mind Styles Model and Style Delineator (GSD) • Benziger’s thinking styles (Benziger & Sohn, 1993) • Sternberg’s [1997] Thinking Styles Inventory: Executive, monarchic, local,

conservative, legislative, judicial, hierarchic, anarchic, global, liberal, etc.

Intelligence Types; models include: • Gardner’s (2011) Theory of Multiple Intelligences • Cattell-Horn fluid-crystallized model (Cattell, 1943) • Carroll’s (1993) three-strata model • Vernon’s (1964) verbal-perceptual model

Personality Traits; taxonomies include:

• NEO: Neuroticism, Extraversion, Openness (Costa & McCrae, 1980) • The basic five-factor model of personality traits: Openness, Conscientiousness,

Extraversion, Agreeableness, and Neuroticism (e.g., Norman, 1963) • The Myers-Briggs Type Indicator (MBTI) (Myers, 1962) • HEXACO Personality Inventory-Revised (e.g., Lee & Ashton, 2004) • The Seven-factor Temperament and Character Inventory (TCI; Cloninger, Svrackic,

& Przybeck, 1993)

Learner Preferences

23

• Language • Reading Level • Information consumption format: (Text, audio, video etc.) • Notification style (Delivery method, Frequency) • Highly vs less structured learning strategy • Superficial vs deep learning

Personal Interests that may be collected by researchers with participants’ informed consent:

• Hobbies • Lifestyle • Affiliations (Professional, Academic, Volunteer, Religious, etc.) • Group Memberships (Newsgroups, Forums, Social Networks, etc.)

Population Characteristics In addition to describing the traits and characteristics of individuals, in future we may also need the ability to describe populations of learners. There are three aspects that deserve further investigation: 1. Teams of learners 2. Computationally observed population patterns identified by aggregating data from specific

individual learners. 3. Generalized population patterns of behavior as described by a learning expert

Whatdataalignmentsareneeded?The types of alignments, the frameworks to which they are aligned, and the purposes they serve are detailed below in Table 4. Table 5 Data Alignments: Learner Trait Framework

Purpose Frameworks Aligned to Proficiency per MOS, Job or Job Task, confidence in the estimate should also be recorded.

competency framework, performance measurement framework

Mastery estimate per KSA, TLO or ELO, confidence in the estimate should also be recorded.

competency framework, performance measurement framework

Goals and learning objectives affiliated with the learner goal & learning objective framework

Level of progress per goal goal & learning objective framework, performance measurement framework

Time spent per goal goal & learning objective Time spent per activity activity trait framework Number of attempts per activity activity trait framework Credentials and learning outcomes achieved credential & learning

outcome framework Learner’s relationships (e.g. currently at, frequently at) with locations context framework Learner’s relationships (e.g., available to, currently using) to device categories

context framework

24

Learner’s relationships (e.g., available to, currently using) to platforms or operating systems

context framework

Learner’s relationships (e.g., available to, currently using) to sensors context framework Learner’s relationships (e.g., available to, currently using) to input modalities

context framework

Learner’s relationships (e.g., available to, currently using) to output modalities

context framework

Learner’s academic, job or military transcript transcript framework Available time to attempt an activity context framework

ExperienceFrameworksAn experience framework should provide the vocabulary for describing moment to moment actions of a learner that impact learning directly or indirectly. They do not need to be limited to only learning experiences. Other life experiences can be relevant to personalizing learning. Whatdataconceptsshouldbeincluded?Fine Grained Learning Experiences

• Minute to Minute Learning Activity Interactions (e.g. xAPI statements) including • Timestamp: Date and time when the experience took place • Experience Description (e.g. xAPI verb statements) • Success: Whether or not the attempt on the Activity was successful. • Duration: Period of time over which the experience occurred. • Score location – where in the xAPI statement score is located (if present)

Non-Learning Experiences Note: Many more possible, what is valuable is source data for that can be used to infer information about personal context that influences how learning should be delivered

• Calendar Interactions • GPS Interactions (like google maps) • Social Media Interactions • Communication Activity • On the Job Activity • Hobby Activity • Event Participation

Whatdataalignmentsareneeded?The types of alignments, the frameworks to which they are aligned, and the purposes they serve are detailed below in Table 6. Table 6 Experience Framework Alignments

Alignment Frameworks Aligned To The leaner that had the experience learner trait framework The activity that produced the experience data activity trait framework The score (and measurement scale) if the experience reports a score performance measurement

framework The maximum possible score if the experience reports a score performance measurement

framework Category of device used for the experience context framework

25

Platform or operating system used for the experience context framework

Sensors used for the experience context framework

Input modalities used for the experience context framework

Output modalities used for the experience context framework

Per KSA relationship (e.g. demonstrated, refuted) competency framework

Per Job Task relationship competency framework

Note1: We have discovered that some of the information already present in xAPI statements could benefit from alignments. For example, the device the experience took place on, or the scale by which score should be interpreted. We recommend that rather than replicating alignment information as additional objects within xAPI, that instead xAPI be extended to support alignments to other frameworks. Note2: Additionally, further exploration in Year2 is required to understand the optimal location for data needed for inferences. Meta-adaptation needs the activities themselves to be annotated with data to assist with recommending the next best activity. In Year1, it was assumed that interpreting the xAPI statement could be done by cross referencing the experience statement with the metadata about the generating activity. For example, to figure out what KSA the experience impacted. However, it has become apparent that the activity metadata is not sufficient to entirely interpret the experience. For example, if an activity teaches more than one KSA or can support more than one device, we need to know which specific KSA or device was actually used during the experience. We have suggested that these become alignments, however, an open question remains regarding which alignments are optimal for providing the necessary clarifications without unduly burdening the Activity developer generating the experience statements.

PedagogicDecisionsFrameworksPedagogic Decision Frameworks should provide the vocabulary for describing pedagogic sequencing models, instructional theories and strategies, learning support mechanisms and types of tasks to be learned, as well as other vocabulary necessary for describing research and theory about learning mechanisms. Note, we are actively working with instructional experts to identify the important aspects to include in this framework, drawing upon decades of instructional research. We anticipate that this area will grow, and that it may need to be broken out into multiple individual frameworks. Whatdataconceptsshouldbeincluded? Pedagogic Sequencing Dimensions for ordering learning objectives, task parts, and other work-decomposition products

• Hierarchical, based on job-duty-task analysis (JDTA) • Easy to difficult • Concrete to abstract

26

• Increasing in conceptual elaboration, procedural elaboration, and/or theoretical elaboration

• Simple to complex o Gagne’s (1977) learning progression: Stimulus-response connections to motor

and verbal chains to multiple discriminations to concepts to simple rules to complex rules

• Task performance timeline • Knowledge to skills

Locus of control • Learner Centered • Teacher Directed

Instructional strategies

• Training Strategies o Scenario or event based (e.g., Fowlkes, Dwyer, Oser, & Salas, 1998) o Compare-and-contrast (e.g., Bransford, Franks, Vye, & Sherwood, 1989) o Cross-training (e.g., Tannenbaum, Smith-Jentsch, & Behson, 1998) o Guided exploration (e.g., Kass, Burke, Blevis, & Williamson, 1994) o Perceptual training (e.g., Biederman & Shiffrar, 1987) o Deliberate practice (Ericcson et al., 1993) o Game based (Prensky, 2000)

• Instructional Design Guidance o Direct Instruction

§ Merrill (1999): Presentation, Practice, Support o Experiential Learning Frameworks

§ Kolb’s (1984): Concrete experience, Reflective observation, Abstract conceptualization, Active experimentation

§ Pfeiffer and Jones (1975): Experience, Publish, Process, Internalize, Generalize, Apply

§ Shank Berman, and Macpherson (1999; learning by doing): Define goals, Set mission, Present cover story, Establish roles, Operate scenarios, Provide resources, Provide feedback

o Problem-Centered Frameworks § Hirumi (2014): InterPLAY - Problems are presented to the learner in

sequence, from simple to complex, and learner accesses learning resources needed to solve the problem as he/she encounters difficulties and knowledge road blocks

§ Merrill (2002): Problem, Activation, Demonstration, Application, Integration

§ Jonassen (2011): Problem solving for decision making - Present problem/case, Compare to similar cases or analogies, Generate options, Analyze options, Make decision, Report selection

§ Wiley and Waters (2005): Real world problem, Demonstration (of problem solving), Practice with feedback, Support

o Collaborative Learning Frameworks § Nelson (1999): Build readiness, Form and norm groups, Determine

preliminary problem, Define and assign roles, Engage in problem-solving, Finalize solution, Synthesize and reflect, Assess products and processes, Provide closure

27

§ Rogoff’s (1995) three planes of activity: Guided participation, Apprenticeship, and Participatory appropriation

Learning Theories

• Transfer Appropriate Processing (Morris, Bransford, & Franks, 1977) • Encoding Specificity Principle (Thompson & Tulving, 1970) • Levels of Processing (Morris, Bransford, & Franks, 1977) • Picture Superiority Effect (Paivio, Rogers, & Smith, 1968) • Theory of Deliberate Practice (Ericcson et al., 1993) • Template Theory of Expertise (Gobet, 2005; Gobet & Simon, 1996) • The Data-Frame Model of Sensemaking (Klein, Phillips, Rall, & Peluso, 2007) • Cognitive Transformation Theory (CTT; Klein & Baxter, 2006) • Cognitive Load Theory (Sweller, 2011) • Social Learning Theory (Bandura, 1977)

Instructional Philosophies

• Constructivism/Interpretivism/Phenomenology • Behaviorism/Positivism • Rationalism • Pragmaticism

Instructional Design Models & Principles

• Van Merriënboer, Kirschner, & Kester’s (2013) ten steps to instructional design for complex tasks: Design learning tasks, develop assessment instruments, sequence learning tasks, design supportive information, analyze cognitive strategies, analyze mental models, design procedural information, analyze complex rules, analyze prerequisite knowledge, design part-task practice

• Systematic Design, e.g., via Analyze, Design, Develop, Implement, Evaluate (ADDIE) • Systemic Design • Interactive Design

Support mechanisms

• Scaffolding; Jumaat and Tasir (2014) identify the following categories: o Conceptual scaffolding (e.g., advanced organizers, Venn diagrams, concept

maps, rubrics) o Procedural scaffolding (e.g., demonstrations, step-by-step instructions, job aids) o Strategic scaffolding (e.g., prompts and activities to find and use alternative

methods) o Metacognitive scaffolding (strategy training, prompts to reflect/think about

learning) • Tutoring

o Peer tutoring § Fixed vs Reciprocal roles (Topping, 1996)

o Computer-based tutoring • Feedback

o Process/performance feedback (“This is what you did wrong”) o Outcome feedback (“You did it wrong”) o Effects feedback (“This is what can happen when you do that”; Wulf, McConnel,

Gartner, & Schwarz, 2002)

28

o Embedded vs. after-action o Comparative feedback (Compares performance and the performance context

with other performances and performance contexts; Bransford, Franks, Vye, & Sherwood, 1989: Gentner et al., 2003)

o Attentional guidance (Identifies cues or “specific characteristics of the problem to which the student should attend”; Marshall, 1995, p. 161)

o Multi-source feedback (Feedback from multiple sources to provide “a more accurate view of multiple facets of performance”; Tannenbaum, Smith-Jentsch, & Behson, 1998, p. 252)

Task Types • Jonassen’s (2011) problem dimensions: Diagnostic, decision making, procedural, trouble

shooting, strategic, policy analysis, design • Ill-structured (Complex task, task performed in a complex environment) vs. well-

structured (Task performed the same way every time; can be performed using primarily procedural and declarative knowledge)

• Isolated/individual vs coordinated team performance • Dominant type of knowledge used in task performance:

o implicit vs explicit o episodic vs semantic o declarative vs procedural

• Multiple parallel or integrated tasks vs single task performance • Primarily physical vs primarily cognitive work • Cognitive dimensions:

o Macrocognitive activities: Sensemaking, decision making, planning, attention management, problem detection

o Cognitive efficiencies of expert performance: Automaticity, perceptual-motor fluency, rich, integrated knowledge base, metacognitive skill, strategic skill (e.g., cognitive heuristics)

o Bloom’s revised cognitive process dimensions (Anderson et al., 1992; Krathwohl, 2002): Remember (recognize, recall), understand (interpret, exemplify classify, summarize, infer, compare, explain), etc.

o Bloom’s revised knowledge dimensions (Anderson et al., 1992; Krathwohl, 2002): Factual knowledge (terminology, specific details and elements), Conceptual knowledge, etc.

Description Languages Overview We have identified two types of description languages that are needed to support meta-adaptation; an Alignment Description Language and a Needs Description Language. • The Alignment Description Language is used to relate objects in one framework to objects in

another framework. This is particularly useful for describing the state of individual learners and individual activities. In Year1 learners and activities were aligned with KSAs to facilitate matching appropriate activities to leaners. The Alignment Description language extends this idea so that learners and activities can be aligned with other relevant science of learning concepts.

• The Needs Description Language is used to describe propositional logic relationships that indicate imperative conditions, or learner needs. The goal is to provide a generalized way to

29

utilize alignments to identify actionable learner needs on a moment to moment basis and allow a formal differentiation between identifying a need and deciding how to resolve a need.

Alignment Description Language Once frameworks have been defined that provide a vocabulary of core concepts relevant to learning in the Science of Learning Data Model, we can describe the state of individual instantiated objects such as Activities and Learners using alignments to the defined vocabulary. Object properties that indicate relationships of the object to one or more frameworks are called alignments. Alignments describe the object using the vocabulary defined in another framework. By using alignments, concepts can be defined once, then reused in a modular fashion to describe many kinds of objects. So far, several types of alignments have been discovered to be important, and it is anticipated that more will be discovered in future. The types of alignment known so far are:

• SimpleTextAlignment • SimpleLinkAlignment • DiscreteMeasurementAlignment • ContinuousMeasurementAlignment • AlignmentToAlignment

A description language is needed that can describe multiple types of alignments, and allow for the addition of new types of alignments as they are discovered. This section will describe the known alignment types and provide illustrative examples. SimpleTextAlignmentA Simple Text Alignment enables relating an object to a concept in another framework using a text relationship. Let’s consider an examples that relate different objects (a learner and an Activity) to the same concept (a KSA). Figure 2 SimpleTextAlignment Activity Example shows relating an Activity to a competency. The alignment object supports linking to a specific KSA (e.g., Perform PenTesting) in a competency framework, and it allows a text string to be used to describe the relationship expressed by the linkage (e.g., Teaches). The relationship expressed by the alignment object is “Activity Teaches Perform PenTesting”.

30

Figure 2 SimpleTextAlignment Activity Example

By defining the competencies separately from the Activity or the Learner, it is possible to refer to them in a modular way. We can therefore describe a Learner relation to a specific KSA as shown in Figure 3 SimpleTextAlignment Learner Example. This alignment relates a Learner to the same KSA we related an Activity to. What is different is the Relationship. The alignment can be summarized as “Learner DoesNot Hold KSA Perform PenTesting”, that is, the learner is not competent at the Perform PenTesting KSA. By having a modular definition of a KSA and aligning the state of an Activity and a Learner to that KSA we now have important information needed for a meta-adaptation algorithm to be able to locate Activities that teach KSAs a specific Learner has not yet mastered.

Figure 3 SimpleTextAlignment Learner Example

31

To encode this concept in a standard so that the standard is long-lived and does not need to change rapidly as new ideas emerge, basic constructs should be included in the standard that allow many ideas to be expressed. For example, a node type used to describe a KSA should be included in a standard, but the specific competency “Perform PenTesting” should not. Similarly, the specific relationships such as “DoesNotHold” or “Teaches” are not encoded in the standard; only the primitive required to express them is in the standard. For example, the SimpleTextAlignment object is included in the standard and it contains the following:

• URI for linking to a specific instance of a competency framework • URI for linking to an instance of a specific KSA • A text field that will hold any relationship that can be expressed as a string.

This is exactly how the LRMI AlignmentObject works as shown in Figure 4 LRMI Alignment Object and discussed at https://blogs.pjjk.net/phil/explaining-the-lrmi-alignment-object/ This is great, since the LRMI Alignment Object is exactly what we need for a SimpleTextAlignment.

Figure 4 LRMI Alignment Object (Barker, 2014)

SimpleLinkedAlignmentWhat about the other types of Alignments we envision? As the blog author of https://blogs.pjjk.net/phil/explaining-the-lrmi-alignment-object/ notes, they would like to see the AlignmentObject extended to allow the alignmentType to support a URL to allow the alignmentType to have a richer description than a simple text string (Barker, 2014). This is precisely the same extension that we are calling a SimpleLinkedAlignment. The relationship can be a link instead of just a text field as shown in Figure 5 SimpleLinkedAlignment Activity Example.

32

Figure 5 SimpleLinkedAlignment Activity Example

DiscreteMeasurementAlignmentOne of the limitations of the LRMI AlignmentObject is that it provides a way to describe an object using a single concept from another framework (e.g., KSA). But some of the relationships we want to be able to express use concepts described in more than one frameworks. Consider the relationship shown in Figure 6 DiscreteMeasurementAlignment Learner Example, which says that the Learner is a Novice at the JobTask Cyber Security.

33

Figure 6 DiscreteMeasurementAlignment Learner Example

What this relationship is expressing is a measurement of the Learner’s proficiency using a discrete scale of measurement. But which scale? Novice/Proficient/Expert (Jonassen, 2011b)? Fundamental Awareness/Novice/Intermediate/Advanced/Expert? (NIH, nd). We would like the TLA to offer the flexibility to work with different discrete measurement scales than already exist and which have differing numbers of steps. If a measurement scale is described in its own framework, separate from the framework where the JobTasks are defined, then they can be combined in a modular way so that JobTask expertise can be measured using any existing scale ontology. To do this, the DiscreteAlignmentObject adds URIs for linking to the scale of measurement in addition to the fields that the SimpleLinkedAlignment has:

• URI for linking to a specific instance of a framework that holds the concept to be measured

• URI for linking to an instance of the specific concept to be measured • URI for linking to the specific instance of a framework that holds the scale of

measurement • URI for linking to an instance of the specific measurement step (e.g., Novice) • URI for linking to the type of relationship (e.g., Is, LessThan, GreaterThan, etc.)

34

ContinousMeasurementAlignmentBuilding on the foundation presented when discussing DiscreteMeasurementAlignment, what happens if the scale of measurement can have different units of measure, but uses continuous values rather than discrete values? Think Fahrenheit vs Celsius as an easy everyday example. For an example, in the TLA object space, consider describing the learner’s IQ. This is still a measurement of a concept (IQ) using a scale (less than 80 to greater than 140), but instead of measuring in discrete increments, we allow continuous values. This relationship is expressed as an alignment in Figure 7 ContinuousMeasurementAlignment Activity Example.

Figure 7 ContinuousMeasurementAlignment Activity Example

AlignmentToAlignmentAlignments are powerful ways to express relationships between objects and concepts. In particular, they help with the hard problem of identifying which features of an Activity match the state of an individual so that their experience can be personalized. In the Figure 6 DiscreteMeasurementAlignment Learner Example, we used a DiscreteMeasurementAlignment to describe the Job Task proficiency of a Learner (Novice at Cyber Security). A meta-adaptation algorithm will then look for which Activity will best suit the learner’s needs. If we give the DiscreteMeasurementAlignment instance that captured that relationship a URI, then we could simply re-use it as shown in Figure 8 AlignmentToAlignment Activity Example. This figure expresses that the Activity is AppropriateFor Novices at the JobTask of CyberSecurity.

35

Figure 8 AlignmentToAlignment Activity Example

AlignmentsandTimeSo far, we have constrained ourselves to thinking about alignments that describe the current state of objects. However, each alignment is implicitly associated with a point in time. There are four points in time that are important when considering how to personalize learning using meta-adaptation, as described in Table 7 Alignments and Time. Personalization requires knowing where the learner has been (historical state), where they are now (current state), where they’d like to go (desired future state), and if they are on track to get there (predicted future state). Table 7 Alignments and Time

Alignment Timepoints Description Examples Current state The present alignment status of

the object with a framework element.

Learner X hold KSA Z now, Activity X teaches KSA Y

Historical state The past alignment status of the object with a framework element.

Learner X held KSA Z at time T

Desired future state The preferable alignment status of an object with a framework element at a specific future time.

Learner X wants to become expert at JobTask Y this semester

Predicted future state A projected (and potentially computationally derived) alignment status of an object with a framework element at a specific future time

Learner X will hold KSA Z at time T, Utility of activity X will be Y at time T

DesiredFutureStatevs.GoalLet’s consider what the difference is between a desired future state and a goal. A goal, vaguely, is a desired future state for a learner. So can we use an alignment to fully express a goal? That is, since we used an alignment to express the learner’s current state (e.g., Job Task Proficiency as show in Figure 6 DiscreteMeasurementAlignment Learner Example), then we could use the

36

same alignment construct to describe the learner’s alignment at a future point in time as shown in Figure 9 Desired Future Alignment.

Figure 9 Desired Future Alignment

So, is this enough to fully express a goal? It turns out that it is not; it doesn’t, for example, capture the priority associated with the goal, or learner’s reason for pursuing the goal. To fully express a goal, we will need a bit more than just desired future state. However, expressing the desired future state with an alignment does help us to express some of the most important aspects of a goal; the expected outcome (e.g., Proficiency Equals Expert) and the associated learning objective (e.g., JobTask Cyber Security). Therefore, this indicates that the goal framework can have an immediate jump start by leveraging concepts built up in other frameworks using alignments. SummaryOur exploration of alignments has lead us to the conclusion that rather than a strange anomaly surrounding competency, they are in fact a primary construct the TLA should support to enable specifications to be decoupled from specific domains and theories. Using alignments, a few alignment objects types could be used to express many different theories about learners and activities that would make it possible to perform complex personalization. NeedsDescriptionLanguageSimilar to the language used to describe alignments, personalizing the learner experience requires understanding the needs of the learner. What is a need? Is a need exclusive to a learner, or can it apply to other types of objects? By exploring some examples, Table 8 Need Definition shows that both Learners and Activities can have needs, and both must be addressed to best serve the learner. Table 8 Need Definition

Term Concept Examples Need What conditional relationships in

object alignments result in imperatives? What possible

• Learner Example • Conditional Relationship: Learner’s proficiency

improvement rate is not on track to pass cyber

37

resolutions map to specific imperatives?

security exam on the test date • Imperative: Need to reach minimum proficiency

by test date • Resolution: Add extra daily study session

• Activity Example

• Conditional Relationship: No Activity X exists that teaches Competency Y

• Imperative: Need an Activity X that teaches Competency Y

• Resolution: Create an activity that teaches competency Y

We therefore envision a Needs Description Language as providing a way to describe propositional logic relationships that can span multiple frameworks to identify alignment conditions that indicate imperative conditions, or needs. That is, what discrepancies between current, historical, predicted future, and desired future states of one or more object's alignments with one or more frameworks can exist that are the indicator conditions that identify needs? What possible resolutions to the need make sense from a learning science perspective? While the formal logic is very propositional in nature, the execution of the framework should be friendly to instructional experts. The expectation is that there will be common types of conditional patterns in formal logic that translate into more intuitive human readable descriptions of need. Table 9 Conditional Patterns Identifying Need shows a sampling of these patterns, it is intended as an illustrative example, not an exhaustive list of patterns. Table 9 Conditional Patterns Identifying Need

Formal Condition Pattern Example of Condition Patterns Example of Identified Need An object's predicted future alignment state does not equal the same object's desired future alignment state

Learner's predicted goal achievement date is later than the desired graduation date

Accelerate learning pace

An object's current alignment state does not equal the same object's desired alignment state

Learner’s frustration level is above the optimal learning level

Reduce frustration

An object's predicted future alignment state is not compatible with the predicted future alignment state of another object

Learner is predicted to be midway through a non-mobile activity when their external context is predicted to switch to traveling in a car

Pause or change activity

No object exists in the population of that type of object with a specific framework alignment

No Activity X exists that teaches KSA Y

Add an Activity X that teaches KSA Y

Note that the conditional patterns can describe needs for populations of objects (e.g., There is no such Activity X that teaches competency Y) in addition to needs of individual objects (e.g., individual learners). Each identified need may have one or more viable resolutions. Similar to the conditions, the expectation is that there will be common resolution patterns. Table 10 Need Resolution Patterns

38

shows some of the possible patterns, again intended as an illustrative example, not an exhaustive list. Table 10 Need Resolution Patterns

Resolution Pattern Examples Add activities to resolve an alignment discrepancy

Add additional practice activities to increase learning rate

Perform an action whose predicted impact modifies an existing object’s alignment status

Play a stress relieving mini-game to reduce frustration, Personal attention from an instructor can reduce frustration

Switch activities to resolve alignment discrepancy

Switch to a mobile equivalent of a non-mobile activity for travel

Add a new object Create an activity X that teaches competency Y

AssetData ActivityStateThe Asset Data model is used to describe the state of individual Activities that are present in a specific TLA instantiation. The Activity State is stored in an Activity Index component. The definition of the vocabulary that is used to describe an Activity belongs in ontologies that reside in the Science of Learning Frameworks. These can be instantiated in separate components (e.g., Competency Framework) as needed and referenced from the Activity State using alignments. The definition of the vocabulary in the Activity Trait Framework reads most easily if the reader thinks about “current state” of an Activity while reading it. However, it is very important to note that we anticipate that the Activity State recorded in the Activity Index needs to be capable of containing state information for the same time points described to be important in the Alignments and Time section, namely:

• Current state • Historical state • Desired future state • Predicted future state

The activity state will also contain needs of an activity using the Needs Description Language. An example of an activity need is Activity X needs outdated images to be updated to address declining utility rating. In addition to maintaining the state of individual activities, we also need to maintain the state information describing populations of Activities. For example, summative information identifying that no Activity that teaches KSA X is available, or no Activities for mobile devices are available.

LearnerData LearnerStateThe Learner Data model is used to describe the state of individual Learners that are interacting with a specific TLA instantiation. The Learner State is stored in a Learner Profile component. The definition of the vocabulary that is used to describe a Learner belongs in ontologies that

39

reside in the Science of Learning Frameworks. These can be instantiated in separate components (e.g., Competency Framework) as needed and referenced from the Learner State using alignments. The definition of the vocabulary in the Learner Trait Framework reads most easily if the reader thinks about “current state” of a Learner while reading it. However, it is very important to note that we anticipate that the Learner State recorded in the Learner Profile needs to be capable of containing state information for the same time points described to be important in the Alignments and Time section, namely:

• Current state • Historical state • Desired future state • Predicted future state

The Learner state will also contain needs of a learner using the Needs Description Language. Some examples of learner needs include the following:

• Need a refresher course in competency X • Need to accelerate progress on goal X for achievement by desired time T • Need an activity with lower attentiveness requirements • Need to switch to a mobile activity for travel

In addition to maintaining the state of individual learners, we also anticipate the need to maintain state information describing populations of Learners.

Inference&PredictionAlgorithmUseCasesSupportingMeta-Adaptation Producing real-time recommendations for meta-adaptation requires reasoning over metadata and making inferences and predictions to

• Understand alignments • Deduce needs • Identify potential resolutions

Now that we have a description of what the data is needed for meta-adaptation, we can turn our attention to how it is produced and consumed. Many types of inferences and predictions are needed to produce meta-adaptation. Each TLA Inference Processor and Prediction Processor used in the meta-adaptation use case is a consumer and/or producer of data defined in the Science of Learning Data frameworks. To examine the data flow, rather than consider each producer/consumer at a software component level, it is more meaningful to examine the use cases when an inference or prediction is needed and examine the input necessary and the output produced. The important aspect is to understand what data needs to be shared and why. In an implementation, each specific software component may handle one or more of the use cases described. InferenceandPredictionUseCaseOverviewWe can divide the inference and prediction use cases into two broad categories.

1. Generation of Asset Data – Inferences and predictions that produce alignments of individual Activities or populations of Activities to Science of Learning frameworks.

40

2. Generation of Learner Data – Inferences and predictions that produce alignment of individual Learners or populations of Learners to Science of Learning frameworks

We explored ~40 Use Cases, summarized by category in figure 13.

Figure 10 Inference & Prediction Use Case Summary

Each use case category supports meta-adaptation either directly or indirectly: • Directly supporting the Learner • Reducing manual entry needed from Instructors and Learning Experts • Assisting with content curation data analysis

The producer/consumer relationships for each use case is shown MetaAdaptationDataFlowMatrix.xls, where each entry in column A represents an individual algorithmic use case. In the following sections, each use case category will be defined, and individual use cases will be described using representative examples. Rather than covering the use cases by category such as grouping all Asset Data use cases, we will cover them in the same order they are covered in the spreadsheet, which lists them in the temporal order in which data is produced and consumed in the meta-adaptation learner user case. A few data flow steps that are not inferences or predictions will be covered to assist in describing the temporal flow of data. Specifically, where do humans input data into the system, and how does data exit the TLA system for use by external systems?

ManuallyGenerateNewDataandAlignmentsCurrently, smart components do not yet exist to help “bootstrap” a TLA instantiation. The system users: learning experts, instructors and even the students themselves, must manually enter information that describes the characteristics of both the Activities available and the Learners (see Table 10 below). One of the goals of the TLA is to reduce this manual burden. Rather than this being an exhaustive manual activity, we envision in the future that a set of manually provided alignments can be used as training data for smart algorithms that can begin to automate generation of new alignments. Most of the following sections will address the goal of automating alignment generation, describing specific use cases for how and when that automation could occur.

41

Table 11 Manual Generation of Data And Alignments

Use Case Description Examples Manual Entry by Learning Expert

• Learning experts manually populate science of learning data models.

• Learning experts manually review learning activities and annotate them with alignments to science of learning data models.

• Populate a competency framework with KSAs.

• Associate individual learning Activities with the KSAs they teach or assess.

Manual Entry by Learner • Learners manually enter information about themselves to enable the system to provide personalization.

• Enter specific learning goals (e.g. obtain a certification in Cyber Security)

Manual Entry by Instructor

• Instructors can provide additional information about Activities and Learners from observing the system in use.

• Needs for new or improved Activities

• Add new Learner goals • Add proficiency observations

InferIndividualActivityAlignmentsfromActivitySourceMaterialThis section is about automating the generation of alignments by examining the source data native to the Activity, that is the activity content itself such as images, text and audio. (See Table 11 below.) Table 12 Inferences from Activity Sources

Use Case Description Examples Infer Competency Framework Alignments

Use algorithms that process Activity source material to align the Activity with Competency Frameworks.

Search activity text, audio and imagery to extract keywords that are indicative of specific KSAs (e.g. search for “Perform Pen Testing”)

Infer Goal Framework Alignments

Use algorithms that process Activity source material to align the Activity with Goal Frameworks.

Search activity text, audio and imagery to extract semantic meaning indicative of supporting specific goals (e.g. phrases such as “at the end of this chapter you will be able to apply all the steps for performing pen testing successfully”).

Infer Context Framework Alignments

Use algorithms that process Activity source material to align the Activity with Context Frameworks.

Search Activity source for evidence of external context compatibility (e.g. audio format is compatible with using a mobile device during travel) and internal context compatibility (e.g. “bite sized modules” is compatible with short attention span)

Infer Learner Trait Framework Alignments

Use algorithms that process Activity source material to align the Activity with Learner Trait frameworks.

Search activity text, audio and imagery to extract semantic meaning indicative alignment with types of personal preferences (e.g. uses baseball images to explain concepts is compatible with learners who like baseball)

Infer Activity Trait Framework Alignments

Use algorithms that process Activity source material to align the Activity with Activity Trait Frameworks.

Search activity text, audio and imagery to extract semantic meaning indicative alignment with types of Activity traits

42

(e.g. uses baseball images to explain concepts is compatible with “uses sports metaphors”)

Infer Performance Measurement Framework Alignments

Use algorithms that process Activity source material to align the Activity with Performance measure frameworks.

Search activity text, audio and imagery to extract semantic meaning indicative alignment with specific assessment paradigms (e.g. awards scores using a 0-100 scale)

InferIndividualActivityAlignmentsfromExistingActivityAlignmentsOnce the source material has been examined and used to populate an initial set of alignments for an Activity, those existing alignments can be used to draw further inferences about the same Activity, (See Table 12 below.) Table 13 Inferences from Activity Alignments

Use Case Description Examples Infer Additional Alignments from An Activity’s Existing Alignments

Use algorithms that process Activity alignments that already exist to translate between different science of learning models that have defined equivalencies.

If an Activity is marked to use percentage based scoring then it could also be marked as supporting pass/fail scoring with a science of learning model providing the threshold percentage that delineates pass from fail.

InferIndividualActivityAlignmentsfromSimilarActivitiesThis section covers use cases that identify similarities between activities and use existing Activity alignments in one Activity to populate gaps in the alignments of similar Activities. (See Table 13 below.) Table 14 Inferences from Similar Activities

Use Case Description Examples Infer Alignments from Alignments of Similar Activities

Use algorithms that process Activity alignments that already exist to identify similarities between Activities and fill in gaps by transferring information present in one Activity to other similar Activities.

If two Activities both are presented in the same audio format, and one is marked as compatible with mobile devices, then the second activity could also be marked as mobile device compatible.

InferActivityPopulationNeedsFrameworkAlignmentsfromAllExistingActivityAlignmentsOnce a population of Activities has been aligned with science of learning data models, it is possible to infer alignments with Needs Frameworks about the population as a whole. (See Table 14 below.) Table 15 Inferences from all Existing Activity Alignments

Use Case Description Examples Infer Gaps in Competency Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Competency Frameworks

No activities exist that teach perform pen testing

Infer Gaps in Goal Coverage

Use algorithms that process Activity alignments that already exist to identify

No activities exist that support obtaining certification in cyber security

43

population needs in respect to Goal Frameworks.

Infer Gaps in Context Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Competency Frameworks.

No activities exist that teach perform pen testing on a mobile device

Infer Gaps in Learner Trait Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Learner Trait Frameworks

No activities exist that teach perform pen testing using social media interactions

Infer Activity Gaps in Activity Trait Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Activity Trait Frameworks.

90% of Activities do not have a human readable description

Infer Gaps in Performance Measurement Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Performance Measurement frameworks.

No activities exist that can assess cyber security skills using a novice/intermediate/expert scale

Infer Gaps in Pedagogic Decisions Coverage

Use algorithms that process Activity alignments that already exist to identify population needs in respect to Pedagogic Decisions Frameworks.

No activities exist that provide practice for perform pen testing

Infer Depth of Coverage

Use algorithms that process Activity alignments that already exist to identify which activities are similar or overlapping in purpose to understand where there is good coverage in available material on a specific topic.

There are multiple activities available for teaching perform pen testing

GenerateNewLearnerDataIn the same way that Activities have “source data”, that is their content, there is also source data about a learner. The source data about a learner is their stream of experiences. This section covers how that data is produced, so we understand how it is introduced into a computational system. (See Table 15 below.) Table 16 Learner Source Data

Use Case Description Examples Learning Activity An Activity that is learning focused

reports on the moment to moment interaction of the Learner.

Learner X answered question 5 on quiz 2 correctly.

Non-learning Activity An Activity that is not learning focused reports on the moment to moment interactions of the Learner.

Learner X entered a new meeting into their calendar.

InferIndividualLearnerAlignmentsfromExperienceFrameworkAlignmentsOnce experiences, or “source data”, about a learner exist, algorithms can begin to make inferences about that data, interpreting its meaning against different science of learning models. Specifically, this section is about generating alignments about a single learner by reasoning over that learner’s experiences. (See Table 16 below.)

44

Table 17 Individual Learner Alignments

Use Case Description Examples Infer Mastery Estimates Use algorithms to generate alignments to

Competency Frameworks from Experience Framework Alignments (current or historical).

The experience Learner X passed Test Y can be used in conjunction with the alignment of Test Y to Competency Z to infer that the Learner’s mastery estimate in skill Z has increased.

Infer Goal Progress Use algorithms to generate alignments to Goal Frameworks from Experience Framework Alignments (current or historical).

If a learner’s experiences include Activities that issue certifications, then the certification can be used to infer a goal completion.

Infer External Context Use algorithms to generate alignments to Context Frameworks from Experience Framework Alignments (current or historical).

If a learner is using an Activity that reports GPS location and it is changing, then it can be inferred that the learner is traveling.

Infer Learner Traits Use algorithms to generate alignments to Learner Trait Frameworks from Experience Framework Alignments (current or historical).

If a learner consistently selects activities in audio format when multiple formats are offered, an inference can be made that the learner prefers for audio.

InferIndividualLearnerAlignmentsfromExistingLearnerAlignmentsAfter alignments have been made for a specific individual learner using the experiences about a learner, those inferences can also be reasoned on and combined to draw further inferences about the same learner. (See Table 17 below.) Table 18 Inferences about Learner Alignments from Existing Learner Alignments

Use Case Description Examples Infer Learner Traits Use algorithms to generate alignments to

Learner Trait Frameworks from Experience Framework Alignments (current or historical).

While external context such as “traveling” can potentially be derived from just the experience stream, inferring internal context like “low motivation” may need to reason over multiple types of alignments such as typical pace at answering quiz questions and current proficiency. An expert might answer multiple choice questions rapidly because they know the answers, but a novice might answer multiple choice questions rapidly because their motivation is low and they just wish to be done as soon as possible.

PredictIndividualLearnerNeedsfromExistingLearnerAlignmentsThe alignments of an individual Learner describe the past, present, and desired future states of the learner. That information can be used to predict their upcoming needs for future times that can be seconds away or on a much longer timescale such as days or weeks away. (See Table 18 below.) Table 19 Inferences about Learner Alignments with Needs Frameworks from Existing Learner Alignments

Use Case Description Examples

45

Predict Competency Needs

Use algorithms to generate individual Learner needs in relation to Competency Frameworks.

Science of Learning models that capture information such as knowledge decay rates can be combined with a learner’s historical alignments to predict when the mastery estimate for a specific competency might drop below the minimum acceptable threshold.

Predict Goal Needs Use algorithms to generate individual Learner needs in relation to Goal Frameworks.

If a competency need has been identified, then a new goal to complete a refresher course might be predicted to resolve the competency need. If a series of failures has been observed in an activity stream, new remediation goals might be introduced.

Predict External Context Needs

Use algorithms to generate individual Learner needs in relation to Context Frameworks.

If the learner’s experiences indicate current and upcoming location, then a context need such as shifting the current learning activity to a mobile equivalent for travel can be predicted.

Predict Learner Trait Needs

Use algorithms to generate individual Learner needs in relation to Learner Trait Frameworks.

If a learner’s internal context has displayed decreasing motivation, then a need for a motivator can be predicted.

PredictIndividualLearnerNeedsfromAlignmentsofSimilarLearnersIn addition to examining the alignments of an individual to predict their upcoming needs, historical population alignment patterns can also be used to predict upcoming needs based on patterns observed in the historical data about previous population groups who are similar. (See Table 19 below.) Table 20 Inferences about Learner Alignments with Needs Frameworks from Similar Learner Alignments

Use Case Description Examples Predict Individual Learner Needs from Similar Learners

Use algorithms to generate individual Learner needs by extrapolating from population level patterns of similar Learners.

Predict specific learning trajectories (or activity sequences) that are most likely to result in meeting the Learner’s goals based on the experiences of similar Learners.

InferIndividualActivityAlignmentsfromLearnerPopulationAlignmentsOnce many learners have experienced a learning Activity, the historical alignments of the learner population can be examined to identify temporal sequences of significance. Inferences about the Activities the aggregated learner population experienced can be made by examining how the alignments of a population of learners tended to change after experiencing an Activity. (See Table 20 below.) Table 21 Inferences about Activity Alignments from Learner Population Alignments

Use Case Description Examples Infer Competency Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Competency Frameworks.

If multiple learners experience an Activity and their mastery estimate in a specific competency increases, then the Activity can be aligned to that competency.

46

Infer Goal Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Goal Frameworks.

If multiple learners experience an Activity and complete a specific goal, then the Activity can be aligned to that goal.

Infer Context Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Context Frameworks.

If multiple learners experience an Activity while traveling, then the Activity can be aligned with mobile contexts.

Infer Learner Trait Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Learner Trait Frameworks.

If multiple Learners with low motivation show increased motivation after experiencing an Activity, that activity can be aligned with increasing motivation.

Infer Activity Trait Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Activity Trait Frameworks.

The utility rating of a specific Activity can be derived by aggregating either (or both) the explicit ratings entered by many individuals (thumbs up/down), or implicit ratings based on observed patterns in mastery estimates that correlate to exposure to the Activity.

Infer Credential & Learning Outcome Framework Alignments

Use algorithms to generate individual Activity alignments from aggregated Learner population alignments in respect to Credential & Learning Outcome Frameworks.

If multiple learners experience an Activity and receive a certification, then the Activity can be aligned to that certification.

Infer Pedagogic Decisions Framework Alignments

Use algorithms to generate population Activity alignments from aggregated Learner population alignments in respect to Pedagogic Decisions Frameworks.

The utility of specific sequences of activities can be inferred from examining the impact of those learning trajectories across many learners

PredictIndividualActivityNeedsfromLearnerPopulationAlignmentsIn addition to deriving alignments that describe current state from the experience of a population, Activity needs can also be derived from examining trends in population experiences with the Activity over time. (See Table 21 below.) Table 22 Predict Activity Alignments with Needs Frameworks

Use Case Description Examples Predict Activities in Need of Refresh

Use algorithms to generate individual Activity needs from aggregated Learner population alignments.

Predict from trends in the movement over time of the rating of positive/negative utility of specific Learning Activities which specific materials might be reaching their end of life and are in need of a refresh.

PredictActivityPopulationNeedsfromLearnerPopulationAlignmentsOnce individual Activitiy needs and needs for a population of Learners has been identified, Activity population needs can be predicted. (See Table 22 below.) Table 23 Predict Activity Population Needs from Learner Population Alignments

Use Case Description Examples Predict Critical Activity Needs

Use algorithms to generate population Activity needs from aggregated Learner population alignments.

If many Activities within the population have a refresh need aligned with them, the urgency of the need can be identified by examining which learners in the

47

population are predicted to need the Activity in the near future.

Predict Content Creation with Largest ROI

Use algorithms to generate population Activity needs from aggregated Learner population alignments.

Predict what kind of specific learning materials should be created to fill gaps to have the largest positive learning impact on the learner population.

UtilizeActivityandLearnerAlignmentsforMeta-adaptationAll of the inferences and predictions discussed in the preceding sections have applications to different consumers (which could be human or software). This section will focus on describing the software consumption of the inferences and predictions for the specific purpose of facilitating meta-adaptation. (See Table 23 below.) Table 24 Activity and Learner Alignments for Meta-adaption

Use Case Description Examples Match Activities to Needs The alignments of Activities can be used

to identify specific Activities that meet individual Learner needs.

• Activities that teach Competency X are appropriate for Learners that need to improve in Competency X.

Identify Best Activity to Match Needs

The alignments of Activities can be used to narrow the set of specific Activities that meet individual Learner needs.

• Only Activities that are aligned with mobile Contexts are appropriate for Learners that are traveling and using a mobile device.

Prioritize Learning Needs Utilize the Learner needs to identify the best Activity for the Learner to pursue at the current time, in the current context.

• Identify which specific learning activities should be prioritized based on predicted near/mid/long term learning needs

AppendixA:DataCentricHardProblemsCrossCuttingHard problems impact all three data models in the TLA (Science of Learning, Learner, Asset):

• Lack of agreement on terminology. Different vendors have different terminology to describe aspects of all three TLA data models. Sometimes two different words from different vendors mean the same thing; other times, the same word is used by different vendors to mean different things. Even with a common vocabulary like an xAPI profile it is very difficult for developers to consistently utilize verbs such as "completed" so they have the same meaning across activities. There is no one best set of terms or any standard way to convert from one vendors preferred vernacular to another. Without a common vocabulary sharing understanding across vendors is difficult or impossible. Yet it is unlikely that the community will be able to agree on a standardized terminology. Therefore, a technology solution is desired that can facilitate vendors sharing information in their own vernacular, but still communicating correctly and effectively.

• The theories represented in the data are not static. New theories will be introduced, existing theories will be refined, and obsolete theories will be identified over time. The timeline for theory development and modification is not synonymous with the timeline of standards development. The lengthy standards development process can hinder the incorporation of new theories. The TLA must support the dynamic nature of theories on a timescale conducive to the creativity that drives innovation. One potential path forward

48

is to develop standards that provide a set of primitives that can be used as a language for expressing many theories. The theories themselves are expressed by application of the standard through means such as a profile rather than encoding the theory itself into the standard.

• Manual curation burden. Currently, much of the data is must be manually curated. Manual curation is not time/cost effective. Automated assistance is needed to enable scaling up TLA instantiations to have thousands of Activities and Learners. A key challenge is how to get past the cold start problem? Where can enough training data be acquired to enable learning algorithms to effectively assist with data curation?

• The granularity and breadth of current data specifications is often too course-grained and too sparse to provide enough input for inference and prediction processors to effectively utilize sophisticated algorithms for meta-adaptation. For example, when inferring context with only sparse data available, it may be impossible to distinguish key edge cases. For example, is the Learner stuck and sitting in front of the computer trying, or did they walk away and leave the computer unattended?

ScienceofLearning

• We are just beginning to understand the number and types of frameworks that are necessary to enable meta-adaptation. In some cases, rich models (e.g., pedagogic strategies) exist in the Science of Learning community, but they are not encoded in a computationally accessible format. In other cases, some aspects of a framework have been captured in an existing standard (e.g., for Activity metadata), but the granularity and breadth of the existing metadata does not match our anticipated needs. And in yet other cases (e.g., Needs Description Language) the concept is still formative and will need to be researched and developed further.

ActivityData

• What data and alignments are important now is hard to quantify, and what data and alignments will be important in future are even harder to predict. It is not yet clear if the Activity Trait Framework should contain anything more than basic intrinsic information such as name and summary description. It is quite possible that all significant detail really belongs broken out into individual frameworks. There could be a metaphors framework, for example. Last year we attempted to make a catch-all set of metadata, and it has become apparent that many of the activity traits in last year’s metadata are really alignments to other complex frameworks (for example, what devices an Activity can run on are context Alignments). Further investigation of the best way to capture activity alignments is needed, in particular focusing on making the architectural mechanism extensible so that new types of alignments can be added in future as they are discovered. Allowing alignments to new frameworks not yet discovered may provide the flexibility needed.

• It is unclear how best to correlate learner experience data with the alignments describing the originating activity in a meaningful way that enables inferring additional alignments such as estimating mastery in a specific competency. Activity metadata as it exists currently tends to summarize for the totality of the Activity (e.g., "Activity X teaches competency Y"). However, the series of learner experiences that describe the moment-to-moment activity corresponding to the activity summary may be long, and each individual element may be very difficult to interpret as contributing to the summation. New algorithms need to be developed to utilize moment to moment experience data, and new ways of structuring Activity alignments need to be developed to enable those algorithms.

49

• Estimating positive/negative utility of specific learning activities presents some unique challenges. Inferring the positive/negative trajectories through the Activity space is relatively straightforward; learner experience data stream sequences are correlated with mastery estimate trends. When an Activity or sequence of Activities is correlated with a positive or negative trend in mastery, the real hard problem is identifying the likely cause of the correlation. Is it due to a characteristic of the learning material or a characteristic of the learner? It is highly probably that it is a complex interaction of both. At its simplest, the learning material could be high/low quality, but more interestingly characteristics of the learning material such as "uses audio extensively" or "uses sports metaphors" might intersect with specific learner alignments such as "low motivation" and "likes sports." Additionally, the utility of and individual learning Activities is not static; it varies over time. An initially useful learning activity can become obsolete (e.g., material it contains becomes dated) and its utility decreases. Recognizing the loss of utility is a good start; even better is predicting the loss of utility before it becomes problematic.

LearnerData• Interpreting learner experiences, such as estimating mastery of a competency, is not

done directly, but inferred from the learner's moment-to-moment experiences that accumulate over time. Currently this is difficult because there is no direct alignment between individual experience entries and the summative nature of current metadata describing the Activity they experienced. For example, the series of learner experiences that describe the moment-to-moment activity corresponding to the activity summary may be long, and each individual element may be very difficult to interpret as contributing to a summation, such as teaches competency X. Additionally, even within one learning activity, the series of experiences is not likely to be serial sequences related to a single competency. Instead, complex interleaving of experiences relating to multiple competencies is highly probable. Adding one further layer of complexity, the activities will all have their own paradigms for reporting on the experiences. New algorithms need to be developed that can interpret the many types of information about a learner, and new ways of structuring Learner alignments need to be identified to facilitate those algorithms.

• Some information about Learners is not directly related to learning but can help to identify the context in which learning is taking place. Existing real-time feeds (e.g., twitter feeds) that might provide clues about the external context of the learner (e.g., location) are sufficiently intensive that they require software integrating with them to be capable of operating at scale; they don't scale down well or at all for inclusion in small scale demonstrations.

• It is not yet clear how best to balance instructor-influenced pedagogic sequencing with automation. What should be automated? How does the instructor or instructional designer influence the learner’s experiences? New paradigms for designing learning that leverages cross-system Activities need to be identified.

• Interpreting learner data in meaningful ways means getting access to the necessary input. However, tracking learner data intersects with privacy concerns. Privacy considerations restrict available data feeds that could be used to infer environmental constraints.

50

ReferencesAnderson, L. W., Krathwohl, D. R., Airasian, P., Cruikshank, K., Mayer, R., Pintrich, P.,

... & Wittrock, M. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy. New York: Longman.

Allinson C.W., & Hayes J. (1988). The Learning Styles Questionnaire: an alternative to Kolb’s Inventory. Journal of Management Studies, 25(3), 269–281.

Arnold, M.B. (1960). Emotion and Personality. Volume I: Psychological Aspects. New

York: Columbia University Press. Bransford, J.D., Franks, J.J., Vye, N.J., & Sherwood, R. D. (1989). New approaches to

instruction: Because wisdom can’t be told. In S. Vosniadou & A. Ortony (Eds.), Similarity and Analogical Reasoning (pp. 470-497). New York: Cambridge University Press.

Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychological review, 84(2), 191-215.

Barbe, W.B., Swassing, R. H., & Milone, M. N. (1979). Teaching through modality strengths: concepts and practices. Columbus, Ohio: Zaner- Bloser.

Benzger.,K., & Sohn, A..(1993).The Art of Using Your whole Brain. Rockwall,Texas: KBA Publishing.

Biederman, I., & Shiffrar, M. M. (1987). Sexing day-old chicks: A case study and expert systems analysis of a difficult perceptual-learning task. Journal of Experimental Psychology: Learning, memory, and cognition, 13(4), 640.

Biggs, J.B. (1987). Student approaches to learning and studying. Hawthorn, AU: Austrailian Council for Education Research.

Bloom, B. (1956). Taxonomy of education objectives: The classification of education goals, Handbook 1: Cognitive domain. New York: McKay.

Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge, England: Cambridge University Press.

Cattell, R. R. (1943). The measurement of adult intelligence. Psychological Bulletin, 40, 153–193.

Costa, P. T., Jr., & McCrae, R. R. (1980). Still stable after all these years: Personality as a key to some issues in adulthood and old age. In E B. Baltes & O. G. Brim (Eds.), Life span development and behavior (Vol. 3, pp. 65-102). New York: Academic Press.

Cloninger C.R., Svrakic D.M., & Przybeck T.T. (1993). A psychobiological model of temperament and character[ Archives of General Psychiatry, 50, 975-990.

Driscoll, M. P. (1994). Psychology of Learning for Instruction. MA: Allyn & Bacon. Dunn R., & Dunn K. (1992). Teaching secondary students through their individual

learning styles. Needham Heights, MA: Allyn and Bacon. Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate

practice in the acquisition of expert performance. Psychological Review, 100, 363–406.

Fleming, N.D; (1995), I'm different; not dumb. Modes of presentation (VARK) in the tertiary classroom, in Zelmer,A., (ed.) Research and Development in Higher Education, Proceedings of the 1995 Annual Conference of the Higher Education

51

and Research Development Society of Australasia (HERDSA), Volume 18, pp. 308 – 313.

Fowlkes, J., Dwyer, D. J., Oser, R. L., & Salas, E. (1998). Event-based approach to training (EBAT). The international journal of aviation psychology, 8(3), 209-221.

Gagne´ (1977). The conditions of learning (3rd Ed.). New York: Holt, Rinehard, and Winston.

Gardner, H. (2011). Frames of mind: The theory of multiple intelligences. Basic books. Gobet, F. (2005). Chunking models of expertise: Implications for education. Applied

Cognitive Psychology, 19(2), 183-204. Gobet, F., & Simon, H. A. (1996). Templates in chess memory: A mechanism for

recalling several boards. Cognitive Psychology, 31(1), 1-40. Gray, J. (1978). The neuropsychology of anxiety. British Journal of Psychology, 69(4),

417- 434. Gregorc, A. E (1979). Learning/teaching styles: Potent forces behind them. Educational

Leadership, 36, 234-236. Hancock, P. A. (1989). A dynamic model of stress and sustained attention. Human

factors, 31(5), 519-537. Herzberg, F., Mausner, B., & Snyderman, B. B. (2011). The motivation to work (Vol. 1).

Transaction publishers. Hirumi, A. (2016). Principles of Learning and Design. University of Central Florida.

Presentation, Orlando, 2018. Hirumi, A. (2014). Grounded Design of Online and Hybrid Learning: Design

Fundamentals. International Society for Teacher Education. Hirumi, A. (1993). An empirical evaluation of a model to determine the relationship

between learners' attitude toward instruction and learners' acquisition of verbal information and intellectual skills. Unpublished doctoral dissertation, Florida State University, Tallahassee, Florida.

Honey P., & Mumford A. (1992). The manual of learning styles. Maidenhead: Peter Honey Publications.

Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem solving learning environments. New York, NY: Taylor and Francis.

Jones, J. E., & Pfeiffer, J. W. (1975). The 1975 Annual handbook for group facilitators. University Associates.

Jumaat, N. F., & Tasir, Z. (2014, April). Instructional scaffolding in online learning environment: A meta-analysis. In Teaching and Learning in Computing and Engineering (LaTiCE), 2014 International Conference on (pp. 74-77). IEEE.

Kass, A., Burke, R., Blevis, E., & Williamson, M. (1994). Constructing learning environments for complex social skills. The Journal of the Learning Sciences, 3(4), 387-427.

Keller, J. M. (1983). Motivational design of instruction. In C. M. Reigeluth (Ed.), Instructional-Design Theories and Models (pp. 386-429). NJ: Lawrence Erlbaum.

Klein, G., Baxter, H. C. (2006). Cognitive transformation theory: Contrasting cognitive and behavioral learning. Proceedings of the Interservice/Industry Training Systems and Education Conference. Washington, DC: NDIA.

52

Klein, G., Phillips, J. K., Rall, E., Peluso, D. A. (2007). A data/frame theory of sensemaking. In Hoffman, R. R. (Ed.), Expertise out of context (pp. 113-158). Mahwah, NJ: Erlbaum.

Kolb D.A. (1999). The Kolb Learning Style Inventory, Version 3. Boston: Hay Group. Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and

development. New Jersey: Prentice Hall. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into

practice, 41(4), 212-218. Krathwohl, D.R., Bloom, B.S., & Masla, B.B.(1964). Taxonomy of educational

objectives, II. Affective domain. New York, McKay. Lee, K., & Ashton, M. C. (2004). Psychometric properties of the HEXACO personality

inventory. Multivariate behavioral research, 39(2), 329-358. Locke, E. A., & Latham, G. P. (1990). A theory of goal setting & task performance.

Prentice-Hall. Marshall, S, P. (1995). Schemas in problem solving. New York, NY: Cambridge

University. Maslow, A. H. (1943). A theory of human motivation. Psychological review, 50(4), 370-

396. McClelland, D., Atkinson, J., Clark, J., & Lowell, E. (1953). The achievement motive.

New York: Appleton-Century-Crofts. Merrill, D. (2002). First Principles of Instruction. Education Technology, Research and

Development, 50(3), 43-59. Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus

transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior, 16(5), 519-533.

Myers, I.B. (1962). ‘The Myers-Briggs 7ype Indicator. Palo Alto, Cal.: Consulting Psychologists Press.

Nelson, L. M. (1999). Collaborative problem solving. Instructional design theories and models: A new paradigm of instructional theory, 2, 241-267.

Rogoff, B. (1995). Observing sociocultural activity on three planes: participatory appropriation, guided participation, and apprenticeship. In J. V. Wertsch, P. Del Rio, & A. Alvarez (Eds.), Sociocultural studies of the mind (pp. 139-164). New York: Cambridge University Press.

Norman, W. T. (1963). Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. Journal ofAbnormal and Social Psychology, 66, 574-583.

Oatley, K., & Johnson-Laird, P. N. (1987). Towards a cognitive theory of emotions. Cognition and Emotion, 1(1), 29-50.

Paivio, A., Rogers, T. B., & Smythe, P. C. (1968). Why are pictures easier to recall than words?. Psychonomic Science, 11(4), 137-138.

Parsons, S. A. (2008). Providing All Students ACCESS to Self-Regulated Literacy Learning. The Reading Teacher, 61(8), 628-635.

Prensky, M. 2000. Digital game-based learning. New York: McGraw Hill. Schank, R. C., Berman, T. R., & Macpherson, K. A. (1999). Learning by

doing. Instructional-design theories and models: A new paradigm of instructional theory, 2, 161-181.

53

Simpson, B.J. (1966). The classification of educational objectives: Psychomotor domain. Illinois Journal of Home Economics, 10(4), 110-144.

Sternberg, R.J. (1997). Thinking styles. New York: Cambridge University Press. Sweller, J. (2011). Cognitive load theory. In Psychology of learning and motivation (Vol.

55, pp. 37-76). Academic Press. Tannenbaum, S.I., Smith-Jentsch, K.A., & Behson, S.J. (1998). Training team leaders to

facilitate team learning and performance. In J.A. Cannon-Bowers & E. Salas (Eds). Making Decisions Under Stress (pp. 247-270). Washington, DC: APA.

Tomkins, S. S. (1984). Affect theory. In K.R. Scherer & P. Ekman (Eds.), Approaches to emotion (pp. 163–195). New York, NY: Psychology Press.

Tulving, E., & Thomson, D. M. (1973). Encoding specificity and retrieval processes in

episodic memory. Psychological review, 80(5), 352-373 Topping, K. J. (1996). The effectiveness of peer tutoring in further and higher education:

A typology and review of the literature. Higher education, 32(3), 321-345. Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a

learner's mind: Instructional design for complex learning. Educational psychologist, 38(1), 5-13.

Vermunt J.D. (1994). Inventory of Learning Styles (ILS) in higher education. Tilburg: University of Tlburg.

Vernon, P. E. (1964). The structure of human abilities. London: Methuen. Wiley, D. A., & Waters, S. H. (2005). Scoping and sequencing educational resources

and speech acts: a unified design framework for learning objects and educational discourse. Interdisciplinary Journal of Knowledge and Learning Objects, 143.

Wulf, G., McConnel, N., Gartner, M. & Scharwz, A. (2002). Enhancing the learning of sport skills through external focus feedback. Journal of Motor Behavior, 34(2), 171‒182.

Zimmerman, B. J. (2013). Theories of self-regulated learning and academic achievement: An overview and analysis. In B.J. Zimmerman & D.H. Schunk (Eds.), Self-regulated learning and academic achievement (pp. 10-45). Routledge.