background key stage 3 and ict: what are they? statute currently demands that all pupils aged 7, 11...
TRANSCRIPT
Background
Key stage 3 and ICT:What are they?
Statute currently demands that all pupils aged 7, 11 and 14 be tested in “core” subjects of English, maths and science. These are known as key stages 1,2 and 3. There is no requirement to test Information Communication Technology but there is a requirement for teachers to report an attainment level for pupils at key stage 3. The project which informed this presentation may lead to a statutory assessment in ICT at key stage 3.
ICT is:
Information Communication Technology
My first assumption is:
That everyone has read my accompanying paper
That we know what data we need, we’ve been testing for a long time
My second assumption is:
That my first assumption is untrue
Pellegrino’s Triangle
Cognition
InterpretationObservationFor this you require: data
Give tasks and observe
For this you need to know what it is you need to understand
Which In turn drives the development of tasks
To deliver value to learners
To provide valid information to stakeholders, to provethe test project meets its business case
To inform the development of the test model andtest construction, marking and reporting, both formatively and summatively,
To improve test delivery infrastructure
To prove validity
Why do we need data?
Questions that have been asked:
Is the on screen test comparable to the paper test?
Are the results reliable?
Do questions perform in the same wayon screen as on paper?
Are pupils disadvantaged by their level ofcomputer literacy?
How do we make the tests secure?
Does having different types of equipmentaffect the results?
How do we present the questions in a different media?
What we are measuring and why
Questions that should be being asked : What can we test using this medium? How do we deliver assessments that are fair to learners?
How can we model this around the known opportunities and limitations the medium offers? How do we build a process model that provides all stakeholders with confidence that their needs are being met?
What information can we elicit from an e-assessment? What data is required to support the information needs of all the stakeholders?
Who are the stakeholders?
The Department for Education and SkillsMinisters; Senior officials within DFES;Transforming Secondary Education Programme Board Members;KS3 Strategy ICT Assessment Programme Board;11-14 Sub Programme Board Members;Pupil Standards Division;KS3 Delivery Team and Strategy Advisors;ICT In Schools Programme;Operational staff in other Divisions within Schools Directorate and across DFES as below: Analytical Services Directorate (ASD);Commercial Services Division – (CSD, Gateway Review Team);Special Educational Needs Division (SEND – National Framework Team); Curriculum Division (CD – Curriculum Innovation);Schools Communications Unit (CSU) – Communications with LEA’s;Project and Programme Management Unit (PPMU- health check);Internal Audit Division (IAD); and Nigel Hall CGEY Consultancy.Qualifications and Curriculum Authority (QCA)Key Stage 3 ICT Test Development Project;Software contractors;QCA Project Staff;External StakeholdersBECTA;Schools (teachers/headteachers/technical and teaching support staff/pupils);LEA’s (Assessment Advisers, LAI including ICT Advisers, ICT Line Managers, ICT Consultants, KS3
Strategy Managers and Standards Fund officers);Managers and Standards Fund officers);Teacher Training Agency (TTA);Office for School Standards in Education (OFSTED).
Indicative stakeholder requirements:
Marking rules refinement, through updating a rules base or marking scheme.
Government or management performance statistics
Contract performance information – hardware/software interaction, screen refresh time, speed of data processing, support etc
Formative and summative report statement development
Marking algorithm, including sufficiency threshold, performance,
Software security performance measures, such as number of hacking attempts.
Teaching strategy management – washback
Business case metrics
On going user acceptance testing, including regression testing of software developments
The pedagogic need for the test, eg a diagnostic maths test for 11 year olds from reporting a standardised score, a strengths / weaknesses profile of performance & %ile ranking
Proof of validity.
Test security
Source = automatically available 1 = Primary data source= automatically transferred SMS= Schppl Management System= by data file transfer CPS= Central Point Server= exception report APS= Administration Point Server
Available to:Data SMS APS CPS QCA DfES Data source or outcome
Administration Pupil biodata and test historyLocal data 1 2 3 School management systems
to SMS 2a Input by school at admin set up to SMS 2b Input by pupil on registering for a test.
National data 1 DC1 / DC2 School level dataSMS 1 2 PLASC data set
Performancewithin a test 1 2 Initial teacher assessment range data
1 2 Confirmation question data1 2 Pupil log data files1 2 Free text response data filesPr
actic
e te
st o
nly
Data flow chart
The way that data is made available to different parts of the system
Data
Test
Pupil
Question
Technical
Task /process
Audittrail
Markingschema
Testdata
Socio –economic
SchoolPLASC
Award
Report
Diagnostics
Subtasks
Testpath
Answer
Timetaken
Changes
Re-visits
Param-eters
Correctanswer
Randomisation
Adapt-Ivity
Algor-ithms
Param-eters
Adaptivity
Curriccover
Stats
Infra-structure
Hardware
Software
Algor-ithms
DeliveryMode audio
textual
Summ-ative
Form-ative
SubjectSpecif.
Non-subSpecif.
Data relationships
GeneratedAudience FrequencyPupil AccessedCreatedStored enabler icaa qca dfes schoolProfiles
ICAA CPS CPS Test/task fly opportunities given vs takenICAA CPS CPS Test/task fly processes demonstrated mapping against rules base (Formative)ICAA CPS CPS ICAA fly sufficiency demonstrated by level descriptionICAA CPS CPS CPS A performance vs ILAICAA CPS CPS ICAA A Boundary set profile (Summative test)
Levels ICAA APS CPS Teacher fly Initial teacher assessment levelICAA APS CPS Test/task fly Confirmation question levelICAA APS CPS Test/task fly Pupil task outcome level (Formative)ICAA CPS CPS Test/task A Pupil task outcome level (Summative)ICAA CPS CPS Test/task A Free text response data files
CPS CPS ICAA A/fly level attainedCPS CPS ICAA A/fly sufficiency of level attainment
test to testNational curriculum year group ICAA CPS CPS CPS MIS A current vs past performance in practice tests
ICAA MIS MIS CPS MIS Post A ICT test performance vs other test performanceClassProfilesLevels ICAA MIS CPS PLASC MIS A Gender
ICAA MIS CPS PLASC MIS A National curriculum year groupICAA MIS CPS PLASC MIS A Ethnicity
Min and max year group ICAA MIS CPS PLASC MIS A First languageICAA MIS CPS PLASC MIS A Free school mealsICAA APS CPS Teacher MIS A SEN provisionICAA MIS CPS PLASC MIS A Pupils in care
Task / contextICAA MIS CPS PLASC A GenderICAA MIS CPS PLASC A National curriculum year groupICAA MIS CPS PLASC A EthnicityICAA MIS CPS PLASC A First language
In APS 50.3 5.6.2.4 pp110 ICAA MIS CPS PLASC A Free school mealsIn CPS ICAA APS CPS Teacher A SEN provisionIn CPS ICAA MIS CPS PLASC A Pupils in careIn CPS
Boundary review test to test ICAA CPS CPS CPS MIS A current vs past performance in practice testsOther subject results ICAA MIS MIS CPS MIS Post A ICT test performance vs other test performance
In APS 50.3 SchoolProfilesLevels ICAA CPS CPS CPS A Gender
ICAA CPS CPS CPS A National curriculum year groupICAA CPS CPS CPS A EthnicityICAA CPS CPS CPS A First languageICAA CPS CPS CPS A Free school mealsICAA CPS CPS CPS A SEN provisionICAA CPS CPS CPS A Pupils in care
Data & reporting schema
Data Sets and Reports as defined in current HLD and DD
Related DocumentsQCA 32.1, QCA 32.3, QCA 50.9, QCA 54.2
Data Stored at
CPS
Data Stored at
APSData not available
Pre-defined report at
CPSCPS Pre-defined report
ref in QCA 50.9
Pre-defined report at
APSAPS Pre-defined
report ref in QCA 50.9Ad-hoc report
Opportunities Taken vs Opportunities Taken
How can you have data partially stored no
How can you have partial data?
no It should be (see comments)
n/a It should be (see comments) n/a n/a yes
Summative Test Formative Statement yes n/a n/a yes CPS: 50.9-5.1.7 CPLR002 no n/a n/a
Practice Test Formative Statement no yes n/a no n/a yesAPS: 50.9-6.2.2 ASLR004 n/a
Sufficiency demonstrated by level yes no n/a yes
Algorithm report 50.9-5.1.5.2 now superceded by 54.2-4.3.3 f inal report w ill be in 51.2.4 no n/a n/a
Performance vs ILA yes yes n/a yes 50.9-5.1.1 CPLR001 yes 50.9-6.2.4 ASLR006 n/a
Boundary Set Profile (Borderline Check) yes no n/a yes
Algorithm report 50.9-5.1.5.2 now superceded by 54.2-4.3.3 f inal report w ill be in 51.2.4 no n/a n/a
Teacher initial level assessment yes yes n/a yes 50.9-5.1.1 CPLR001 yes 50.9-6.2.4 ASLR006 n/aConfirmation Question Level (ILCT) yes no n/a yes 50.9-5.1.10 CPLR004 no n/a n/a
Pupil Task Outcome (Formative Test) no yes n/a yes CPS: 50.9-5.1.7 CPLR002 yes 50.9-6.2.2 ASLR004 n/aPupil Task Outcome (Summative Test) yes no n/a yes CPS: 50.9-5.1.9 CPLR003 50.9-6.2.2 ASLR004 n/aFree text response data f iles (summative) yes no n/a yes 54.2-4.7 no n/a n/a
Level attained yes yes n/a yes CPS: 50.9-5.1.9 CPLR003 yes 50.9-6.2.2 ASLR004 n/a
Sufficiency of level attained yes yes n/a yes CPS: 50.9-5.1.9 CPLR003 yes 50.9-6.2.2 ASLR004 n/a
Current vs past performance in practice tests partial partial partial no n/a no n/a yes
ICT test performance vs other test performance partial partial partial no n/a no n/a yes
Genderyes - summative
yes - practice n/a
no - but available by pupil n/a
no - but available via MIS n/a yes
National curriculum year groupyes - summative
yes - practice n/a
no - but available by pupil n/a
no - but available via MIS n/a yes
Ethnicityyes - summative
yes - practice n/a
no - but available by pupil n/a
no - but available via MIS n/a yes
First languageyes - summative
yes - practice n/a
no - but available by pupil n/a
no - but available via MIS n/a yes
Free school mealsyes - summative
yes - practice n/a
no - but available by pupil n/a
no - but available via MIS n/a yes
Levels
Class
Data Set / Report
Profile
Levels
Test to Test
PUPIL
Level 5
ProcessesCheck accuracy
Elaborationse.g. looks for
more than 1 sourcee.g. checkscalculations
e.g. models answer
Evidencese.g. finds alternative
matching setse.g. finds non-matching
sets and validatese.g. looks butcannot find alternatives
Atomic evidences
e.g. opens 3 alternativepotential sites
e.g. cuts and pastes‘matching’ data
e.g. keys in matchingdata
Pupils select the information they need for different purposes,check its accuracy and organise it in a form suitable for processing. They use ICT to structure, refine and present information in different forms and styles for specific purposes and audiences. They exchange information and ideas with others in a variety of ways, including using ….
Opportunities are clusters of Atomic Indicators, clusters of opportunities within a task provide evidence
Example of the rules base structure
Generic rules
Micro and Macro level data
Where the micro looks in detail atwhat the pupil did and how the test itselfperformed –
the macro defines and looks at the more holistic,esoteric measures, such as efficiencyand appropriateness
Process Indicator 9Accuracy
Elaboration 2Accuracy of Text
Elaboration 1Accuracy of data
Selected text
Use of hyperlink
Modelling formulae
All text
Used appropriately throughout the document
etc
Spell check
Grammar check
Corrected text
Elaboration 3Accuracy of content
Used appropriately say, at the end of the document
Test development – skills needed to deliver data requirements
Test writers require a knowledge of:
the subject domain
the detailed workings of the processes and the rules base
the marking / decision algorithms to be employed in the test engine
metadata and ambient data requirements
the use of the media in the assessment itself
adaptivity of the test items or tasks
accessibility of language
special educational needs accessibility tools and software
technical issues
test security
Software schema for theTest Package Set
ParametersMetadataAlternate forms
CA MappingAmbient dataMetadata
Commands DLL
Test Package Set
Captured Actions DLL
Atomic Indicator DLL
Toolkit
Marking data
Test
Word ProcessorSpreadsheetWeb browsereMailFile ManagerDatabaseGraphics
Tasks
Virtual worldRandomisationAdaptivity
Level 4Level 5Level 6, 7, 8
Clusters of functions or Captured pupil actions= Atomic IndicatorCopies & pastes data set
Paste
Copy
Data
Metadata & ambient data
Ctrl C
Edit paste
Metadata:Application,Time, etc
- <event name="MessageSend (end)" time="13/05/2004 10:49:37">- <eventData>- <CapturedAction Type="MessageSend"> <Field Name="InvokedInApplet" Value="eM" /> <Field Name="SentSubject" Value="Re: L3 Some advice please" /> <Field Name="Time" Value="0:35:44.46" /> <Field Name="FlagUrgent" Value="No" /> <Field Name="InvocationIdentifier" Value="SendMessageCommand" /> <Field Name="CellIdentity" Value="" /> <Field Name="SentAddress" Value="[email protected]" /> <Field Name="SentCCName" Value="" /> <Field Name="HandledInApplet" Value="eM" /> <Field Name="Text" Value="> As your work on this task is getting close to its end, I would
like you to help me understand some of the things you have done. Please could you email the answers to these two questions before you continue with the work? > > > > 1. You have created several files. How do you keep track of them all? > i organise them into folders with appropriate names so i know where to find them! > > > 2. I have noticed that people use the copy command. Please could you explain why that is? > so you don't have to write out long pieces of information, you can easily and quickly copy and paste them! > > > Felicity Murray > > Editor" />
<Field Name="SentCCAddress" Value="" /> <Field Name="Name" Value="MessageSend" /> <Field Name="InvocationMethod" Value="toolbar" /> <Field Name="SentName" Value="Editor Daily Trumpet research" /> <Field Name="RequestReceipt" Value="No" /> <Field Name="Task" Value="vmx://execution/TASK_3_3.xml" /> <Field Name="AssetIdentifier" Value="vemail://local/Drafts/DTN_4_SEM_0" /> <Field Name="SelectionText" Value="" /> </CapturedAction> </eventData>
Example of meta- and ambient data
<?xml version="1.0" encoding="UTF-8"?><!-- edited with XMLSPY v5 rel. 3 U (http://www.xmlspy.com) by John Taylor (3T Productions Limited) --><atomicindicators ilct="1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation=".\ais.xsd">
<!-- test series identifier --><series year="2004" stage="KS3" subject="ICT" issue="S" version="1"/><!-- marking scheme identifier --><scheme context="DTN" level="4"/><atomicindicator name="AI_ILCT_G0006_0083" catype="Copy" threshold="1">
<fields><field name="Task" datatype="STRING" comparator="IN" values="ILCT"/><field name="InvokedInApplet" datatype="STRING" comparator="IS" values="eM,sH"/><field name="InvocationMethod" datatype="STRING" comparator="IS" values="accelerator"/><field name="SelectionText" datatype="STRING" comparator="IN" values="Get hold of data about the town population, teachers and the police force, from different contacts"/>
</fields></atomicindicator><atomicindicator name="AI_ILCT_G0006_0084" catype="Copy" threshold="1">
<fields><field name="Task" datatype="STRING" comparator="IN" values="ILCT"/><field name="InvokedInApplet" datatype="STRING" comparator="IS" values="eM,sH"/><field name="InvocationMethod" datatype="STRING" comparator="IS" values="accelerator"/><field name="SelectionText" datatype="STRING" comparator="IN" values="put together a model containing all the data from the three areas"/>
</fields></atomicindicator><atomicindicator name="AI_ILCT_G0006_0085" catype="Copy" threshold="1">
<fields><field name="Task" datatype="STRING" comparator="IN" values="ILCT"/><field name="InvokedInApplet" datatype="STRING" comparator="IS" values="eM,sH"/><field name="InvocationMethod" datatype="STRING" comparator="IS" values="accelerator"/><field name="SelectionText" datatype="STRING" comparator="IN" values="put together the complete article"/>
</fields></atomicindicator>
Software manifestation of the atomic indicators
DfES
Contractor
QCA
Primary Local Data
Other subject results
Plasc return
Centralserver
Pupil biodataresults
Test performance
Test metadata
System performance
SchoolMIS
Task / assetby context
Initial LevelAssessment
Initial LevelConfirmation
Free text
Context
Practice test results
Localserver
Rules base mapping
Marking algorithms
Class
Demographics
Service levelagreement
Primary data
Data
Machine diagnosticsSoftware diagnostics
Reporting
Test Results Package
Sessioninformation
Audit logs
TRP manifest(index of files)
Event audit log
Results.xml
Diagnostics
Captured actions
Free text
Index of CA / Diagnostic / Free text mappings
Detail of CAs invoked and associated metadata minute by minute
Artif Int marked
Questions asked
Responses r’cvd
Human markedCommands for apps
Test State Info Handles AI/CA mappings & ADQs passes info to results.xml
FormativeSummative
Count data
DfES
Contractor
QCA
Report generation points
Centralserver
SchoolMIS
Test / task / asset performance
Pupil / class / school performance
System performance
Performance vs other subjects
Management performance
Practice test performance
Process & test evaluation
LEA / National level performance
Class / class performance
Infrastructure performance
Fairness & validity
Marking performance
PSA & strategy performance
Reports data flow
Central PointSystem
Admin PointSystem
DeliveryPoint
System
School MIS
QCAICAA
3T
ICAA
Pupil Schools
AuthoredTests
PLASCData
TestResponse
ResultsReports
ResultsReports
ResultsReports
Tests forSchools
Test forPupil
PupilResult
AggregatedPupil
Results
Total "confidence of achievement" from the three tasks
Wholly directed Independent workingLevel "THEMES" Develop ideas & Combine tools Integration Scope info flow System design implement for
solve problems Critical evaluation & efficiency Define impl refine others to use1 100% 1 3 100% 3 5 100% 5 6 0% 0 7 0% 0 8 0% 01 100% 1 3 100% 3 5 100% 5 6 10% 1 7 0% 0 8 0% 01 100% 1 3 100% 3 5 86% 4.3 6 0% 0 7 0% 0 8 0% 01 100% 1 3 100% 3 5 24% 1.2 6 0% 0 7 0% 0 8 0% 01 100% 1 3 100% 3 5 0% 0 6 10% 1 7 0% 0 8 0% 0
3 100% 3 5 30% 1.5 6 0% 0 7 0% 0 8 0% 02 100% 2 3 100% 3 5 0% 0 6 0% 0 7 0% 0 8 0% 02 100% 2 3 100% 3 5 33% 1.7 6 10% 1 7 0% 0 8 0% 02 100% 2 Combine refine 5 33% 1.7 6 0% 0 7 0% 0 8 0% 02 100% Interpret plausibility 5 20% 1 6 10% 1 7 0% 0 8 0% 02 100% 2 4 100% 4 5 33% 1.7 6 0% 0 7 0% 0 8 0% 02 100% 2 4 100% 4 5 35% 1.8 6 0% 0 7 0% 0 8 0% 02 100% 2 4 100% 4 5 90% 4.5 6 10% 1 7 0% 0 * Level2 100% 2 4 100% 4 5 70% 3.5 6 0% 0 7 0% 0 TA2 100% 2 4 100% 4 5 60% 3 6 10% 1 7 0% 0 Norm ref Q's
2 4 100% 4 5 30% 1.5 6 10% 1 7 0% 0 Open taskAspects 4 100% 4 6 0% 0 7 0% 0 Direct Q's
Find 4 100% 4 7 0% 0Dev 4 100% 4 Total 41.14 7 0% 0 LEVELEx 4 100% 4 Total weighted Level 7 0% 0Rev 4 100% 4 132 5 7 0% 0Level 5
Skills / techniques Skills/ tech & app Application
QCA TeLogo Name: Joe Bloggs Class: 9A
In the Tourist Information Centre, you were asked to:oMake a presentation for use in the Pepford Tourist Information Centre.oWork out how much it will cost to stay in a hotel.oSend an email with an attachment.
When working in Pepford Tourist Information Centre, you showed that you could:
Using data and information sources decide which sources of information were most accurate and relevant, e.g. you compared data from two websitesdecide which data is most suitable for the job you are doing, e.g. you put the address of the tourist information centre in your poster
Searching and selecting search data to help you prepare your presentation, e.g. you used efficient searches in the web browserexperiment with rules of a model to see what happens, e.g. you changed the hotel pricing model
Organising and investigating work out a new cost when your customer asked you to e.g. you entered customer information into the model to find out the price of the hotel.make changes to your presentation to make it better e.g. you used autorun for the presentation
Analysing and automating processes create a model for a system e.g. you created an efficient model to calculate the cost of a hotel bookingautomate simple processes e.g. you used a master slide for your presentation
Models and modelling use a spreadsheet or model e.g. you used a spreadsheet to create a model to work out hotel costschange a spreadsheet model, e.g. you changed the hotel cost model to make it easier to use
Control and monitoring N/A
Fitness for purpose change your work to make it suitable an audience e.g. you changed your presentation to make it suitable for younger peoplepresent your ideas in a way which suited the audience e.g. you used a large font to make it suitable for young people
Refining and presenting information combine information from a wide variety of sources to solve a problemchoose the most suitable data from the sources available and combine them in your presentation e.g. you choose an appropriate logo for the Tourist Information Centre
Communicating use email and other tools to share ideas with groups of people e.g. you sent your presentation to Mr. Jones, the manager.use ICT in a businesslike way e.g. you used the letter template to write a letter to Mr. Harris.
Validitya fine balance
Context
Security
Adaptivity Randomisation Construct
Face
Content
Di/convergent
Predictive
Concurrent
Statistical
TESTTESTTESTTESTTESTTESTTEST
DATADATADATADATADATADATADATADATADATADATADATA
RELIABIL
ITY R
ELIABIL
ITY R
ELIABIL
ITY R
ELIABIL
ITY R
ELIABIL
ITY R
ELIABIL
ITY