making the most of piaac - literacy and numeracy for adultset+al... · understanding of complex...

Preliminary investigation of adults’ numeracy practices through secondary analysis of the PIAAC dataset

Making the most of PIAAC:

Diana Coben, National Centre of Literacy & Numeracy for Adults, University of Waikato, New Zealand [email protected] Paul Satherley & David Earle, Ministry of Education, New Zealand Barbara Miller-Reilly, University of Auckland, New Zealand ALM22 conference: Opening Our Mathematical Eyes

1-2 pm, 13th July 2015 http://www.alm-online.net/home/alm-22-conference-page/

What is PIAAC? •  Programme for the International Assessment of Adult

Competencies, aka The Survey of Adult Skills www.oecd.org/site/piaac/

•  PIAAC assesses key cognitive and workplace competencies in: –  Literacy –  Numeracy –  Problem solving in technology-rich environments

•  Collects information on how often people undertake a range of (literacy and) numeracy activities at work and in everyday life

•  New Zealand’s results due 2016 (PIAAC Round Two)

What are we doing?

•  Exploring what secondary analysis of PIAAC data can tell us about what adults do regarding numeracy

•  Reviewing accessibility and user-friendliness of the data

•  Encouraging beginners and experienced practitioners and researchers to explore the PIAAC dataset

Making the most of secondary data sources

“A number of secondary data sources [!] are currently under utilised for education research on adults. One reason for this is that a huge time investment is needed by the researcher to analyse such data, as the data sets are large and complex. Making available better guidance specific to education researchers would be helpful to encourage greater use of these data sets.”

(Vignoles, 2007)

Numeracy in PIAAC “the ability to access, use, interpret and communicate mathematical information and ideas in order to engage in and manage the mathematical demands of a range of situations in adult life!” “Numerate behaviour involves managing a situation or solving a problem in a real context, by responding to mathematical content/information/ideas represented in multiple ways.”

(OECD, 2012, p. 34)

PIAAC data related to numeracy practices

•  PIAAC directly measures numeracy skill through a numeracy assessment

•  Provides average scores on the PIAAC numeracy scale for whole 16-65 year old population, or subgroups

•  Background Questionnaire - administered face-to-face in the respondent’s home by an interviewer who enters the answers into a laptop computer

PIAAC Background Questionnaire Covers educational background, skills used at work (for those currently or recently in employment) and in other contexts such as the home and the community, e.g.:

–  Demographic characteristics – age, gender, ethnicity, country of birth, first language, language spoken most often at home

–  Other personal characteristics – learning dispositions, self-assessed health status

–  Education and training characteristics –  Work characteristics –  Self-assessed mathematics skills for work –  Self-assessed reading and writing skills for work –  Skill use in everyday life – literacy activities, ICT activities

May help answer interesting questions when analysed together with numeracy practice information

The numeracy practices questions in the PIAAC Background Questionnaire

Interviewer asks respondent how often s/he undertakes 7 different numeracy practices in work or everyday context: •  In your job, how often do you usually calculate prices, costs or

budgets? •  Use or calculate fractions, decimals or percentages? •  Use a calculator – either hand-held or computer based? •  Prepare charts, graphs or tables? •  Use simple algebra or formulas? •  Use more advanced mathematics or statistics such as calculus,

complex algebra, trigonometry, or use of regression techniques?

The same questions are asked, in the past tense, for people who are not currently working, but who worked in the last 12 months

Frequency! and its limitation

•  The five frequency options are: –  never –  less than once a month –  less than once a week but at least once a month –  at least once a week but not every day –  every day

•  Limitation: questions only measure frequency; they do not measure intensity or complexity of numeracy activity

Types of analysis and analysis issues •  A range of types of analysis are possible to help answer

different questions about adults’ numeracy practices. •  First steps often simple univariate tabulations - provide a

‘big picture’ view •  Next steps may be multivariate analysis - allows better

understanding of complex associations between factors •  PIAAC dataset can provide in-depth understanding of

associations between various factors •  Multivariate analyses:

–  can show which factors are most strongly associated with frequent numeracy practice, and

–  can provide numerical measures of, e.g., how much increase in frequency of numeracy practice is associated with one unit of measured numeracy skill, whilst holding other factors constant

Secondary analysis of PIAAC data Secondary analysis of PIAAC data is: •  undertaken later in time than primary analysis •  more in-depth and multivariate •  less headline-focused and descriptive •  brings together different parts of PIAAC dataset to generate new

knowledge and understanding of associations and relationships •  provides empirical findings to build or refine conceptual or statistical

models •  often theme-based, rather than general in focus

Primary PIAAC analysis: •  includes average scale scores in PIAAC skill domains for key

demographic and socioeconomic groups •  aligns with international headline reporting

What PIAAC data cannot tell us •  PIAAC data cannot tell us to what extent frequent

numeracy practice causes high numeracy skill •  Analytical writing should clearly distinguish associations

from causes and not use ambiguous language such as leads to, brings about, influences, is linked with, etc.

•  It may be that numeracy practice, opportunity or requirement to undertake numeracy practice, and numeracy skill, are all mutually reinforcing

•  Even where the data show a strong association between two factors (e.g., numeracy practice and numeracy skill) whilst controlling for other factors, we cannot infer that a change in one factor will result in a change in the other – either on an individual or a group level

What PIAAC can tell us about adults’ numeracy activities

•  How often do people perform numeracy activities at work?

•  How often do people perform numeracy activities in everyday life?

•  What are the patterns for different groups in numeracy activities at work?

•  What are the patterns for different groups for numeracy activities in everyday life?

•  What similarities or differences exist for numeracy activity at work and in everyday life?

•  What similarities or differences exist for numeracy and literacy activity at work?

What PIAAC can tell us about adults’ numeracy activities (continued)

•  What similarities or differences exist for numeracy and literacy activity in everyday life?

•  What are the patterns of work-related numeracy activity together with ICT activities?

•  What are the characteristics of people who seem to be matched or mismatched on numeracy skill at work and numeracy activity? Is ‘field of study of highest qualification’, or ‘occupation’ related to the match or mismatch?

•  What changing patterns for young people’s numeracy activities can we see on their pathways from school to tertiary education to work?

•  How have patterns of numeracy activity at work changed over time?

What PIAAC data are publicly available and how can they be accessed?

•  Information on PIAAC ‘Public Data & Analysis’ is at http://www.oecd.org/site/piaac/publicdataandanalysis.htm

•  OECD PIAAC Gateway website ‘Data Tools › Datasets and Tools’ http://piaacgateway.com/datasets/ gives users tools needed to analyse the PIAAC dataset

•  Guide: What You Need to Consider (AIR PIAAC Team, nd) https://piaac.squarespace.com/s/WorkingWithPIAACData_022015.pdf

•  Online Distance Learning Data Training (DLDT) modules http://nces.ed.gov/training/datauser/#PIAAC cover statistical procedures and methods of analysis

•  FAQ webpage http://nces.ed.gov/surveys/piaac/faq.asp •  Interactive ‘Q & A. Your questions, our answers’ at

http://piaacgateway.com/your-questions-our-answers/

PIAAC Data Explorer (PDX) •  PIAAC Data Explorer (PDX)

http://piaacdataexplorer.oecd.org/ide/idepiaac/variables.aspx -web-based application for accessing PIAAC data, does not require advanced statistical knowledge or specialist software

•  point-and-click interface for creating statistical tables and charts and exploring levels of adult skills and demographics

•  4 steps to go through when using PIAAC Data Explorer, supported by DLDT modules

PIAAC International Data Explorer Step 1

•  We chose “Numeracy” as our Subject •  Note that “Adults (16-65)” only option for Age at this point •  After choosing “Numeracy”, fuller screen appears with a

choice of dataset to use: “PIAAC 2012”, “ALL 2003” or “All years/studies”. We chose “PIAAC 2012”

•  Then choice of 72 Categories and Groups. –  Under the Category “Scale Scores”, and Sub-Category “Skills”,

the Measure “PIAAC Numeric: Numeracy” is already selected. –  Under the Group “International”, we chose the Jurisdiction

“OECD average”

•  Choose “Select Variables” to move to Step 2 to select the independent variable(s)

Language: English

STEP 1:Select criteria from each drop-down menu to begin. Additional options related to your selections will appear. Then select measures, jurisdictions, and years based onavailable data.

Help

Subject: Numeracy Age: Adults (16-65) Reset

View all (72) View Selected (1) Search: Go

Category Sub Category MeasureAll

Years/Studies PIAAC 2012 ALL 2003

Scale scores Skills PIAAC Numeric: Numeracy details

Skill use Reading

components Reading

component timings Full population

estimates International

backgroundquestionnaire

Derivedvariables

Group JurisdictionAll

Years/Studies PIAAC 2012 ALL 2003

International OECD Average

Selected countries

OECD National Entities

OECD Sub-National Entities

Partners

Select Variables

About Accessible Version Data Explorer Technology

Select Criteria Select Variables Edit Reports Build Reports

STEP 1

X


•  Under the Category (and Sub-Category) “Major reporting group” we chose the variable “All Adults”.

•  In addition, under the category “International Background

Questionnaire”, and Sub-Category” “Skill use – literacy & numeracy”, we chose the following six variables:

•  “Skill use work - Numeracy - How often - Calculating costs or budgets” •  “Skill use work - Numeracy - How often - Use or calculate fractions or percentages” •  “Skill use work - Numeracy - How often - Use a calculator” •  “Skill use work - Numeracy - How often - Prepare charts graphs or tables” •  “Skill use work - Numeracy - How often - Use simple algebra or formulas” •  “Skill use work - Numeracy - How often - Use advanced math or statistics” •  Now that the variables are selected, choose “Edit Reports” to

move to Step 3.

STEP 2

All Years/Studies

STEP 2:Select at least one variable from the category list below. View the list of all available variables, view by selected variables only, or search variables by keywords. Yearsselected will override previous selections.

Help

Subject, Age: Numeracy, Adults (16-65)Jurisdiction: OECD AverageMeasure: PIAAC Numeric: NumeracyYear/Study: PIAAC 2012

Reset

View all (334) View Selected (7) Search: Go

Category Sub Category VariablePIAAC 2012 ALL 2003

Majorreporting groups

Majorreporting groups

All adults details

Person resolved gender from BQ and QC check(derived) details

Education - Highest qualification - Level details

Current status/work history - Subjective status details

Current work - Employee or self-employed details

Background - Born in country details

Age groups in 5-year intervals based on LFSgroupings (derived) details

Age in 10 year bands (derived) details

Internationalbackgroundquestionnaire

Education

Current status Current work Last job Skill use at

work Skill use -

literacy &numeracy

Skill use work - Literacy - Read directions orinstructions details

Skill use work - Literacy - Read letters memos ormails details

Skill use work - Literacy - Read newspapers ormagazines details

Skill use work - Literacy - Read professionaljournals or publications details

Skill use work - Literacy - Read books details

Skill use work - Literacy - Read manuals orreference materials details

Skill use work - Literacy - Read financialstatements details

Skill use work - Literacy - Read diagrams maps orschematics details

Skill use work - Literacy - Write letters memos ormails details

Skill use work - Literacy - Write articles details

Skill use work - Literacy - Write reports details

Skill use work - Literacy - Fill in forms details

Skill use work - Numeracy - How often -Calculating costs or budgets details

Skill use work - Numeracy - How often - Use orcalculate fractions or percentages details

Skill use work - Numeracy - How often - Use a

Edit Reports



PIAAC International Data Explorer STEP 2 (later choices)

International background questionnaire Education -->

etc Skill use at work --> Skill use - literacy & numeracy --> Skill use work - Literacy - --- etc

Skill use work - Numeracy - How often - Calculating costs or budgets Skill use work - Numeracy - How often - Use or calculate fractions or percentages Skill use work - Numeracy - How often - Use a calculator Skill use work - Numeracy - How often - Prepare charts graphs or tables Skill use work - Numeracy - How often - Use simple algebra or formulas Skill use work - Numeracy - How often - Use advanced math or statistics

Skill use work - ICT - --- etc


•  The PIAAC International Data Explorer then generated seven possible reports.

•  These seven reports can be reviewed, edited, deleted or copied. New reports can be created, new format and statistics chosen.

•  We selected Averages and Percentages from the drop down menu “Statistics Options”.

•  We chose “Build Reports” to move to Step 4 to view the completed reports.

STEP 3:Preview and edit existing reports using the action links next to each report name. Create new reports, set format and statistic options. (New and copied reports will appearat the bottom of the report list.)

Help

Subject, Age: Numeracy, Adults (16-65)Jurisdiction: OECD AverageMeasure: PIAAC Numeric: NumeracyVariables: All adults, Skill use work - Numeracy - How often - Calculating costs or budgets, Skill use work - Numeracy - How often - Use or calculate fractionsor percentages, Skill use work - Numeracy - How often - Use a calculator, Skill use work - Numeracy - How often - Prepare charts graphs or tables, Skill usework - Numeracy - How often - Use simple algebra or formulas, Skill use work - Numeracy - How often - Use advanced math or statisticsYear/Study: PIAAC 2012

Reset

Create New Report Format Options Statistics Options

Report All Action Measure Variable Year/Study Jurisdiction Statistic

Report 1 Preview | EditDelete | Copy

PIAAC Numeric: Numeracy All adults PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Calculating costs or budgets PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use or calculate fractions or percentages PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use a calculator PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Prepare charts graphs or tables PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use simple algebra or formulas PIAAC 2012 OECD Average Averages


PIAAC Numeric: Numeracy Skill use work - Numeracy - How often - Use advanced math or statistics PIAAC 2012 OECD Average Averages

Build Reports



STEP 3


•  For each of the six variables of Numeracy Skills used at work, a report is generated.

•  Each report lists Averages of the Numeracy scale scores and Percentages of the population (each with Standard Errors) for the response categories (Never; Less than once a month; Less then once a week but at least once a month; At least once a week but not every day; Every day).

•  At Step 4 each report table can be viewed by selecting the report name from the drop-down menu. It is possible to generate charts and complete significant tests for any table.

•  We chose “Export reports” to download selected reports as Excel worksheets (or Word document or HTML).

STEP 4

STEP 4:View each report table by selecting the report name from the drop-down menu. Create report types to edit and preview, each tab created represents one report type toexport.

Help

Subject, Age: Numeracy, Adults (16-65)Jurisdiction: OECD AverageMeasure: PIAAC Numeric: NumeracyVariables: All adults, Skill use work - Numeracy - How often - Calculating costs or budgets, Skill use work - Numeracy - How often - Use or calculate fractions orpercentages, Skill use work - Numeracy - How often - Use a calculator, Skill use work - Numeracy - How often - Prepare charts graphs or tables, Skill use work -Numeracy - How often - Use simple algebra or formulas, Skill use work - Numeracy - How often - Use advanced math or statisticsYear/Study: PIAAC 2012

Select Report: Report 3 Export Reports

Chart Significance Test

Averages and percentages for numeracy, age by Skill use work - Numeracy - How often - Use or calculate fractions or percentages [GQ03C], year and jurisdiction: PIAAC 2012

Year/Study Jurisdiction

Never Less than once a monthLess than once a week but at least once

a month At least once a week but not every day

AverageStandard

Error PercentageStandard

Error AverageStandard






Error Average

NOTE: The Numeracy scale ranges from 0 to 500. Detail may not sum to totals because of rounding. Some apparent differences between estimates may not be statistically significant. SOURCE: Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of Adult Competencies (PIAAC), 2012.

PIAAC 2012 OECDAverage 257 (0.3) 46 (0.2) 283 (0.6) 9 (0.1) 291 (0.6) 8 (0.1) 293 (0.5) 13 (0.1) 290



Table

Analysis of the reports generated on this run

•  Firstly, analysing statistics on Numeracy skill-use in a work context for the first three skills on the IDE list of 6: –  Calculating costs or budgets –  Use, or calculation of, fractions or percentages –  Use of a calculator

•  Then, in the next 2 tables, we’re analysing statistics for the last three Numeracy skills –  Preparation of charts, graphs or tables –  Use of simple algebra or formulas –  Use of advanced math or statistics

NB: se is Standard Error

Average of Numeracy scale scores and Percentage of adults shown for 3 frequency options of 3 Numeracy skill use at work variables

Never Less than once a week but at least once a month Every day

Average (se) Percentage (se) Average (se) Percentage (se) Average (se) Percentage (se)

Frequency of Calculating costs or budgets in a work context

264 (0.3) 49 (0.2) 290 (0.7) 8 (0.1) 280 (0.4) 21 (0.1)

Frequency of Use, or calculation of, fractions or percentages in work context

257 (0.3) 46 (0.2) 291* (0.6) 8 (0.1) 290* (0.4) 23 (0.2)

Frequency of Use of a calculator in a work context

254 (0.4) 30 (0.2) 284* (0.7) 8 (0.1) 283* (0.3) 37 (0.2)

!  For each Numeracy skill differences between all percentages and between most averages are statistically significant (not those marked with *)

!  Percentages of people in work who never use a calculator at work, or use one everyday are markedly different from those for the other two Numeracy skills.

!  These statistics for Numeracy skills may indicate that adults with the lowest skill are choosing (or being chosen) to work in jobs that entail infrequent activity. This is in keeping with research on the British Cohort Studies, which also showed that people’s skills diminish with lack of use (Bynner & Parsons, 1998, 2000).

Average of Numeracy scale scores and Percentage of adults shown for 3 frequency options of the other 3 Numeracy skill use at work variables

Never Less than once a week but at least once a month Every day

Average (se) Percentage (se) Average (se) Percentage (se) Average (se) Percentage (se)

Frequency of Preparation of charts, graphs or tables in a work context

261 (0.3) 60 (0.2) 298 (0.5) 10 (0.1) 294 (0.8) 6 (0.1)

Frequency of Use of simple algebra or formulas in a work context

260 (0.3) 55 (0.2) 296 (0.7) 7 (0.1) 292 (0.5) 17 (0.2)

Frequency of Use of advanced math or statistics in a work context

270 (0.2) 86 (0.1) 306 (1.0) 3 (0.1) 297 (1.4) 2 (0.0)

!  For these Numeracy skills differences between all percentages and between all averages are statistically significant within each skill.

!  For Use of advanced math or statistics the percentages of adults choosing ‘Never’ is highest, and for ‘Everyday’ lowest, frequency options – similar results occur for the other two more complex skills above.

!  Averages of Numeracy scale scores for Use of advanced math or statistics are higher within most response categories which suggests that people with strong numeracy skills have jobs that entail frequent numeracy activity.

Some concluding thoughts

•  Overall, we found our exploration of the PIAAC dataset stimulating and challenging in equal measure

•  While the scale and complexity of the dataset may appear daunting we would encourage you to undertake your own exploration

•  Our advice is: start with something simple and take it from there

References •  AIR PIAAC Team. (nd). What You Need to Consider Before Working with PIAAC

Data. Retrieved 3 May, 2015, from http://static1.squarespace.com/static/51bb74b8e4b0139570ddf020/t/54e773d7e4b0328389a39b36/1424454615327/WorkingWithPIAACData_022015.pdf

•  OECD. (2011). PIAAC Conceptual Framework of the Background Questionnaire Main Survey. Paris: OECD. http://www.oecd.org/site/piaac/PIAAC(2011_11)MS_BQ_ConceptualFramework_1%20Dec%202011.pdf

•  OECD. (2013). Technical Report of the Survey of Adult Skills (PIAAC). Paris: OECD. www.oecd.org/site/piaac/_Technical%20Report_17OCT13.pdf

•  OECD, PIAAC, www.oecd.org/site/piaac/ •  OECD, PIAAC, Public Data & Analysis (includes link to PIAAC International Data

Explorer) http://www.oecd.org/site/piaac/publicdataandanalysis.htm •  PIAAC Numeracy Expert Group. (2009). PIAAC Numeracy: A conceptual framework

OECD Education Working Paper (Vol. 35). Paris: OECD Publishing. http://www.oecd-ilibrary.org/education/piaac-numeracy-a-conceptual-framework_220337421165

•  PIAAC Gateway: Datasets and Tools http://piaacgateway.com/datasets/ •  Vignoles, A. (2007). The Use of Large Scale Data-sets in Educational Research.

Retrieved 10 July, 2015, from http://www.tlrp.org/capacity/rm/wt/vignoles

making the most of piaac - literacy and numeracy for adultset+al... · understanding of complex...

Documents