data quality concepts_ 20th jan 2015
TRANSCRIPT
Accurately Measuring Progress
Khulisa Management Services (Pty) Ltd
DATA QUALITY CONCEPTS & PRINCIPLES
Khulisa Management Services
1st Feb 2015
Accurately Measuring Progress
WHAT IS DATA?
Data
• result of measurements
• building block for information and knowledge
• refers to a collection of numbers / other outputs produced from activities
• can be qualitative or quantitative
DATA INFORMATION KNOWLEDGE
Accurately Measuring Progress
DATA is deemed to be of Good Quality if it accurately represents the situation to which it relates
GOOD QUALITY DATA
Accurately Measuring Progress
DATA QUALITY
Data Quality
• Refers to the worth or accuracy (Value) of the information collected
• Focuses on ensuring that data management processes are of a high standard
• Data reflects TRUE performance
• Data is: Appropriate
Organised
Timely and Available
Accurate and Complete
Cost Effective
Accurately Measuring Progress
DATA QUALITY - VCRIPT
1. VALIDITY / ACCURACY
• Data is considered correct• Data measures what it intended to measure
• Minimizes errors: during, recording, transcribing etc.
5.PRECISION
• Data has sufficient detail• Error measurements: source bias / error; instrumentation error; sampling error; transcription error and manipulation error
4.INTEGRITY
• Data generated is protected from deliberate bias or manipulations (personal or political)
• Truthfulness of data
2. COMPLETENESS
• Information system providing the results is appropriately inclusive
• Represents the complete list of eligible persons or units and not just part thereof
3. RELIABILITY
• Data generated is based on protocols and procedures that do not change according to who is using it
• Data is measured and collected consistently
• CONSISTENCY + INTERNAL QUALITY CONTROL + TRANSPARENCY
6.TIMELINESS
• Data that is up-to-date (current) and when info is available on time
• Timeliness is affected by rate of updating; rate of activity changes and when info is used or required
• FREQUENCY AND CURRENCY
Accurately Measuring Progress
DATA QUALITY DIMENSIONS CRITERIA DEFINING CHARACTERISTICS
ValidityPrecise definitions for data to be collectedValidation of data sourcesDesign of data tools
Completeness
Completeness of received filesNo missing informationNo blank fieldsNeed procedures for addressing incomplete and missing data
Reliability
Consistency of tools and processes including TechnologyDocumentation of DMS and Standard Operating ProceduresDocumentation of Data Quality PlanImplementation of quality control checksCommunication of data problemsHR recruitment and training
Professionalism – objectivity in analyses policies and practicesTransparency – documentation and QC stepsEthical standards – avoids incentives, confidentiality of data
Accurately Measuring Progress
Accurately Measuring Progress
Quality• Quality in ISO 9000:2000 is defined as
the degree to which a set of inherent characteristics fulfils the requirements.
• With this definition the implication is that quality is relative to what something should be (requirements) and what it actually is. (To all stakeholders and not only to some.)
Accurately Measuring Progress
DATA MANAGEMENT SYSTEM
SOURCE
COLLECTION
COLLATION
ANALYSIS
REPORTING
USE
Definition•Where the data comes from
Examples• Registers or filesDefinition
•Process of gathering data e.g. Excel Files or Activity Forms etc Definition
• Aggregation of data into summarized formats
Examples• Excel Spreadsheets’ formulae
Definition•Review and analyse data
• Examples• Trend Analyses• Issues Analyses
Definition•Descriptive information, •presenting data
Definition• Evidence based decisions
Accurately Measuring Progress
BMGF - INDICATORS
Accurately Measuring Progress
Poorly Defined Indicators
• Undermine the validity of data such that
(i) incorrect data may be collected or
(ii) the data collection process is not correct for the data that is meant to be gathered.
(iii) each person interprets the indicator in his/her own way and goes about their activities according to his/her individual understanding.
Accurately Measuring Progress
INDICATOR USEFULNESS - CRITERIA
CRITERIA DEFINING CHARACTERISTICS
RelevanceIndicators are in line with Project/Programme activities, objectives, goals and visionDetermines what data is produced and required resources
InterpretabilityProvides sufficient information to allow users to properly interpret statistical information e.g. the concepts, variables, classifications and methodology
AccessibilityReflects how readily the indicator data can be located and accessedLabeling, filing and dissemination process are adequate
Coherence
The coherence of indicators reflects the degree to which it can be successfully brought together with other data within a broad analytical framework and over timeThe use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across stakeholders
Accurately Measuring Progress
Common Risks to Data Quality: Manageable Risks
• Manageable risks = risks whose mitigation is within an organisation’s control.
• Common Manageable Risks : Absence of (or incomplete) SOPs Absence of (or poor) clear, operational definitions of
indicators Manual aggregation of data (using calculators & MS Word) Poorly designed data collection tools No backup of electronic data Absence of clearly defined margins of error for data quality Failure to keep an error log Absence of (or inconsistent) data quality checks Absence of (or incomplete) documentation of the DMS Absence of (or incomplete/inconsistent) Version Control
Mechanisms Data Management Staff shortages or high staff turnover Lack of (or poor) training in Data Quality/management)
Accurately Measuring Progress
Common Risks to Data Quality: Acknowledged Risks
• Acknowledged risks are those risks that are beyond an Organisation’s control.
• Examples Data collection & reporting for indicators
outside an organisation’s normal operations. Delays in external data collection Lack of data quality control for external data High turnover of staff
Accurately Measuring Progress
Data Collection• Designing data collection tools as simply and clearly as possible, for example Using tick boxes to minimise the need to write
Colour coding forms Dates /Time Period for the data being recorded on the tool
Including version numbers and date of release on tools
Accurately Measuring Progress
Data Collection• Limiting data collection manipulation. • Including signoff fields (and dates) for all
handlers of the data collection tools (e.g. data collectors, verifiers/supervisors, and “receivers” of the tool)
• Operationalising indicator definitions to improve Validity of data collected
• Using the same data collection tools across all sites implementation. Where different tools are used e.g. at different
types of facilities - inter-tool reliability should be established.
Accurately Measuring Progress
Data Collation:• Developing simple, clear, easy-to-understand, and easy-to-use
collation tools include version number and dates of release for tools include quality-check sign-off fields, and fields for dates of collation Use electronic collation tools, e.g MS-Excel, MS-Access – to the maximum
extent possible -- Minimise manual collation processes• Documenting all collation processes and procedures, through SOPs,
including Write detailed instructions on how to complete the tool /collation process how to handle missing data timeframes required for collation.
• Train data handlers in basic computer literacy so they can comfortably use applications like Excel.
• Incorporate cross-checks (verifications) by a second individual of all collations
• Using the same collation processes over time.• Keeping records of all collation processes including any errors found. • Data to be securely stored either under lock and key or password
protected
Accurately Measuring Progress
Data Analysis: Good Practices
• Applying appropriate analysis methods, including correct arithmetic formulae.
• Securing expert advice on the most applicable analysis methodology to use,
• Using current data. • Disclosing all conditions and assumptions affecting
interpretations of data. • Training staff in analysis methodologies. • Recording dates when analyses are carried out • Second person cross checking all analysis before data are
used or submitted • Maintaining an audit trail of analysis procedures
conducted – including documentation on analysis procedures and outputs.
• Adequate storage of records - both soft and hard - of all analyses procedures and outputs.
Accurately Measuring Progress
Data Reporting: Good Practices • Checking all data before final reporting• Copying other people with an programme when submitting final
reports • Reporting /Feedback to all levels within a programme. • Developing and adhering to reporting formats.• Developing and adhering to strict schedules for completing and
sharing reports at each level of the DMS. Include time for informing sites/offices of incomplete /missing
data Sending regular reminders to sites/offices for timely submission
of data required for reports.• Keeping copies of all data used to prepare reports.• Recording dates when reports are submitted • Informing report recipients of data limitations (e.g. missing data,
weaknesses)• Reporting format aligned with type of data received
i.e. if qualitative data received then the report mostly qualitative.
Accurately Measuring Progress
Accurately Measuring Progress
Khulisa Management Services (Pty) Ltd
Khulisa Management ServicesPhone: +27 (0)11 447-6464Email: [email protected]: [email protected]