Download - ABS Statistical Databases
![Page 1: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/1.jpg)
ABS Statistical Databases
Session 6
Mark Viney
Australian Bureau of Statistics
6 June 2007
![Page 2: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/2.jpg)
INPUT THRUPUT OUTPUT
INPUT THRUPUT OUTPUT
INPUT THRUPUT OUTPUT
"Stove Pipe" approach
![Page 3: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/3.jpg)
INPUT
THRUPUT
Standardised
interface
Standardised
interface
INPUT
INPUT
OUTPUT
OUTPUT
OUTPUT
OUTPUT
"Clearing-House" Approach
Standardised
interface
OUTPUT
IDW ABSIW
![Page 4: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/4.jpg)
e-Census
![Page 5: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/5.jpg)
e-Census 2006
Conducted 2006 Population Census with the option of electronic submission of responsesƒ drop-off/ pick up
drop-off/mail back in 201110.2% of returns were electronic
ƒ no edits incorporated into electronic formƒ less visits to pick up paper formsƒ less paper forms
less scanning/repair
![Page 6: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/6.jpg)
ABS Secure Deposit Box
![Page 7: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/7.jpg)
Secure Deposit Box
An externally facing database to allow respondents to lodge their raw data electronicallyƒ Excel spreadsheet (essentailly replacing a paper form)
ƒ Administrative datasets
![Page 8: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/8.jpg)
ABS Statistical Databases
ABS Input Data Warehouse (ABS IDW)ABS Information Warehouse (ABSIW)
![Page 9: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/9.jpg)
ABS Input Data Warehouse(ABS IDW)
![Page 10: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/10.jpg)
Input Data Warehouse
Used as a repository for data as soon as it is entered into ABS computer systemsƒ Initially used for data received electronicallyƒ Now used to load (and process) survey data
![Page 11: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/11.jpg)
Input Data Warehouse
Structureƒ Star schema
1 fact table and several dimension tableseach data cell is stored as 1 row in the fact table
![Page 12: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/12.jpg)
Star Schema
![Page 13: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/13.jpg)
ABS Input Data Warehouse- What it allows us to do
Keep a historical record of what each cell was at every point in the processingƒ Reason for the changeƒ when it changedƒ who changed itƒ change in value
Ready access to both current and historical data
![Page 14: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/14.jpg)
ABS Input Data Warehouse- What it allows us to do
A data store for use with :-ƒ editingƒ imputationƒ winsorisationƒ estimation
Quick easy analysis and confrontation of data:-ƒ across time ƒ across dataitemsƒ across data sources
![Page 15: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/15.jpg)
ABS Input Data Warehouse- Flow of Information
I m p u ta tio nW in s o r iza tio nE d it in g
C o lle c t io n 1data
C o lle c t io n 3data
C o lle c t io n 2data
O utputP ro c e s s ing
![Page 16: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/16.jpg)
What we hope to achieve from IDW
Reduced costsImproved data qualityTools to assist with management of data providers
Better understanding of Editing processesƒ Significance Editing
One single source of microdataƒ for all statistical collections
Well managed and secure data storage
![Page 17: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/17.jpg)
ABS Information Warehouse (ABSIW)
![Page 18: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/18.jpg)
ABS Information Warehouse
Need to make both data and metadata:-ƒ Visibleƒ Relatableƒ Accessibleƒ Understandableƒ Reliableƒ Media Independent
![Page 19: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/19.jpg)
ABS Information Warehouse
Visibleƒ central known location
Relatableƒ across collections
Accessibleƒ tools to allow extraction and manipulation
![Page 20: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/20.jpg)
ABS Information Warehouse
Understandableƒ data fully described by metadata
Reliableƒ single sourceƒ high availability
Media Independentƒ single source for outputs
paper publicationselectronic releasesad - hoc requests
![Page 21: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/21.jpg)
ABS Information Warehouse
Define and manage metadataLoad lightly aggregated data Validate data as compliant with metadata Manipulate dataProduce statistical outputsMake data publicly available
![Page 22: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/22.jpg)
ABS Information Warehouse- Flow of information
Data from a collection
Load info on how to categorize data
Load info on what data items mean
Load info about collection
Load data to the
ABSDB Closed DB
Sign-off data to the
ABSDB Open DB
Disseminate output tables
Derive ad-hoc client
data requests
Disseminate time series
Processing System Information Warehouse PPW
![Page 23: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/23.jpg)
ABS Information Warehouse- Define and Manage Metadata
Interfaces to manage metadataƒ load, amend, validate, extractƒ dataitems,classifications, collections,datasets,publications
Application Program Interfaces (API) to link with other systems/programs ƒ increasingly using XML
![Page 24: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/24.jpg)
ABS Information Warehouse- Loading data
Load data from major sourcesƒ Input Data Warehouseƒ SASƒ FAMEƒ SuperCROSS
![Page 25: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/25.jpg)
ABS Information Warehouse- Generating New data Cubes
Passing data through one or more steps to derive a new tableƒ aggregationƒ drop dataitemsƒ calculate new items
![Page 26: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/26.jpg)
ABS Information Warehouse- Other Manipulations
Seasonal Adjustment ƒ SeasABS (X-11)
Chain Volume Measuresƒ FAME (timeseries)
SupertablesConfidentialisation
ƒ Disclosure Avoidance Analysis System
![Page 27: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/27.jpg)
ABS Information Warehouse- Data Delivery
Data combined with metadataOutput formats created tailored to specific useƒ spreadsheetsƒ timeseriesƒ supertablesƒ paper publicationsƒ electronic release
![Page 28: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/28.jpg)
ABS Information Warehouse- Public Release
Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time)ƒ This data is then available to ABS Statistical Consultants to satisfy customer requests
Feed data to websiteƒ www.abs.gov.au
![Page 29: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/29.jpg)
ABS Website
www.abs.gov.au
![Page 30: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/30.jpg)
![Page 31: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/31.jpg)
National Data Network (NDN)
www.nationaldatanetwork.orgwww.nationaldatanetwork.org
![Page 32: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/32.jpg)
We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service
Australian Bureau of Statistics
![Page 33: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/33.jpg)
National Data Network
Website that raises visibility of statistical dataƒ regardless of publishing agency
A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia
![Page 34: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/34.jpg)
National Data Network
One central websiteƒ descriptions of dataƒ quality statementƒ references to other data
Several websites (Nodes) owned and maintained by other agencies
www.nationaldatanetwork.org
![Page 35: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/35.jpg)
![Page 36: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/36.jpg)
National Data Network
![Page 37: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/37.jpg)
National Data Network
Current FocusCurrent Focusƒ Publish / Search / AcquirePublish / Search / Acquire
Future FocusFuture Focusƒ Design / Capture / ProcessDesign / Capture / Processƒ Analyse / ReportAnalyse / Report
![Page 38: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/38.jpg)
We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service
Australian Bureau of Statistics
![Page 39: ABS Statistical Databases](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681598e550346895dc6d819/html5/thumbnails/39.jpg)
Questions?