nordisk statistikermøde i københavn 11.-14. august 2010 the archive statistical method 1960-2010...

Nordisk Statistikermøde i København 11.-14. august 2010

The archive statistical method 1960-201050 years - A Summary

bySvein Nordbotten

8/11/2010 1Svein Nordbotten

8/11/2010 Svein Nordbotten 2

The situation 50 years ago

• Separate statistical organizational units for limited subject matter domains• Few possibilities to link data within and between established domains• Reuse of micro data expensive and time consuming• Creating longitudinal micro data sets rarely possible

• Published statistics followed a narrow and restricted tabulation plan• Analysis limited to aggregated macro data

• Statisticians were experts in their subject matter domains• Little exchange of experience and information among different groups


Two important events in the 1950s

Electronic, programmed computers made processing of large data sets more effective for administrative and statistical purposes.

Electronic recording made establishing, maintaining and sharing comprehensive population/object registers possible.


The information technology

The UNIVAC 1 computer entered the official statistical scene in the US Census Bureau in the early 1950’s, and when the processing of the 1960 censuses were due, a number of NSIs were prepared with their own computers.

When the information technology later merged with communication technology, it changed the time and space dimensions for statistical processing.

The new technical potensial inspired to think about all activities of an NSI asinterrelated parts of a production system in contrast to the traditional separate statistical processes, and the term statistical production became common.


Administrative registers

Local object identifiers had been used for a long time in many admistrative processes. These specialized systems were expensive to maintain , represented a vast duplication of efforts and were of limited usability as statistical data sources.

The efficiency of shared, public object identifiers were obvious. About 1950 the first steps to establish general, permanent, unique object identifier registers for the economic units as well as for the population were taken in the Nordic countries with active participation by statisticians.


Statistical ideas in the 1950s

Unique and permanent identifiers permitted compilation of the ’life lines’ of statistical objects including interobject events by data linkage.

Models for statistical micro systems named statistical archive systems, statistical file systems, register-based statistical systems, etc. were discussed in international fora from about 1960.

The next slides illustrate the popular version of ideas discussed. Ten years later the generic version became known as data base managementsystems.

8/11/2010 Svein Nordbotten 704/18/23 Svein Nordbotten 7

Objects

Time Location

The 3 basic concepts


Objects

Attributes

Time

Censusattributes

Censusday

Humanpopulation

Archive data container

8/11/2010 Svein Nordbotten 9Time

Objects

Timeplane

Object plane

Attribute plane

Attributes

The 3-dimensional space model

Object:BTime: t-1

Object type Atype ATime tTime tObject type ATime t

Object: CTime: t-1

Object: CTime: tOOOOO

Object A:Time: t-2

Object: CTime: t-2

Object : ATime: t

Object: BTime: t

Object: BTime: t-2

Object: ATime: t-1

Link H

Link HLink H

Link H

Link H

Link H

Link V

Link V

Link V

Link V

Link V

Link V


Latitudinal and longitudinal integration


An example


Summary of the 50 year old visions

Comprehensive official register systemsGeneral utilization of systems

Fast and high capacity ITExtensive and userfriendly communicationSoftware for data processing, saving, retrieval and communication

Data collection independent of specific usageUse of communication network for collectionFunctional organization

Wider statistical coverage by more intensive reuse of dataStatistics on-demandAnalytical interest in and tools for exploiting the new possibilities

Legislation regulating the use of micro data


Utilization of public identification systems

• Population register systems in general use• Economic unit register systems in general use by administrative organizations• Register systems for geographical location of properties established and in use


Sample of present applications

Censuses (population, location registers)

Demographics (demographic registration)

Foreign trade (custom data)

Income (taxation data)

Social statistics (registration of services)

Employment (unemployment registration)

Education (registration of students)

Health (medical registrations)

Crime ( Police, court, prison registrations)

Business (Business registers)


Methodological challenges

The present intensive utilization of administrative data in officialstatistics has also required work on methods in several fields:

1.Electronic processes for collecting administrative data2.Quality control of data received from external administrative sources3.Integrating data from different statistical and administrative sources4.Evaluating quality of integrated data sets5.Editing data using associated administrative background data6.Estimating population parameters using administrative background data7.Confidentiality and security in systems based on administrative data8.Statistics on demand from systems based on administrative data9.Evaluation of costs and benefits from using administrative data


Privacy issues

•Statistical legislation•Privacy legislation•Misuse of information

•Value of knowledge•Regulatory use and research

•Solutions? A political question

AD1 ADM 1

AD2 ADM 2

SD SPNSI

Information privacy law

Statistical law

Dat

a co

llecti

onD

ata products



Anticipations/realizations

Comprehensive official register systems……………………………………………... . YesGeneral utilization of systems………………………………………………………………. Yes

Fast and high capacity IT………………………………………………………………………. YesExtensive and userfriendly communication…………………………………………… YesSoftware for data processing, saving, retrieval and communication……… Yes

Data collection independent of specific usage…………………………………….. ? Use of communication network for collection......................................... YesFunctional organization……………………………………………………………………….. Yes?

Wider statistical coverage by more intensive reuse of data…………………. YesStatistics on-demand…………………………………………………………………………… ?Analytical interest in and tools for exploiting the new possibilities……… Yes?

Legislation regulating the use of micro data………………………………………… Yes?


What next?

1. Electronically observed data2. Mobile use of statistics

Require legislative, technical and methodoligical preparations.


Thank you all for your attention!

I look forward to seeing you all

in another 50 years.

nordisk statistikermøde i københavn 11.-14. august 2010 the archive statistical method 1960-2010...

Documents

statistical systems

statistical processing

statistical data sources

statistical micro systems

statistical archive

svein nordbotten slide

statistical purposes

statistical file systems