foresight, insight. hindsight us statistical observations fritz scheuren norc university of chicago

Foresight, Insight. HindsightUS Statistical Observations

Fritz Scheuren

NORC University of Chicago

Reminder on Definitions

• Hindsight reflecting on the past –Personally/Collectively

• Insight, where it all comes together, like this Conference

• Foresight future seeing or shaping, also familiar but bears discussion

A Small Statistical Corner

• US Official Statistics• Censuses and Surveys• Administrative Records• Focus on lived experiences,

ala Deming

Times They are a Changing

• Relevance of our Discipline?

• Responsiveness of Statistics?

• Information Age?

• Misinformation Age?

• Service Partnerships?

Children’s Teaching Game?

• High?

• Low?

• You’re Too Slow!

• How Avoid Being Too Slow?

But Change is Accelerating!

• Is our Discipline Keeping Up?

• Certainly Computationally!

• Tool/Theory Building too!

• Practice Slower?

• Organizational Issues?

How To Keep Up/Catch Up?

• Google (of course)

• Metadata Revolution -- Still more Promise than Practice

• Meta-Analytic Reuse -- Still Often Too Hard

High-Clockspeed Trend(use of cell phones, portable devices)

Responsiveness to Trends

• How Well Do We Play?

• High?

• Low?

• You’re Too Slow!

Response Times to Trends(organizational clockspeed = rate at which an organization introduces new products, adopts

new production processes, or reorganizes itself; Sources: Charles H. Fine, 1998; David W. Rejeski, 2003)

Technological Mega-Trends

• Faster and Faster Computing(Slower for Official Statistics)

• Descriptive to Analytic• Randomization-based to Model-

based • Producer Dominated to Customer

Shared

Typical Grief Response to Change Still Often True

• Denial

• Anger

• Bargaining

• Depression

• Acceptance

Examples of Hindsight, Insight, Foresight

• Nonresponse Circa 1980

• US Census Taking Circa 1990

• Paradata Circa 2000

• Visualization Circa 2010

• Next Steps Together?

Nonresponse

Hindsight

Example

40 Year CPS Income Trend • Insignificant Missingness in 1962 • Now Nearly Half of Interviews

have Some Missingness• About a Third of the Amount is

Imputed• But Still using the Same Basic

“Hot Deck” Methods Today

Greater Bias Concerns

• Possibility of Greater Nonresponse Bias

• Potentially More Income Understatement

• Also Characteristics of Poor Blurred

Variance More Understated

• Growing Variance Not Directly Reduced

• Rubin’s Multiple Imputation Solution Still Not Used in CPS

• Remains Descriptive Rather than Analytic

Paradata Modeling

Insight

Example

Meta-Data Revolution

• Applying Computing to Documentation and Training

• Including Measurement Process or Paradata

• Achieving Full Systems Thinking

Unify Meta/Paradata

• Bringing All Survey Aspects together electronically

• Sharing with All Stakeholders

• Breaking Down Barriers between Departments

Unify Meta/Paradata

• Bringing All Survey Meta-Data together electronically

• Sharing with All Stakeholders

• Breaking Down All Barriers Between Departments

Manage System as a Whole

• Not Just Conformance to Requirements Quality

• But Total Fitness for Use Quality• From Sampling/Nonsampling to

Total Survey Inference• Record linkage Example

Total Systems Thinking

• Turning Sample “Models”

• Into Full Survey “Models”

• Using Paradata and Experience

• Politz-Simmons Example

Visualization

Foresight

Example

Complex Survey Graphics

• Clustering and Weighting Distort

• Analytically these can be “solved” Approximately

• Design Effect Example

Restoring Visual Metaphor

• Inverse Sampling Algorithm

• Works for Many Designs

• Satisfactory Analytically

• Works Graphically too but not yet Always

Simulation Alternatives

• Capture Essential (Sufficient?) Conditions

• Simulate Graphics Analytically

• Retain Real Sample?• Apply Empirical Residuals?

Regression Diagnostics

• Design-Based and Analytic Alternatives Being Examined Now

• Both Have Merits• May Imbed both in an Open-

Source, like R

Next Steps?

Further Considerations and Examples

Expected White Swan?

Unexpected Black Swan

Addressing Evolving Trends