Sampling, Validity and the Longitudinal Archive

Mike Savage

University of Manchester

Imagining Modern England: popular identities and the post-war social sciences

How do we use different – even incommensurate - qualitative sources to construct accounts of social change?

• My forthcoming book presents an analysis of social change in England using a range of archived qualitative social science data.– Mass-Observation Directives, 1939-1950– Material from Geoffrey Gorer archives– Elizabeth Bott, Family and Social Networks, 1950-54– John Goldthorpe and David Lockwood, Affluent Worker Studies, 1961-

1964– Brian Jackson, Working Class Community, 1960-1965– Richard Brown, Tyneside Shipbuilding Workers, 1969– Ray Pahl studies on Hertfordshire suburbanites, ‘managers and their

wives’ 1961-1969.• The use of these diverse studies, even aided by dedicated research

time, poses the challenge of doing justice to the wealth of archived material whilst presenting a coherent and rigorous account.

Sampling archived qualitative social science data

longitudinally: four issues

• Practical: how do we deal with extensive, often un-hierarchical, data repositories?

• Methodological: how do we systematically compare archived data from separate, autonomous archives, in different historical periods

• Historical: How do we see the deployment of research methods themselves as agents of social change?

• Political: How do we challenge positivist conceptions of sampling yet in a way which avoids justifying slap-dash research?

1: Political Issues

Challenging orthodox conceptions of ‘trend analysis’

Conventional survey based longitudinal analysis dominates social science, but strips out contextual detail to allow strict formal comparison between aggregate entities (nations, social groups, etc) over discrete time periods– Economic indicators (inflation, wealth, unemployment, occupational

structure, etc)– Psephological studies of voting using British General Election Surveys– Studies of changing attitudes (e.g. Inglehart, Post-materialism)

• Archived qualitative material does not allow this abstraction from context and hence cannot deploy ‘representative’ or ‘aggregate’ findings.

• It is however possible to use the messiness of qualitative inquiries as themselves evidence for hidden processes.

• This offers an historical riposte to Mark Abrams who in 1948 attacked M-O for its unrepresentative samples. 60 years later, we learn little from opinion poll or survey findings about contextual historical issues – but the sprawling material in the M-O archive offers rich pickings indeed.

Example 1: Bott’s Family and Social Network sample

• Bott and her colleagues intensively interviewed 20 husbands and wives about their marital roles and relationships.

• Bott emphasises that it is not possible to generalise from these small numbers, yet develops an influential analytical distinction between working class families with role segregation, and middle class families with more fluid roles. These differences are then explained by contrasting the social networks.

• The fieldnotes indicate a further revealing feature of the sample: that nearly all the households were pre-disposed towards research, and especially the social sciences, and hence that the project itself reveals the formation of a new kind of social group committed to technical and scientific values .

Example 2: Mass-Observation on social class identities

• Various directives (1939, 1948, 1949, 1990) ask about class identities, though in different ways.

• It is not possible to use the responses to give an account of representative trends in class awareness, given the skewed social composition of M-O

• However, it is possible to explore what they say about literate middle class awareness– Leftist and critical perspectives 1939– Reaction against post-war Labour government 1948– Emergence of a managerial identity, 1949– Individualised and reflexive class identities 1990

• This exercise involves focusing on form as well as content, and reading the material as accounts of specific, located, social groups, not ‘the population as a whole’.

2: Practical issues

The problem of sampling• My project raises the issue of how to deal with such a

wealth of archive material. The modern archive proliferates data, so preventing comprehensive knowledge: how does one proceed?

• Much historical archival works takes a hierarchical form of focusing on key sources, e.g. of powerful organisations, individuals, decisions, unusual case studies, etc. The operation involves working with and through the power biases of the sources

• By contrast, archives such as Mass-Observation and those with qualitative data on ‘popular’ groups lack an obvious hierarchy. There are often extensive batches of observations, interviews, directives, etc. It is not clear that any particular record is more worthy of examination than anything else, and one cannot rely on working with the biases of the archive.

• Should one look at the ‘whole population’ (i.e. all the sources) or a sample. And if the latter, how is the sample to be drawn?

Whole populations?• This initially sounds the most convincing route. It can be practical in

some cases, e.g. Elizabeth Bott, Family and Social Network, where details of 20 household case studies are kept in one box.

• As soon as the scale of the work increases, this strategy is problematic. One becomes faced with the need to abstract: e.g. John Goldthorpe and David Lockwood’s Afffluent Worker project, with 229 interviews, at home and at work. One strategy here is to just focus on a particular part of the research. Even so, how does one take notes?

• Mass-Observation poses another order of magnitude. Its directive have several hundred responses, often lengthy. It is not possible to read or analyse them all. Does one reduce the narrative accounts to simple counts or frequencies? Does one focus on a few ‘exemplary’ cases (in which case, what is the reason for treating them as exemplary?)

• We are faced with a similar issue to that which confronted the early social sciences themselves….. As soon as one studies ‘popular’ groups, who exactly does one talk to?

The politics of sampling• Consider the following strategic uses of Mass-

Observation material in recent historical work with different sampling and analytical strategies– David Kynaston in Austerity Britain– Tony Kushner in We Europeans– Simon Garfield, Diaries– Nick Hubble, Mass-Observation and Everyday Life

• All indicate the problems of sampling such a vast source. How does one deal with issues of coding and extracting relevant material? The danger of the ‘juicy quote’ syndrome.

• I will return to this issue but I want to consider now how this debate is not just one for us, but resonates with historical and political concerns. The formation of sampling methodologies are themselves a key part of socio-cultural change in the post-war period

3: Historical Issues

Methods and Socio-Cultural Change

• The rise of the post-war social sciences is one of the most important, yet un-researched, aspect of post-war change. – Social scientists are 3% of UK academics 1948 > c. 45% 2001– Social scientists play pioneering role in elaborating new

‘technologies of the social’ in post-war years• National sample survey (from 1930s)• The ‘user questionnaire’ (from 1960s)• The qualitative interview (from 1950s)

– Social scientists generate ‘epoch descriptions’ which come to embed change into the social itself

• ‘Affluence’ (1960s)• Post-industrialism (1970s)• Globalisation (1990s)

• The social sciences are key agents in the emergence of ‘knowing capitalism’ (Thrift).

Sampling in historical perspective Let us not consider the history of qualitative research from the high ground of the national sample survey, but instead consider the history of the national sample survey from the standpoint of the situated, qualitative researcher. In these terms, what does the sample survey achieve?

1. It constructs ‘public opinion’ from the 1930s (Osborne and Rose), though there is considerable early resistance and scepticism.

2. It becomes a key government technology from Second World War (1942)

3. It provides an inscription device for ‘individual’ development (Cohort studies).

4. It allows the isolation of ‘social groups’ as definite bounded entities (notably in 1960s ‘white heat’).‘

5. The survey co-produces the social scientific habitus itself (through enlisting ‘educated’ research subjects and agents),

1: Surveys and public opinion

• Rose and Osborne (2000) argue that polling creates a new conception of ‘public opinion’

• This jostles with, and comes to displace, notions of ‘national character’ (c.f. Mandler) which had predominated beforehand.

• However, until the 1960s, there was considerable suspicion of, and resistance towards polls (e.g. no one believed their prediction that Labour would win the 1945 election).

• (Partial) respectability only comes during the 1950s. As late as 1964 it was noted that educational research was – ‘something of a bandwagon (to which) individuals and bodies with

scanty experience and minimal competence are now turning their attention to… some of these are commercial interests who skilfully conceal their origins or convince a reputable educational organisation that they will finance research without strings. Others are self appointed pressure groups with innocent and high sounding titles… questionnaires are a favourite instrument of such groups, since they appear simple to construct and interesting to fill’

• Consider also Goldthorpe and Lockwood’s critique of ‘tick-box’ research which initiated the ‘Affluent Worker’ project, as well as Bourdieu’s concerns with survey research.

2. As governmental technology • The first national survey, under government aegis, in 1942. The

OPCS becomes the main body to conduct national surveys till the 1960s.

• Key devices of post-war government begin to deploy survey measures, for example the ‘retail price index’, linked to the Family Expenditure Survey (1957>). Surveys construct notions of the nation as ‘modern imagined community’.

• The survey becomes the key device for establishing ‘government departmental’ expertise in the ‘white heat’ of the 1960s Labour government, defining the client groups of departments (e.g. the ‘poor’, the ‘ill’)

• supplementing the ‘gentlemanly’ Royal Commissions. – Fulton Commission (Civil Service reform)– Plowden Report (educational reform) – Radcliffe-Maud Report (local government reform)

• Surveys were not seen as ‘re-usable’, but as ‘one-off’ inquiries • Academic social scientists still work at some remove from surveys

Up until the 1950s many teachers, most educationalists, and nearly all politicians envisaged educational research as a mildly interesting and marginal activity… the periodic reports of the Central Advisory Council… formulated their recommendations by the time honoured means of canvassing opinion and seeking a consensus. Recent reports, however have illustrated a revolutionary change. Crowther, Robins and Plowden were not content with mainly canvassing opinion… they proceeded to seek out facts.

(National Foundation of Economic Research, 1967-68)

3. As device for eliciting the ‘individual’• Until 1970s public surveys rarely focus on national

random samples, but focus on specific ‘problem’ groups, notably children.

• Cohort Study (1946) and National Child Development Study (1958) pioneer studies of the ‘developmental individual’ using innovative panel study design

• During the 1960s, extensive surveys of children and young people were common.– Youth Survey 1961, 1962, 1963– Child Chest Survey, 1966– Buckinghamshire child survey, 1961– Politics and the English child, 1969

• Educational reform and ‘comprehensivisation’ depends on survey research (e.g. consider the National Foundation for Educational Research)

4: As elaborating on ‘social groups’

• In contrast to field analysis, surveys permit the sampling of pre-specified social constituencies, often remarkably fine-grained, e.g.– Interviews with the ‘poor’ (numerous)– interviews with ‘handicapped’ (1968) (N = 12738)– Attitudes on International Affairs among African Students in

Britain (1963) (N= 291), – Survey of aircraft noise near Heathrow (1961, 1969, N = 4699)– Surveys of phone users.

• The national random sample survey only has marginal position and the impetus comes from political science (British Election Survey 1964>; Butler and Stokes; Runciman; Nuffield Mobility Study)

5. As eliciting the social scientific habitus itself

• Numerous surveys are about students themselves, including surveys at Cambridge, Essex, Edinburgh, Manchester, Nottingham, UCL, Exeter

• National Survey of 1960 University graduates, surveys of trainees, of the ‘impact of schemes’

• Remarkable enthusiasm for surveys of University teachers (1964 Halsey; 1969, Ministry of Labour)

• MENSA plays a role in pioneering user surveys (e.g. 1969 survey to see if MENSA members are ‘upwardly mobile’)

• New Society pioneers the ‘user questionnaire’.

The social scientific habitus is elicited around concerns with change, development and technological modernity.

Lessons for the history of the sample survey

• We should not take the random, representative sample as the scientific model from which the problems of the qualitative archive deviate.

• We should instead see the sample survey as having their own kinds of agency which it has the capacity to conceal.

• We should be more confident in developing strategies for sampling which are appropriate for qualitative data

4: Methodological issues

Systematic analysis?

• We can be critical of positivist accounts, but still need to reflect on how best to use the ‘messiness’ of archived data rather than a simple appeal to genealogical or deconstructive methods. – Sampling based on ‘theoretical saturation’– Sampling based on ‘adequate numbers’– Analysis based on the ‘telling exception’ – Analysis based around the formation of coherent

typologies– Use of content analysis, frame analysis, and narrative

analysis, as a means of emphasising position-taking.

Methodological resources• Feminist and qualitative arguments about

research methods

• Andrew Abbott’s attention to ‘battles over jurisdiction’

• Walter Benjamin’s critique of historicism

• Franco Moretti’s mapping of narrative and plot.

• Pierre Bourdieu’s use of field analysis as a means of exploring ‘stakes and positions’

Conclusions

• There is huge potential in the use of qualitative archived data

• The fact that this data cannot be analysed in similar ways to archived quantitative data is not an inherent problem so long as researchers build the biases of the inquiry into the analysis

• We need to develop clear analytical strategies which allow the rigorous – yet partial - examination of such data to avoid the problems of (a) juicy quote syndrome and (b) false completeness.