jaki mccarthy and denise abreu usda’s national agricultural statistics service
DESCRIPTION
Identifying Sources of Error: the 2007 Classification Error Survey for the US Census of Agriculture. Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service Presented at the International Total Survey Error Workshop Tallberg, Sweden June 2009. Target: Census of - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/1.jpg)
Identifying Sources of Error: the 2007 Classification Error Survey for the US Census of
Agriculture
Jaki McCarthy and Denise AbreuUSDA’s National Agricultural Statistics Service
Presented at the International Total Survey Error Workshop
Tallberg, Sweden
June 2009
![Page 2: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/2.jpg)
Errors in one survey can be measured with matching information from other sources Target:
Census of Agriculture
Alternate Source of Information:
June Agricultural Survey
![Page 3: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/3.jpg)
Error of Interest: Scoping Errors, i.e. Census Misclassification Census farms incorrectly classified
as non-farms
Census non-farms incorrectly classified as farms
![Page 4: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/4.jpg)
Errors in one survey can be measured with matching information from other sourcesCensus of Agriculture Census of Agriculture
conducted every 5 years
Count of all US Ag operations ($1000 or more in sales)
Primarily mail data collection
Data collected December - March
June Agricultural Survey (JAS)
Annual area frame based sample survey in June
JAS is primarily face to face interviews
Data collected in first 2 weeks of June
JAS has been used to measure undercoverage and misclassification on census
![Page 5: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/5.jpg)
JAS – Area Frame BasedJAS – Area Frame Based
![Page 6: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/6.jpg)
![Page 7: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/7.jpg)
7
![Page 8: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/8.jpg)
NASS Area Frame - SEGMENTNASS Area Frame - SEGMENT
Theoretically complete sampling frame
No overlap or gaps
Segments of land sampled
8
![Page 9: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/9.jpg)
NASS Area Frame – Segment NASS Area Frame – Segment EnumerationEnumeration
Sampled segments divided into tracts representing unique land operating arrangements
In-person interviewers screen for whether a tract is part of an agricultural operation and, if so, collect crop and livestock information
9
![Page 10: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/10.jpg)
10
Background:Previous Classification Error Studies
Measured census classification error – records incorrectly classified as farms or non-farms and duplication
Census records matched to JAS
JAS was assumed as truth; differences between the two sources were designated as census misclassification
Overall census misclassification error was estimated
![Page 11: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/11.jpg)
11
Background:Previous Classification Error Studies
Net classification error was small and was not used to adjust census numbers
For these reasons, shift in study’s primary objective
![Page 12: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/12.jpg)
12
Current Classification Error Survey
To identify REASONS for discrepancies between the JAS and the Census
Qualitative examination of why errors occur Classification errors Reporting errors also examined
To provide information to improve quality of the data, reduce analyst review and editing
![Page 13: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/13.jpg)
13
2007 CES Objective
Determine whether acreage/scoping differences are legitimate changes or errors
Determine why people report incorrectly
Determine if the forms were correctly processed
![Page 14: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/14.jpg)
14
Methods Census records matched to JAS records
Respondents records with scoping or acreage discrepancies were identified
Respondents re-interviewed and asked to resolve discrepancies
Census farm Census non-farm
JAS farm Match Misclassification - undercount
JAS non-farm Misclassification - overcount
Match
![Page 15: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/15.jpg)
Identifying Groups with Discrepancies
Group Description Action Total
MATCH:Classification in
agreement, acres comparable
Census farm/ JAS farm OR
Census non-farm / JAS Non-farm
No Action 1,629
MATCH:Classification in
agreement, acres not w/in 25%
Census farm / JAS estimated farm No Action; JAS Incorrect
240
Census farm / JAS farm Re-interview 1,122
Potential Scoping Errors:
Classification Conflict
Census non-farm by NASS / JAS farm FO Review Only 158
Potential Undercount:Census non-farm / JAS farm
Re-interview 185
Census non-farm / JAS estimated farm
No Action; JAS Incorrect
53
Potential Overcount:Census farm/JAS Non-farm
Re-interview 279
Total 3,666
![Page 16: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/16.jpg)
16
Discrepancies between Census and JAS
Scoping differences: 18.4% of matched records had discrepancies in classification (~3% net classification error)
Acreage differences: 37.2% of matched records had acreage differing by more than 25%
![Page 17: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/17.jpg)
17
Methods 67 respondents were re-interviewed by
enumerators in July 2008
Respondents reviewed questionnaires from both the 2007 Census and the 2007 JAS
Then asked to identify which was correct and why they were different
![Page 18: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/18.jpg)
Census 59.7%
Neither 11.9%
Both 13.4%
JAS 15.0%
Scoping Differences Which Source is
Correct?
![Page 19: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/19.jpg)
Scoping Differences Which Source is
Correct?TRUE Census
Misclassification
![Page 20: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/20.jpg)
20
Scoping Differences – Census is Correct
Number of Responses(n=39)
![Page 21: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/21.jpg)
21
Scoping Differences – JAS is Correct
Number of Responses(n=10)
![Page 22: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/22.jpg)
22
Scoping Differences – Both Sources Correct
Number of Responses
4
2
1
1
Different Operation
NASS O/S
Land Sold
Land Purchased
Re
aso
ns
for
Dis
crep
anc
ies
(n=9)
True Change – reported correctly
True Change – reported incorrectly
![Page 23: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/23.jpg)
23
Scoping Differences – Neither Source Correct
Number of Responses(n=9)
![Page 24: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/24.jpg)
24
Scoping Differences – Overall Summary by Category
Number of Responses
(n=67)
True Change - Incorrect
True Change – Correct
![Page 25: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/25.jpg)
25
Summary – Scoping Differences Very few of these cases are real changes between
JAS and the Census
Census was correct more often than June
Most discrepancies are actual errors June tracts screened out incorrectly Proxy respondents reporting incorrectly in JAS Specific types of land excluded (government program land,
woods, rented)
![Page 26: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/26.jpg)
26
Source Used to Report Acres
Source Used to Report Acres* Percent
(n=67)
No Records, I know my acreage 50.8%
Tax records 10.5%
FSA records 6.0%
Operation books 14.9%
Other records (ie., deed, GPS #s) 1.5%
* Multiple answers allowed
![Page 27: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/27.jpg)
27
What did we learn about Census misclassification? Classification error remains minimal
and is probably smaller than previous estimates
JAS cannot be used as “truth” Re-interview with resolution shows both
the Census and the JAS have errors JAS is not the GOLD STANDARD --
personal interviews not always best way to get accurate responses
Some errors due to respondents and won’t be eliminated
![Page 28: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/28.jpg)
28
Our external source had more errors than our target:
Recommendations to improve the JAS: Avoid proxy respondents in JAS Review of screening in JAS
Intensive re-screening of all non-ag tracts is in progress
Estimation of farms missing from JAS Capture/Re-capture
estimates in progress
![Page 29: Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service](https://reader030.vdocuments.us/reader030/viewer/2022032612/5681333e550346895d9a35af/html5/thumbnails/29.jpg)
To examine errors, you need a good measure