empirical project 3 measuring the effect of a sugar tax

CORE PROJECTSThis empirical project is related tomaterial in:• Unit 3 (https://tinyco.re/

4935560) of Economy, Society,and Public Policy

• Unit 22 (https://tinyco.re/4379576) of The Economy.

EMPIRICAL PROJECT 3

MEASURING THE EFFECT OFA SUGAR TAX

LEARNING OBJECTIVESIn this project you will:

• use the differences-in-differences method to measure the effects of apolicy or program, and explain how this method works

• use line and column charts to visualize and compare multiple variables• create summary tables to describe the data• interpret the p-value in the context of a policy or program evaluation.

Key concepts

• Concepts needed for this project: mean, frequency table, and p-value.• Concepts introduced in this project: natural experiment, differences-in-

differences, and conditional mean.

INTRODUCTIONIn Empirical Project 1, we mentioned that natural experiments can helpus determine whether one variable causes another variable. A usefulapplication of natural experiments is assessing the effects of a policy. To doso, we compare the outcomes of two groups, both before and after thepolicy took effect:

• The treatment group: those who were affected by the policy• The control group: those who were not affected by the policy.

51

https://tinyco.re/4935560




natural experiment An empiricalstudy exploiting naturally occurringstatistical controls in whichresearchers do not have the abilityto assign participants to treatmentand control groups, as is the case inconventional experiments. Instead,differences in law, policy, weather,or other events can offer theopportunity to analyse populationsas if they had been part of anexperiment. The validity of suchstudies depends on the premisethat the assignment of subjects tothe naturally occurring treatmentand control groups can beplausibly argued to be random.

differences-in-differences Amethod that applies anexperimental research design tooutcomes observed in a naturalexperiment. It involves comparingthe difference in the average out-comes of two groups, a treatmentand control group, both before andafter the treatment took place.

Specifically, we take the difference in outcomes of the treatment andcontrol group before the policy was implemented, and compare it with thedifference in outcomes after the policy was implemented. This method isknown in economics as differences-in-differences. We need to compareoutcomes before the policy has happened, because in a natural experimentwe cannot choose exactly who receives the treatment (whereas in the lab wecould randomly assign the treatment). Since the two groups are notrandomly chosen, we need to account for any pre-existing differencesbetween the two groups that could affect the outcomes, for example differ-ences in age (for people) or characteristics (for products). If these otherfactors remain constant over the period considered, then we can reasonablyconclude that any observed changes in the outcome differences between thegroups are due to the policy. Natural experiments therefore allow us tomake causal statements about policies and outcomes.

We will use the 2014 sugar tax in the US to learn how before-and-aftercomparisons are done in practice. Sugar-sweetened beverages (SSBs) areconsidered unhealthy because of their link to conditions such as diabetesand obesity. In November 2014, the city of Berkeley in California becamethe first US jurisdiction to implement a tax on SSB distributors, with theaim of discouraging SSB consumption. The tax of one cent per fluid ouncemeant that if retailers raised their prices to exactly counter the effects of thetax, a $1 can of soda (12 oz) would now cost $1.12. But did sellers actuallyrespond this way? And what effects did the tax have on shoppers’expenditure on sugary beverages?

A group of researchers did a differences-in-differences study of theeffects of this SSB tax, which you can read about in a Forbes article(https://tinyco.re/1312449). Figure 3.1 summarizes the timeline of the tax,and the data that they collected and published in a 2017 PLoS Medicinejournal paper (https://tinyco.re/8147535). We will make before-and-aftercomparisons using the data they collected, in order to learn about theeffects of the sugar tax.

Figure 3.1 Berkeley sugar-sweetened beverages tax implementation and evaluationtimeline.

Lynn D. Silver, Shu Wen Ng, SuzanneRyan-Ibarra, Lindsey Smith Taillie, MartaInduni, Donna R. Miles, Jennifer M. Poti,and Barry M. Popkin. 2017. Figure 1 in‘Changes in prices, sales, consumerspending, and beverage consumptionone year after a tax on sugar-sweetenedbeverages in Berkeley, California, US: Abefore-and-after study’(https://tinyco.re/8147535). PLoS Med 14(4): e1002283.

EMPIRICAL PROJECT 3 MEASURING THE EFFECT OF A SUGAR TAX

52











EMPIRICAL PROJECT 3

WORKING IN GOOGLESHEETS

GOOGLE SHEETS-SPECIFIC LEARNING OBJECTIVESIn addition to the learning objectives for this project, in this section you willlearn how to create summary tables using Google Sheets’ PivotTable option.

PART 3.1 BEFORE-AND-AFTER COMPARISONS OFRETAIL PRICESWe will first look at price data from the treatment group (stores inBerkeley) to see what happened to the price of sugary and non-sugarybeverages after the tax.

• Download the data from the Global Food Research Program’s website(https://tinyco.re/5269171), and select the ‘Berkeley Store Price Survey’Google Sheets dataset.

• The first tab of the Google Sheets file contains the data dictionary. Makesure you read the data description column carefully, and check that eachvariable is in the Data tab.

1 Read ‘S1 Text’ (https://tinyco.re/9522240), from the journal paper’ssupporting information, which explains how the Store Price Survey datawas collected.

(a) In your own words, explain how the product information wasrecorded, and the measures that researchers took to ensure that thedata was accurate and representative of the treatment group. Whatwere some of the data collection issues that they encountered?

(b) Instead of using the name of the store, each store was given a uniqueID number (recorded as store_id on the spreadsheet). Using GoogleSheets’ filter function, verify that the number of stores in the datasetis the same as that stated in the ‘S1 Text’ (26). Similarly, each productwas given a unique ID number (product_id). How many differentproducts are in the dataset?

53




Following the approach described in ‘S1 Text’, we will compare the variableprice per ounce in US$ cents (price_per_oz_c). We will look at whathappened to prices in the two treatment groups before the tax (time =DEC2014) and after the tax (time = JUN2015):

• treatment group one: large supermarkets (store_type = 1)• treatment group two: pharmacies (store_type = 3).

Before doing this analysis, we will use summary measures to see how manyobservations are in the treatment and control group, and how the twogroups differ across some variables of interest. For example, if there arevery few observations in a group, we might be concerned about theprecision of our estimates and will need to interpret our results in light ofthis fact.

Instead of calculating summary measures one by one (as we did inEmpirical Project 2), we will use Google Sheets’ PivotTable option to makefrequency tables containing the summary measures that we are interestedin. The tables should be in a different tab to the data (either all in the sametab, or in separate tabs).

2 Use Google Sheets’ PivotTable option to create the following tables:

(a) A frequency table showing the number (count) of store observations(store type) in December 2014 and June 2015, with ‘store type’ as therow variable and ‘time period’ as the column variable. For each storetype, is the number of observations similar in each time period?

(b) A frequency table showing the number of taxed and non-taxedbeverages in December 2014 and June 2015, with ‘store type’ as therow variable and ‘taxed’ as the column variable. (‘Taxed’ equals 1 ifthe sugar tax applied to that product, and 0 if the tax did not apply).For each store type, is the number of taxed and non-taxed beveragessimilar?

(c) A frequency table showing the number of each product type (type),with ‘product type’ as the row variable and ‘time period’ as thecolumn variables. Which product types have the highest number ofobservations and which have the lowest number of observations?Why might some products have more observations than others?

EMPIRICAL PROJECT 3 WORKING IN GOOGLE SHEETS

54

GOOGLE SHEETS WALK-THROUGH 3.1

Making a frequency table using the PivotTable option

Figure 3.2 How to make a frequency table using Google Sheets’ PivotTableoption.

1. The dataThe data will look like this. We will be making a pivot table using Column C(store type) and Column K (time). It will show how many observations of eachstore type there are in 2 time periods (Dec2014 and Jun2015).

2. Insert a blank pivot tableAfter step 2, a new sheet named ‘Pivot Table’ will appear.

PART 3.1 BEFORE-AND-AFTER COMPARISONS OF RETAIL PRICES

55

3. Insert a blank pivot tableA sidebar will appear in the newly created sheet. The pivot table is currentlyblank. In order to create the table, we will select the relevant row variable(s),column variable(s), and values in the sidebar.

4. Choose the variables to put in the pivot tableAfter step 5, your pivot table will look like the one above.

5. Filter the values of each variableThe table shown does not have any blank cells, but if there are any, then youcan remove them by filtering the data. You can also filter the data so that yourtable will show specific time periods only.


56

conditional mean An average of avariable, taken over a subgroup ofobservations that satisfy certainconditions, rather than allobservations.

Non-taxed Taxed

Store type Dec 2014 Jun 2015 Dec 2014 Jun 2015

1

3

Figure 3.3 The average price of taxed and non-taxed beverages, according to timeperiod and store type.

Besides counting the number of observations in a particular group, we canalso use the PivotTable option to calculate the mean by only usingobservations that satisfy certain conditions (known as the conditionalmean). In this case, we are interested in comparing the mean price of taxedand non-taxed beverages, before and after the tax.

3 Calculate and compare conditional means:

(a) Create a table similar to Figure 3.3, showing the average price perounce (in cents) for taxed and non-taxed beverages separately, with‘store type’ as the row variable, and ‘taxed’ and ‘time’ as the columnvariables. To follow the methodology used in the journal paper, makesure to only include products that are present in all time periods, andnon-supplementary products (supp = 0).

(b) Without doing any calculations, summarize any differences orgeneral patterns between December 2014 and June 2015 that youfind in the table.

(c) Would we be able to assess the effect of sugar taxes on product pricesby comparing the average price of non-taxed goods with that of taxedgoods in any given period? Why or why not?


57


Making a pivot table with more than two variables

Figure 3.4 How to make a pivot table with more than two variables.

1. The dataThe data will look like this. We will be making a pivot table using Column C(store type), Column K (time), and Column I (taxed). It will show the averageprice of taxed and non-taxed beverages in Dec2014 and Jun2015, according tostore type.

2. Count the number of times each product appears in the datasetWe only want to look at products that were present in all time periods, toensure we are comparing the same group of products over time. We will createa new variable (called ‘Number’, shown in Column M) that shows how manyperiods of data are available for each product. The COUNTIFS function willhelp us count the number of observations that satisfy certain conditions.


58

3. Insert a blank pivot tableWe will make and store the frequency table in a new tab in the spreadsheet.

4. Fill in the Pivot TableAfter step 6, your pivot table will look like the one above. By default, GoogleSheets uses all the available data.

5. Round cell values to two decimal placesTo make the table easier to read, we will round the cell values to two decimalplaces.


59

6. Filter the values inside the tableWe will filter the data according to the values of ‘store_type’, ‘time’, ‘Number’and ‘supp’.

7. Filter the values inside the tableWe will filter the data so that only the store types we want (1 and 3) are visible.

8. Filter the values of each variableWe will filter the data so that only the time periods we want (Dec2014 andJun2015) are visible.


60

9. Filter the values of each variableWe now filter the data so that only products that are available in all timeperiods (Number = 3) are visible.

10. Filter the values of each variableWe filter the data so that only non-supplementary products (supp = 0) arevisible.

11. Remove the grand total row and columnAfter step 15, your table should look like the one above.


61

In order to make a before-and-after comparison, we will make a chartsimilar to Figure 2 (https://tinyco.re/8127041) in the journal paper, toshow the change in prices for each store type.

4 Using your table from Question 3:

(a) Calculate the change in the mean price after the tax (price in June2015 minus price in December 2014) for taxed and non-taxedbeverages, by store type.

(b) Using the values you calculated in Question 4(a), plot a column chartto show this information (as done in Figure 2 of the journal paper)with store type on the horizontal axis and price change on the ver-tical axis. Label each axis and data series appropriately. You shouldget the same values as shown in Figure 2.


Making a column chart to compare two groups

Figure 3.5 How to make a column chart to compare two groups.

1. Create a table showing differences in meansWe will display the calculated differences in the table highlighted in blue. Thelabels on the rows are in the same order as in the pivot table, but the rows areflipped, since 0 corresponds to ‘Non-taxed’ and 1 to ‘Taxed’ in the pivot table.


62


2. Create a table showing differences in meansFill in the table by using cell formulas to calculate the differences required.After step 2, your table will look like the one shown above.

3. Draw a column chartWe will use the table we created to make a column chart.

4. Add chart and axis titles, and move the chart legendAfter step 7, your chart will look like the bottom chart of Figure 2 in the journalpaper.


63

statistically significant When arelationship between two or morevariables is unlikely to be due tochance, given the assumptionsmade about the variables (forexample, having the same mean).Statistical significance does not tellus whether there is a causal linkbetween the variables.

To assess whether the difference in mean prices before and after the taxcould have happened by chance due to the samples chosen (and there areno differences in the population means), we could calculate the p-value.(Here, ‘population means’ refer to the mean prices before/after the taxthat we would calculate if we had all prices for all stores in Berkeley.) Theauthors of the journal article calculate p-values, and use the idea ofstatistical significance to interpret them. Whenever they get a p-value ofless than 5%, they conclude that the assumption of no differences in thepopulation is unlikely to be true: they say that the price difference isstatistically significant. If they get a p-value higher than 5%, they saythat the difference is not statistically significant, meaning that they thinkit could be due to chance variation in prices.

Using a particular cutoff level for the p-value, and concluding that aresult is only statistically significant if the p-value is below the cutoff, iscommon in statistical studies, and 5% is often used as the cutoff level. Butthis approach has been criticized recently by statisticians and socialscientists. The main criticisms raised are that any cutoffs are arbitrary.Instead of using a cutoff, we prefer to calculate p-values and use them toassess the strength of the evidence against our assumption that there are nodifferences in the population means. Whether the statistical evidence isstrong enough for us to draw a conclusion about a policy, such as a sugartax, will always be a matter of judgement.

According to the journal paper, the p-value is 0.02 for largesupermarkets, and 0.99 for pharmacies.

5 Based on these p-values and your chart from Question 4, what can youconclude about the difference in means? (You may find the discussion inPart 2.3 (page 44) helpful.)

PART 3.2 BEFORE-AND-AFTER COMPARISONS WITHPRICES IN OTHER AREASWhen looking for any price patterns, it is possible that the observedchanges were not due to the tax, but instead were due to other events thathappened in Berkeley and in neighbouring areas. If prices changed in asimilar way in nearby areas, then what we observed in Berkeley may not bea result of the tax. To investigate whether this is the case, the researcherscollected price data from stores in the surrounding areas and comparedthem with prices in Berkeley.

Download the following files:

• The Excel file (https://tinyco.re/6625315) containing the price data theycollected, including information on the date (year and month), location(Berkeley or Non-Berkeley), beverage group (soda, fruit drinks, milksubstitutes, milk, and water), and the average price for that month.

• ‘S5 Table’ (https://tinyco.re/7724734) comparing the neighbourhoodcharacteristics of the Berkeley and non-Berkeley stores.

1 Based on ‘S5 Table’, do you think the researchers chose suitablecomparison stores? Why or why not?


64





We will now plot a line chart similar to Figure 3 (https://tinyco.re/6367345) in the journal paper, to compare prices of similar goods in dif-ferent locations and see how they have changed over time. To do this, wewill need to summarize the data using Google Sheets’ PivotTable option, sothat there is one value (the mean price) for each location and type of good ineach month.

2 Assess the effects of a tax on prices:

(a) Create a table similar to Figure 3.6 to show the average price in eachmonth for taxed and non-taxed beverages, according to location. Use‘year and month’ as the row variables, and ‘tax’ and ‘location’ as thecolumn variables. (You may find Google Sheets walk-through3.2 (page 58) helpful.)

(b) Plot the four columns of your table on the same line chart, withaverage price on the vertical axis and time (months) on the horizontalaxis. Describe any differences you see between the prices of non-taxed goods in Berkeley and those outside Berkeley, both before thetax ( January 2013 to December 2014) and after the tax (March 2015onwards). Do the same for prices of taxed goods.

(c) Based on your chart, is it reasonable to conclude that the sugar taxhad an effect on prices?

How strong is the evidence that the sugar tax affected prices? According tothe journal paper, when comparing the mean Berkeley and non-Berkeleyprice of sugary beverages after the tax, the p-value is smaller than 0.00001,and it is 0.63 for non-sugary beverages after the tax.

3 What do the p-values tell us about the difference in means and the effectof the sugar tax on the price of sugary beverages? (You may find thediscussion in Part 2.3 (page 44) helpful.)

Non-taxed Taxed

Year/Month Berkeley Non-Berkeley Berkeley Non-Berkeley

January 2013

February 2013

March 2013

…

December 2013

January 2014

…

February 2016

Figure 3.6 The average price of taxed and non-taxed beverages, according tolocation and month.

PART 3.2 BEFORE-AND-AFTER COMPARISONS WITH PRICES IN OTHER AREAS

65



The aim of the sugar tax was to decrease consumption of sugary beverages.Figure 3.7 shows the mean number of calories consumed and the meanvolume consumed before and after the tax. The researchers reported the p-values for the difference in means before and after the tax in the lastcolumn.

4 Based on Figure 3.7, what can you say about consumption behaviour inBerkeley after the tax? Suggest some explanations for the evidence.

5 Read the ‘Limitations’ in the ‘Discussions’ section (https://tinyco.re/6616217) of the paper and discuss the limitations of this study. Howcould future studies on the sugar tax in Berkeley address theseproblems? (Some issues you may want to discuss are: the number ofstores observed, number of people surveyed, and the reliability of theprice data collected.)

6 Suppose that you have the authority to conduct your own sugar taxnatural experiment in two neighbouring towns, Town A and Town B.Outline how you would conduct the experiment to ensure that anychanges in outcomes (prices, consumption of sugary beverages) are dueto the tax and not due to other factors. (Hint: think about what factorsyou need to hold constant.)

Pre-tax(Nov–Dec 2014),n = 623

Post-tax(Nov–Dec 2015),n = 613

Pre-tax–post-taxdifferenceUsual intake

Caloric intake (kilocalories/capita/day)

Taxedbeverages

45.1 38.7 −6.4, p = 0.56

Non-taxedbeverages

115.7 147.6 31.9, p = 0.04

Volume of intake (grams/capita/day)

Taxedbeverages

121.0 97.0 −24.0, p = 0.24

Non-taxedbeverages

1,839.4 1,896.5 57.1, p = 0.22

Models account for age, gender, race/ethnicity, income level, and educationalattainment. n is the sample size at each round of the survey after excludingparticipants with missing values on self-reported race/ethnicity, age, education,income, or monthly intake of sugar-sweetened beverages.

Figure 3.7 Changes in prices, sales, consumer spending, and beverage consumptionone year after a tax on sugar-sweetened beverages in Berkeley, California, US: Abefore-and-after study.

Lynn D. Silver, Shu Wen Ng, SuzanneRyan-Ibarra, Lindsey Smith Taillie, MartaInduni, Donna R. Miles, Jennifer M. Poti,and Barry M. Popkin. 2017. Table 1 in‘Changes in prices, sales, consumerspending, and beverage consumptionone year after a tax on sugar-sweetenedbeverages in Berkeley, California, US: Abefore-and-after study’(https://tinyco.re/8147535). PLoS Med 14(4): e1002283.


66









empirical project 3 measuring the effect of a sugar tax

Documents