network propagation and air congestion policies

54
Network Propagation and Air Congestion Policies Tom Lam, Clemson University Christy Zhou, Clemson University Jan. 4, 2021, ASSA/TPUG Session Lam and Zhou Network Propagation 1 / 41

Upload: others

Post on 28-Nov-2021

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Network Propagation and Air Congestion Policies

Network Propagation andAir Congestion Policies

Tom Lam, Clemson UniversityChristy Zhou, Clemson University

Jan. 4, 2021, ASSA/TPUG Session

Lam and Zhou Network Propagation 1 / 41

Page 2: Network Propagation and Air Congestion Policies

Congestion in Air Transportation

Jets line up for takeo�, March 27, 2006 at O'Hare International

Airport in Chicago, Illinois (Photo by Tim Boyle/Getty Images)

I Congestion ine�ciency wastes time: Busy airports, such as JFK

and EWR tend to be congested 10 to 20 percent of the time. At

Newark, planes average taxi times of 52 minutes during

congested periods versus 14 minutes during less busy times.

Pushback times for planes can exacerbate the situation. - Forbes

Lam and Zhou Network Propagation 1 / 41

Page 3: Network Propagation and Air Congestion Policies

Air Travel Time Has Been Increasing10

011

012

013

014

015

0m

inut

es

2010 2011 2012 2013 2014 2015 2016 2017year

Elapsed Time + Delayed DepartureElapsed TimeAirborne Time

A. Delayed, Elapsed, and Airbourn Time

510

1520

25m

inut

es2010 2011 2012 2013 2014 2015 2016 2017

year

Taxi TimeDelayed DepartureDelayed Arrival Time

B. Delayed and Taxi Time

Sources: Chu and Zhou (2019) using DOT On-Time Performance

Lam and Zhou Network Propagation 2 / 41

Page 4: Network Propagation and Air Congestion Policies

Sources of Inefficiency in Air Travel Delays

Underprovision of modern infrastructure

I A public good problem

I Positive e�ect of increasing FAA expenditure (Morris and

Winton 2017), adopting the Next Generation Air Transportation

System (NextGen) (Chu and Zhou 2019)

Congestion externality (this paper)

I Airline companies do not have incentives to schedule their �ights

in a way that internalizes the cost of delays they impose on their

competitors, causing a congestion externality

I E.g., Delta does not need to internalize the cost of delay it

imposes on United

I This e�ect can further propagate via the air tra�c network

Lam and Zhou Network Propagation 3 / 41

Page 5: Network Propagation and Air Congestion Policies

Congestion also matters for competition

Usual forms of competition

I price

I non-price channels such as product di�erentiation along attribute

space to substitute consumer away from competing products

Congestion - another plausible non-price channel

I Could be unintended or strategically intended

I A�ects rival demand: Consider delay as a dimension in the

product space. Delay would make rival �ights would look bad on

websites such as �ightstats.com

I A�ects rival marginal cost: delay per se is costly

I Empirical evidence: Mayer and Sinai (2003 AER); Morris and

Winston 2007 AEJ Pol)

Lam and Zhou Network Propagation 4 / 41

Page 6: Network Propagation and Air Congestion Policies

Hub airlines bunch flights into peaks at hubHub vs. Airport Total Flights at DFW

Sources: Mayer and Sinai (2003, AER)

Lam and Zhou Network Propagation 5 / 41

Page 7: Network Propagation and Air Congestion Policies

Hub airlines bunch flights into peaks at hubDeparture Density for Hub and Non-hub Carrier at DFW

Sources: Mayer and Sinai (2003, AER)

Lam and Zhou Network Propagation 6 / 41

Page 8: Network Propagation and Air Congestion Policies

Mostly for departure flights...Departure Density for Hub and Non-hub Carrier at DFW

Sources: Mayer and Sinai (2003, AER)

Lam and Zhou Network Propagation 7 / 41

Page 9: Network Propagation and Air Congestion Policies

This paperGoal: To quantify the e�ect of congestion on air delay and air travel

time - the congestion e�ect

I important for designing air tra�c policies

I important for learning the proportion of congestion that are not

internalized and how they vary across airports

Challenge - the presence of network

I Congestion e�ects are heterogeneous

I E.g., the external cost �ight A imposed on other �ights might be

higher than �ight B

I The heterogeneity arises from (1) The characteristics of the

�ights a�ected (e.g. number of passengers, destination) (2) to

what extent this e�ect propagates in the network (e.g. peak

time, busy airport)

Lam and Zhou Network Propagation 8 / 41

Page 10: Network Propagation and Air Congestion Policies

Suggestive evidence of heterogeneous effects

Measure the importance of a �ight using the eigenvalue centrality

using �ights from March 31 to April 10, 2018

I For a departing �ight A (as a node), its centrality depends on

the other �ights that share the runway with �ight A and

proceeding operations of those �ights. We allow all proceeding

�ights to connect to this node if they share the airport within a

90 minute window.

Lam and Zhou Network Propagation 9 / 41

Page 11: Network Propagation and Air Congestion Policies

Approach

Step 1. Estimate the contemporaneous e�ect of having more �ights

on departure delay and taxi-out time - the direct congestion e�ect.

The identi�cation arises from:

I (1) Variation in number of unscheduled �ights

I (2) A selection-over-unobservable approach (SOO). Exhaustive

sets of �xed e�ects using the rich high-frequency air travel data:

OAG data from 2014 to 2017

Step 2. Quantify how the direct e�ect propagate via the network -

the indirect congestion e�ect

I Simulating a shock of �ight delay to the �ght network

The summation of them represent the overall marginal congestion

cost of an unexpected �ghts on delay and taxi-out time

Lam and Zhou Network Propagation 10 / 41

Page 12: Network Propagation and Air Congestion Policies

Approach

Step 1. Estimate the contemporaneous e�ect of having more �ights

on departure delay and taxi-out time - the direct congestion e�ect.

The identi�cation arises from:

I (1) Variation in number of unscheduled �ights

I (2) A selection-over-unobservable approach (SOO). Exhaustive

sets of �xed e�ects using the rich high-frequency air travel data:

OAG data from 2014 to 2017

Step 2. Quantify how the direct e�ect propagate via the network -

the indirect congestion e�ect

I Simulating a shock of �ight delay to the �ght network

The summation of them represent the overall marginal congestion

cost of an unexpected �ghts on delay and taxi-out time

Lam and Zhou Network Propagation 10 / 41

Page 13: Network Propagation and Air Congestion Policies

Our estimates would allow us toAssess how second-best congestion price can approximate the �rst

best

I Methods as in Sallee (2019)

I Current policies: airport-by-weight-speci�c landing fee at

speci�c airports

I Are centrality good enough to approximate the social cost?

I Other policy options regarding unscheduled �ights

Improve the delay multipliers in the FAA's Cost-Bene�t Guideline

I Current delay multipliers by FAA: Airport-speci�c parameters

for top 20 airports using simple counting

I Should construct from estimated marginal e�ect of congestion

I Should be more heterogeneous; should account for the network

e�ects

Lam and Zhou Network Propagation 11 / 41

Page 14: Network Propagation and Air Congestion Policies

Empirical Model

Step 1: Estimate the direct congestion e�ect

For a �ight i departing from the origin airport o in a given year y and

month m on a given outgate time t (down to a particular minute):

yiot = βnactot + φob + φym + εifot (1)

I yiot - departure delay (min), taxi-out (min)

I nactot - the number of unscheduled �ights taxiing out from the

airport o at the actual outgate time tI Data: all domestic and international �ights from and to top 40

US airports from 2014 to 2017 using OAG historical �ight

dataset

Lam and Zhou Network Propagation 12 / 41

Page 15: Network Propagation and Air Congestion Policies

Unscheduled flightsI Charter airlines and unscheduled airlines do not set months or years

ahead of the time as scheduled carriers

I Commercial charter �ights - can buy a seat or reserve for a groupI Private charter �ights - can rent a trip or own a private jetI Our data: 5.3% are unscheduled, within which 32.9% are private

Lam and Zhou Network Propagation 13 / 41

Page 16: Network Propagation and Air Congestion Policies

Summary Statistics: Main X Variable

Lam and Zhou Network Propagation 14 / 41

Page 17: Network Propagation and Air Congestion Policies

Summary Statistics: Other X Variables

Lam and Zhou Network Propagation 15 / 41

Page 18: Network Propagation and Air Congestion Policies

Empirical ModelStep 1: Estimate the direct congestion e�ect

For a �ight i departing from the origin airport o in a given year y and

month m on a given outgate time t (down to a particular minute):

yiot = βnactot + φob + φym + εifot (1)

I φob - the main airport-timeblock �xed e�ects:

origin airport o × time block b (which is year × quarter-in-a-year

× day-of-a-week × hour-of-a-day × 15-minute-slot-in-an-hour)

- e.g., compare

�ights at ATL Friday Nov. 01 12:00-12:15pm

�ights at ATL Friday Nov. 22 12:00-12:15pm

I φym - year by month �xed e�ects

Lam and Zhou Network Propagation 16 / 41

Page 19: Network Propagation and Air Congestion Policies

Identifying assumptions

No additional unobservables correlated with unexpected number of

unscheduled �ights and air tra�c delay

I Airports and terminal towers do not schedule a dynamic shift of

sta� members to respond to unexpected increases of operations

every day

I Peak time blocks (15-min) vary across days

I Airlines have to schedule their �ights to the computer reservation

system (CRS) every quarter ahead of time (Forbes 2008 IJIO)

Lam and Zhou Network Propagation 17 / 41

Page 20: Network Propagation and Air Congestion Policies

The Direct Congestion EffectDeparture Delay

Lam and Zhou Network Propagation 18 / 41

Page 21: Network Propagation and Air Congestion Policies

The Direct Congestion EffectTaxi-out

Lam and Zhou Network Propagation 19 / 41

Page 22: Network Propagation and Air Congestion Policies

RobustnessAlternative and Additional Fixed E�ects

I Add route-by-carrier �xed e�ects Go

I Add route-by-carrier-by-year-quarter �xed e�ects Go

I Add aircraft model (e.g., B777) �xed e�ects Go

I Use scheduled outgate time to de�ne �xed e�ects Go

More Conservative Clustered Standard Error

I Origin airport-by-quarter Go

I Origin airport-by-year Go

I Origin airport Go

To be added

I Add route-by-terminal �xed e�ects

I Control of number of runways

Lam and Zhou Network Propagation 20 / 41

Page 23: Network Propagation and Air Congestion Policies

Additional resultsAlternative Speci�cations: Include

I nact (unscheduled �ight at actual outgate time) and

nactreg (regular �ight at actual outgate time) Go

I nact (unscheduled �ight at actual outgate time) and

ninair (unscheduled �ight at actual in-air time) Go

Additional heterogeneous e�ects

I By domestic vs international �ight Go

I By domestic top 40 airports vs others Go

I By if the �ight is the �rst operation of an aircraft in a day Go

I By hub vs non-hub carrier at hub airports

To be added

I Consider capacity constraint: Allow for hockey-stick style of marginal

e�ect

Lam and Zhou Network Propagation 21 / 41

Page 24: Network Propagation and Air Congestion Policies

The indirect effectGoal: Simulate how delays and taxi-out time propagates in

subsequent minutes within the airport

Lam and Zhou Network Propagation 22 / 41

Page 25: Network Propagation and Air Congestion Policies

The indirect effectIn the future: Add how delays and taxi-out time propagate across

locations

Lam and Zhou Network Propagation 23 / 41

Page 26: Network Propagation and Air Congestion Policies

Simulate the indirect effect from τ to {τ + 1, τ + 2...}

Arrange all the scheduled �ights in an airport into 1-minute slots. Shock

the airport by adding extra �ights at a given timeslot τ by ∆�ights

I 1. At τ , set ∆�ights = 1I 2. Compute the expected outgate time and in-air time of this

additional ∆�ight using �xed e�ects and residuals

I 3. Use step 2 to compute in which time slots we will have ∆nact = 1I 4. For each a�ected time slots, simulate direct congestion e�ect of

�ights in those time slots. Using β̂ and ∆n to compute the new

outgate time and in-air time for these �ights

I 5. Re-calculate ∆n caused by step 4 all time slots

I 6. Repeat steps 4 � 5 until no additional �ight is a�ected

I 7. Sum up all repeated steps 4 � 6 (except for the �rst-round direct

e�ect) as the indirect e�ect

Lam and Zhou Network Propagation 24 / 41

Page 27: Network Propagation and Air Congestion Policies

Simulate the indirect effect from τ to {τ + 1, τ + 2...}

Arrange all the scheduled �ights in an airport into 1-minute slots. Shock

the airport by adding extra �ights at a given timeslot τ by ∆�ights

I 1. At τ , set ∆�ights = 1I 2. Compute the expected outgate time and in-air time of this

additional ∆�ight using �xed e�ects and residuals

I 3. Use step 2 to compute in which time slots we will have ∆nact = 1I 4. For each a�ected time slots, simulate direct congestion e�ect of

�ights in those time slots. Using β̂ and ∆n to compute the new

outgate time and in-air time for these �ights

I 5. Re-calculate ∆n caused by step 4 all time slots

I 6. Repeat steps 4 � 5 until no additional �ight is a�ected

I 7. Sum up all repeated steps 4 � 6 (except for the �rst-round direct

e�ect) as the indirect e�ect

Lam and Zhou Network Propagation 25 / 41

Page 28: Network Propagation and Air Congestion Policies

Simulate the indirect effect from τ to {τ + 1, τ + 2...}

Arrange all the scheduled �ights in an airport into 1-minute slots. Shock

the airport by adding extra �ights at a given timeslot τ by ∆�ights

I 1. At τ , set ∆�ights = 1I 2. Compute the expected outgate time and in-air time of this

additional ∆�ight using �xed e�ects and residuals

I 3. Use step 2 to compute in which time slots we will have ∆nact = 1I 4. For each a�ected time slots, simulate direct congestion e�ect of

�ights in those time slots. Using β̂ and ∆n to compute the new

outgate time and in-air time for these �ights

I 5. Re-calculate ∆n caused by step 4 all time slots

I 6. Repeat steps 4 � 5 until no additional �ight is a�ected

I 7. Sum up all repeated steps 4 � 6 (except for the �rst-round direct

e�ect) as the indirect e�ect

Lam and Zhou Network Propagation 26 / 41

Page 29: Network Propagation and Air Congestion Policies

Simulate the indirect effect from τ to {τ + 1, τ + 2...}

Arrange all the scheduled �ights in an airport into 1-minute slots. Shock

the airport by adding extra �ights at a given timeslot τ by ∆�ights

I 1. At τ , set ∆�ights = 1I 2. Compute the expected outgate time and in-air time of this

additional ∆�ight using �xed e�ects and residuals

I 3. Use step 2 to compute in which time slots we will have ∆nact = 1I 4. For each a�ected time slots, simulate direct congestion e�ect of

�ights in those time slots. Using β̂ and ∆n to compute the new

outgate time and in-air time for these �ights.

I 5. Re-calculate ∆n caused by step 4 all time slots

I 6. Repeat steps 4 � 5 until no additional �ight is a�ected

I 7. Sum up all repeated steps 4 � 6 (except for the �rst-round direct

e�ect) as the indirect e�ect

Lam and Zhou Network Propagation 27 / 41

Page 30: Network Propagation and Air Congestion Policies

Simulate the indirect effect from τ to {τ + 1, τ + 2...}

Arrange all the scheduled �ights in an airport into 1-minute slots. Shock

the airport by adding extra �ights at a given timeslot τ by ∆�ights

I 1. At τ , set ∆�ights = 1I 2. Compute the expected outgate time and in-air time of this

additional ∆�ight using �xed e�ects and residuals

I 3. Use step 2 to compute in which time slots we will have ∆nact = 1I 4. For each a�ected time slots, simulate direct congestion e�ect of

�ights in those time slots. Using β̂ and ∆n to compute the new

outgate time and in-air time for these �ights.

I 5. Re-calculate ∆n caused by step 4 all time slots

I 6. Repeat steps 4 � 5 until no additional �ight is a�ected

I 7. Sum up all repeated steps 4 � 6 (except for the �rst-round direct

e�ect) as the indirect e�ect

Lam and Zhou Network Propagation 28 / 41

Page 31: Network Propagation and Air Congestion Policies

Numbers of ∆flights to Shock

I We will set ∆�ights = 2, 4, or 10 to illustrate

I 2 ≈ the s.d. of nact

I 10 << the s.d. of nactreg

Lam and Zhou Network Propagation 29 / 41

Page 32: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsATL Saturday March 1, 2014

I Outcome: number of �ights a�ected (delay to the next minute,

or have to wait more on the runway)

I ∆�ights = 2

050

100

150

200

250

Num

ber o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 30 / 41

Page 33: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsATL Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 2

010

020

030

040

0M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 31 / 41

Page 34: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsATL Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

010

0020

0030

0040

0050

00M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 32 / 41

Page 35: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsATL Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 10

050

0010

000

1500

0M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 33 / 41

Page 36: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsATL Monday March 3, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

050

0010

000

1500

0M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 34 / 41

Page 37: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsORD Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

010

0020

0030

0040

00M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 35 / 41

Page 38: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsSEA Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

010

020

030

040

0M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 36 / 41

Page 39: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsAUS Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

020

4060

8010

0M

inut

es o

f flig

hts

affe

cted

(del

ay +

taxi

out)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 37 / 41

Page 40: Network Propagation and Air Congestion Policies

Direct and indirect congestion effectsMDW Saturday March 1, 2014

I Outcome: minutes of �ights a�ected (minutes of additional

departure delay or additional taxi-out time)

I ∆�ights = 4

050

100

150

Min

utes

of f

light

s af

fect

ed (d

elay

+ ta

xiou

t)

5pm 6am 7am 8am 9am 10am11am12pm 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm11pm12amOutgate time in a day

Direct + propagated effectDirect effect

Lam and Zhou Network Propagation 38 / 41

Page 41: Network Propagation and Air Congestion Policies

Summary + next (for simulation)

Summary:

I Sizable direct congestion e�ect

I Sizable and (very) heterogeneous propagated e�ect

Construct �ner network structure:

I Include propagation across airports, i.e., allow �ight A delayed

at its departure airport in t to delay at its destination airport

I Include propagation across operations for an aircraft, i.e., allow

operation n to a�ect its next operation n + 1I Include propagation across �ights sharing the same departure

gate (using the newly acquired data)

I Account for heteroskedasticity

Lam and Zhou Network Propagation 39 / 41

Page 42: Network Propagation and Air Congestion Policies

Next for policy implications

Next for policy implications

I Fraction of delay internalized vs externalized

I Second-best congestion prices

I Delay multipliers

Thank you!

Lam and Zhou Network Propagation 40 / 41

Page 43: Network Propagation and Air Congestion Policies

Alternative Specification I

Back

Lam and Zhou Network Propagation 1 / 12

Page 44: Network Propagation and Air Congestion Policies

Alternative Specification II

Back

Lam and Zhou Network Propagation 2 / 12

Page 45: Network Propagation and Air Congestion Policies

Additional FE I: Carrier-by-Route FE

Back

Lam and Zhou Network Propagation 3 / 12

Page 46: Network Propagation and Air Congestion Policies

Additional FE II:Carrier-by-Route-by-Year-Quarter FE

Back

Lam and Zhou Network Propagation 4 / 12

Page 47: Network Propagation and Air Congestion Policies

Additional FE III: Aircraft Model e.g.,Boeing 777

Back

Lam and Zhou Network Propagation 5 / 12

Page 48: Network Propagation and Air Congestion Policies

Alternative FE Timing Definition: Use theScheduled Outgate Time for the Main FEs

Back

Lam and Zhou Network Propagation 6 / 12

Page 49: Network Propagation and Air Congestion Policies

Alternative Cluster SE I: Airport-by-Quarter

Back

Lam and Zhou Network Propagation 7 / 12

Page 50: Network Propagation and Air Congestion Policies

Alternative Cluster SE II: Airport-by-Year

Back

Lam and Zhou Network Propagation 8 / 12

Page 51: Network Propagation and Air Congestion Policies

Alternative Cluster SE III: Airport

Back

Lam and Zhou Network Propagation 9 / 12

Page 52: Network Propagation and Air Congestion Policies

Heterogeneous Effect: Domestic vs Int’l

Back

Lam and Zhou Network Propagation 10 / 12

Page 53: Network Propagation and Air Congestion Policies

Heterogeneous Effect: US Top 40 vs Others

Back

Lam and Zhou Network Propagation 11 / 12

Page 54: Network Propagation and Air Congestion Policies

Heterogeneous Effect: First Flight in a Day

Back

Lam and Zhou Network Propagation 12 / 12