Comments on Analysis submitted by Chairman-CBSE

Debasis SenguptaIndian Statistical Institute

April 14, 2013

General remarks

• It is good to see some concrete analysis.• Summary of findings was missing.• I have summarized as I have understood.• I do not have access to the 516,416 sets of matched

data (about 48.6% of AIEEE 2012 candidates).• I have done additional analysis from what I could get

from the presentation.• I used same definitions and notations.• No contradiction with findings of Normalisation

Committee.

A major issue: How the two procedures alter current scenario

• The NIT’s are concerned with top 3% of rank list.• Chairman-CBSE has computed, for each state,

as a percentage.• ‘%age under procedure 1’ minus ‘%age under current

scenario’ is called P1 – P0.• P2 – P0 is similarly defined.• Comparative plot is reproduced here from page 6 of

circulated presentation.

statethefromappearingcandidatesno.Total

95statethefromcandidatesofNo. percentilethabove

Plot of P1–P0 and P2–P0 for 25 boards

AM AP BR CBSE CG CISC GO GT HP JH JK KK KL MG MN MP MR MZ NIOS NL PB RJ TN UK UP

-4%

-2%

0%

2%

4%

6%

8%P1-P0 P2-P0

• Main conclusion: Maximum deviation is only about 2%.

• No problem (on this aspect) with either procedure.

• Larger deviations for P1 – P0 is found in other percentile ranges, which do not matter to NIT’s.

• Board scores often have less than 0.5 correlation with AIEEE scores (see page 21 of report). A procedure giving 40% weight on board scores is expected to produce some changes.

• This fact was known to the normalisation committee from earlier analysis of over 2 lakh matched data (subset of present data), and included in the report (pages 22-23, F12-F21).

Another percentage

• Consider, for each state

expressed as a percentage.• This gives the composition of the top 5% students.• This is a matter or direct relevance to NIT’s.• P1 – P0 and P2 – P0 can be defined as before.• Comparative plot is shown next.

percentile95thabovecandidatesno.Total

percentile95thabovecandidatesofNo. tatefrom the s

Change in representation of boards in top 5%, for the two procedures

• Main conclusion: CBSE gains from Procedure 2; AP board and CBSE lose from Procedure 1.• Under Procedure 1,

– AP board share declines from 20.5% to 17.5%; CBSE share declines from 61.7% to 59.4%.– Both boards have above-par representation even after decline,

as 10.8% of candidates in matched data set are from AP board; 39.6% from CBSE.

• Under Procedure 2, rise of CBSE share is from 61.7% to 65.7%, which is further above-par.

AM AP BRCBSE CG

CISC GO GT HP JH JK KK KLMG MN MP MR MZ

NIOS NL PB RJ TN UK

UP

-4%-3%-2%-1%0%1%2%3%4%

P1-P0P2-P0

Tip of the iceberg

• This finding (increase in CBSE’s already large share among top 5%) is not new.

• Same phenomenon was observed from simulation study based on the 2 lakh matched cases available earlier.

• The present finding (though based on more data) is not necessarily the ultimate one, as the matched cases have different composition than the ‘set of all candidates’.

• In AIEEE 2012, 30% students were from CBSE; in matched part of data, 40% were from CBSE.

• Simulations indicate more dramatic increase (through Procedure 2) in CBSE’s share in top 1%, 0.1% etc.

Simulation conditions

• 10% Random samples chosen from AIEEE 2012 scores data.

• Presumed correlation between AIEEE and board scores:– Observed correlations for AS, CBSE, JK, MR, MZ,

UK (available till Feb’13)– 0.4 for other boards (observed value for most

boards).• 100 separate simulation runs.

Actual performance of CBSE students in AIEEE 2012

• 30.1% of all AIEEE 2012 candidates were CBSE students.

• However, this groups constitutes– 66 of the top 100 AIEEE rank holders,– 608 of the top 1,000 AIEEE rank holders,– 5,977 of the top 10,000 AIEEE rank holders,– 54,430 of the top 100,000 AIEEE rank holders.

• Slide 12 of Chairman-CBSE reports the corresponding values for the matched data set, which show similar trend. (Figure in that slide is correct, the numbers listed below are swapped.)

Share of boards in different percentile ranges of AIEEE 2012

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90 to 99

99 to 99.9

99.9 to 99.99

above 99.99

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CBSEMR AP BR RJUPAll other

Rest of the iceberg: How Procedure 1 or 2 would change the shares

• Procedure 1 reduces marginally the dominance of CBSE students in merit list ,

• Procedure 2 increases drastically that dominance.

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%AIEEE 2012 - Procedure 1

CBSEMR AP BR RJUPAll other

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%AIEEE 2012 - Procedure 2

UPCBSEMR AP BR RJAll other

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%

AIEEE 2012 – Current procedure

CBSEMR AP BR RJUPAll other

…and the numbers behind the charts

Procedure 1: Share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100CBSE 15% 18% 21% 24% 26% 29% 33% 38% 44% 53%MR 17% 16% 14% 13% 12% 11% 10% 8% 7% 5%AP 7% 8% 8% 8% 8% 8% 9% 9% 10% 13%BR 10% 9% 9% 8% 8% 7% 6% 6% 5% 3%All other 51% 49% 48% 47% 46% 44% 42% 39% 35% 26%

Procedure 2: Share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100CBSE 11% 14% 17% 21% 25% 29% 35% 42% 48% 59%MR 20% 18% 16% 14% 12% 11% 9% 7% 5% 4%AP 6% 6% 7% 8% 8% 9% 9% 10% 11% 13%BR 11% 10% 10% 9% 8% 7% 6% 5% 4% 2%All other 53% 51% 50% 49% 47% 44% 41% 36% 31% 22%

Current scenario: share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 OverallCBSE 14% 17% 20% 22% 26% 28% 33% 39% 46% 54% 30%MR 17% 17% 15% 14% 12% 11% 9% 8% 6% 5% 11%AP 7% 8% 8% 8% 8% 8% 9% 9% 10% 13% 9%BR 10% 9% 9% 8% 8% 7% 6% 5% 4% 3% 7%All other 51% 49% 48% 48% 46% 45% 42% 38% 33% 25% 43%

Findings from simulationPresent scenario Procedure 1 Procedure 2

Top 0.01% 65.1% 61.9% 82.7%

Top 0.1% 60.5% 60.4% 80.1%

Top 1% 60.0% 60.0% 69.6%

Top 10% 54.4% 53.4% 58.8%

• Procedure 1 produces a marginal reduction in CBSE’s share at the top; 2% reduction at the top 0.01%.

• Procedure 2 brings a great windfall to CBSE students.• CBSE students gain more and more from Procedure 2 as one goes up the merit list

– share at top 1% increases by 10% , – share at top 0.1% increases by 20%.

• Table given in Slide 16 of Chairman-CBSE is in line with this finding. (Slide 15 erroneously shows CBSE’s share in top 0.01% under Procedure 1 as 88.2%. This goes against common sense and also against the trend of the adjacent column.)

How does Procedure 2 dramatically increase CBSE’s share of top-rankers?

• It does so by practically turning the board score into another AIEEE score.

• The composite score for each board has same mean as AIEEE scores from that board, but has much smaller spread.

• The set of scores from each board get more concentrated around their respective means.

• It is as if every student gets a second chance to perform at the same mean level.

• As a result, top CBSE students gain conspicuously.

Why does Procedure 1 have only a mild effect on CBSE’s share of top-rankers?

• For CBSE, the students who do well in AIEEE are generally those who do well in board exam.– This may be guessed from the higher rank correlation (see page

21 of report) of CBSE and AIEEE scores.

– They mostly hold on to their position in merit list.

• For boards with less correlation with AIEEE, this is not the case.– Many AIEEE top-rankers from other boards move towards

middle in terms of composite score.

– If their board exams had been better aligned with AIEEE, they would have gained more from Procedure 1.

Will smaller boards gain unfairly from Procedure 1?

• Consider the percentage of candidates from a board making it to top 5%.

– Plot in Slide 6 and Table in Slide 8 of Chairman-CBSE shows that the maximum change effected by Procedure 1 is 2%.

– Table in slide 7 of Chairman-CBSE shows that these percentages are below 5% (par-score) for all small boards under Procedure 1.

(In fact, all boards except AP, CBSE, CISC and KK have below par representation.)

– If boards with less than 1000 matched samples are regarded as ‘small’ and the rest are ‘large’, the difference in mean changes in the two groups are statistically insignificant at the 5% level.

• Consider the percentage of the top 5% candidates coming from a board.

– Slide 5 of this presentation shows that the changes effected by Procedure 1 are well below 2% for all boards except AP and CBSE.

– ‘Small’ and ‘large’ boards have statistically insignificant difference of mean change.

• Answer to the question is NO.

Fate of students with common scores

• In 2012, five students (4 from CBSE, 1 from Maharashtra board) were tied at– AIEEE aggregate marks of 130 ,– board percentile of 93.1.

• According to Procedure 1, they shared rank 20,352.• According to Procedure 2

– CBSE students had shared rank 18,077,– Maharashtra board students had rank 34,175.

• The differential treatment is entirely due to relatively poor performance of other students of Maharashtra board in AIEEE.

• This goes against the principle of fairness.• Chairman-CBSE has tried to give a counter-example (Slide 11), involving 19

matched scores tied exactly as above. It is claimed that Procedure 2 would preserve the equivalence but Procedure 1 would not. This is impossible, as equal percentiles are treated equally by Procedure 1.

Assumptions• Chairman-CBSE mentions twice in his email message: Procedure 1 assumes

same merit distribution across boards, Procedure 2 does not.• This is a half truth. The other half is: Procedure 2 assumes that the disparity of

merit distributions across boards can be precisely measured (and hence adjusted for) by the AIEEE/JEE-Main score.

• In reality, AIEEE/JEE-Main performance disparity across boards can result from– Difference in merit/ability distributions,– Difference of board examination pattern with AIEEE (rank correlations indicate this),– Lack of instruction in English and Hindi (only available languages for AIEEE/JEE-Main),– Less access to coaching,– Load of an extra subject in board (for some boards).

• Adjusting only for one factor is impossible – in presence of confounding factors.• Disparity may worsen after adjustment.• AIEEE performance patterns are VERY disparate. If that is used to calibrate, 80%

of MH board would be deemed equivalent to 50% of CBSE. The implied assumption of LARGE merit disparity among large boards is risky.

How students with same AIEEE score move under two procedures

• Slide 10 of Chairman-CBSE shows the movements of ranks of candidates tied at AIEEE aggregate marks 130 (rank 27493).

• Claim : Procedure 1 shows greater movement of rank than Procedure 2.• Fact : Barring stray cases, the range of movements differ only minimally.• In Slide 10, a large movement of rank of a UP candidate through Procedure 1 has been

marked with an arrow. Greater movement can be seen in the case of some CBSE candidates for Procedure 2. The other marked case is not too far.

• The table given in Slide 13 gives an overall picture of rank movements. However, it is hard to draw conclusions from a large table. The normalization committee had looked at scatter plots of the ranks for a pictorial overview (Figure 3.5, page 23). The scatter for Procedure 2 was found to be marginally less dispersed.

• As explained in the report, this may be attributed to the board-specific distortions inflicted on board percentiles by Procedure 2, which bring them closer to AIEEE score.

• The low rank correlation between AIEEE percentiles (restricted to boards) and board percentiles, mentioned in page 21 of the report, imply that large movements in some cases are normal – if board percentiles are indeed given 40% importance.

In summary…

• Barring some errors (e.g., in slides 11, 12 and 15), the computations reported by Chairman-CBSE make sense.

• These computations corroborate the findings of the normalisation committee led by Prof. S.K. Joshi.