The ability of Cambridge applicants to predict their interview scoresTheloniouss
[/field]Post-interview stress is common among Cambridge (and Oxf*ord) applicants, in significant part because many applicants leave their interview with a very poor idea of how well they have performed. In 2016, user @Doonesbury conducted some research on this and found that the interview score predictions of unsuccessful and successful applicants are not meaningfully different, see Figure 1
. It is my aim, with this research, to conduct the same test on a larger sample size. I also aim to account for both predicted score and perceived difficulty in order to determine whether this might serve as a more effective predictor of success.
I hypothesise that, in accordance with @Doonesbury’s results, there will be no significant difference between the predicted scores of successful and unsuccessful applicants. I expect that there will also be no difference between the groups in terms of the perceived difficulty of their interviews.
[field]Results and Analysis
[/field]1. Results at a glance
The initial survey received 154 responses, of which 115 could be paired with a response to the second survey. 16 of the 154 responses contained errors making it impossible to contact the initial respondent and a further 23 responses could not be paired with a response to the second survey. The responses transcribed from Doones’ graph amounted to 63 complete responses. The below tables (Figure 2
contains key summary statistics from the main groups of interest, and the accompanying graph (Figure 3) shows the distributions of predicted scores and reported difficulties, divided into successful and unsuccessful applicants.
Figure 3: Bean plots of interview score and difficulty, allowing for easy comparison of the distributions.2. Data clean-up
The data above then requires some further consideration before we can analyse it. The first question is how to treat the non-respondent and error categories. Both categories appear to differ from the successful and unsuccessful categories in potentially significant ways, such as in the mean score of non-respondents and the median difficulty for errors.
I would tend to ignore the errors, as I expect these would not be more likely to receive or not receive offers (though it could be argued that the inability to type your own email address suggests you are not Cambridge material). The non-respondents, however, are probably more likely to have not received offers, as applicants who were deselected are likely to be less willing to respond. I have discounted them anyway because it would be difficult to account for them.
Next we should consider whether to include Doones’ data or not. While its inclusion would increase our sample size, without knowing how the data was collected it’s probably not sensible to include it, as different methods of data collection might result in different styles of result. For example, Doones’ data uses a different scale to mine, which has resulted in fractional score predictions. As a result of this, and because there is no way to account for perceived difficulty, I will not be using Doones’ data in this analysis.3. Data analysis
In order to analyse this data, I have used logistic regression. This accounts for the binary nature of the response variable and means certain assumptions typical of regression analysis can be ignored, like normality and homoscedasticity (equal variance). The assumptions, and extent to which each is met, of logistic regression are below:
1. Binary response variable – the response variable is whether or not an offer is received
2. Independent observations – almost certainly met, I can’t see how offer-holders could have influenced one another’s responses
3. No correlation between explanatory variables – This assumption is slightly violated, as the two explanatory variables show weak correlation, as determined by Kendall’s tau correlation test, which was used because it accounts for tied ranks and doesn’t require normality (tau=-0.18, p=0.017). The correlation, however, is low and so logistic regression should still be valid.
4. Large sample size – The sample size here is 115-138, which should be large enough for logistic regression.3.1. Fitting the model
In fitting the model, I have initially assumed an interaction between interview difficulty and score – it’s likely that these will influence each other so it makes sense to include an interaction. Since the AIC for this model is lower than the AIC for the model which excludes it (153.7 with the interaction, 154.3 without it), I have left the interaction in the model. The coefficients, std. errors and p-values for this model are below:
The plot below visualises these results, as well as the data which was fitted:
Figure 4: The shade of the graph indicates the predicted offer likelihood, with darker squares indicating higher likelihood of receiving an offer. The points plotted show the collected data, with offers indicated in blue and rejections in red. Larger points indicate a greater number of respondents giving that exact combination of answers.
The model, however, does not find any significant results. None of the p-values are below 0.05, though the values for both score and score:difficulty are close. McFadden’s r-squared for this model is 0.085, suggesting very poor explanatory power.