Interviews Research

The ability of Cambridge applicants to predict their interview scores
Theloniouss

[field]Introduction[/field]Post-interview stress is common among Cambridge (and Oxf*ord) applicants, in significant part because many applicants leave their interview with a very poor idea of how well they have performed. In 2016, user @Doonesbury conducted some research on this and found that the interview score predictions of unsuccessful and successful applicants are not meaningfully different, see Figure 1. It is my aim, with this research, to conduct the same test on a larger sample size. I also aim to account for both predicted score and perceived difficulty in order to determine whether this might serve as a more effective predictor of success.

I hypothesise that, in accordance with @Doonesbury’s results, there will be no significant difference between the predicted scores of successful and unsuccessful applicants. I expect that there will also be no difference between the groups in terms of the perceived difficulty of their interviews.

[field]Methods [/field]
Data was collected via survey, with the initial round asking for applicants’ predicted interview scores and how difficult they found the interview, on a scale of 1 to 10. Applicants were identified by their email or TSR username, both of which were provided.

Invalid responses (where an applicant’s email address and TSR username could not be used to contact them, usually because of invalid addresses or email addresses for internal use only (such as school emails) to this survey were removed, and can be found, anonymised, in Appendix 1.

A second survey, available after decisions had come out asked whether applicants had received offers as well as for their course and college. Course and college data will not be used in this analysis due to the small sample sizes, but if similar surveys are conducted in future the data could be collated.

Second survey responses were matched to first survey responses by the email and TSR username provided. Where first survey responses could not be matched, they were classed as “non-respondents” and can be found in Appendix 1. The full dataset of complete responses may also be found in Appendix 1.

In addition to the novel data collection, I have also transcribed Doones’ data from the graph (Figure 1 and this transcription is also available in Appendix 1.

All results have been anonymised and individual data points are identified by ID number.

[field]Results and Analysis [/field]
1. Results at a glance
The initial survey received 154 responses, of which 115 could be paired with a response to the second survey. 16 of the 154 responses contained errors making it impossible to contact the initial respondent and a further 23 responses could not be paired with a response to the second survey. The responses transcribed from Doones’ graph amounted to 63 complete responses. The below tables (Figure 2 contains key summary statistics from the main groups of interest, and the accompanying graph (Figure 3) shows the distributions of predicted scores and reported difficulties, divided into successful and unsuccessful applicants.








2. Data clean-up
The data above then requires some further consideration before we can analyse it. The first question is how to treat the non-respondent and error categories. Both categories appear to differ from the successful and unsuccessful categories in potentially significant ways, such as in the mean score of non-respondents and the median difficulty for errors.

I would tend to ignore the errors, as I expect these would not be more likely to receive or not receive offers (though it could be argued that the inability to type your own email address suggests you are not Cambridge material). The non-respondents, however, are probably more likely to have not received offers, as applicants who were deselected are likely to be less willing to respond. I have discounted them anyway because it would be difficult to account for them.

Next we should consider whether to include Doones’ data or not. While its inclusion would increase our sample size, without knowing how the data was collected it’s probably not sensible to include it, as different methods of data collection might result in different styles of result. For example, Doones’ data uses a different scale to mine, which has resulted in fractional score predictions. As a result of this, and because there is no way to account for perceived difficulty, I will not be using Doones’ data in this analysis.


3. Data analysis
In order to analyse this data, I have used logistic regression. This accounts for the binary nature of the response variable and means certain assumptions typical of regression analysis can be ignored, like normality and homoscedasticity (equal variance). The assumptions, and extent to which each is met, of logistic regression are below:
1. Binary response variable – the response variable is whether or not an offer is received
2. Independent observations – almost certainly met, I can’t see how offer-holders could have influenced one another’s responses
3. No correlation between explanatory variables – This assumption is slightly violated, as the two explanatory variables show weak correlation, as determined by Kendall’s tau correlation test, which was used because it accounts for tied ranks and doesn’t require normality (tau=-0.18, p=0.017). The correlation, however, is low and so logistic regression should still be valid.
4. Large sample size – The sample size here is 115-138, which should be large enough for logistic regression.

3.1. Fitting the model
In fitting the model, I have initially assumed an interaction between interview difficulty and score – it’s likely that these will influence each other so it makes sense to include an interaction. Since the AIC for this model is lower than the AIC for the model which excludes it (153.7 with the interaction, 154.3 without it), I have left the interaction in the model. The coefficients, std. errors and p-values for this model are below:


The plot below visualises these results, as well as the data which was fitted:



The model, however, does not find any significant results. None of the p-values are below 0.05, though the values for both score and score:difficulty are close. McFadden’s r-squared for this model is 0.085, suggesting very poor explanatory power.

[field]Discussion [/field]The results are inconclusive given how close to 0.05 the p-values are, and that different methods of analysis would have returned a significant result. The explanatory power of even the inclusive model is very poor. However, the key takeaway here should be comparisons between the mean predictions of the successful and unsuccessful applicants, as well as the high variance of both groups. It would be unreasonable for an applicant to assume they had failed because their interview was difficult or because they felt it went poorly (and vice versa).

It is reasonable to conclude that an applicant’s prediction of their interview score has very little bearing on whether their application is likely to be successful or not. This analysis finds the predictive power of applicant estimates to be insignificant.