This year, as we know, 'real' exams are back. But here are four facts you may not be aware of about the 2022 results in England:
- Around 800,000 A level grades were announced on August 18 – of which around 200,000 are likely to be wrong.
- About 5,000,000 GCSE grades will be announced on August 25 – of which around 1.25 million are likely to be wrong.
- For every 10 students taking 8 GCSEs, only circa 10% will receive a certificate on which all 8 grades are right; the remaining 90% will receive a certificate on which at least one grade is wrong.
- All scripts, in all subjects, at all levels, marked at, or very close to, any grade boundary has approximately a 50:50 chance of being awarded the right grade – and a 50:50 chance of being awarded the wrong grade.
For clarity, 'wrong' in Ofqual speak means: 'would have been awarded a different grade had the script been originally marked by a senior examiner'. That's important, for Ofqual refers to the grade as determined by the mark given by a senior examiner as 'definitive' or 'true' (implying that any different grade must be 'non-definitive' or 'false').
I have opted for the simpler word 'wrong' – for if it were possible for a script to be re-marked by a senior examiner on appeal, then any originally-awarded 'non-definitive' grade would be changed to the senior examiner's 'definitive' one. If faced with this scenario, I suspect any student would be more likely to say "The original grade must have been wrong!" rather than "The original grade must have been 'non-definitive'".
In case you're sceptical, the facts stated above are not fantasy. They are direct inferences from Ofqual's own research, in which it marked entire cohorts of scripts; once by an 'ordinary' examiner – thus simulating what usually happens – and then by a senior examiner. Consequently, each script had two marks.
You would be forgiven for expecting that the vast majority of the resulting grades would have subsequently been the same, with a few random exceptions. Not so. In fact, across the 14 subjects studied by Ofqual, for approximately 1 script in 4, the grade awarded by an 'ordinary' examiner was different from the 'definitive' grade awarded by a senior examiner.
There were wide variations by subject too. Taking Physics and History as examples: for the former, around 12 grades in every 100 were different; for the latter, that number rose to nearly half (in the region of 44 grades in every 100).
All this information was published by Ofqual in 2018 (for example, see Figure 12 on page 21 here, and here). However, the results have not been widely publicised, so, if these facts are 'news' to you, you're not alone. Nor has Ofqual ever admitted that '1 grade in 4 is wrong', although it has come quite close, most notably at a hearing of the Education Select Committee, held on 2 September 2020 in the aftermath of the 'mutant algorithm fiasco'. Below is an exchange between Ian Mearns, the Labour MP for Gateshead, and Dame Glenys Stacey, Ofqual's Chief Regulator at the time:
Q1059 Ian Mearns: I am just wondering, therefore, whether one of the issues that you really should address, Dame Glenys, is that 25% of grades on an annual basis are regarded as being unreliable, either up or down.
Dame Glenys Stacey: Thank you. I'll certainly keep that in mind, and I look forward to speaking to you about it, Mr Mearns. It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think Michelle mentioned, that we have faith in them, but they are reliable to one grade either way. We have great expectations of assessment in this country.
That is the Chief Regulator of Ofqual admitting that exam grades are 'reliable to one grade either way' and in essence saying that the grade B on a young person's certificate might be a B, but it might be a C – or even an A. No-one really knows which.
I wonder what a student - who, on 18 August, was distraught about having missed the university place of their dreams -might think about that. Such disappointment, of course, is inevitable on results day, but what can be done about it? Here are four potential courses of action:
- Ofqual should be lobbied, now, to ensure that every certificate includes: OFQUAL WARNING: The grades on this certificate are only reliable to one grade either way. Those, after all, are Ofqual's own words. And they are the truth.
- A student missing a place by one grade might contact the appropriate admissions officer and ask about the institution's policy in relation to Ofqual's admission that grades 'are reliable to one grade either way'.
- Just before the algorithm's results were to be announced on 13 August 2020, Gavin Williamson instructed Ofqual to allow a 'valid mock' result as the grounds for appeal. James Cleverley could follow this precedent and instruct Ofqual to allow any script to be re-marked by a senior examiner to ensure the award of the 'definitive' grade - at no cost to the appellant.
- And surely, for 2023, and all years thereafter, this problem should be fixed. In the meantime, students, parents, teachers, colleges, universities and employers should complain, very loudly, that grades should not be 'reliable to one grade either way', but 'reliable, full-stop'.
This article was written by Dennis Sherwood and is based on this findings documented in his book, Missing the Mark: Why so many school exam grades are wrong, and how to get results we can trust, available via the Canbury Press
A background note from the author
My interest in this was triggered in 2016 when Ofqual went out to consultation on its proposed changes to the rules for appeals – changes that would make it harder to appeal, and harder to discover and correct grading errors. That was reported in the press, and when I read it, that struck me as most odd. Why would a regulator make it harder for the consumer to have access to justice? But that's what happened and the rules were changed in 2016.
In November 2016, just a few months after the rule change, Ofqual published its report Marking Consistency Metrics, which was the first evidence that, on average across all subjects, about 1 grade in every 4 is 'wrong' (my words, as mentioned in my article, Ofqual terms it 'non-definitive'). This was first reported in Schools Week within a few days of the report's publication. That's about 1.5 million 'wrong' grades, year in, year out. With – since the rule change – no right of appeal.
Ofqual has hotly disputed this ever since, culminating in its acknowledgement that grades are 'reliable to one grade either way'. I think that's important. Especially to young people who are denied a place for want of that single grade.
Not many people have read Marking Consistency Metrics, or its successor, Marking Consistency Metrics - An update, nor the transcript of the Education Select Committee hearing of 2 September 2020. So generally, people don't know that 1 grade in 4 is 'wrong' (my words)/grades are reliable to one grade either way (Ofqual's words).
That's why I wrote 'Missing the Mark' – which is in essence the 'collected works' of the 50+ blogs and articles I've written on this topic over the past few years.