BU Research Blog

So it’s back to school for pupils and teachers, and Parliament is back (although still mostly virtually). What’s in the news?

Ofqual fight back

The House of Commons Education Committee grilled Ofqual this week in a fascinating session – the transcript is here. Before the session, Roger Taylor, the Chair of Ofqual, submitted a written statement, which you can read here.   We thought we would summarise the good bits for you.

Before you skip, though, the obvious question is “does it matter” – or is it all just a witch-hunt?  Clearly it does matter, because some of the same issues that led the government to cancel exams this year still apply – missed school time, uneven opportunities to learn, the implications of a second wave.  In our next segment, we look at the hints about next summer.

If you want to skip the next bit, the conclusion seems to be: Ofqual were handed an impossible brief by the Minister, who made it harder by changing policy on the hoof without asking them, they had a solution to it all in the form of a better appeals process to address outlying results (like high performing students in schools with poor previous performance) but never got a chance to roll it out because of the mocks fiasco, that they always thought exams should have gone ahead, and that the algorithm was fair and has been unfairly criticised by people who don’t understand the data!  Gavin Williamson is giving evidence soon, so that will be worth reading.  And Ofqual are going to publish correspondence so everyone can see that it wasn’t their fault….

David Kernohan has written about it for Wonkhe here.

The written statement  starts with an apology to students, teachers, and HE and FE providers.  As widely reported on the news channels yesterday, it confirms that Ofqual didn’t want the exams to be cancelled – they wanted them held in a socially distanced way.  Gavin Williamson decided to cancel them because of concerns about lost schooling and the risks with getting students back into schools.  So the well known solution and the well known moderation process was adopted. 

You will recall this decision was announced on 18^th March – which was very early – and might be said to have shown decisiveness and the desire to provide certainty in a complex situation.  But of course that assumes that the alternative was going to be a good and not a mutant one, which we all hoped it would be…..

In the evidence session, Roger Taylor said that after Ofqual offered advice on options:

• It was the Secretary of State who then subsequently took the decision and announced, without further consultation with Ofqual, that exams were to be cancelled and a system of calculated grades was to be implemented. We then received a direction from the Secretary of State setting out what he wished Ofqual to implement.

In the statement, Ofqual say:

• The principle of moderating teacher grades was accepted as a sound one, and indeed the relevant regulatory and examination bodies across the four nations of the United Kingdom separately put in place plans to do this. All the evidence shows that teachers vary considerably in the generosity of their grading – as every school pupil knows. Also, using teacher assessment alone might exacerbate socio-economic disadvantage. Using statistics to iron out these differences and ensure consistency Written submission from Roger Taylor, Chair of Ofqual looked, in principle, to be a good idea. That is why in our consultations and stakeholder discussions all the teaching unions supported the approach we adopted. Indeed, when we consulted on it, 89% of respondents agreed or strongly agreed with our proposed aims for the statistical standardisation approach.

And they knew there were risks but on the whole the averaged out effect was correct:

• We knew, however, that there would be specific issues associated with this approach. In particular, statistical standardisation of this kind will inevitably result in a very small proportion of quite anomalous results that would need to be corrected by applying human judgment through an appeals process.
• For example, we were concerned about bright students in historically low attaining schools. We identified that approximately 0.2% of young peoples’ grades were affected by this but that it was not possible to determine in advance which cases warranted a change to grades. That is why the appeals process we designed and refined was so important. But we recognise that young people receiving these results experienced significant distress and that this caused people to question the process.

In the evidence session, Roger Taylor was asked about this and he said:

• It was clear that to make a valid judgment would require a degree of human judgment and therefore a form of appeal would be necessary to make this work, but we were also exploring with the exam boards how we could implement a system of outreach to those students through the exam boards to let them know on the day, “Look, we think you’ve probably got a very good case for appeal.” That was the direction we were moving in. When the mock appeals route came in, that question became less relevant.

And they are still defending it:

• The statistical standardisation process was not biased – we did the analyses to check and found there was no widening of the attainment gap. We have published this analysis. Indeed, ‘A’ and ‘A*’ grade students in more disadvantaged areas did relatively better with standardised results than when results were not standardised.

They were challenged on this in the evidence session.

• Robert Halfon, the chair, asked about it: The Department for Education confirmed on 14 August that pupils from lower socioeconomic groups were more likely than their peers to have their centre assessed grades downgraded by Ofqual’s algorithm at grades C and above. The difference between Ofqual’s moderated grades and teacher centre assessed grades for lower socioeconomic groups was 10.42%. In contrast, the difference between Ofqual’s moderated grades and teacher centre assessed grades for higher socioeconomic groups was 8.34%.
• Michelle Meadows, Executive Director for Strategy and Research, replied: We had done a full equalities analysis, looking at the grades not just by socioeconomic status but by other protected characteristics such as ethnicity, gender and so on, and what we were able to see and we were very confident about was that any fluctuation in outcomes seen for these various groups this year was extremely similar to the small changes in outcomes we had seen in previous years. In other words, there was nothing about the process that was biased.

And when challenged about the impact on individual students, Roger Taylor said in the evidence session:

• I disagree with the notion that this algorithm was not fit for purpose or that a better algorithm would have produced a different result; but I strongly agree with your statement that to say this was fair just fails to recognise what happens to students—just the level of accuracy that was fundamentally possible with the information that was available was too low to be acceptable to individuals, and we recognised this right at the outset. We identified this as a risk.

And on small class sizes etc

• However, the impossibility of standardising very small classes meant that some subjects and some centres could not be standardised, and so saw higher grades on average than would have been expected if it had been possible to standardise their results. This benefitted smaller schools and disadvantaged larger schools and colleges. It affected private schools in particular, as well as some smaller maintained schools and colleges, special schools, pupil referral units, hospital schools and similar institutions. We knew about this, but were unable to find a solution to this problem. However, we still regarded standardisation as preferable because overall it reduced the relative advantage of private schools compared to others.
• Ultimately, however, the approach failed to win public confidence, even in circumstances where it was operating exactly as we had intended it to. While sound in principle, candidates who had reasonable expectations of achieving a grade were not willing to accept that they had been selected on the basis of teacher rankings and statistical predictions to receive a lower grade. To be told that you cannot progress as you wanted because you have been awarded a lower grade in this way was unacceptable and so the approach had to be withdrawn. We apologise for this.

And here is the killer statement:

• With hindsight it appears unlikely that we could ever have delivered this policy successfully.

And whose fault is it?

• Understandably, there is now a desire to attribute blame. The decision to use a system of statistical standardised teacher assessments was taken by the Secretary of State and issued as a direction to Ofqual. Ofqual could have rejected this, but we decided that this was in the best interests of students, so that they could progress to their next stage of education, training or work.
• The implementation of that approach was entirely down to Ofqual. However, given the exceptional nature of this year, we worked in a much more collaborative way than we would in a normal year, sharing detailed information with partners.
• We kept the Department for Education fully informed about the work we were doing and the approach we intended to take to qualifications, the risks and impact on results as they emerged. However, we are ultimately responsible for the decisions that fall to us as the regulator.
• …. The blame lies with us collectively – all of us who failed to design a mechanism for awarding grades that was acceptable to the public and met the Secretary of State’s policy intent of ensuing grades were awarded in a way consistent with the previous year.

Autumn exams:   It was clear to everyone that autumn exams would be a problem for those intending to start university this year.  No plan or proposal was made for this, apart from ministerial exhortations that universities should be flexible, and vague references to a January start.  Put on top of an absolute prohibition on unconditional offers, it was hard to see what universities were meant to do. Ofqual say:

• “the original policy was adopted on the basis that the autumn series would give young people who were disappointed with their results, the opportunity to sit an examination. However, the extended lockdown of schools and the failure to ensure that such candidates could still take their places at university meant that this option was, for many, effectively removed. This significantly shifted the public acceptability of awarding standardised grades”

I have no idea what that means….but it looks like blaming the context for the problems.  Roger Taylor clarified it in the evidence session:

• When the decision was originally made, there was a strong belief that the autumn series would be the compensation for that—that people would be given a chance and that university places could be held open for them that they could take in January, and that that would limit that damage. At the time, it was felt that it was a fair offer, but of course, over time, schools did not reopen; there were no arrangements for late entry to university; and by July, it was clear that the autumn series did not represent any sort of reasonable alternative that candidates felt would make up for being given an inaccurate calculated grade. At that point, we were in a situation where it was difficult to see how people would accept it as a fair way to have their grades awarded.

Autonomy and influence

• Roger Taylor: The relationship is one in which the Secretary of State, as the democratically accountable politician, decides policy. Ofqual’s role is to have regard to policy and to implement policy, but within the constraints laid down by the statute that established Ofqual. Those constraints are that the awarding of grades must be valid, it must maintain standards year on year, and it must command public confidence. We can decide not to implement a direction from the Secretary of State if we feel that it would directly contradict those statutory duties, but if the policy does not directly contradict those statutory duties, our obligation is to implement policy as directed by the Secretary of State.

There was a bit more about this in the evidence session when Roger Taylor was asked about the mock appeals policy (see below) and he said:

• It is important, in trying to manage public confidence, that we do not have a Secretary of State stating one policy and Ofqual stating a different policy. It also struck us that the way to resolve this was to move at pace and it needed to be negotiated and managed in an orderly fashion. But we were acting with full independence.

The comings and goings about the use of mock results in appeals were discussed at length:

• Roger Taylor:… the Secretary of State informed us that, effectively, they were going to change policy. Until that point, the policy had been calculated grades plus an appeals process. The Secretary of State informed me that they were planning to change this policy in a significant way by allowing an entirely new mechanism by which a grade could be awarded through a mock exams appeal. Our advice to the Secretary of State at this point was that we could not be confident that this could be delivered within the statutory duties of Ofqual, to ensure that valid and trustworthy grades were being issued. The Secretary of State, as he is entitled to do, none the less announced that that was the policy of the Government.
• That having been announced as the policy of the Government, the Ofqual board felt—I think correctly—that we should therefore attempt to find a way to implement this in a way that was consistent with our statutory duties. We consulted very rapidly with exam boards and other key stakeholders. We were very concerned that this idea of a valid mock exam had no real credible meaning, but we consulted very rapidly and developed an approach that we felt would be consistent with awarding valid qualifications. We then agreed that with the Department for Education and, to our understanding, with the Secretary of State’s office. We then published this on the Saturday. We were subsequently contacted by the Secretary of State later that evening and were informed that this was in fact not, to his mind, in line with Government policy.
• ….It was published about 3 o’clock on the Saturday. I think the call from the Secretary of State was probably at around 7 o’clock, 8 o’clock that evening. The Secretary of State first phoned the chief regulator. …
• The Secretary of State telephoned me and said that he would like the board to reconsider. ….given the Secretary of State’s views, it felt appropriate to call the board together very late that evening. The board convened at, I think, around 10 o’clock that evening. I think at this stage we realised that we were in a situation which was rapidly getting out of control—that there were policies being recommended and strongly advocated by the Secretary of State that we felt would not be consistent with our legal duties, and that there was, additionally, a growing risk around delivering any form of mock appeals results in a way that would be acceptable as a reasonable way to award grades….

Grade inflation

• Ian Mearns asked: This is the problem: Ministers are regularly telling us that we have more good and outstanding schools, with the most highly professional teaching profession that we have ever had. Given that process, that improvement and that continuing improvement, should there not be some increase in the levels of achievement by youngsters year on year that cannot be put down as grade inflation?
• Roger Taylor replied: On your point about grade inflation, we were very aware that being very strict about grade inflation would only make this situation worse. That is why, in the design of the model, at every point where we could reasonably do this, we erred in the direction of making decisions that allowed grades to rise. Consequently, the final result of the moderated grades did allow for between 2% and 3% inflation in grades which, in assessment terms, is very significant and larger than would represent the sorts of effects that you talked about resulting from improvements in teaching, but we felt that that was appropriate in these extremely unusual circumstances, given the disruption happening in people’s lives as a result of the pandemic.

Issues with CAGs:

• David Simmonds MP said that he has had more complaints about the u-turn and the fairness of the CAGs than the original grades. There is concern about the lack of opportunity for students to appeal these grades.
• Roger Taylor said: It goes to the nature of the problem: there is not an independent piece of information that can be used to determine between these two competing claims. That is why the lack of any form of standardised test or examination makes this a situation that people find very hard to tolerate.

On private students (who have to take exams in the autumn):

• Roger Taylor: I have huge sympathy with these people. Clearly, they have been some of the people who have lost out most as a result of the decision to cancel exams. I will hand over to Julie to say a little bit more about this, but once the decision had been taken to cancel exams, it was very hard to find a solution. We explored extensive solutions, but ultimately the situation was one in which, once exams had been cancelled, these people had lost the opportunity to demonstrate their skills and knowledge in a way that would enable them to move forward with their lives. That was the situation we were in.

On the tiering problem (students getting a higher grade than permitted by the exam, i.e. foundation students at GCSE who can’t get higher than a 5, who got a 6, for example):

• Michelle Meadows: In the absence of papers this year, we felt that the fairest thing to do was to remove those limits on students’ performance. So there were a very small number of cases where, for the tiered qualifications, less than 1% of foundation tier students received higher grades and, for the higher tier, less than 0.5% received lower grades than they would normally achieve. We felt that it was a decision in favour of students—that they would not be constrained in the normal way.

And on BTECs:

• Roger Taylor: It was not inevitable that there would be a domino effect, because the use of calculated grades inside the BTEC system was completely different from what had gone on with general qualifications. They were two completely separate pieces: one Ofqual was closely involved with and where we had the authority to make a decision; and the second was one that Pearson were responsible for and where we had no authority to determine how they were going to respond to the situation. That was their call.

And did the algorithm mutate?

• Ian Mearns: At what point did the algorithm mutate?
• Dr Meadows: I don’t believe that the algorithm ever mutated.

So what about next year

There are already discussions about delaying the exams, some elements have been changed, there are discussions about having an online option with open book exams, etc.  Ofqual have now made it extremely clear in the evidence session referred to above that they didn’t want to cancel exams this summer and they certainly don’t want to next summer, but also that they don’t want to rely on moderated CAGs again.  So some form of formal assessment seems likely.  But this one has some way to run.

For what was announced in August, Schoolsweek have a nice round up of the changes to A levels and for GCSEs here.  The Ofqual statement about A levels, AS levels and GCSEs is here.

In their statement referred to above, Ofqual confirm that amongst the lessons learned from this year are some things that will influence next year:

• any awarding process that does not give the individual the ability to affect their fate by demonstrating their skills and knowledge in a fair test will not command and retain public confidence
• a ‘better’ algorithm would not have made the outcomes significantly more acceptable. The inherent limitations of the data and the nature of the process were what made it unacceptable

And there should have been better comms and not just by them.

In the evidence session, Roger Taylor said:

• I think we have been very clear that we think that some form of examination or standardised test, or something that gives the student an ability to demonstrate their skills and knowledge, will be essential for any awarding system that the students regard as fair. We have done some consultation, and have published the results of that consultation, but it is obviously a fast-moving environment, and the impact of the pandemic remains uncertain over the future, so it is something that we are keeping under constant review……I want to be really clear that, absolutely, we raised it in our initial consultation, and we are very conscious of the enormous benefit that would come from delay. We recognise the value in trying to find a way of making this work.

And Julie Swan said:

• Content for GCSEs, AS and A-levels is of course determined by Ministers, and Ministers, as I am sure you will know, have agreed some changes to content for a couple of GCSE subjects—history, ancient history and English literature. We have published information about changes to assessment arrangements in other subjects that will free up teaching time, such as making the assessment of spoken language in modern foreign languages much less formal. …..as well as allowing, for example, GCSE science students to observe practical science, rather than to undertake it themselves….We are working with the DFE to get to conclusions within weeks, rather than months.

Gavin Williamson’s position

Gavin Williamson gave a statement to the House of Tuesday, on the first day back.  He said very, very little, really.  He apologised and then moved on quickly to talk about schools going back.  David Kernohan has written about this for Wonkhe too.

• The problem with having a Prime Minister who will only sack officials is that we are forced to watch senior politicians descent into near-Grayling levels of farcical inadequacy without hope of respite. Williamson’s haunted soul screams for release, but still he has to field questions about next summer while struggling to get through the next five minutes.

Research Professional cover it here.

Meanwhile in HE

The Office for Students have today launched a call for evidence into Digital teaching and learning in English Higher Education during the pandemic.  It closes on 14^th October 2020.

The review will consider:

1. The use of digital technology to deliver remote teaching and learning since the start of the pandemic and understand what has and has not worked.
2. How high-quality digital teaching and learning can be continued and delivered at scale in the future.
3. The opportunities that digital teaching and learning present for English higher education in the medium to longer-term.
4. The relationship between ‘digital poverty’ and students’ digital teaching and learning experience

If you are interested in contributing to a BU institutional response please contact policy@bournemouth.ac.uk as soon as possible.

Inquiries and Consultations

Have you contributed to a Parliamentary Inquiry?  Many colleagues from across BU have done so over the last year, and inquiries can be relevant for both academic and professional services colleagues.  Your policy team (policy@bournemouth.ac.uk) can help you prepare and submit a response – there are some important rules to follow about content and presentation, but a good submission might result in a call to give oral evidence (by video, these days) or get people talking about your submission.

You can find the list of open Parliamentary inquires here.  They include (just a few examples):

• Police conduct and complaints (accepting written evidence until 14^th September 2020)
• Digital transformation in the NHS {(until 9^th September)
• Reforming public transport after the pandemic ?(until 24^th September)
• Biodiversity and ecosystems (until 11^th September)
• Black people, racism and human rights {(until 11^th September)

And you can also find Secre – a small selection (these have longer dates):

• A call for evidence on a future international regulation strategy
• Pavement parking
• Marine energy projects
• Distributing Covid and flu vaccines
• Recognition of professional qualifications
• Marine monitoring
• Deforestation in UK supply chains
• Waste management plan for England
• Front of pack nutrition labelling
• Review of the Highway Code to improve road safety for cyclist, pedestrians and horse riders

Let us know if you are interested in responding to these or any others.MinisSecre

Subscribe!

To subscribe to the weekly policy update simply email policy@bournemouth.ac.uk.

Did you know? You can catch up on previous versions of the policy update on BU’s intranet pages here.. Some links require access to a BU account- BU staff not able to click through to an external link should contact eresourceshelp@bournemouth.ac.uk for further assistance.

External readers: Thank you to our external readers who enjoy our policy updates. Not all our content is accessible to external readers, but you can continue to read our updates which omit the restricted content on the policy pages of the BU Research Blog – here’s the link.

JANE FORSTER                                            |                       SARAH CARTER

Policy Advisor                                                                     Policy & Public Affairs Officer

Follow: @PolicyBU on Twitter                   |                       policy@bournemouth.ac.uk