How are universities protecting degree standards?

The statements:

• review trends over several years
• bring together all the tools providers use to protect degree standards – such as the latest data and policies
• outline activities identified by providers on what more they can do
• must have approval from the provider’s governance board, giving scrutiny at the highest level
• are publicly available so universities are transparent about their practices   

While not required, 81% of recently surveyed providers set actions in their statements. Most commonly to:

• Review their degree algorithm design against the newly published principles. For example, providers have looked at the rules around the grade borderlines, reduced the use of multiple algorithms and limited discounting (ignoring lower grades). Providers have also taken steps to explain how the algorithm works through student-centred resources such as online calculators.  
• Analyse demographics or subject areas. This is to better understand differences in degree outcomes. From this, the statements have prompted conversations within departments on assessment design.
• Look at the impact of Covid-19. For example, looking at the impact of temporary policies and capture lessons from the pandemic, such as the closing of awarding gaps.
• Expand the responsibility of external examiners to comment on degree standards over time. We are also seeing more providers appoint chief or principal external examiners to oversee standards across the whole institution.

We have asked providers to update their statements annually with the latest data and progress on actions. From the providers we spoke to, 95% will update their statements in 2022 to reflect outcomes in the 2020–21 academic year.

We are committed to working with GuildHE and QAA to continue to receive feedback on the usefulness of the statements and any support that will help providers engage with them better.

In Scotland, universities review degree outcomes as part of programme reviews. In addition, QAA Scotland staff meet each Scottish institution annually to discuss quality, standards and education enhancement. 

The principles hold providers to a strong set of obligations that make sure a student’s grade is a reliable summary of their performance. The principles have helped produce algorithms that are consistent, transparent and reliable.

We are seeing action to:

• Look at the combined impact of an algorithm. This is to ensure parts do not overly reinforce each other. For example, this could happen where the final level of a student’s study is accounted for multiple times by weighting more heavily on the final year, discounting marks from previous years, and uplifting students on the borderline.
• Tighten borderline policies to protect boundaries. For example, ensuring that a 2:1 is typically 60–69 or a 2:2 50–59 (in the conventional 0–100 grading approach).
• Explain the algorithm to students. This has proved valuable where the design of algorithms differs across different subject areas.

The pandemic has meant many universities had to put their algorithm reviews on hold. This is because emergency changes needed to be introduced. We understand that plans are now back on track and expect more action on this in the future.

The descriptors work alongside other frameworks such as FHEQ, FQHEIS and subject benchmark statements. Their status as a sector-owned reference point is important. They are not a strict formula – this would stifle the freedom of providers to innovate their assessment practice. Instead, they should be seen as a guide to help with transparency and consistency.

Feedback suggests that 80% of institutions are using or will use the standards. We are seeing the descriptors used in the following ways:

• To help existing institution-level or subject-level descriptors align more with the sector ones.
• Support training and development for academics and external examiners. This enables the descriptors to support consistent marking judgements and exam committees.
• Embed the descriptors from the course approval stage through to assessment design, marking practice and to the student receiving their result.

For providers in England, the Office for Students (OfS) has consulted on making the descriptors a sector-recognised standard. It intends to use them to challenge providers where they have concerns about degree standards. Their status as a reference point needs to be protected, but going forward, they will be a useful addition to how quality and standards are protected in England.

We need a strong external examiner system that can support providers to make further changes to learning in the post-pandemic environment. The external examiner system must empower examiners to call out any concerns they have about degree standards.

We are seeing providers expand the role of external examiners. This includes taking on responsibilities related to degree standards, approving degree outcomes statements and examining data in more detail.

In August 2021, along with GuildHE, we asked QAA to launch work to strengthen the external examining system. QAA have formed an advisory group and kick-started a series of engagements with the sector. From this, they will develop a series of principles which will build a shared understanding of the external examiner role.

Within this work, we also want to consider where external examiners have a role in addressing the challenges of grade inflation.

AUB reviewed its degree algorithm against the principles and examined the data to ensure changes didn’t disadvantage particular groups. It noted some areas where its approach didn’t align, and made minor changes which would meet the principles without undermining the intention of their algorithm. The new algorithm will be brought in for students enrolling in 2022.

As a young university, Hartpury University chose to publish a degree outcomes statement as part of its ongoing work to ensure awards reflect industry and professional standards and expectations. The university monitors achievement annually, at programme and department level, through an achievement report and a quality report through its committee and governance structure. It makes evidence-informed decisions about how to amend and enforce its academic regulations. The statement of intent and connected research has helped the university contextualise its practice with other providers. This led to more confident decision-making when enacting regulations during the Covid-19 pandemic.

Over the last few years, NTU has developed its grade-based assessment framework in three key ways. First, the university has enhanced the rigour of the degree algorithm. A student is awarded a first if they have performed at a first-class standard on the majority of their modules.

Second, different award boards had historically used different variations of the rules on borderline classifications to enable students with profiles at the top of a classification band to achieve a higher classification. There is no longer any exam board discretion to do this.

Third, NTU has shifted to a ‘linear’ system of numbers that stand in for each grade (0-16, instead of 0-100). In the 0-100 (non-linear) system some students benefit from the inflationary effects of the traditional ‘70-100’ first-class range. This range is used in different ways across the sector and is an important factor in any analysis of grade inflation.

Following the publication of the paper on algorithm principles, UCEM undertook a review of how its practice aligns with each of the principles. The exercise reassured UCEM’s quality standards and enhancement committee that its practice aligns with the principles. It also assured UCEM that its choice of algorithm model and approach to discounting and classification at borderline are all appropriate.

The review recommended areas for improvement, including committing to a standard five-yearly cycle of algorithm review and providing a summary within the student handbook of how marks will be used to classify a degree and why this approach is right for UCEM programmes.

There is surprisingly little research on exit velocity in degree algorithms – where the weight is placed on a student’s marks later on in their degree. The University of Chichester wanted to understand this more. Previous research shows that of 100 responding institutions, 87 used exit velocity in their algorithm for calculating degree classification. The standard definition is based on a belief that student marks generally improve from year two to year three. Data from 970 students completing in 2019–20 showed that the level 6 mark was, on average, higher by just 2.6% for an individual student. Exit velocity was therefore unlikely to change a student’s classification, unless at the boundary. Interestingly, the data showed differences across academic areas, leading us to conclude that further analysis is needed. You can read more about this work on Wonkhe.

In December 2020, following an in-depth review of the degree classifications awarded to its graduates, the University of Exeter published its first degree outcomes statement.

During the 2020–21 academic year, the university also undertook research and analysis to determine the impact on degree outcomes of its no detriment policy brought in during the pandemic. The policy introduced a ‘safety net’ for individual students by calculating a benchmark based on their assessment and examination grades before the pandemic. The university found that most students did not need to make use of the ‘safety net’, as they achieved a higher final credit-weighted stage mean. For those that did rely on their benchmark, just 1.4% of UK domiciled final year students saw a change in their degree classification. The policy achieved its aim as students more likely to have been impacted socially and economically by the pandemic benefitted the most.

The University of Worcester has taken several steps in response to the statement of intent. The university updated its generic grade descriptors to align with the classification descriptors. The descriptors inform a review of local grading criteria. Through this process, the university gained a greater awareness of the demands on academic staff when writing and navigating grade descriptors and recognised that no one size fits all. The university’s classification method was mapped against the algorithm principles and it’s currently reviewing revisions to the method.