Home > Policy and analysis > Forum for Responsible Research Metrics

Forum for Responsible Research Metrics

​A group of research funders, sector bodies and infrastructure experts working in partnership to promote the responsible use of research metrics.​

​​​About the forum

The Forum for Responsible Metrics is developing a programme of activities to support the responsible use of research metrics in higher education institutions and across the research community in the UK. It focuses on the recommendations made in the Metric Tide report. 

The Forum will advise on, and work to improve, the data infrastructure that underpins metric use and the culture of research metrics.

The Forum will offer advice to the UK higher education funding bodies on how quantitative indicators might be used in assessing research outputs, environments and impact in REF2021. 

Who is involved with the Forum for Responsible Metrics?

The forum is a partnership between the Higher Education Funding Council for England, Research Councils UK, the Wellcome Trust, Universities UK and Jisc.

The forum is chaired by Professor David Price, Vice-Provost (Research), University College London (UCL). Its members are representatives and experts from university management, academic leadership, research administration and research metrics.

The five partners recognise the wide interest in this work and expect to contact and involve other stakeholders in their discussions. Regular updates will be posted here.

Membership List (Dated November 2017):

  • Professor David Price, Vice-Provost (Research), University College London (Chair)
  • Steven Hill, Head of Research and Health Policy, HEFCE
  • Ian Viney, Director of Strategic Evaluation and Impact, Medical Research Council (representing Research Councils UK)
  • Chonnettia Jones, Head of Insight and Analysis, Strategy, Wellcome
  • Chris Hale, Director of Policy, Universities UK
  • Rachel Bruce, Director open science and research lifecycle, Jisc
  • Stephanie Bales, Director of Research and Innovation Services at Teesside University (representing the Association of Research Managers and Administrators)
  • Professor Roger Kain, Dean and Chief Executive, School of Advanced Study, University of London (representing the British  Academy)
  • Professor Linda King, Pro Vice-Chancellor, Research and Global Partnerships, Oxford Brookes University
  • Professor Ottoline Leyser, Director, Sainsbury Laboratory, University of Cambridge (representing the Royal Society)
  • Keith McDonald, Assistant Director, Research and Innovation, Scottish Funding Council (representing the UK higher education funding bodies outside England – the Scottish Funding Council, the Higher Education Funding Council for Wales, and the ​Department for the Economy Northern Ireland)
  • Professor Mike Thelwall, Professor of Information Science, University of Wolverhampton
  • Professor James Wilsdon, Professor of Research Policy and Director of Impact and Engagement, University of Sheffield
  • Dr Ian Carter, Director of Research and Enterprise, University of Sussex
  • Professor Derek Woollins, Vice Principal (Research and Innovation), University of St Andrews ​
  • Claire Fraser, Research Policy Advisor, HEFCE (secretary)

What are 'responsible metrics'?

Metrics form part of an evolving and increasingly digital research environment, where data and analysis are playing an ever greater role. However, the current description, production and use of these metrics are experimental and open to misunderstanding, and can lead to negative effects and behaviours as well as positive ones.

Responsible metrics can be defined by the following key principles:

  • Robustness – basing metrics on the best possible data in terms of accuracy and scope
  • Humility – recognising that quantitative evaluation should support, but not supplant, qualitative, expert assessment
  • Transparency –that those being evaluated can test and verify the results
  • Diversity – accounting for variation by research field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
  • Reflexivity – recognising and anticipating the systemic and potential effects of indicators, and updating them in response  

​​​Culture of research metrics

Survey on the culture of research metrics in research organisations: respond by Monday 11 December.

The Forum for Responsible Research Metrics are surveying research organisations to understand how they are implementing policies on the use of responsible research metrics. The Forum would like to understand the extent to which research organisations are implementing the principles/frameworks outlined in the San Francisco Declaration on Research Assessment (DORA http://www.ascb.org/dora/), the Leiden Manifesto (http://www.leidenmanifesto.org/) and The Metric Tide (http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/). 

Responses will be used in order to develop advice to the sector on practical ways to implement the culture of responsible research metrics using the principles/frameworks outlined above. It will also inform any recommendations the Forum makes to UKRI. Based on the responses received the Forum will consider whether to develop an agreement with similar ambitions to DORA, utilising The Metric Tide report, which aligns with the UK research base.

Universities UK are hosting this survey on behalf of the Forum.

How to respond

We seek one response per research organisation. To respond to this survey (which will be reported on at aggregate level ensuring anonymity) please follow this link: The culture of research metrics in UK research organisations ​ - a survey​ (a PDF form​ is also available​​).

Please contact Claire Fraser for any questions regarding this survey.  

​​​Research Excellence Framework 2021 (REF 2021)

One key aim of the Forum for Responsible Research Metrics is to provide advice to UK HE funding bodies, via the REF team, on the use of metrics in REF2021 in the three elements of the assessment.

Outputs

In February 2017 the Forum provided advice on the use of quantitative indicators in the assessment of outputs in REF2021. This advice is given in response to questions circulated by HEFCE as a basis for discussion at the Forum's first meeting.

9th February 2017 - Metrics in REF2021: Advice from the UK Forum for Responsible Research Metrics

Environment

The REF consultation, published in December 2016, sought sector comment on:

  • the use of more quantitative data in the assessment of environment (question 34a and 34b)
  • the development of guidelines for the use and standard of quantitative data as evidence for impact (question 32b)

Summaries of the REF consultation responses can be found on the HEFCE website.

The REF initial decisions document, published 1 September 2017, announced that the UOA level environment element of REF2021 will be assessed on the basis of a more structured template, including the use of more quantitative data to evidence narrative content.

The Forum, in May 2017, requested that a working group should be set up to provide detailed advice on the use of metrics/indicators in environment. They have been asked to advice on the use of more quantitative data that is already collected and held by institutions. The working group will provide advice to the FFRRM, which will make recommendations to the REF team, REF panel chairs and the UK funding bodies.

Advice to the REF team and REF panels will be published on this webpage in spring 2018.

Impact

The working group have also been asked to provide advice on the development of guidelines for the use of standard quantitative data as evidence for impact.

Papers

Blogs

How open and accessible is university research? – An update

27 October 2017
As Open Access Week draws to a close, Professor Adam Tickell analyses the university sector's recent activities and achievements in the field of open access.

Research misconduct: how universities are addressing the issue

30 March 2017
Jamie Arrowsmith, Programme Manager at Universities UK, discusses the progress universities have made in addressing research misconduct.

The team

Max Hastings

Max Hastings

Policy Researcher
Universities UK

Martina Tortis

Martina Tortis

Policy Analyst
Universities UK
  
  
collapse Meeting date : 7 December 2016 ‎(6)
FRRM - minutes of first meeting.pdf
  
FRRM - Paper 0101 - Aims and objectives.pdf
  
FRRM - Paper 0102 - REF.PDF
  
FRRM - Paper 0104 - JIF survey.pdf
  
FRRM - Paper 0104a - annex to JIF survey paper.pdf
  
FRRM - Paper 0105 - comms and engagement.pdf