To objectify excellence it needs to be measured. And this produces a host of problems:

1) Simplifying
Obviously the first step in measuring something is to simplify it in order to produce a table and rank people, departments and organisations. This simplification produces highly spurious measures. Thus half a decade of research is assessed on the basis of our ‘four best publications’. While a semester of teaching may be assessed on the basis of a standardised battery of questions asked of those students who turn up to lecture in week 9.

And then, of course, how do we determine when excellence is really excellent? The REF (Research Excellence Framework) provides the following not entirely helpful help, which suggests that even excellence may not be truly excellent.

See a nice discussion of the imprecision of REF measurement by Derek Sayer. And, in case you think that more precise metrics are an answer, there is a an extensive critique by Martin et al here.

2) Over-determination of small differences.
Most of the ways in which excellence is measured and tabulated exacerbates very small differences between individuals and institutions. For instance, the National Student Survey (NSS) confidence intervals are often as large as the difference between the upper and lower quartiles:

• e.g. actual score of 91: Confidence interval 78-96   [18 points].
• e.g. actual score of 72: Confidence interval 57-83   [26 points]
  [scores from City Sociology – CIs vary by dept/institution]

Similarly, the majority of departments entered into any REF panel have scores remarkably similar to one another (see visualisation here). This is especially the case when looking at ‘total 3 or 4 star’ (both categories that the REF definitions suggest involve ‘excellence’). Given the small numbers of items scored (impact case studies in single digits; outputs in double digits) this suggests that resulting rankings are hardly reliable.

3) Institutional effects
There are large institutional affects that impact on of what is measured (grants, teaching scores etc). For instance, institutions restructuring/prioritisation in response to ranking (Hazelkorn 2008).

4) Inattention to reality
Across the sector, newly appointed Vice Chancellors are charged with developing five year ‘Missions’ or ‘Visions’ (often named in a way resonant of space travel: Vision 2020 or Mission 2018). These visions/missions usually specify and quantify the institutional excellence aspired to: typically being ‘above average’ (or ‘top quartile’ or top ten…) in the UK or, more ambitiously, world and become institutional blue-prints. As we are often told, the success of a university vision/mission depends on our institution following an ‘innovative’ (yet widespread) strategy that (typically) involves, restructuring professional service staff, investment in real estate and, of course, recruitment of ‘excellent’ academics. Most institutions are currently recovering from, or still going through the turmoil that this involves. While many individuals, not rated as ‘excellent’ have recently left institutions they had worked in for decades, sometimes following undisclosed pay-offs and gagging orders.

Simple maths tells us that not everyone and not all institutions can be ‘above average’, they can’t all ‘move up’ the tables and everyone doing the same thing won’t differentiate anyone. As such, and looked at across the sector, this is a doomed strategy. And one that serves merely (or largely) to introduce management by fear.

See a critique of the imposition of measurable targets by John Holmwood.

5) Ranking
At the end of the day, even if all of the above problems were resolved (which they won’t be), ranking itself is the most serious problem. Ranking produces, and is designed to produce, winners and losers. Ranking with consequences reproduces those winners and losers (as any study of streaming in schools shows). All measurement of excellence in academia is consequential – in terms of funding, student recruitment, personal careers. This ranking systematically reproduces elites and elite institutions and disadvantages others, starving them of resources and opportunities (an instance of what Robert Merton called The Matthew Effect). By involving academics (for example, as REF panelists and teaching peer reviewers) it legitimates this reproduction and bifurcation.

We need to resist the desire to measure and rank, not just the methods by which ranking is achieved.

We cannot simply oppose the order or rank of any particular person, department or institution (as this presupposes that a fair basis for ranking is achievable, even desirable), but should instead direct our energies against the ranking of our work, lives and educational institutions.