PRODUCTION/OPERATIONS MANAGEMENT

Michael J. Showalter, Feature Editor, Florida State University

Measuring Academic Quality: The UK Experience

by Steve New, Manchester School of Management, UMIST

UK academics are facing unprecedented pressure from the government about the quality of research and teaching. Many--including the author--fear that this emerging regime of measurement and evaluation may erode those valuable features which make the university system distinctive. In a rush towards `managerialism', it seems that the authorities may actually reduce quality.

Universities in the UK are almost all public institutions, and most are ultimately dependent on taxpayer's money. Cash comes by several routes: block grants, student fees (largely paid by local government), and specific research grants (from special funding councils) make up most of a typical university's income. Of course, revenue is also generated by endowments, work for industry and from students paying their own feesþbut most institutes would not survive if reliant on these sources. The universities are therefore really reliant on the stateþand, of course, we all want public funds to be spent wisely. Why not have standard measures of what universities do?

A key idea in the dominant UK political ideology is the notion of the quasi-market in the public sector. Institutions--or maybe even different bits of the same institution--should compete for funds according to set rules. This needs systems of measurement to allow ranking and resources allocation according to clear criteria. The key principle is the biblical maxim of "To him who hath..."; successful institutions get more, losers get less.

In recent years, this logic has been applied in the triennial measurements of research performance and it now is being extended to teaching. But both systems may have unintended consequences; the trouble is that it is difficult to think of viable or better alternatives.

Research

For research, the current system is based on a complex and laborious process of peer review. Each university department has to make a lengthy submission in a pre-determined format. This includes statements of strategy, discussion of recent highlights, plans for development and so forth. This is accompanied by various statistical data about the amount of money raised from research grants and industrial research contracts, the number of research students, and the number of publications in a variety of categories produced by faculty. This list includes the quantity of output under 19 headings, including edited books, refereed journal articles, conference papers, non-refereed professional journal articles and so on. This department-wide information is then supplemented by two tables with entries for each member of academic staff. The first of these gives a selection of straight-forward informationþage, starting and leaving dates (if in assessment period) and so onþand also an aggregated number of publications. This figure is an unweighted sum of output in different categories. The second table gives the references of two 'best' publications for each member of staff during the assessment period.

The submissions in each discipline are then chewed over by a panel of eminent academics appointed by the main funding council--obviously, each member of the panel does not get to assess his or her own department. After a lengthy (and confidential) process, each department is given a score from 1 (worst) to 5 (best); this is then combined with a grade A to C which measures the percentage of staff who are research active. This score is then a direct input to the formula which calculates the block grant, and is also considered when the research councils give out specific grants.

The system has some obvious flaws: although the submissions are packed full of numbers, the assessment is clearly subjective. Furthermore, the panel members have to wade through vast quantities of data, and have no official algorithm to tell them how to interpret the information or resolve disagreements. It is widely believed, for example, that a key criterion is simply the number of refereed journal articles, but there is no clue as to how this is combined with or traded off against other elements. Nevertheless, the panels meet, the wheels grind, the machinery whirs and out pops a ranking for each department.

The good points are that at least the scheme combines quantitative and qualitative data, and does not rely entirely on crude metrics of output. The trouble is, because there is no official method of analyzing the returns, there is no guarantee on what the panel will do with all the data. Or that they will do the same next time.

Teaching

The evaluation of university teaching is just getting underway, and mixes self-evaluation with external inspection. The first of these elements allows a university department to declare itself as unsatisfactory, satisfactory or excellent, and to support this with whatever evidence it chooses. This may include data on outputs (e.g., the employability and reputation of its graduates), inputs (such as the number of professional qualifications held by its staff, the size of the library and so on), and its processes (such as procedures for course design and review and teaching evaluation). Departments cannot go overboard on this, though, and the narrative report is limited to about ten pages.

The external assessment involves a team of assessorsþsome drawn from other institutions, others drawn from industry and the civil serviceþvisiting a department and using a number of methods to collect data. These include meeting staff and students; observing lectures and seminars; examining documents and systems; and reading external examiners reports and examination scripts and coursework. All 'unsatisfactory' and 'excellent' submissions will provoke an external assessment; some 'satisfactories' will be selected too. The outcome is that the external inspection will either confirm or revise the self-assessment. Reports from the visits will be published.

Only a few departments have been assessed in this way so far, and the process is still developing. In the future, however, it seems that this mechanism will--like the research evaluation--have a major impact on block funding, and on the approval of particular courses.

Unintended Consequences

Both of these strands of official measurement will generate consequences beyond the original goal of helping to allocate money. Departments with top research ratings (such as my own) are already using this in promotional material; the same will happen with teaching evaluations. Thus, as well as being crucial to financial health, the ratings become vital to the sustainability of reputation. This affects recruitment of staff and students, industrial contracts and charitable donations. If you are at the top, you get in a 'virtuous circle'; at the bottom, you may end up in a spiral decline. The introduction of formal rankings may accelerate the development of a small 'super-league' of UK universities; others may have to re-invent themselves as teaching-only establishments. In as much as this fits the 'market' model of the public sector, it could be argued that this effect is not unintended at all.

For faculty members, there is concern that the pressure on institutions will unduly affect academic freedom and integrity. Eventually, the pressure to publish may introduce a counter-productive short-termism, and squeeze out the chance to read widely and reflect deeply. Furthermore, the ambiguity in what the system demands may introduce conflict and stress between staff as they work out their 'corporate' priorities. For the teaching assessment, there is a danger that increasing standardisation and systemisation of the educational process may tie up universities in futile bureaucracy.

Another issue is whether the system will actually work in the long run. It is rumoured that there may be a tacit agreement amongst universities to all claim 'excellent' teaching quality; this would then drastically overload the system's capacity for external assessment. If this does not happen, then it is rumoured that one top UK university has considered 'privatising' itself to avoid the possible embarrassment of not doing well in the teaching quality assessment. One institution has recently sought a judicial review of its research evaluation score, claiming that it was unfairly treated.

For those academics involved with the management sciences, these current developments are rather salutary. The UK experience should serve as a reminder that no system of measurement or control--quantitative or qualitative--can be separated from issues of human behaviour and human values.

STEVE NEW is a lecturer at the Manchester School of Management, UMIST--one of the four institutions to win a `5A' in Business and Management Studies. His research interests focus on supply chain management and production systems.