How valuable are evaluations?

About: Clémentine Gozlan, Les valeurs de la science. Enquête sur les réformes de l’évaluation de la recherche en France, ENS Éditions

Since the 2000s, many reforms have been undertaken to transform higher education and research. They have given rise to many studies of the transformation of systems of professional governance, the restructuring of professional groups, the effects of evaluation on knowledge production, and the ways in which professors appropriate classification systems and indicators. In this context, the originality and interest of Clémentine Gozlan’s book lie in its consideration of researchers’ role in the development of managerial instruments and policies and its examination of how these instruments are made, rather than simply focusing on the reforms’ effects.

For researchers, evaluation is a familiar activity. Through peer evaluation of publications and evaluation of laboratories and research centers by specific authorities, it is woven into research practices. Even so, the creation in 2007 of the Agence d’Évaluation de la Recherche et de l’Enseignement Supérieur (the Agency for the Evaluation of Research and Higher Education, or AERES) triggered considerable debate and reticence on the part of the academic community, which Gozlan attributes to major changes in the governance of scholarly work.

Starting from this insight, Gozlan studies AERES, in the five years it existed (2007-2012), as a standpoint from which to observe controversies and power struggles in the academic community—a place where professional norms and judgment practices are constructed. To do so, her study focuses on one of AERES’ three sections—devoted to the evaluation of laboratories and research units—and on the “social and human sciences” (SHS), domains that are under-studied by the history and sociology of science (Lamont, 2009). In these fields, the introduction of new modes of evaluation proved the most controversial (Aust, Gozlan, 2018), which Gozlan ascribes to the fact that certain evaluation instruments seemed contrary to conceptions of research quality (because their criteria included, for example, project financing and rankings). Gozlan’s study consists of in-depth interviews and direct observation of the section’s work, in addition to qualitative and quantitative analyses of many written sources.

The stakes of this book are many. We will dwell on three: the conception of evaluation tools; challenges to these tools; and their appropriation by evaluators.

The agency and its evaluation tools

The study makes it possible to get beyond the idea that AERES was simply the result of a reform imposed on professors “from above,” even if the agency was a political initiative. While actors from outside the academic profession (managers, top civil servants, consultants, etc.) were involved, Gozlan shows that faculty engaged in both teaching and research were central to the process. Consequently, the agency was neither a “relay of the state, governing the academic community it” (p. 42), nor an entity entirely under political control.

The contribution of this argument is that it transcends the inside/outside-the-profession dichotomy, as well as the scholar/manager alternative, by carefully describing the relationships between different internal actors in AERES (administrators, scientists, and quality experts [1]) and their varying ability to influence the definition of evaluation procedures.

A graph of the trajectories of forty social and human science representatives and of various coordinating representatives who served between 2007 and 2011, made possible by a multiple correspondence analysis (MCA) [2] sheds lights on their similarities and differences. Gozlan identifies three types: the “academic” (who participates little in the profession’s governing bodies), the “university leader” (with responsibilities at the level of the campus, research center, or field), and the “political-administrative leader” (with responsibilities at the national or political level). Participants in the elaboration of evaluation procedures tend to be colleagues rather than superiors: they are neither experts from outside the academic world nor former colleagues who have permanently left the academic profession and its professional structures. Gozlan calls them “middling elites” to refer to their position midway between the leadership class and rank-and-file professionals, as they continue to be researchers or teacher-researchers, in practice as well as in their own self-image.

Gozlan next considers why some professors wish to participate in the elaboration of AERES judgment tools (that is, the standards, norms, charters, and best practices in evaluations, from preparing a visiting committee to writing an evaluation report). Once the bureaucratization of academic policies has been situated in a longer timeframe, two issues appear to explain this trend towards rationalization. First, it responds to criticism of existing evaluation authorities, who are suspected of being too close to the people they are evaluating and having conflicts of interests. Second, the desire to establish valid standards irrespective of disciplinary specificities and demand for homogeneous and comparable evaluation reports is based on a conception of procedural justice that implies that “equity of judgment [is] guaranteed by the scrupulous respect of transparent and identical procedures for all” (p. 82).

Yet despite the efforts of the agency’s directors to implement these standards, the concrete work of AERES employees consists rather in a form of bricolage—a process of trial-and-error and internal reflection. Bureaucratization leads to ideas about the “right way to evaluate” (p. 80)—that is, the capacity of evaluators to honor pre-established rules and norms. This attitude is not always well received.

Reinventing and contesting evaluation tools

Using two case studies, Gozlan shows that the capacity to contest evaluation instruments is unequal. This is due to several factors: the characteristics of critics, the ability to build a common oppositional position and pursue legitimate routes, the characteristics of the instruments, and proximity to professional ideals and practices.

The first case concerns mobilization against the classification of journals by the individuals representing, to AERES, literary research (the chairpersons of learned societies, the sections of the National Committee of Universities, and the National Committee of the CNRS [France’s main research institute]; journal editors in affected fields; and various academic officials at AERES). Gozlan explores the motivations of this criticism: by challenging the objectivity and legitimacy of this classification, the representatives sought to re-politicize the instrument and highlight its multiple biases; by calling attention to their field’s particularities, they opposed the classification and demanded professional autonomy. This discursive work was not enough to successfully oppose the initiative, which was the result of “power relations between competing governmental and professional institutions” (p. 129). The disciplinary representatives insisted on their own expertise and managed to appeal to influential contacts to contest the implementation of the classification, with the help of representative bodies (including learned societies such as the French Society for General and Comparative literature, for example) which had become “essential to the creation of educational and research policies” (p. 133), if only because some of them design the curriculum for the agrégation (a prestigious state-sponsored competitive examination).

The second case study considers an internal consultation process in 2011 that took place over several months, in which Gozlan participated and which resulted in evaluation guidelines defining “the scholarly conditions to which researchers’ writings must adhere to be considered scholarly” (p. 143). The book recounts the process for establishing evaluation norms, defining scholarship in the very course of its development, determining scholarly value, and deciding “what ’doing research’ means” (p. 147). A first effort at defining the boundaries of scholarly production resulted in the distinction between scholarship and non- scholarship (that is, vulgarization, creative endeavors, and activism).

This posture consisted of remaining neutral on the content of academic work, while constructing an instrument allowing for a formal and procedural evaluation of its scholarly character, acknowledging the wide range of rules for evaluating scholarly production in the social and human sciences. In this way, AERES sought to declare its independence from scholarly controversies that are unique to each field and avoid favoring a particular practice or conception of academic work, to the detriment of minority or marginal views. Gozlan does note, however, that if the guidelines do not judge the quality of a research project’s results, the protocol’s formalization nonetheless impacts publication practices and “the laws of scientific recognition” (p. 176).

The appropriation of evaluation tools

Finally, the evaluation process and its performative effects are analyzed by comparing two fields, literary studies and geography, through several methods (quantitative studies of AERES reports, interviews, written sources, and the observation of a visiting committee—that is, peer groups that visit research units and teams to meet personnel and evaluate their work). This comparison is productive, as it shows how different instruments are appropriated: while literary scholars disparage AERES’ tools, geographers use them much more systematically. Gozlan attributes this to these fields’ respective positions in the academic world: geography is a “dominated” field (Bourdieu, 1984, p. 145), whereas in the human and social sciences, literature lies at the high end of the prestige scale (Bourdieu, 1984). This field benefits from far more recognition in France than in the United States (Duell 2000), as it does not seem to face the legitimacy crisis experienced by geography faculty. Another explanatory factor relates to whether reforms are perceived as being in a field’s interest: expectations of collective work and international publications align with the organization of academic work in geography but are seen as burdensome in literary studies.

Conclusion

Gozlan concludes that the establishment of AERES resulted in a “redistribution of the power to articulate the norms that count in the academic community” (p. 224). AERES’ tools reform scholarly practices “from a distance,” since they tend to target procedures rather than scholarly quality.

The book provides material for understanding the debates that accompanied AERES’ creation. Its interest is not confined to scholars working on these questions, as it opens onto many important issues that help one to understand and think about our system of research, higher education, and scholarly work. Though it deals with the years 2007-2012, the questions it raises are all the more contemporary in that the new guidelines by the Haut Conseil de l’Évaluation de la Recherche et de l’Enseignement Supérieur (High Council for Evaluation of Research and Higher Education, or HCERES, which replaced AERES), released in November [2021] also provoked vigorous reaction on the part of the Assembly of Laboratory Directors, which denounced the trend towards extreme bureaucratization that was undermining collegiality and the academic mission of the evaluated units.