The language of human rights permeates discourses on international policy, development, foreign assistance, and social justice. Concomitantly, political pressure grows to hold policy makers and corporate actors to account for the human rights consequences of their decisions. Faced with growing investments in human rights and persistent flagrant violations, researchers, practitioners, policy-makers and funders insistently ask whether programs and policies actually effect the changes they are designed to promote and what unintended consequences they engender. Human rights organizations face demands for accountability; scholars, increasingly focused on human rights as a field of inquiry, re-read disciplinary debates as they gauge the way in which particular policies or interventions do or do not work and why; international organizations and governmental institutions simultaneously take steps to mainstream human rights and inquire into the effectiveness of the measures undertaken.
Stakeholders debating the value-added of investments in human rights and those seeking to establish accountability for violations require the capacity to assess human rights, determine baselines, define indicators, determine standards against which to validate data and measure change. In fields contiguous to human rights — and increasingly interpenetrated within it — such as development and humanitarian assistance, impact assessments constitute a well-established practice. Yet, despite burgeoning attention of practitioners and academics, there is little agreement on how methods devised for other contexts may be applied to the evaluation of human rights policies and programs or how human rights-specific evaluations can be conducted and their results interpreted.
To address the “question of impact,” in April 2009 the then Center (now, Institute) for the Study of Human Rights at Columbia University convened a workshop that drew together researchers from numerous disciplines (anthropology, economics, political science, history, sociology and law) and advocates from major human rights organizations. That workshop highlighted several key questions (summarized below), including the appropriateness — for specific ends — of particular research methodologies, the difficulty of controlling for the effects of monitoring on the conduct (and hence impact) of interventions, the complexity involved in identifying and measuring unintended consequences, and the divergent — and sometimes conflicting — approaches to impact assessment of activists, academic researchers, policy-makers, and funders.
The April 2009 workshop addressed the impact of human rights programs and policies in general terms. But “human rights” is a highly articulated field, divided into varying kinds of organizations — domestic and international advocacy and service-delivery organizations, think tanks, public administrations, corporations, academic institutions as well as international organizations — and operating in many arenas, from children’s rights to the promotion of transparency. That organizational frameworks and issue clusters affect the ways in which human rights are defined, impact is assessed, and the appropriateness of particular methodologies is determined appears intuitively evident, but is in fact a hypothesis that remains to be tested. What is known in relation to impact within particular sectors? Do the issues raised by our April workshop and in other settings assume different configurations in relation to different sectors? To address these questions, this symposium will focus on four distinct sectors: the freedom of the press; women's rights; international criminal courts; and, development, in particular in a rights-based perspective.
Moreover, as the April workshop and other debates have highlighted, both sectoral issues and general themes relating to evaluation and impact assessment often appear differently to differently situated actors. The cost-benefit analysis of conducting an evaluation when resources for interventions are perceived to be urgently needed, the time-frames within which results are to be expected, the "thickness" of the descriptions gathered, the recourse to narratives and testimonies, the attention to the singularity of a particular case in a given context in contrast to the isolation of measurable traits across a sample that allows comparability, the evidentiary value of any datum and, indeed, the definition of what does or does not constitute data may vary depending on whether one is an advocate or a policy-maker, a donor or an academic researcher. Even the identification of the results of one intervention may vary: where advocacy generally aims to change policies or conditions on the ground, in the first instance it often proceeds by drawing attention to a particular situation. The salience gained — in the eyes of opinion leaders, policy makers, political gatekeepers, media, and public opinion generally — constitutes an objective of an intervention and not simply an incidental result. Analogously, gains in legislative and policy frameworks — the signature and ratification of a treaty, the decision to establish a tribunal — often inform the emergence of constituencies which press for implementation and alter the political landscapes in which human rights issues are negotiated. But if policy-makers, donors or scholars focus evaluations on immediate results then the extent to which opinion change has been effected or the bases for social mobilizations have been strengthened may not be registered. The challenge is to establish fruitful dialogue among interlocutors while respecting divergent points of view. In short, one must ask whether and how scholars' analyses can help inform advocates' choices; how advocacy can lead to a reconstruction of theoretical frameworks; how the lessons policy-makers derive may be of use to advocates, donors or scholars. To help nurture this conversation, the symposium convenes advocates and academics, policy-makers and donors.
But underlying all sectoral inquiries remain several general questions, which require collective, multi-perspectival, consideration. The symposium will focus on two such questions: how can human rights be measured, and appropriate indicators established; and, what are the ethical issues that impact assessments must confront and how can they be addressed. The measurement of human rights, and hence any assessment of change, is naturally dependent on the interpretation given to such rights and the identification and weighting of factors related to these interpretations. Consensus in these areas appears provisional at best. But alongside definitional matters, practical considerations also limit approaches to measurement and the development of indicators. As has long been recognized, data collection is shaped by numerous factors, including governmentally (or inter-governmentally) generated knowledge demands, prevailing scientific paradigms and strategic choices regarding data collection techniques. Ultimately, research "on the ground" requires local negotiations over meanings which, inter alia, bring into play respondents' perceived obligation to supply (or data-gatherers' perceived right to demand) information. Furthermore, if the experience gained in the course of an intervention changes the advocates', or the policy-makers', own understandings of the problems to be dealt with, the methods to be utilized, and even the meanings of the intervention as initially defined, how is such change factored into an evaluation? Is this change itself — perhaps best posited as the result of a functioning feed-back loop, however informally structured — not an objective of any intervention? But if objectives and methods change, how can evaluations be effectively conducted? How, then, do such definitional variations and fluctuations and implementation constraints affect the measurement of human rights?
Ethical issues are similarly critical to evaluation and impact assessment. Methodologies involving randomized samples, are not only difficult to apply and may not be contextually appropriate, they also raise such questions as the justice of either withholding benefits from control populations (in relation to which a particular intervention has not been effected) or exposing pilot communities to potential harm before risks have been fully assessed. But this is just one of many issues that range from the potential endangerment of informants to the fractured loyalties that may be involved when impacts are assessed in relation to the differing needs of an intervention's beneficiaries, funders, or implementers. How can evaluations and impact assessments neutralize potential biases introduced by their own reliance on external funders or adherence to particular theoretical and methodological frameworks? And to what extent have participatory and other methodologies been successful in transforming the relationships between "intervenors" and "intervened-upon" into ones more closely resembling parity, including in reference to the establishment of accountability for the consequences of interventions and the evaluation of their impacts? Finally, the question of whether evaluation is "worth" the resources devoted to it in the face of the urgency of so many situations in which human rights violations are at issue underlies many difficult discussions, especially for practitioners. What principles are at stake when the decision to "do the right thing" even in the face of likely failure is weighed against the decision to husband resources as one awaits a more propitious moment or a "better" idea? What principled approaches can guide the choice of projects and beneficiaries and how should the evaluation of any single — or even any sample — program or policy affect such choices? What does — and should — evidence-based human rights entail?
Ultimately, the "question of impact" goes beyond the purpose, methodology or objectivity of any one evaluation. Rather, it leads us to ask: who needs to know what, when, in the context of which interpretive framework, using what evidentiary language, and so that what kind of lessons can be gleaned?
The symposium will convene a broad spectrum of participants — academics, policy-makers, advocates, and donors — to examine the ways in which the impact of human rights policies and programs is defined and assessed. Sessions will focus on a few sectors and on two cross-cutting themes: the measurement of "human rights" and the development of appropriate indicators, and the ethical issues that arise when impact analyses are structured into human rights interventions. In terms of sectors, discussions will center on development (in particular in a rights-based perspective), women's rights, international courts, and press freedom. Through both the sector-based and cross-cutting thematic sessions, we hope to foster conversations among participants addressing the same general topic from different perspectives and to ground discussion of methodological issues in empirical analysis. We also hope that taking a sectoral approach can help illuminate whether (and how) sectors matter to impact analysis.
Summary of questions raised by the April CSHR workshop on the impact of human rights policies and programs:
Goals and motivations: What are the goals of impact assessment and how do they differ among sectors and organizations? What are the underlying factors that influence impact assessment initiatives? For example, for practitioners, the need for evaluation is immediately situated in operational contexts: knowledge applicable to strategic decision-making, programme development, and implementation is key. Studies that do not meet these criteria may fail the cost–benefit calculus of allocating scarce resources for urgent interventions vs. ‘diverting’ them into evaluation. For researchers, questions about impact often arise from disciplinary debates. The underlying aim (and the form that the ultimate product takes) is less a contribution to immediately practical knowledge than to further scholarly endeavours — although ‘translation’ from one realm to the other is often an important objective.
Methodological approaches: What are the most appropriate methodologies? Can and should methodologies developed to analyze interventions in one general field, for example humanitarian assistance or development, be applied to impact assessments relating to human rights advocacy or service-delivery NGOs?
How do methodological approaches and challenges vary among actors (scholars, practitioners, donors, etc.)? Do such actors conduct impact assessments with different goals and needs in mind, and if so how do such differences affect their methodological approaches?
Data and units of analysis: How can one define appropriate indicators and gather baseline data? What issues are raised by the availability of quantitative v. qualitative data? What are the most appropriate units of analysis (global, international, regional, national, or local).
Causation and attribution: How can one trace outcomes to a specific intervention?
The definition of time horizons: The effects of interventions may become evident only years after they are undertaken, but the longer the timeline between the intervention and the measurement of its impact, the more complex the issues of causation and attribution appear. Moreover, longer time-lines may not meet donors' reporting requirements or practitioners' own needs to assess their strategies.
The circular effects of monitoring: The decision to monitor and evaluate a particular intervention may affect the conduct of the intervention itself.
The ‘information paradox': once monitoring of impacts is established, more phenomena that correspond to a particular description may be registered.
Unintended consequences of human rights interventions: Incorporating into evaluation and impact assessment methodologies collateral effects and unpredicted consequences requires monitoring of areas which may not fall within the scope of the original interventions. What are the potential negative effects raised by and ethical concerns associated with impact assessment? How are these measured?