Tracing the evolution of evaluation using the evaluation tree

Anyone can do program evaluation. It can be as simple as distributing a survey to participants to get their feelings towards a program, or as complicated as tracking the implementation and outcomes of a program over many years. However, to become an evaluator, one needs to understand evaluation theories.

An evaluation theory is a set of guidelines that describe what a good evaluation is. Different evaluation theories provide different perspectives of how evaluation should be conducted. Selecting one or multiple evaluation theories can guide us on what methods and processes are appropriate for a specific evaluation. One framework which has helped me sort these evaluation theories is the evaluation tree by Marvin C. Alkin.

In this blog post, I will explain what the evaluation tree is and how it is useful for learning how to conduct an evaluation.

Introducing the evaluation tree

Marvin C. Alkin is a Professor Emeritus of Education at the University of California, Los Angeles (UCLA). After receiving his doctorate in education from Stanford University in 1964, he moved to UCLA where he has been working in the Graduate School of Education and Information Studies ever since. He is best known for his research on the use of evaluations and comparing different evaluation theories. He, along with Christina A. Christie, first introduced the evaluation tree as part of the book Evaluation Roots in 2004. The evaluation tree has been updated numerous times, with the latest iteration published in 2023 as part of the third edition of Evaluation Roots. 

The evaluation tree is a conceptual framework that highlights the foundations of evaluation and sorts different evaluation theories into three groups (represented by the three branches) depending on which part of program evaluation they emphasise the most: 

1. Methods: The study design describing how the evaluation will be conducted.
2. Valuing: How and by whom evaluations can make judgements on a program.
3. Use: How the evaluation will be utilised.

These groups are interrelated to each other, with each evaluation theory being placed in a primary branch and aligned to a neighbouring branch. For example, evaluation theories that are placed to the left of the valuing branch are also closely aligned to the methods branch.  

The foundations of evaluation

The evaluation tree is supported by a trunk describing three foundations of evaluation. These foundations provide the “why” and the “how” of conducting any program evaluation.

Evaluation foundation #1: Social accountability

The first foundation is social accountability. Evaluation provides information on what has been accomplished as a result of implementing the program. Policymakers and program managers can use this information to make decisions on their programs, while advocates and end-users can use the same information to hold decision-makers accountable. According to Alkin (1972), there are three types of accountability, each of them covered by an evaluation type:

1. Goal accountability which assesses whether appropriate goals and objectives of the program were set. This is associated with formative evaluation which is designed to generate a program theory, a description of how a program should work to produce outcomes. 
2. Program accountability which assesses whether appropriate procedures have been implemented as intended to work towards the program goals and objectives. This is associated with process evaluation which assesses how the program is being implemented and whether that is following the plan. 
3. Outcome accountability which assesses whether and how much the goals and objectives of the program have been achieved. This is associated with summative evaluation which measures the short-, medium- and long-term outcomes of a program.

Evaluation foundation #2: Social inquiry

The second foundation is social inquiry. This describes the application of social research philosophies and methods to study how people behave in social groups and settings. This can range from quantitative studies that assess whether a program is effective in producing the desired outcomes to qualitative studies that explore the experiences of people participating in a program. 

Evaluation foundation #3: Epistemology

Relating to social inquiry is epistemology, the third foundation of evaluation. Epistemology describes what counts as knowledge and how reality can be interpreted. There are three main schools of thought in epistemology:

1. Post-positivism: The goal of post-positivists is to measure truth within a single reality with some degree of uncertainty. Post-positivism is associated with identifying and controlling values and biases to study the link between causes and outcomes as cleanly as possible.
2. Constructivism: Constructivists believe that there is not one single reality, but rather multiple realities that are ‘constructed’ from the subjective beliefs of individual people. In contrast to post-positivists who try to control bias, constructivists embrace bias to consider how different people may view a program.
3. Pragmatism: Pragmatists sit in the middle between post-positivists and constructivists. While they agree with constructivists in that there are multiple views of reality, they also align with post-positivists in identifying a view that is most aligned to what is happening in reality. 

The next few sections will describe each branch in detail and highlight, in my opinion, some of the most prominent evaluation theories in each branch, along with the principal people behind the theories.

The methods branch

Emerging from the social inquiry foundation of evaluation, the methods branch is concerned with applying appropriate research methods to evaluation in order to generate knowledge. Most evaluation theories in this branch use or adapt experimental methods that were pioneered in the social sciences and applied research. Prominent evaluation theories in the methods branch include:

1. Experimental methods (Donald Campbell): Campbell is best known for introducing experimental and quasi-experimental designs into social science research and evaluation. These study designs aim to reduce bias when studying the effects of programs on outcomes. The difference between the two study designs is that while participants are randomised into a control and treatment group in experimental designs, no such randomisation occurs in quasi-experimental designs. 
2. Quasi-experiments (Thomas Cook): Cook extended the concept of quasi-experimental designs in evaluation, arguing that evaluators need to take the context of the program and the views of stakeholders into consideration when planning quasi-experimental designs (which can range from pre/post-tests to comparing groups without randomisation).
3. Tailored evaluation (Peter Rossi): Tailored evaluation contends that research methods should be adapted to what is currently happening in a program. For instance, if a program was currently running, the evaluation would focus less on designing the program and more on measuring what is currently happening. 
4. Theory-driven evaluation (Huey-Tsyh Chen): Rossi and Chen developed the idea of theory-driven evaluation, where a program theory is designed first describing how a program should work. This identifies potential areas of investigation for an evaluation to measure intended outcomes, as well as identify unexpected consequences. 
5. Evidence-based policy use (Carol Weiss): This theory stresses the importance of running a methodically sound evaluation to produce results that are reliable and generalisable. It also introduces the notion that evaluation is a political activity, where the political context of a program, including vested interests, negotiation, supporters and critics, influences how evaluation results are interpreted and received.

The use branch

The use branch is concerned with producing evaluations that are not only used to inform decisions about a program, but also make an impact to decisions and changes seen in a program. Evaluation theories within this branch emphasise who and how evaluation results will be used to ensure evaluation is utilised in a meaningful way, as well as empowering individuals to conduct evaluations themselves. Prominent theories in the use branch include:

1. CIPP model (Daniel Stufflebeam): Standing for Context, Input, Process and Product, the CIPP model incorporates evaluation in the program design process to continually provide decision makers with information to improve their programs. The CIPP model is also associated with the establishment of a representative stakeholder panel that works with the evaluator to tailor the evaluation to their needs. 
2. Utilisation-focused evaluation (Michael Patton): One of the most prominent evaluation theories in the use branch, utilisation-focused evaluation stresses the need to identify primary intended users who are likely to use the evaluation or to have a stake on the results generated. From there, the evaluator engages primary intended users at all stages of evaluation to foster buy-in to use the evaluation results. Evaluations in utilisation-focused evaluation are adaptive, with evaluation questions and designs altered in the face of changing conditions and results generated. 
3. Learning-oriented evaluation (Hallie Preskill and Rosalie Torres): The goal of learning-oriented evaluation is to motivate individuals, teams and organisations to learn from the evaluation. The evaluator acts as a facilitator to identify needs within the organisation and to guide staff to learn from the evaluation, given the organisation’s capacity for learning.  
4. Interactive evaluation (Jean King): Interactive evaluation guides the evaluator to continually engage stakeholders, create a participatory environment and foster community leaders in order to build trust, increasing the chances of evaluation use.  

The valuing branch

Deriving from the epistemology foundation of evaluation, the valuing branch is concerned with how we judge programs. The valuing branch can be split into two sub-branches, with different epistemologies influencing each one. 

The first sub-branch is objectivist-influenced valuing which is aligned to the methods branch of the evaluation tree. Theories in this sub-branch focus on the evaluators being able to make objective judgements about a program based on what the evaluation has found. Prominent theories in the objectivist-influenced valuing sub-branch include:

1. Goal-free evaluation (Michael Scriven): This theory emphasises the role of the evaluator in deciding which program outcomes to examine and presenting a single value judgement of whether a program is good or bad based on their own set of criteria. 
2. Educational connoisseurship (Elliott Eisner): The evaluator is positioned as a connoisseur, similar to an art critic. As a connoisseur, evaluators use their area of expertise to identify important areas of the program to evaluate and describe and make judgements on a program.
3. Responsive evaluation (Robert Stake): Responsive evaluation advocates on the use of case studies that are tied to specific contexts. Here, the evaluator themselves collects and interprets the beliefs and values of stakeholders to provide a thick description of a program.

The second sub-branch is subjectivist-influenced valuing which is aligned to the use branch of the evaluation tree. Theories in this sub-branch reject the notion of a single reality, instead arguing that reality can be interpreted differently by individual people. Applied to evaluation, these theories focus on the stakeholders to understand what they value and how this affects the evaluation. Prominent theories in the subjectivist-influenced valuing sub-branch include:

1. Deliberative democratic evaluation (Ernest House): Placed in the perspective of social justice, this theory emphasises that evaluators have to be responsive to the needs of stakeholders and inclusive of those who would be powerless and disadvantaged during program evaluation.
2. Values-engaged evaluation (Jennifer C. Greene): Values-engaged evaluation highlights the idea of engaging with multiple stakeholders and the program context to define a set of stakeholder-driven criteria that determine the value of a program.
3. Fourth-generation evaluation (Egon Guba and Yvonna Lincoln): This theory pushes the responsibility of valuing a program to stakeholders who have different perspectives on the program. The evaluator considers the values of different stakeholders when planning and conducting the evaluation.
4. Transformative evaluation (Donna Mertens): Transformative evaluation emphasises the need to involve a diverse range of people in evaluation, particularly those from marginalised communities, and to tailor evaluations to challenge the status quo by being inclusive of, building positive relationships with and improving the lives of marginalised communities.

Which branch do I belong to?

Out of the three evaluation branches, I am most closely associated with the methods branch. That is because I place a huge emphasis on planning and conducting rigorous, ethical evaluations to produce results that end-users can trust to make decisions on programs. This has been derived from my science and public health training, where I not only have an understanding of experimental designs and qualitative and quantitative research methods, but a commitment to generate evidence to support my conclusions. As an evaluator, I am aligned to theory-driven evaluation as I create program logic maps to design evaluation frameworks and tools that measure how programs are being implemented and how effective they are in producing outcomes. I am also a fan of quasi-experimental designs due to its flexibility in designing evaluations in public health and education where randomisation is either impractical or unethical. Specifically, I use pre-post test designs within a group to more easily track changes in outcomes during the program. 

Looking at the use branch, I am gradually thinking more about how my evaluations can be utilised more by various stakeholders to make changes to their programs. In my work, I have started to engage with primary intended users to learn more about the programs they run and how my evaluations can be tailored to deliver useful information to them. I am also starting to think about how my evaluations can be widely disseminated to other people and groups and how I can be involved in implementing the recommendations that I have written to ensure that the evaluation is used.

Going to the valuing branch, I am not a huge fan of evaluation theories in the objectivist-influencing valuing sub-branch. Even though valuing programs is a central activity for evaluators, judging programs under my own set of criteria can be harsh. Furthermore, not involving stakeholders in planning and conducting the evaluation can reduce the chances of them being receptive to the evaluation findings and recommendations, instead increasing their resistance. In some cases, commissioners or program managers may not appreciate the judgements or recommendations I pass down on a program, particularly if they are heavily invested in a program and may not want to change course. 

On the other hand, I can see how evaluation has evolved under evaluation theories placed in the subjectivist-influencing valuing sub-branch. These theories emphasise the importance of getting stakeholders involved in the evaluation in order to understand their views. Often, the end-product is a set of criteria that is aligned to the values of stakeholders, leading to conclusions that would be considered fair to stakeholders. Consequently, these theories have made evaluation more inclusive of stakeholders, particularly those who historically have been underrepresented such as indigenous people. 

As evaluation continues to evolve, I see the increasing importance of engaging and working with stakeholders to design evaluations that not only balances rigour with context, but would also be useful to stakeholders. That is something I am developing in my line of work. 

Conclusion

Learning different evaluation theories is important because it provides some guidelines and principles of how to conduct a good evaluation. They provide the foundations for designing evaluations that are trustworthy, adapted to context and useful to stakeholders. The evaluation tree provides one framework in which various evaluation theories can be sorted into different groups. The parts of the evaluation tree work together to outline what makes a good evaluation.

In the process of doing research for this post, I can see how evaluation has evolved to be grounded in strong foundations, yet adaptable to the changing demands of stakeholders. I learnt that while I am most closely associated with the theories in the methods branch, there are other theories in the valuing and use branches that closely resonate with me. Writing this post to summarise the evaluation tree has sparked my interest in learning more about evaluation theories and how I can adapt them in my work. I may showcase some evaluation books and theories that I find interesting, and summarise them in my future blog posts.

Resources

Alkin, M.C., & Christie, C.A. (Eds.). (2023). Evaluation roots: Theory influencing practice (3rd ed.). Gilford Publications