|
Program evaluation is essentially a method to determine if a program 'works'. That is, does the program do
what it is supposed to do.
Behind this seemingly simple question are a host of other more complex questions. For example, the first question is, what is
a program supposed to do? It is often difficult to define what a program is supposed to do, so indirect indicators may be used
instead. For example schools are supposed to 'educate' people. But what does 'educate' mean? Give knowledge? Teach how to think?
Give specific skills? If the exact goal cannot be defined well, it is difficult to indicate whether the program 'works'.
Another question about programs is, what else do they do? There may be unintended or unforseen consequences of a program. Some
consequences may be postive and some may be negative. These unintended consequences may be as important as the intended
consequences. So evaluations should measure not just whether the program does what it should be doing, but what else it may be
doing.
Perhaps the most difficult part of evaluation is determining whether it is the program itself that is doing something. There
may be other events or processes that are really causing the outcome, or preventing the hoped for outcome. However, due to the
nature of the program, many evaluations cannot determine whether it is the program itself, or something else, is the 'cause'.
One main reason that evaluations cannot determine causation involves self selection. That is, people select themselves to participate in a program. For example, in a jobs
training program, some people decide to participate, and others, for whatever reason, do not participate. It may be that those
who do participate are those who are most determined to find a job, or who have the best support resources, thus allowing them to
participate. The people who participate are somehow different from those who don't participate, and it may be the difference, not
the program, that leads to a successful outcome for the participants, that is, finding a job.
If programs could, somehow, use random selection, then they could determine causation. That is, if a program could randomly assign people to participate or to not participate in the
program, then, supposedly, the group of people who participate would be the same as the group who did not participate, and an
evaluation could 'rule out' other causes.
However, since most programs cannot use random assignment, the outcomes of the program can be described, but causes cannot be
assigned. So for example, the evaluation can say something like, "People who participate in program xyz were more likely
to find a job, while people who did not participate were less likely to find a job." The evaluation, however, cannot clearly show
that the program was the cause.
If the program is fairly large, and there are many participants, and there is enough data, statistical analysis can be used
sometimes to make a 'reasonable' case for the program by showing, for example, that other causes are unlikely.
Another approach is to use the evaluation to analyse the program process. So instead of focusing on the outcome (for example,
did people in a jobs training program get jobs), the evaluation would focus on what the program was doing. For example, did
people seem to learn the skills being taught? Did people stay in the program or did they drop out part way through? Were the
teachers teaching appropriate skills? And so forth. This information could help how the program was operating.
People who do program evaluation can come from many different backgrounds, such as sociology, psychology, economics, social work or many other areas. Some graduate
schools also have specific training programs for program evaluation.
Program evaluations can involve quantitative methods of
social research or qualitative methods or both.
External Links
|