One of the biggest debates that we have in the field of social science
research is the role of evaluation (e.g., evaluation of a policy, the completion
of treatment goals or of the program itself) and where independence sits within
it. Should organisations and governments conduct their own evaluations, or does
that create a conflict of interest? Is all independent research really
independent, or is it just independent of the organization/provider? How
independent is it of the researcher’s own biases? It is important to consider
these factors when deciding to do research, validate programmes, conduct
evaluation and implement change.
The place to start is asking the right questions: What is the purpose of
evaluation, research, and development of an evidence base; is it to justify
what is being done, a required part of a larger grant or project, to see if
what is being done is working, to justify an existing programme/venture or to
implement something new in a broader fashion? Is it about “evidence based
policy” [research driven] or “policy based evidence” [ideologically driven]? On
the other hand, is an organisation implementing an evidence-based practice or
developing practice-based evidence in earnest?
The reasoning behind the purpose of the research is as important as the
research being conducted, especially if the research is being used to justify
or implement something; therefore, is the research question trying to see if
something is working or if something is going to work in a different setting.
This means that evaluation can often usher in change to policies and
programmes, which can be minor or major but often seen as significant by people
on the ground. This can, in turn, bring about new questions: Should an
organisation implement a top-down approach or method, or integrate it in
accordance with the unique characteristics and contributions of those who work
and benefit from it? It is important that everyone involved in the process recognizes
the importance of evaluation in the evolution of our research and working; it’s
not a threat but rather an opportunity to see if something works, to identify
good practice and identify bad practice. Therefore, who should conduct
evaluation research; organisations themselves or external organisations/researchers?
It is generally viewed that good research is independent research as
organisations are often considered to be biased in how they evaluate their own
programmes and practices. The notion that organisations cannot conduct their
own critical, methodologically sound and unbiased research is outdated,
problematic and questionable. The issue is not with research or methodology,
but rather with how organisations understand, priorities and train their staff
in research. If staff do not fully understand research and its impact, then
their ability to develop, conduct, justify and roll out research becomes
problematic. This is a training issue and investment issue, not an independence
issue. This is reinforced by the fact that not all research and researchers are
independent with the same researchers being used by the same organisations
frequently and/or with research tending processes being subjective. Which
means that so called independent research can be as subjective and biased as
research conducted by the organisation itself; particularly, if the
organisations do not have their own in house understanding of research, their
own critical reflection of research and how to process it.
The processes involved in research involve interaction between the
researcher and the organisation, regardless of whether the research is
independent or not, with both needing to be able to work together to collect
the data and implement change. A decision not to engage in the research process
or implement changes is problematic regardless of who conducts the evaluation
and has nothing to do with independence, but rather it’s about quality control.
The most important part of doing evaluation research is the process and
methodology, it is about accepting the good and the bad on both fronts
[researcher and organisation]; it’s about us and us, not them and us.
Evaluation is about change and evolution, not revolution and condemnation.
Evaluation is about maintaining funding, generating funding, maintaining good
practice and eliminating bad practice. It’s a process, not an end point.
Kieran McCartan, PhD, & David Prescott, LICSW
No comments:
Post a Comment