The evaluation received the 2012 outstanding evaluation award from the american evaluation association. The text further deals with one form of metaevaluation. The metaevaluation confirms the need to reorganise the responsibilities and tasks of the evaluation advisors in the mela team, in conformity with the draft of a recently developed mela framework. Metaevaluation example metaevaluation of quality and coverage of usaid evaluations 20092012 executive summary definition of u. Use the kec to generate a new evaluation or apply the checklist to the original evaluation as a product. Metaevaluation is the process of evaluating another evaluation process, thereby concreting the results and removing potential problems which would arise from a study. A detailed description of how an evaluation will be done. The contribution of metaevaluation to program evaluation sfu. He lists the parts a kec involved in a meta evaluation. This metaanalysis is the first exercise to examine the adequacy and synthesized findings of a large number of unicefsupported evaluations in the area of nutrition. Metaevaluation can be done formatively or summatively or both. In this sense, metaevaluation should be part of an open dialogue between various parties in the evaluation process.
This is to ensure a systematic and structured approach to the metaanalysis. This metaanalysis report was prepared by the independent evaluation unit ieu of the united nations office on drugs and crime unodc. Dp173 the use of metaanalysis in economic evaluation. The success case method scm involves identifying the most and least successful cases in a program and examining them in detail. In line with the gbv aors mandate, the primary focus of this study was on settings with internallydisplaced persons idps. There are a variety of evaluation designs, and the type of evaluation should match the development level of the program or program activity appropriately. The historical roots of metaanalysis can be traced back to 17th century studies of astronomy, while a paper published in 1904 by the statistician karl pearson in the british medical journal which collated data from several studies of typhoid inoculation is seen as the first time a metaanalytic approach was used to aggregate the outcomes of multiple clinical studies. Metaevaluation example this evaluation of evaluations, or metaevaluation, was undertaken to assess the quality of usaids evaluation reports. Exante evaluation aims to ratify, to research or to emit a precise. Metaevaluation, as defined in this paper that is the evaluation of evaluation, different from evaluation synthesis, has a first meaning as. Mipds also define specific objectivespriorities at a lower level which, unlike strategic objectives, are potentially measurable. Metaanalysis is a statistical procedure for analyzing the combined data from different studies, and can be a major source of concise uptodate information. Metaevaluation is the professional imperative of evaluation. The overall conclusions of a metaanalysis, however, depend heavily on the quality of the metaanalytic process, and an appropriate evaluation of the quality of metaanalysis metaevaluation can be challenging.
This approach was developed by robert brinkerhoff to assess the impact of organisational interventions, such as training and coaching, though the. Glasss approach standardizes various effect measures and. Meta analysis refers to a process of integration of the results of many studies to arrive at evidence syn thesis normand, 1999. Meta definition is showing or suggesting an explicit awareness of itself or oneself as a member of its category. Program eval models checklist western michigan university. Metaevaluation as a tool for learning sage journals.
Failure to specify the underlying questions and criteria guiding an evaluation remains problematic. It is considered as specific evaluation of the operation and result of all assessment processes that take place in higher education institutions. It can and should be done in the first place by an evaluator on his or her own work. United nations office on drugs and crime vienna international centre p. The independent evaluation unit of the united nations office on drugs and crime can be contacted at. Metaanalysis is an analytical tool that permits the evaluation of a diagnostic or therapeutic modality through the appropriate use of previously published smaller studies. From meta, a wikimedia project coordination wiki definition. This evaluation of evaluations, or metaevaluation, was undertaken to assess the quality of usaids evaluation reports. The purpose of the metaevaluation was to provide an overall. This is a resource file which supports the regular public program areol action research and evaluation on line offered twice a year beginning in midfebruary and midjuly. It includes thematic areas of methods, reporting, reproducibility, evaluation, and incentives how to do, report, verify, correct, and reward science. Using metaanalysis and data visualization to inform policy inspired by metaanalysis metaanalysis is a technique for creating a quantitative summary of the evidence surrounding intervention effectiveness across studies.
The evaluation also makes the case that the lack of strong watsan was a downfall of orad ii, and did. Metaevaluation a synthesis of evaluation studies 2005 2016 december 2016 evaluation series no. This paper examines the practice of metaevaluation as indicated by. Metaanalysis involves the summarization of research findings to come to some general conclusions regarding effects or outcomes of a given treatmentprojectprogram. Afterwards, they define the terms evaluation and metaevaluation. The study builds on usaids practice of periodically examining evaluation quality to identify opportunities for improvement. What is a metaanalysis in 1976, glass coined the term metaanalysis metaanalysis refers to the analysis of analyses the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.
This variance explicitly describes the extent of the heterogeneity and has a crucial role in assessing the degree of consistency of effects across studies, which is an element of randomeffects metaanalysis that often receives too little attention. In its metaevaluative use, the main and perhaps only source of information is the evaluation report itself. Types of evaluation centers for disease control and. When the treatment effect or effect size is consistent from one study to the next, metaanalysis can be used to identify this common effect. Metaresearch is an evolving scientific discipline that aims to evaluate and improve research practices. In 1976, glass coined the term metaanalysis metaanalysis refers to the analysis of analyses the. Keep the evaluation open to serve newly identified stakeholders 6 excellent. An evaluation plan sets out strategies to systematically collect the information needed to tell the story of a program and its success this can include who is involved in a program evaluation a designated member of a wikimedia user group might be the leader in this evaluation, or volunteers who produced an editathon might work. Metaanalysis, metaevaluation and secondary analysis. This mfa metaevaluation covered both aspects of this definition. Metaevaluation of usaiduganda evaluation reports page 6 metaevaluation of usaiduganda evaluation reports executive summary and recommendations uganda monitoring and evaluation management services umems is a fouryear usaiduganda project 20082012 whose purpose is to provide a comprehensive performance. Using metaanalysis and data visualization to inform policy.
It also involves an evaluation of the quality of this series of evaluations and its adherence to established good practice in evaluation. Thus, the dashboard will gather only the information reflected in that report. Development of a protocol it is generally recommended as with any scientific endeavour, that the methods of the metaanalysis should be established beforehand in the form of a protocol. Several aspects of alnaps meta evaluation are similar to other metaevaluation initiatives, for. Much of this data may not be clearly stated in the report, and so will have to be searched for throughout the entire document, analysing the evaluators. Agency for international development usaid an independent federal agency of the united states that provides aid to citizens of foreign countries. Metaanalysis is the statistical procedure for combining data from multiple studies. This definition includes evaluative activities that come before the implantation of a program. The evaluation looks at how the principles of aid effectiveness have been put into practice by international development partners and the. Apart from providing synthesized findings against evaluation key evaluation criteria and cross.
Metaevaluation a critical assessment of the strengths and weaknesses of the evaluation itself e. Types and uses of evaluation in order to plan the evaluation in accord with the most appropriate evaluation method, it is necessary to understand the difference between evaluation types. Section 12 provides conclusions and implications for research and practice on performance auditing of tourismmarketing programs, as well as effective strategies for managing such programs, are provided. Review evaluation do metaevaluation better evaluation. A metaevaluation is an instrument used to aggregate findings from a series of evaluations. Users of this checklist are advised to consult the full text of the joint committee 1994 program evaluation. When the effect varies from one study to the next, metaanalysis may be used to identify the reason for the variation. Such an example is the metaevaluation of the evaluation of the paris declaration on development aid. The metaevaluation assessed the quality of 45 evaluative reports published since 2004. The metaevaluation will feed into management planning by providing a better understanding and insight of performance areas requiring management attention and will help with the development of the next ocha strategic framework 201020. The major addition to the joint committee standards for evaluation, when revised in 2010, was the addition of evaluation accountability standards focused on metaevaluation. Metaevaluation of quality and coverage of usaid evaluations. Meta evaluation are food and nutrition security teamsector descriptor hereare putting evaluations into practice. Metaevaluation and analysis of project evaluations 2016.
The metaevaluation dashboard visual brains by sara vaca. Metaanalysis, metaevaluation and secondary analysis are methods of summarizing, examining, evaluating or reanalyzing data in research and evaluation efforts. Some use it to describe aggregating information from several individual evaluations, others define it as a systematic tool. Metaevaluation example concurrent metaevaluation a critique 12. Evaluation metaevaluation of project and programme evaluations in 20122014 geraldine cooney and katrina rojas mariane arsenault kelly babcock 20153 this evaluation was commissioned by the ministry for foreign affairs of finland to the universalia management group ltd. Meta analysis his study adopted a mixed methods approach, including an analysis of multiple quantitative data sources and 45 key informant interviews. Metaanalysis is a technique for creating a quantitative summary of the evidence surrounding intervention effectiveness across studies. Arrange to involve stakeholders throughout the evaluation, consistent with the formal evaluation agreement.
550 199 1103 1082 876 437 834 1287 263 1014 1386 374 555 262 1516 64 339 1419 186 1264 1339 1190 854 1136 719 475 553 1533 403 286 1215 1391 357 10 105 160 821 83 1106 1285 557 299 1244 506 932 883 768