Transcription
The Evaluation Processin 10 Steps –a Guideline
Published byBrot für die Welt –Evangelischer EntwicklungsdienstEvangelisches Werkfür Diakonie und Entwicklung e. V.Caroline-Michaelis-Straße 110115 Berlin, GermanyTelephone 49 30 65211 deAuthorsNicole Derbinski,Thomas ReinhardtEditorDietmar MälzerLayoutNORDSONNE IDENTITYBerlin, June 2017
The Evaluation Processin 10 Steps –a Guideline
Table of ContentsForeword 5Steps of an Evaluation 6Evaluation – Introduction 9Step 01Planning and Budgeting 14Step 02Terms of Reference (ToR) 16Step 03Obtaining Offers 23Step 04Selecting Evaluators 24Step 05Concluding the Contract 28Step 06Kick-off and Clarification Meeting 30Step 07Inception Report 33Step 08Debriefing/Presentation of the Results 35Step 09Assessment of the Final Report 37Step 10Dealing with the Results of Evaluations 41
ForewordEvaluations are becoming increasingly important in international development cooperation.External evaluations are used to provide accountability e.g. towards donors and – much moreimportantly – to learn from the experiences and assessments of external experts.In order for evaluations to satisfy both the requirements of the implementing organisation anddonors’ requirements, it is important that the process is properly planned and supported andthat the scheduled timescale is adequate.When do I start planning? What constitutes high-quality Terms of Reference? How do I assesswhether a report is good or bad? These are typical questions that arise during an evaluation process. These guides have been produced to provide employees and partner organisations of Brotfür die Welt – Evangelischer Entwicklungsdienst (Bread for the World – Protestant DevelopmentService) with practical assistance during the evaluation process.The evaluation process has been divided into ten steps. A guide has been produced for each ofthese process steps. The guides can be used independently of one another. There is also an introductory guide that contains basic information about the topic of evaluation. The data-collectionstep is performed by the external evaluators, meaning that there is no guide for this step.Each guide explains what needs to be considered during this step, why it is important and whoneeds to be involved. The aim is to provide important tips and assistance as to how the individualsteps can be implemented. The guides are specifically tailored to external evaluations, but theyalso contain relevant information for other evaluation forms. All guides are available in the fivelanguages of communication of Bread for the World – Protestant Development Service (Bread forthe World).The guides and their attachments/tools should be understood as aids: there is no obligation touse them. For details on Bread for the World’s stipulations regarding evaluation, please refer tothe document “Requirements for implementation of project evaluations” in the enclosure tocooperation agreement. However, please note that other donors may have other stipulations thatalso need to be taken into account.Bread for the World is always available for any questions and suggested improvements youmay have.5
Steps of an Evaluation01Planning and Budgeting0305Obtaining OffersConcluding the Contracton application/planning of the projectafter finishing the Termsfollowing the selection ofor as soon as the need is identifiedof Reference (ToR)the evaluatorsapprox. one weekapprox. 4 weeks (inone day to two weeks consideration of the submission period)review donors’ stipulations regardingdraft the contract evaluationshave the contract signeddistribute the Terms ofsecure funding, fix budgetby all contracting parties Reference – to knownbroadly define the objectives and form evaluators, via Internet forums, networks etc.specify time point taking into account theframework conditions (access to projectregion, availability of target groups andemployees etc.)0102030405Preparation0204Drawing up the Terms of ReferenceSelecting Evaluatorsat least ten weeks prior to thepromptly after submissionplanned field phaseperiod endsapprox. 4-6 weeksat least one weekidentify the stakeholders andreview offers received interested parties and how they willselect the evaluators, potentiallybe included over the course of thein conjunction with otherevaluation (point in time, form) stakeholdersprepare the Terms of Reference,document and justify the decisionpotentially in conjunction withother stakeholdersLegendWhenDuration6Evaluation Guideline Version as of June 20171As the data is collected by the external evaluators, there is noguide for this step.
0709Inception ReportAssessment of the Final Reportapprox. one week followingfollowing receipt of the draftthe Kick-off and clarificationreportmeetingup to two weeks followingapprox. 1-2 weeks receipt of the draft reportdiscuss and approve the in-review the draft report andception reportrequest any correctionsapprove the final reportData-collation phase1the commissioning organisation supportsevaluators as agreeddocuments and information are made availableIf needed, help with logistics, appointmentsand communication is providedReporting by the evaluators0607080910Implementation06Kick-off and tation ofthe results10Dealing with the Resultsfollowing receipt of the final reportif stakeholders or target groupsat the end of the field phase or priorneed to be visited, a run-up of atto submission of the final reportleast two weeks prior to the fieldvisit must be scheduledapprox. one dayimplementation planhalf a day to one daydiscuss the results and recommendaat the end of the field visits the evaluators present the collated datacompile material, documents,and preliminary results to the target contacts and provide them to thegroupevaluatorsapprox. two months to draw up anthe results and recommendations aretions with stakeholdersdraw up an implementation planwhere relevant: make the evaluationreport available to a broad public(e.g. on the Internet)discuss dates with the evaluatorspresented to the commissioningand other stakeholders, as wellorganisation and potential othernotify relevant, interested actors ofas logistical and other necessarystakeholders and interested partiesresults and recommendations and support plus content-related,(representatives of the target group,where relevant of planned utilisationmethodological, formal expecta-state actors etc.) at a debriefing ses-tions, boundaries and optionssion/presentationimplementation and monitoring ofthe implementation plan7
8Evaluation Guideline Version as of June 2017
Evaluation – IntroductionDefinition:To evaluate means to assess, analyse, appraise.In international cooperation evaluation is defined as “The systematic andobjective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness,impact and sustainability.An evaluation should provide information that is credible and useful, enablingthe incorporation of lessons learned into the decision-making process of bothrecipients and donors.”OECD-DAC Glossary1Why evaluation? a higher-level programme comprising multiple projects (e.g. all projects in a country)Evaluations have two key objectives: a (sector) policy Learning by all stakeholders, i.e. target groups, partner organisations, Bread for the World, donors and an instrument, an approach to workother actors, often in conjunction with other institutions and organisations in development cooperationwith whom the results will be shared, and an organisation or individual directorates or processes of an implementing organisation (e.g. PME,finance systems, HR or management structure). Accountability towards donors (private donors, co- financers, etc.).Who evaluates?What is evaluated?Evaluations may be conducted in different ways, i.e.The object of the evaluation, i.e. what is being evalu- as a self-evaluation by the people responsible for theated, may vary substantially. Examples of this mayimplementation of the project/programmeinclude: as an internal evaluation, i.e. by people who, although one or more individual project component(s)they work in the same organisation that is implementing or financing the project/programme, are not a project (project evaluation) several projects with the same thematic thrust (overarching evaluation)1involved in its execution as an external evaluation by external, independentevaluatorsOECD DAC: Glossary of Key Terms in Evaluation and ResultsBased Management, 20099
Evaluation – IntroductionOverview: Who will realise the evaluation and which points should be considered10Self-evaluationInternal evaluationExternal evaluationCosts are relatively low as normallyno fee for an external evaluatoris incurred. are relatively low as normallyno fee needs to be paid for anexternal evaluator. are relatively high because anexternal evaluator needs to bepaid for, often including theirtransport and accommodation.Non-monetary overhead (time,personneletc.) can be high as at least oneperson from the project/organisation will need to be heavilyinvolved in the preparation(methodology, questions etc.)and realisation. is relatively high for theimplementing person, but theoverhead is limited given thatthe person already knows theorganisation and the framework conditions. are not necessarily lower asthe Terms of Reference need tobe drafted, the tender procedure organised and the selection made. Additionally, external evaluators often do notknow the project and the stakeholders, meaning that a comparatively large amount of timeneeds to be invested in information, communication andorganisation.Possiblenegativeeffects to beconsideredDepending on cultural conditions and prerequisites withinthe team, it may be difficult oreven impossible for theteam-members to engage critically with themselves and theirwork and any potentialimprovements.Within the team care must betaken to ensure that the resultsand recommendations arerelated to the project/organisation and accepted in that spiritand not taken amiss from theperson conducting the evaluation. Similarly, the neutrality ofthe person conducting the evaluation must be guaranteed.Depending on the personalityand prior knowledge of theexternal evaluator, it can be difficult for him/her to understandthe project as a whole and itsframework conditions. Similarly, it must always be ensuredthat all stakeholders are treatedwith the respect due and thatanonymity is guaranteed.Possible positiveside-effectsThe team-members can learnnot only that they are responsible for success, but also thatthey can influence the design.“Learning from each other”within the organisation and anexchange of ideas can be fostered, with the result that moreof colleagues’ existing insightsand knowledge can be fed intothe implementation process.An impartial outsider’s viewcan reveal new aspects, raisenew questions, generate newideas and by doing so providenew impetus for the project/organisation.ProductsNormally no formal evaluationreport is produced. The way inwhich the results are documented must be agreed inadvance.The form of the evaluationreport must comply with internal stipulations; if no stipulations exist, an agreement mustbe reached in advance.An evaluation report thatanswers the questions set downin the Terms of Reference andcomplies with the contract andthe OECD/DAC guidelinesmust be prepared.Evaluation Guideline Version as of June 2017
Evaluation – IntroductionIt is impossible to say whether one type of evaluation Mid-term evaluations, i.e. an evaluation is con-is better than any other – organisations should optducted at some point during the course of the projectfor one or the other form depending on the donors’to ensure that everything is moving in the directionrequirements, objectives and specifications. Theof achieving the defined objectives and to identifyprevious points should only be regarded as a decision- any potential for improvement. Additionally, themaking aid.results can be used as a learning tool for a comingfunding phase or similar projects;When is the evaluationconducted? Final evaluations, i.e. to establish whether the definedobjectives have been met at the end of a project (funding period), and which effects are already apparentThe point at which an evaluation is conducted mayin order to learn for a coming funding phase or sim-vary; there are:ilar projects; Ex-ante evaluations, i.e. the framework conditions Ex-post evaluations, i.e. once a certain amountand prerequisites are compiled and evaluated beforeof time has passed since the end of the project thea project begins – generally to obtain a better picturechanges effected by the project and also their impactof the situation and requirements;beyond the end of the project implementation aresurveyed.When to Evaluate – the Project CycleFilnaElvauEx - ponatiostEval uationStrategic Planning,Identification,ProgrammingProject ClosureExua t i onEv al31Mid2ProjectProposall ua t i onrm-te te Ev aImplementation,Monitoringand Reporting- an4Negotiationand Approval11
Evaluation – IntroductionHow is the evaluation realised? –Quality standards Feasibility – an evaluation should be planned andconducted in a realistic, thoughtful, diplomatic, andcost-effective manner.Evaluation standards describe how the process of anevaluation should be structured to achieve a high qual- Fairness – in the course of the evaluation all peopleity. Their principle purpose is to act as an orientationand groups involved should be treated with respectfor the realisation, selection process and appraisal ofand fairness; the safety, dignity and rights of the peo-evaluations.ple involved in an evaluation must be protected. Theevaluation results should also be made available to allThe most important standards2 are: Impartiality and independence – For external evaluations in particular the external evaluators need tostakeholders and target groups to the extent possible.Cross-cutting issuesbe as independent of the commissioning organisation, the target group and any other stakeholders asCross-cutting issues are topics that run through the workpossible in order to avoid conflicts of interest. Also,of the commissioning organisation like a red thread andthe evaluation should present stakeholders’ and tar-consequently also through the evaluation of a project/get groups’ different perspectives and the assess-programme/organisation. Cross-cutting issues should bements should be made fairly and as far as possibletaken into consideration as far as possible – dependinguninfluenced by personal feelings.on the investigatory interest, framework conditions andany stipulations from donors or stakeholders. Credibility of the evaluators – those conducting theevaluations should be methodologically and techni-Gender, environmental impact and inclusion arecally competent, impartial and independent in ordercross-cutting issues for Bread for the World. This meansto attain the optimum level of credibility and accept-for example that an evaluation should be able to answerance for the evaluation results.the question as to how the intervention(s) investigatedwill impact women, men and people with a disability. As Precision of the data – an evaluation should produceregards the cross-cutting issue of environmental impacts,and communicate credible information and resultsthe question as to the extent to which the measures beingrelating to the evaluation object and questions of theevaluated will contribute to the protection and mainte-evaluation. Key ways of guaranteeing this are:nance of the natural environment should be investigated–– to deploy adequate methodologies,during the evaluations. To achieve this, these cross-cut-–– to take account of the perspective of all relevantting issue should be consistently referred to in the Termsstakeholders,of Reference (ToR).–– to collect sufficient data for a generally validstatement and appraisal to be made. Participation – As far as possible all stakeholdersin a project/programme – target groups, implementing organisations, donor institutions, other actors –need to be listened to and considered – from planning through execution up to the implementation ofthe recommendations of the evaluation. Usefulness/Utility – An evaluation should addressthe objectives of the evaluation and the users’ information requirements. Evaluation reports should contain all necessary information and should be easy tounderstand and comprehensible.212Evaluation Guideline Version as of June 2017Based on the OECD-DAC quality standards for developmentevaluations and the DeGEval evaluation standards. Referencedocuments with further explanations, cf. link list.
Evaluation – IntroductionNotesIn all cases, it should be ensured that the evaluationstandards are complied with and the cross-cutting issuetaken into account at every step from the very outset – i.e.when formulating the Terms of Reference through to thepreparation of the implementation plan – to ensure thatthe quality of the evaluation corresponds to expectationsand the interests of all stakeholders are considered.Linkson self-evaluationIDRC: Enhancing Organizational Performance. A Toolbox for Self-Assessment. 1999. dle/10625/22953on external evaluationRecommendations for Clients of Evaluations ts-of-evaluationsAustrian Development Agency: Guidelines for Project and Programme Evaluations. 2009. on/dcdndep/47069197.pdfon evaluation standardsDeGEval: Standards for evaluation. 2008. (English)http://www.degeval.de/fileadmin/user upload/Sonstiges/STANDARDS 2008-12 kurz engl.pdfOECD-DAC: Principles for Evaluation of Development Assistance. 1991. n/50584880.pdfOECD-DAC: Quality Standards for Development Evaluation. 2010. n/qualitystandards.pdf13
Preparation01Step 01Planning andBudgetingKey points in brief:1. The evaluation plan is a component of project planningand budgeting.2. The duration of the evaluation process needs to be consideredand its commencement scheduled at an early stage.BackgroundThe need to conduct an evaluation may also arise during the course of the project, for example ifEvaluations need to be considered as early as the project planning stage in order to ensure sufficient time for substantial delays occur,preparation, realisation and financing.The following aspects need to be assessed when objectives will not (foreseeably) be achieved,planning a project and deciding whether an evaluationshould be conducted: key framework conditions have changed. Are there any stipulations (e.g. from donors) as toWhen planning the evaluation period it is important towhich projects need to be evaluated and when?remember that both preparation – particularly reachingagreement with the stakeholders (including the target If any of the following points applies, it makes sensealso realisation and corresponding tasks will take time.–– the project is an innovative project and the feasi-Similarly, the availability of target groups needs to bebility or effectiveness of the approach used needtaken into account. For example it is unlikely that farm-to be reviewed after a certain period,ers will be able to support the evaluation during the har-–– the framework conditions are uncertain and aresubject to extreme change, meaning that thevest period.An item for the evaluation must also be included in feasibility or effectiveness under the altered con-the budget. Depending on the size of the project a roughditions need to be reviewed,estimate of how many days the evaluators will require–– the project is to be expanded,and what costs (e.g. travel costs) will be incurred may–– the project is of particular strategic or politicalbe made (see checklist at the end of the document).significance14groups) when drawing up the Terms of Reference – andto conduct an evaluation:Evaluation Guideline Version as of June 2017
Planning and Budgeting01Function within anevaluationThe purpose of drawing up a specific plan and makingadequate provision in the budget is to ensure that theThe evaluation plan should be agreed with donorsand other stakeholders in the project.Notesevaluation can be conducted at the right time, with adequate methods and in the defined scope and the ques-The planning phase includes a provisional budgettions asked can be answered.prior to the start of the project. This plan will need toIn any event the process of drawing up the ToRbe modified subsequently during the process of drawingshould be started in good time in order to ensure thatup the Terms of Reference (ToR) or if framework condi-the evaluation can be concluded by the scheduled pointtions change, additional questions arise, evaluatorin time (see overview “Steps of an evaluation”).teams are deployed etc. pproach and involvedApartiesAs a rule, the project implementing organisation drawsup the budget for the evaluation. It also needs to beensured that the people in the organisation who areresponsible for the evaluation have enough time to prepare the evaluation content and to support the evaluation process.ChecklistThe following issues need to be considered when planning the budget for an evaluation: Should the evaluation be conducted by one or several evaluators? Will additional costs be incurred due to e.g. taxessuch as VAT? Is an international evaluator required? Will the costs of insuring the evaluator need to becovered? (e.g. for field trips to hazardous areas) Which areas/project regions need to be includedin the evaluation and possibly also visited? Will the evaluation be translated into one or multiple languages? Will the evaluation involve one or more field trips? Roughly how many (evaluator) working days willbe required? Will rooms have to be rented for group discussionsor the presentation? What is the average local and/or internationaldaily rate for evaluators? Will travel costs and/or refreshments for representatives of target groups, government representatives, experts etc. have to be paid? Roughly what costs will be incurred by transport,accommodation, visa, other logistics? Will translators be required for the data collation?What costs will this incur?15
Preparation02Step 02Terms of Reference(ToR)Key points in brief:1. ToR provide the reference framework for the evaluators and areenclosed as an attachment to the contract.2. ToR should be created by the commissioning organisation inconsultation with other stakeholders.3. The evaluation questions should take into account both the fiveOECD DAC criteria and cross-cutting issues.According to the OECD-DAC3 Terms of Reference (ToR) are aBackgroundToR are a description of the performance to be renderedfor an evaluation and constitute the framework of refer-“Written document presentingence for evaluator(s).ƀƀ the purpose andards, evaluations need to assess the respective subjectIn accordance with international quality standƀƀ scope of the evaluation,matter under the five DAC criteria (relevance, effec-ƀƀ the methods to be used,tiveness, efficiency, sustainability and impact). For thisƀƀ the standard against which performance is tobe assessed or analyses are to be conducted,reason, it is helpful to formulate questions relating to allƀƀ the resources andmay also be formulated.five criteria in the ToR. Of course, additional questionsƀƀ time allocated,ƀƀ and reporting requirements.”3OECD DACGlossary of Key Terms in Evaluation and Results BasedManagement, 2009Function within anevaluationParticipatory (see below) development of the ToR is akey process towards discussing and clearly defining theinvestigatory interest of the various stakeholders (i.e.what should be found out?). This enables all stakeholders to clarify what needs to be done under what framework conditions (available time, financial resources,size of target group, etc.) and with what objective.16Evaluation Guideline Version as of June 2017
02Terms of Reference (ToR) pproach and involvedAparties Questions regarding the management of the implementing organisation can also be included in theevaluation questions during a project evaluation.ToR are usually produced by the commissioning organ-Caution: Questions regarding the organisationisation. To obtain maximum benefit from the evalua-(management, organisational development etc.)tion, as many stakeholders in the project as possible (e.g.require specific expertise on the part of the eval-representatives of the target group, the partner organisa-uators which needs to be taken into account whention, the funding organisation) and, if it appears useful,drawing up the key qualifications in the ToR.other actors (e.g. representatives of a relevant local ministry or public authority or specialist organisations) In some cases the stipulations of the commissioningshould be involved in developing the evaluation ques-organisation and/or the donors may have to be takentions. This increases openness and interest in theinto account (e.g. minimum components of the finalresults as well as the willingness to provide informationreport). This may be noted in the ToR and detailsto the evaluators. If not all interested stakeholders arediscussed at the kick-off and clarification meetinginvolved in developing the ToR, they should at least be(see also attachment “Sample structure for the eval-given an insight so they can keep abreast of matters.uation report” in guide “6. Kick-off and clarificationThis serves to allay any fears prior to the evaluation.meeting”).CSS (consultancy and support services) can support the creation of the ToR. The evaluators may be instructed via the ToR to pre-Evaluators are not involved in the creation of thepare an implementation plan with all recommenda-ToR. However, the ToR should always be discussedtions made in the final report. Any additional costsduring the kick-off and clarification meeting and maythat may be incurred must be planned for.be amended subsequently (any amendments will beImportant: The implementation plan per se is notrecorded in the inception report or a protocol). Thea component of the final report that the evaluator isinterests of the commissioning organisation should berequired to draw up.paramount in the formulation of the ToR, which meansthat the ToR should not be drafted to align with thecompetences of pre-selected evaluators. It should be assessed whether it makes sense to workwith evaluator teams. This may be the case if spe-Cross-cutting issues must be included in the ToRcific linguistic or technical knowledge is required.depending on the requirements of the commissioningRepresentation of men and women in the team mustorganisation or donors in order for them to be includedbe ensured. This is absolutely essential if for exam-in the evaluation.ple men are not permitted to visit women’s groups incertain cultures. It is also desirable for internationalNotesand local evaluators to work together because theexchange of knowledge and experience about evaluation methods and cultural specifics can enhance Questions about previous projects may be includedin evaluations of ongoing projects. The primary aimthe quality of the evaluation. If an evaluation team isrequired, this should be noted in the ToR.of this is to get information regarding their long termimpact and sustainability. Gender as well as the other cross-cutting issues suchas environmental impact and inclusion should be To enable maximum participation, the final reporttaken into account depending on the objective and(or at least the summary) should be available in theframework conditions of the subject matter of therespective language of communication and possiblyevaluation or the internal regulations of the com-also in the local language. This should be includedmissioning organisation or donor. Questions aboutin the ToR.the cross-cutting issues may be integrated into thequestions under the various DAC criteria. The aim There are a number of ways of handling the plannedbudget in the ToR (see table on next page)of including cross-cutting issues in the ToR is forstatements on gender, inclusion and environmentalimpact to feed into conclusions, recommendations,lessons learned etc.17
02Terms of Reference (ToR) The ToR should refer to the fact that the evaluationquestions – wherever possible and practical – should beanswered differentiated by age, gender and other criteria such as socio-economic affiliation, disability etc.Options for handling the planned budget in the ToR:Option for handling the budgetAdvantagesDisadvantages1In their offer, prospective evaluators provide a proposal asto how the questions in theToR can best
Brot für die Welt – Evangelischer Entwicklungsdienst Evangelisches Werk für Diakonie und Entwicklung e. V. Caroline-Michaelis-Straße 1 10115 Berlin, Germany Telephone 49 30 65211 0 [email protected] www.brot-fuer-die-welt.de Authors Nicole Derbinski, Thomas Reinhardt Editor Dietmar Mälzer Layout NORDSONNE IDENTITY Berlin, June 2017
