After defining evaluation questions, the next step consists of making them operational by turning them into evaluation instruments. In this context, the evaluation instrument refers to the evaluation question plus the information collection method applied. The evaluator has to ensure that the data collection methodology will produce valid and reliable information.

Various methods for collecting evaluation data are explained in Module 8 - Data Collection Methods of the International Program for Development Evaluation Training’s (IPDET) handbook. A toolbox for evaluators: Evaluation tools is available on the EuropeAid’s website on evaluation; it details 13 key evaluation tools explaining what, why and when they should be used and how to implement them. The EC website’s section on How to choose the right mix of tools provides suggestions on how to identify the right combination of tools in order to answer evaluation questions. Guidance on Evaluation and Review for DFID Staff offers a chart (p:80) on advantages and disadvantages of each data collection method.

The methodology for collecting data should produce reliable data that allows valid evaluative judgments that are useful for learning and making decisions.

• Validity is a measure of the extent to which, taken together, the evaluation’s design, data collection methods and analyses provide a reasonable basis for conclusions about the evaluation’s questions.
• Reliability is a measure of the quality of measurement. Information is reliable if the measurement procedure yields the same results if applied repeatedly. One of the most commonly used techniques for determining the reliability of data is triangulation, applying the same measurement procedure to three or more different sources to obtain data that can be compared for similarity.