Step 1: Creating your Analysis Plan
Even if you have planned the evaluation well and have designed excellent instruments and data collection plan, you need to pay close attention to planning your data analysis:
you start collecting the data and During
Important: Never start data collection until you know how you will analyze your data and tested out the data collection instruments and analysis plan.
Task 1: Develop your analysis plan
If you have access to someone with experience in data analysis, hopefully you already have invited that person to be part of the M&E Team (see Module 1) even if she or he has not been involved at every moment. If you are going to contact an evaluation expert, do not wait until you already have the data collected. Make sure you carefully plan the analyses yourselves, or do it with an expert before you start to collect the data.
There are two types of data analysis questions: one type is focused on understanding how your program was delivered and the other is focused on your program’s impact. You will be focusing your data analysis on understanding, or evaluating, the impact you’ve had- the change you have brought about in your target population. In order to understand your program’s impact you will also need to pay attention to how the program was delivered- your monitoring data.
=> Review the evaluation questions and the program objectives. Focus on the comparisons that will answer your evaluation questions and tell you whether or not the program achieved its objectives.
The most basic comparisons are:
- Pre- vs. Mid,- and/or Post-program
- Post-program vs. Comparison group
Within each of the “rows” on this chart there may be sub-groups you need to explore according to dimensions that are relevant to the program; particularly you may want to find out if certain people benefited more from the program than others.
Within each “column” there may also be variations you will want to explore. Did people who attended the program more times, were exposed to its messages more intensely, who perceived it more positively or had more skills initially end up having better scores on the immediate or intermediate results measures?
List all the comparisons that are linked to your evaluation questions and objectives.
=> Check the kind of data you have and decide which analyses you can and should use
Remember that your indicators will either be numerical or non-numerical.
- You can transform qualitative data into numerical or non-numerical indicators
- Your quantitative data will be analyzed as numerical indicators
If you are analyzing non-numerical indicators, you will
- be able to describe your findings in terms of themes and typologies and demonstrate your findings with quotations and testimonies. For more details, Tips: Forms of Qualitative Analysis
- need to indicate how many data sources (e.g. people or documents) you assessed showed the findings you report out of the entire group you studied, e.g. 5 out of 7 newspapers reported stories about domestic violence in negative ways that denigrated or criticized the woman and defended the man.
If you are analyzing numerical indicators, you will be able to calculate differences in means, medians, or percents.
To choose the kind of analysis you can do according to the level of indicator or type of numerical data you have, see Tips: Kind of Analysis According to Level of Numerical Indicators
For more details about the principle kinds of descriptive statistics you will use with numerical indicators see Tips: Descriptive Statistics for Analysis of Numerical Indicators
=> Decide what intra-group comparisons are needed to address your evaluation questions
- Using your monitoring data explore how aspects of the program and attendance or exposure to the program may have affected the evaluation results
- Using socio-demographic data you collected and evidence you used to construct your Theory of Change (Module 2) and Causal Pathway (Module 3), explore if there were different results within each of your groups, such as by age, sex, marital status, economic level, sexual and/or reproductive history, level of motivation, certain pre-dispositional factors, etc. Are there other comparisons needed that better reflect a rights-based social justice perspective? Tips: Rights-based Social Justice Considerations for Both Quantitative and Qualitative Analyses
=> For each set of indicators, fill in the Analysis Planning Worksheet
Worksheet: Analysis Planning
Task 2: Practice run- Test your analysis plan
Why is this so important?
- To help you improve your data collection tools before it is too late
- To help you know whether or not you are collecting all the information you need, or gauge if you are collecting information that won’t be useful
- To help you find out if you are collecting information from the correct data sources
=> Tabulate the data you obtained in the practice run-through (pilot) of your data collection instruments and data collection plan using the same system you will use for data analysis.
Tips: What If You Have No Pilot Test Data?
- See if there are problems with:
- missing data
- instruments filled out incorrectly
- answers that are drastically different than you expected which might indicate
- the questions were not understood as intended
- they were not coded correctly
- you are asking the wrong people
Ask the data collectors for their impressions about this.
You may need to
- rephrase the questions
- clarify the instructions
- refresh the training of your data collectors
=> As you test out your analysis plan, check off each piece of information you use in your data collection instrument. Are there any unchecked pieces of information left after you have specified analyses for each objective? If so, review whether they are necessary and if so, why they are not addressed in your objectives.
Example: In a needs assessment about sexual rights conducted among young people, interviews were carried out to see which rights were most important. One of the choices was: “this right is indispensable”. After the pilot test, the team saw that very few people checked this option. The interviewers had observed that many respondents seemed confused by the word ‘indispensable.’ The questions had been initially tested with the peer educators, but not those similar to the interviewees.
=> For each of your evaluation questions, use the data from your practice run to try out the comparisons that you plan to use according to your analysis plan. Check off each piece of information that you use in your data collection instrument.
- Do you have the data you need?
- Are they in the form needed for the analyses you plan?
- Are there any unchecked pieces of information left after you specified analyses for each objective or evaluation question? If so, are they necessary? Find out why they were not addressed in your comparisons and correct the mistake. If they are not necessary, eliminate them if possible.
Example: If you expect that the impact of your program may be different for women who have had more than one abortion, as compared to those who have had only one or none, but you only asked the question if she ever had an abortion (yes vs. no) you would not be able to make the comparison. Now is the time to adjust the question and answer options needed for your analysis. Tips: What If You Have No Pilot Test Data?
Important: After considering how you would use the data, cut out items that are not useful.
=> Make the modifications as needed.
- Consult your data collectors to base changes on their insights and observations.
- the instruments
- how data are collected and coded, and/or
- how data are tabulated
- When adjustments are significant enough to change the way people might respond, exclude pre-revision responses from the analysis.
Worksheet: Checklist for Good Analysis Practice
Task 3: Adjust your Logical Framework as necessary
Worksheet: Logical Framework (Word, PDF)