Indicators

What are they?

So, what are indicators and how can we think about them? Well, I chose the above image because it was misleading. I know, I’m a monster. Indicators are fairly self-explanatory: they indicate. They are components of a results framework that tell you whether your programme is on the right track and achieving success.

I see them as like checkpoints that have a beautiful, symbiotic relationship with your theory of change like sharks and remora fish.

If you see your Theory of Change as a journey you have planned, your indicators are just checkpoints along the way. If you’re taking your journey and you reach a bridge, you know you’re half way. Similarly, you know you’re going the right way if the mountains remain to your left. Indicators function the same way: the help you see if your programme is doing what it should be doing as helpful check-points. They needn’t be seen as these punitive measures to appease an eye of sauron.

The reason I separated this section from the results framework section is because I think basic indicator knowledge is a world unto its own, and I frequently run this workshop in isolation for teams who are old hats at a LogFrame but want to know how to design a good indicator.

These are just my thoughts from desk research as well as my approach when designing these. I’ve always had positive feedback from donors, responsible officers, and programme teams alike so wanted to share this thinking more broadly.

Stop bean counting

You will have seen me write a lot about the perils of bean counting and my programme teams and training participants hear it a lot as well. But what do I mean by this?

The term was borrowed from my director who said one of the reasons he he hired me because I am one of the rare evaluators who ‘doesn’t bean count’. I loved the phrase as I immediately knew what he meant: I don’t just use quantitative measures for the sake of it.

Bean-counting is using measures like: #of people attending workshops, % of attendees are female, #policies developed, % of people feeling safer.

I will go one-by-one and explain the problems with these types of measures, but first I want to talk a little bit about how quantitative measures and stats are really not our friends sometimes.

Quantitative measures behaving badly

Now. Quantitative measures and statistics are slippery fish. Let me show you why.

Fun fact: cows are 300x more likely to kill you than coyotes. Minor sidenote to this statistical fact: if it was common for people to keep hundreds of coyotes on their property and routinely chase them into corral and handle them… this statistic would be different.

This is a great representation of conditional probability which is a statistical property many struggle to grapple with. Another example is that coconut trees and vending machine feature quite highly in reasons for people dying… these are also things that people shake vigorously and frequently. See what I mean?

The point is: statistics are tricksy and raw numbers often fail to tell you the real story. Another example of this is the following: 93% of people agree crime reduced in their area. Now that looks really good, doesn’t it?

Well. It does look good on paper… but not in reality. This actually means that 14/15 people asked said this, and they all live in the same locale. That means very little if this was a national programme. Your data pool and sampling approach really affects things. By data pool, I mean the total number of people/goods/cats/etc that you have included in that total from which you take a proportion.

Now, a result of 2 policies developed looks really meaningless… or is it? Actually this is a pretty bad way to measure something that took a lot of work – the quantitative in this case is right, but just really fails to capture the significance.

Quantitative measures aren’t wrong, you just have to be really careful with how you use them and what parameters you set around them. Let me show you why and how in the other examples mentioned earlier …

Number of people attending workshop

The first question is… why. Is this really a checkpoint for change or is it just easy to count people?

In some contexts, just getting people in a room might be meaningful – for example in a police reform programme in DRC. Getting people in the room so that you can reach them with your workshops – referring back to our Useful ToCs – is a lot of work and does meaningfully checkpoint a change that matters.

In others, it’s not, such as a leadership course attendees have paid for. What might matter instead is who you get in the room. The other issue is that not only is this a meaningless measure but it also falls into the dangers of false positives and coercion. By coercion, I mean that if your milestone is 40 odd people and you only have 30 signed up, you may end up forcing people to attend who either don’t want to be there, or won’t benefit from it, therefore wasting their time because you’ll miss the milestone otherwise. In workshops this is less serious, but in, say, a family planning programme you could end up forcing women to use contraceptives they don’t want.

See the problem? As for false positives, you could end up meeting that milestone and have a room full of people, but this doesn’t mean change has happened on the ground as they may not have understood etc. it takes away from the change you want to see.

% of attendees are female

% are tricky as you’ll remember – it all depends on your data pool. There are several risks with this one.

The first risk is falling results by year, the second is supply. The falling results issue is due to data pools. Let’s say your workshop in year one (see image in side bar) has 15 attendees, then year two sees 25, then year three sees 49 with the associated disaggregated numbers of female-identifying attendees. The thing is, you do have more female-identifying attendees each year, but your percentage falls because it’s proportional. Now, this isn’t necessarily a problem because it does demonstrate that there’s a gender issue in a sense, but you need to be wary that this can be an issue with % measures when your data pool grows. This is particularly true with GESI measures because there is often a supply issue. Before I talk about this, the other thing to bear in mind is that this way of measuring is also little tokenistic and can lead to coercion issues as mentioned.

Now, supply is quite self-descriptive. Sometimes you just don’t have enough female-identifying people to fill this gap – there aren’t enough female-identifying geologists or security staff and so you set yourself up to fail immediately. Fortunately there are great indicators that tackle these GESI issues that we will talk about later.

Number of policies developed

Now, this one is a doozy. Seems ok, right? Well, no. like discussed in the last section this fails to capture the change. As we all know policy takes years of effort and skill to develop. This measure may capture the change in some senses, but ignores the journey it took to get there. Fortunately we have indicators for these situations but this is a typical bean counting situation I see that is so avoidable.

% of respondents feeling safer.

Another percentage! We know a bit about this right? There is a total risk of falling results, again, if you expand your survey pool each year.

The other issue is methodology causing false positives. There is a big difference between a question saying ‘do you feel safer yes/no’ and a question asking ‘how do you feel?’ with ‘safer’ as an option to choose out of a basket of options. By asking the former you prompt a respondent to state that they feel safe when they may not, or may be in a grey area. Offering it as an option out of a basket of feelings means that you have people self-identifying that feeling and more reliable data. Secondly, this measure relies on a shared understanding of ‘safer’ between the implementer, the evaluator, the donor, and the beneficiaries. You need to carefully define the word and your methodology.

So, you’ve heard me talking about beans now. Please familiarise yourself with results-reality divergence as touched on in my Results Framework section. Otherwise, let’s talk about indicators.

Better indicators

There are better ways to measure and design an indicator, and it is all about being Socratic. Ask: What do you need to know? What matters?

There are roughly 7 types of indicators. The tricksy thing is that often these terms are used incorrectly or not at all and you feel like you’re trying to pull a rabbit out of several hats. Fear not, I have listed the definitions out below with my favourite crossing a river analogy to tie into this concept of being on a journey.

Bedrock Indicators

What are they? These are a core set of indicators that remain fixed, usually at the outcome or impact levels but not exclusively. These allows lower levels to remain flexible.

When do I use them? These are for when you have a clear idea of what is to be achieved at the outcome level but requires a higher degree of flexibility in how to achieve it.

Say you’re going to cross a river, and you know there is a bridge. Thus, you know you can use a bridge to cross that river and can measure your river-crossing-success in terms of using the bridge, you just need flexibility in how to get to it.

An example is: Number of VPO senior leaders using the GCF accreditation toolkit. Maybe we know for sure the VPO senior leaders will use that toolkit, you just need a blended approach to get them there.

Menu Indicators

What are they? Relatively open, these indicators are for when you are bringing about change with an uncertain pathway. In effect they offer a menu of different results that can be achieved, allowing flexibility in the program to decide which ones and how. In effect it is an indicator that says you will bring about a type of change, but the detail depends on what is possible in the context. Menu indicators are when you need to back a few horses knowing one or two may fall – you need a menu of options to choose from to account for that risk.

When do I use them? These are for when you are working in complex systems where the exact impact pathways cannot be known in advance and a number of pathways to success are possible. This means that you can adapt to bring it about.

Say you’re going to cross a river… but there is less certainty about how. You know from research that there is a bridge, but you also have a boat and can swim and speak sea monster. This means that when you get to the river, if the bridge is broken you can take another option across. You still cross the river, you’re just less clear on HOW until you get there – you take one of a specified menu of options with the promise that you will use one of these to achieve the river crossing.

An example is: number of top 9 cross-sectoral constraints to transformational and inclusive growth significantly eased. We know there are 9 constraints, but the ones we target depends on relationships and the enabling environment. As such, we will aim to ease 6 of them by the end of the project, but not specifying which of the 9 those are so that we can adapt the programme as needed and not be trapped into a route that has a dead-end.

Open-Ended Indicators

What are they? These are even more open indicators. This is a little like a menu indicator, but for more uncertain contexts. Open-ended indicators are when you need to deliver a type of change, like with menu indicators, but how that change materialises is even more uncertain: you have a fixed idea of the type of change needed, but don’t target a specific route.

When do I use them? These are for when you are working in complex systems where the exact impact pathways cannot be known in advance and a number of pathways to success are possible. This means that you can adapt to bring it about.

Say you’re going to cross a river… but there is even less certainty about how. We know we have a boat, can swim, and can speak sea monster but know nothing about that river – maybe there are no sea monsters to negotiate with, or it’s toxic to swim in. You’ll have to figure out how to cross it when you get there, and this indicator focuses on the fact that you will do so.

For example: Number of tangible examples of EPU staff using deliverology techniques to advance policy. This is a classic context for needing open-ended indicators: staff using skills. People can be really creative: don’t box in how they will use the skills specifically, just indicate that they will use them in a way relevant to advancing the change pathways.

Steps-Based Indicators

What are they? These are indicators that recognise the journey is as important as the end product, measuring the process to achieve the end result. One crucial slip-up point I see is counting policies delivered or systems established, as earlier. These things are so complicated and it misses out the hard work it takes to get there, as well as the potential for failure. However, getting ¾ of the way still matters – you may have learned something. As such, steps-based indicators measure the journey by which you get to an end result as well as the end result, rather than just making it binary.

When do I use them? These are best for when a programme aims to deliver something that will require a predictable, but complicated and resource-intensive, journey.

Say you’re going to cross a river… but it is very, very broad and choppy. Simply saying you crossed the river misses the effort you put into crossing such a vast body of difficult water. It is not the same as crossing a stream. As such, we will measure the journey taken to cross the river, rather than just being on the other side!

For example: Number of steps taken towards establishing databases, and number of which are significant. This allows you to account for the work done in sorting data, digitizing it, developing a system, etc. which is just as important as having the system there when it comes to the change.

Learning Indicators

What are they? These are indicators which measure progress around ideas and learning.

When do I use them? These are great for when adaptation and learning are critical to the success of the program, such as those that are experimental, or trying to develop an evidence base.

Say you’re going to cross a river… and think there is a bridge across… But you get there and it’s broken and you have to swim. You still crossed the river! However, you learned how to do so, and can inform others approaching the river so that they can ensure they’re prepared to swim or bring a boat.

For example: Number of ideas and/or lessons generated on ‘what works’ in transformational grown. If a programme is trying to learn how or why something works, holding it to account for actually developing learning and documenting it can be really powerful.

Principles-Focused Indicators

What are they? These are a little more interesting and unusual. These indicators are for setting indicators that measure the degree to which a program follows its stated principles: e.g., takes a gender-sensitive approach to implementation.

When do I use them? These are for when how the program achieves its results are as important as the results achieved.

Say I’m going to cross a river… but what really matters is how I cross that river. Maybe being environmentally sensitive is important. As such, this would mean that I only achieve against this indicator if I don’t swim or take a boat, as my sunscreen will harm the fish and the boat will cause noise pollution. To succeed I may need to negotiate with the sea monster.

For example: Degree to which Freetown Justice programme is gender sensitive. Rather than being tokenistic, or risking coercion or supply issues with your traditional ‘% of women’ indicators, this allows us to ensure that everything the programme does is done in a gender-sensitive manner.

Sentinel Indicators

What are they? Similar to a principles-focused indicator, a sentinel indicator measures how you programme. Perhaps programming adaptively is crucial to the programme success? Whack a sentinel indicator in there as they measure the degree to which a program follows a specific approach: e.g. is adaptive.

When do I use them? These are for when how the program achieves its results are as important as the results achieved.

Say I’m crossing a river… but what really matters is how I cross that river. Maybe being adaptive is important. This would mean that if my boat sinks I’m only successful if I find another appropriate way to finish the journey.

For example: Level of effectiveness of AIM4R Programme in delivering an adaptive programme with partners. If this is a programme with a large consortium of partners, it may make sense to use a sentinel indicator to ensure collaborative, cooperative working with partners. this allows for all parties to buy into this being an important way to programme.

Wrap-up

That was a lot of information, but I hope it felt clear. As said, this isn’t necessarily the only way to do things, but it’s what I and my teams/trainees have found helpful. For more practical advice on how to represent these in LogFrames and how to display the more unusual ones like principles-focused or sentinel indicators please see my Results Framework section. Or, if all else fails, feel free to reach out.