Philosophy for evaluation

Hello world,

It has been a long time since I wrote on this blog! I have mostly been writing on my employer’s blog, and also have just been stretched thinly due to racing between projects and workstreams. However, I wanted to take some time to write on something I have been pondering for a few years now as I would love to hear others’ thoughts. 

I had the great fortune earlier this year to be on a podcast called Tolerable Risk hosted by the wonderful Sabrina Segal, who asked me how I got into MEL (ah! The age-old By-What-Means-Did-You-Fall-Into-This-Profession). As part of my response, I noted that I have a rather abnormal degree for an international development sector person: analytical philosophy. I ruminated on how this was in fact the perfect degree for an evaluator given the study of causation, epistemology, concepts of necessity and sufficiency, and logic. It is a thought I have been nursing for some time now, and Sabrina gave me the opportunity to articulate it out loud for the first time.

Fast forward ~7 months and I am hosting a fishbowl at the UK Evaluation Society Conference on the topic of learning from other disciplines in evaluation. I was expecting folk to talk about learning from disciplines like neuropsychology or evidence-based disciplines like forensics, but instead we found ourselves on the topic of philosophy! It seems I am not alone in thinking it is a natural discipline to draw on for our profession.

It really has urged me to write on this, as I think it is fascinating (and I wish I could go back in time to my recently-graduated self who was worried if she would ever use this strange degree she had dedicated time to). I wanted to share my reflections on the value of philosophy in my profession and how I have used it. Now, I note that I explicitly studied analytical philosophy which is a different philosophical tradition to continental philosophy or similar disciplines. I also studied it in the Cambridge tradition which really worships at the altars of Bertrand Russell and similar characters. With that in mind, personally, there are five key areas that I keep finding myself going back to.

1. Epistemology: the study of knowledge.

This was quite possibly one of my favourite lines of study. Epistemology is endlessly fascinating and covers a breadth of topics such as what we truly consider knowledge to be, as well as what we can reasonably prove about our world. In the philosophical tradition there have been so many winding and fun roads, such as the rollercoaster of thinking we knew how to define ‘knowledge’ (a justified, true, belief) until Gettier came along and ruined it all for us by disproving it.

I have been surprised how often I use this in my day-to-day and I know I am not alone in this, having seen it referenced a lot at both UKES and AEA conferences this year. This helps immensely with how I think about and understand strength of evidence (though I will note Tom Aston has gone and made that much easier for us with his work), as well as how I approach proofs. This can be done formally or informally, with one of my favourite ways I apply epistemological concepts being workshop-based: when engaging knowledge holders in (e.g.) a problem analysis workshop, I like to probe them to explain if something is ‘knowledge’ (i.e. evidenced somewhere in this context), a ‘justified belief’ (i.e. hard to evidence, but known, such as an actor group’s criminal affiliations ), or an ‘opinion’ (i.e. a speculation). When I am facilitating, all opinions get marked up (and can even be weighted for just how speculative they are) and put into what I call a ‘research parking lot’ to ground truth later, with priority following weighting if I use it. This is such an invaluable way to identify and draw out solid understanding as well as areas for further evidencing. This is just one example, but I genuinely find myself using these concepts and my learning frequently.

2. Causation: the study of cause-and-effect

I think this is the most obvious one, and one I have seen before! A little while ago I invited myself into a twitter thread between a few evaluators (forgive me, I forget who was on it) where they were discussing interesting texts that could be useful for evaluation… texts that were my entire first year syllabus on causation. I am so grateful for this line of study, and in particular I remember this intersected in my third year with metaphysics modules focused on time, where the ideas of circular causation and the implications for the circularity of time were discussed. This really opened my mind to not just think about causation, but to really notice it and to appreciate its depth and complexity. I do think this set me up to be a systems thinker and a lifelong member of the John Mayne fan club for his Useful Theory of Change, because the idea of a simplistic and linear representation of causation felt so repugnant and went against my every instinct, understanding, and experience.

I also remember having endless seminars and a focused supervision on the concepts of necessity and sufficiency and how they relate to cause-and-effect, something I found deeply interesting but was a little unsure of how useful it was. How wrong I was! As a deep lover of contribution analysis and approaches inspired by it, these are the concepts I use regularly. I think this is also why I have always been so drawn to process tracing as well, though also why I have not applied it like I have contribution analysis (as it tries to rule out other explanations which was less of the focus of my training). I even use them when doing behavioural analysis with groups, such as really thinking through a COM-B model and ensuring all of the causally relevant factors have been included.

There is a lot to say on this one, but so much of it is self-evident that it is probably not useful to ramble on about. However, I am very interested in how others use philosophical teachings on causation in their work.

3. Ethics and political philosophy: I think this is fairly obvious.

Fairly obvious, but important. While the areas of ethics that interested me the most and where I subsequently spent the most time were largely not the most relevant to my current work (i.e. acts vs omissions, perfectionism), broader ethical thinking and concepts of ethical responsibility towards future generations has been incredibly useful. My analytic grounding taught me to interrogate ethical parameters and ensure consistency in what was being applied, and my nature ensured that this was instilled with an understanding of people and sensitivity . Analytical philosophy – unlike its European sibling, continental philosophy – can get so stuck in rigour that it completely forgets the people-centred nature of its ethical reflections. While at times this is useful for thought experiments – such as the infamous trolley dilemma or Rachels’ age old but brutal Smith and Jones adventures – it is not always the most, er, sensitive. I find the analytical grounding terribly useful in how I think through ethical quandaries, but I think what has been most valuable in its application is ensuring that the people whom the quandary is focused on are at the centre. It must also be noted that inconsistency and abstraction can be important at times as well, and understanding when that is needed, the analytical Cambridge tradition is less helpful with regard to that.

Political philosophy was certainly transformational for me, and taught me to consider positionality, concepts of borders, localisation, gender, and feminism in ways that are crucial to how I work. It really teaches you how to ask the right questions, and how to bring in different analytical lenses when navigating delicate topics. I remember a winding discussion with a professor who complimented my shoes, then asked if I knew where they were made. When I did not, he proposed that they were made in a sweatshop by a struggling family in an emerging economy and asked me how I felt about my shoes under those circumstances. After a worried response from me, he then supposed what that family would do without the sweatshop income and if that would really be a better life for that family. This went on and was honestly quite jarring, but important, as it led us to discuss poverty traps and the importance of context and circumstance. I learned an immense amount from the reading and discussions on this topic; I could write an entire blog on this one in and of itself but I shall hold myself back for now.

4. Logic: formal logic and more commonly understood logic.

I spent literal years studying logic, specifically in the form of second order quantitative logic, set theory, and similar items. This has been utterly invaluable to me. I still to this day use logical form to think through certain claims and I infamously use it during workshops as shorthand to the eyebrow raises of participants who wonder why I am writing an ‘E’ backwards or using a horseshoe on its side.

Even the other day, in an email-based conversation with Alan Hudson, we were looking at applicability or relationships between Michie et al’s COM-B framework and the Power, Capabilities, Interest framework which lies at the heart of the SOAS-ACE approach to understanding and addressing corruption. I used set theory to sketch out on a napkin (yes actually) how I could see an alignment between the two in terms of set members that could be in common. It was a useful application, and again reminded me how this way of thinking creeps into my work regularly.

I really thrived in my logic classes, finding ways to set order to wandering thoughts and really understanding how to form powerful arguments and deductions. Critically, it teaches you to spot a weak or problematic argument and how to counter it: something I also find useful as an evaluator. I still remember my first class on this when we were presented with the argument along the lines of the below (I have altered the original as it had some problematic and dated references):

All English people are British,

Some British people eat fish and chips,

Therefore, all English people eat fish and chips.

We had to translate it into formal logic, find the fallacy and propose a better argument. Quite basic, but it was fun, and it got increasingly difficult from there and we were really put through our paces (in particular I fondly remember my Ancient Philosophy professor proving to me that bread is in fact a form of cake and that several signs in my university town were logically incorrect). The logical approach has really stayed with me, and I really do enjoy testing and constructing causal claims as a result of it.

Funny story, but about 8 years later I was given a mug that said ‘all unicorns are awesome. I am awesome. Therefore, I am a unicorn.’ which irked me to a laughable degree. Oh dear. You can imagine I am a right barrel of fun with dodgy causal claims.

5. Being the observer and being uncomfortable.

Did you see this one coming? I think this was the most valuable lesson of all for me. In analytic tradition, you sometimes have to come to conclusions that feel wildly at odds with our intuitions. The first time this happened to me really stands out in my memory. I was in my professor’s supervisory office, and I had finally found a definition of a ‘person’ that felt comfortable and outlined how this was distinct to the concept of a ‘human’… then realised this meant that children did not meet the bar, and neither did those in comas. The concept of non-people humans was uncomfortable, but logically I had to accept it because the definition and deduction held. I could see the twinkle in his eye as he saw that penny drop, and we discussed how sometimes a conclusion lines up against all intuition and we have to stay with that thought, test if it feels comfortable, and then determine if you wish to keep the new conclusion or reject it and find a new way to have consistent logic. He explained that the philosopher’s role was to understand and observe, and to see what logic tells us, not to cling to what we wished was true. This applies to accepting that we cannot prove that we are not brains in vats being stimulated to hallucinate the world, or to accept that perhaps omitting to do something is as morally wrong as actually doing something if your intentions are the same in both circumstances.

This has been unbelievably useful as an evaluator. Not because I am running around accusing children of not being people or worrying I am a brain in a vat, mind you. This is more because my role is an observer: someone who takes in all of the evidence available and comes to a conclusion that makes the most sense in-line with it. Sometimes that means concluding, and therefore explaining, something really uncomfortable such as a programme failing to achieve impact, or – even worse – that it has done harm.

What I think is distinct is of course how this is done. I speak often about the importance of kindness and I wholeheartedly stand by that: we are not rogue philosophers crashing symposiums brandishing chickens, we are people trying to inform other people who are doing their best to do good in the world. As such, learning to be uncomfortable and to observe was critical, but what was most critical was learning to do that with empathy, humanity, and kindness.

There are of course many more areas that are useful: the very practice of appropriately defining things, Bayesian probability, and modules focused on motivation and will being several that come to mind. However, these are the five areas that I have found most pertinent in my day-to-day. I find this amusing given philosophy is often touted as a degree that is hard to ever apply in a work context.

However as I have touched on, I think this is to be drawn into practice rather than directly applied: we must never forget to be human centred, kind, and honest about the reality of the world. Philosophical principles and approaches are so valuable, but also stem from a discipline that applies them in thought experiments and abstract scenarios. Unfortunately the world is not quite so neat, and so while it is incredibly powerful and valuable to draw on logic and epistemology, we must also recognise that the world is not quite as ordered as we would wish it to be and that we must always ensure to bring in realism, as well as kindness and sensitivity to those we engage. After all, evaluation engages people and all of the multifaceted elements that make up a person: we can be logical while also being inclusive and responsive to context. I do think the balance of logic and rigour with being human-centred and culturally responsive is something we struggle to balance in many ways in evaluation and is a much larger topic than can be covered here, so I am still putting a pin in this thought but still acknowledging it as it is one that I have been discussing more broadly.

I told myself I would keep this short and I have not, but I am really curious about what others think. So, tell me: do you think the same? Have you used philosophy in your work? Fellow philosophers-turned-evaluators, how have you applied your training? I am very curious to see what others are up to in this space.