‘Good training’ is simply not good enough, but how do you measure it? Initial reflection on this question showed that we can measure either the quality or the impact of educational activities. Because they are distinct, they lead to a different way of investing resources, depending on how relevant they are for the educational activity or for the overall organisational development.
If we consider the society as a system, the role of educational processes is to create a change in the state of the system, from an initial state to a desired state. In this sense, the impact of the educational process represents the change in between the two states whereas the quality refers to the degree to which a set of inherent characteristics fulfills requirements.
In recent years, the NGO sector has been focusing on the concept of quality in non-formal education. The quality of educational processes has been seen as a way to ensure the recognition of education and of the providers by the society. Although the work related to quality is just at its beginnings, many organisations have embraced the concept and stopped looking beyond the quality of the process or disconnecting the process from the rest of the system (the society and its impact in our case).
Looking at the business sector, education (non-formal education, training etc.) is built around the concept of Return on Investment (ROI) which basically refers to the comparison in between the resources invested in the educational process and the results, both at a personal and organisational level (the impact that the educational process has), giving a lower importance to the quality of the process.
Unfortunately, there is no simple mathematical relation in between quality of the educational process and the impact it has (a high quality process can have a big impact but it can also not have any impact at all – or can have a negative impact).
So, are they connected? And if yes – how? What are the variables of the system that, once modified, change either the quality or the impact or both? I have, therefore, decided to take on the challenge of investigating what can we modify in order to increase quality, impact or both at the same time in a controlled and predictable manner (more to follow).
What is clear at this point is that collecting data (assessment), quantifying that data (measurement) and making judgements (evaluation) are the three steps that are used to research both quality and impact, based on a set of variables (indicators). It is relevant to note that neither impact nor quality evaluations can be completed without an analysis of the present state (bench-marking).
So, do you measure the impact of YOUR educational activities? How about the quality? Do you have a RESOURCE to share?
Hey, thanks for challenging us to think on the subject!
I found a lot on evaluating impact (ROI) in a book written by Paul Kearns – Evaluating the ROI from Learning, 2005, printed by CIPD.
He talks about ROI, adds a critical analysis of the Philipps’ model, of Kirkpatrick’s model and he has tools and techniques for evaluation: the probability check, the input/output/outcome test, the added value scanner, the 3 box system :), business objective and training objective integrator.
I know it sounds like “this is only for business”… it’s not… everything can be applied in NGOs and public sector.
On the subject of quality, I would bet on quality of the teachers / trainers; (self-)assessment, CPD – Continuous Professional Development, being part of a professional association… that sort of things. In my opinion, if you assure the quality of the teacher / trainer / facilitator you have a very big chance that the quality of materials, the sense of the experience and all the others will follow.