SCEM and Kirkpatrick: Learning Program Evaluation

I have been doing some browsing and reading to come up with a more comprehensive model for learning program evaluation. I'm familiar with Kirkpatrick's four step model, and I recognize it as a valuable taxonomy that guides practitioners in what to evaluate, including transfer of training and the effect of changed behavior on organizational bottom lines. However, I often think that Kirkpatrick's model overlooks that there are many other factors in an organization that effect whether or not a learning intervention carries over into the workplace and results in improved performance. So I went looking for critiques of Kirkpatrick.

Here's what I found. Robert Brinkerhoff, of Western Michigan U. came up with the “Success Case Evaluation Method” (SCEM) that includes qualitative data and storytelling to identify and describe successful adoptions of skills and knowledge in the workplace. Here is a synopsis of the SCEM.

I like that SCEM includes rich description of the environment and that it includes storytelling. I'm convinced that storytelling is one of the most effective methods for passing along information and values. In SCEM, the results of the study are communicated in story form. I like that. The Kellogg Foundation has a white paper describing SCEM here.

I don't think, at this point, that the SCEM replaces Kirkpatrick's model, rather it fits into it. Kirkpatrick's model is more of a taxonomy – describing different levels that should be considered in a program evaluation – and not prescribing methods for carrying out those different level assessments. Brinkerhoff's SCEM, on the other hand, seems to be a specific tool that takes into account the complexities of an organization, recognizing its culture and dynamic nature. Then I found a press release from a training company, Knowledge Advisors, that they had added Brinkerhoff to their Board of Advisors and were integrating SCEM tools into their software Metrics That Matter. See press release.

This interests me for various reasons, not the least of which is the possible integration of Metrics That Matter into learning management systems such as MOODLE, although they don't specifically mention MOODLE. Here's a PDF describing that integration.

Note: Both these previous links are marketing.

I would be interested in hearing from any of you that have experience with either SCEM or Metrics That Matter. Or any other program evaluation tool that can integrate with a learning management system.