Beyond Reporting: Gaining Actionable Insight from Learning Data Logs
Previously I’ve blogged that no business should want an LMS. One of the problems I’ve highlighted is the lack of actionable intelligence that you get from a Learning Management System – it just doesn’t solve any real-world problems.
And so, if our mission is to actually get some actionable intelligence that can influence the way your business operates, we’ve got to change our methodology and move towards an ecosystem approach.
Not to report on how many people have completed a training activity, but to understand what activities actually make a difference in the performance of our companies.
Which brings me, of course, to a Captain’s Log…
The Case for Keeping a Log
Logs are a fascinating way to document history, and I’d first like to share with you a personal example of why this is so…
During World War II my wife’s Grandfather, Desmond Brown, kept a meticulous log that we still have in the family today. Whilst he was training to become a navigator with the Royal Air Force, Desmond noted down every flight he took, his role on the flight and the pilot in command.
By coincidence my own Grandfather, Harry Betts, also served with the Royal Air Force in WWII, but he was a pilot and was a few years older than Desmond. In fact, by the time Desmond joined up, Harry had flown his thirty missions over enemy territory and had been sent to train new recruits in Canada.
Which, as it turned out, was where Desmond had been sent as a new recruit to earn his wings. The potential coincidence dawned on us one Christmas day whilst talking about family history.
My wife’s father still kept his father’s WWII things and so he went upstairs to find his old logbook.
And there it was…
On the 18th July, 1943, Harry met Desmond. 72 years later and a few thousand miles away, their grandchildren would meet and marry. A wonderful insight into a remarkable coincidence.
Gaining Insight from Learning Data Logs
I haven’t counted how many flights are logged in Desmond’s book. That would take a while, as is the way with logs.
But just because I can’t give you the obvious facts and figures in quick order, doesn’t mean the log entries themselves aren’t insightful.
Logs like this are useful for all of their entries, beyond a tally of how much progress has been made or the final entry that marks training as ‘complete’.
In many regards a tally would be a better way of storing this data; a log is tedious for summary figures.
But summary figures tend to be what most people ask for when they first get their hands on a Learning Record Store (LRS).
The Need to Look Beyond Reporting
One of the most common requests we get with regards to adopting the xAPI and Learning Locker is reporting: What are the reporting functions? Are the reporting functions better than my LMS? Can I create customised reports?
First of all: yes to all of the above, obviously.
But just because you can use an LRS to do reporting, should you? I think perhaps not.
You see logs make for lousy ways of reporting quickly on summary pieces of data. Logs aren’t like a tally that gets overwritten. Each entry is immutable and they stack on top of each other, sequentially.
If you want to find one log entry in particular, you’ve got to go back through every log you’ve ever stored until you find it.
And if you want to count how many times certain things occur in the logs you’ve got to count through each stored record, even if the record in question isn’t a match for your count, before you get to your answer.
This can be slow and tedious at the best of times, especially when your log is a big one (our biggest logs record hundreds of millions of individual entries!).
Whilst we create indexes and caches and all manner of shortcuts to make this process easier, it’s never going to be as fast as simply storing a tally for certain outcomes and overwriting it every time it gets updated.
This, as far as xAPI is concerned, is a problem; a mismatch of expectations versus reality.
You see, when it comes down to it, the Learning Record Store is a log: A time-sequenced record of what happened.
This is why an LRS that follows the xAPI specification is immutable (xAPI statements cannot be deleted) – a log is a record of what occurred and cannot be changed retrospectively.
This is a bit of a problem for reporting.
If someone is 67% of the way through a learning activity, what I want to know is that the current progression total is “67%”, not how it came to be 67%. And when the learner has progressed a bit further and I run the report again, I want it to say “75%”.
I don’t want some long-winded report that says that it used to be 67% but now it’s 75%. That’s a bit pointless.
And yet that’s exactly what a lot’s of people trying out the xAPI for the first time want to know: They want to know the current state of something, given very recent changes.
But why would you go to the time and expense of logging all sorts of activity using the xAPI if you were only really interested in knowing one or two data points? Why not just track those two points and be done with it?
Using a log for only its most recent entry is wasting 99.9% of the stored data.
In our xAPI world, a key difference between the LMS and the LRS is in the nature of knowing how things are, versus how they came to be. The LMS stores things as they are; the LRS tells you how things got to be in that position.
To get the most out of the xAPI world you need to be interested in deriving insights from logs.
Deriving Insight from Logs
If you use an LRS for reporting alone, you are missing the point.
Sure, there’s a chance your LRS will do this better than your existing system, but that’s more a reflection on the poorness of existing reporting systems than a benefit of xAPI.
Reporting is the tip of the iceberg when it comes to the real potential of xAPI and the Learning Record Store.
A Learning Record Store does have a genuine advantage over incumbent LMS software where the data comes from multiple sources.
Used in this way, an LRS can create aggregated reports which an LMS might not be capable of doing. (But frankly it’s still a poor use of a log and other systems can probably do it better.)
Reporting like this is the most obvious use case of aggregating data. This is what I would call the ‘above the water line’ stuff. It’s visible and immediately obvious – but it’s not the bulk of the benefit you can get from a Learning Record Store.
You might find it harder to get out a ‘simple’ report from a log than you would a relational database and, as such, if you don’t look below the water line at the bulk of the iceberg you might find yourself somewhat disappointed with an LRS.
Any action you take as a result of reporting data is going to be reactive; the report will tell you when a deadline has been missed or a KPI fallen short. And, sure, you can react to that. But you can’t prevent it from happening – it’s already happened.
Going below the surface of the water line we get deeper levels of insight and into the possibility of becoming more proactive with your actions based on data.
Ironically, sometimes in order to be more proactive in the future you must first look into the past. This is why our second layer down in the Iceberg is called Pattern Recognition.
A log shows you all the data points that have been collected along a journey, not just the most recent result. That gives you a unique ability to spot patterns in historical data.
This could be based on times of the day / week / month / year, or based on the previous logs of people who have been down a path before.
A Learning Record Store gives you this advantage right away. You can use the data to see patterns that have occurred again and again.
Trends & Predictions
Of course, by understanding what others have done previously, and with a bit of allowance for change, you can have a fair go at predicting where a result will end up.
You can extrapolate past patterns into future trends.
And so we call the next level down on the iceberg ‘Trends & Predictions’ – the ability to accurately predict the future.
There are a number of schools of thought in terms of how accurate predictions can be obtained from patterns and trends, but the two most EdTech software falls into are deterministic and probabilistic:
- Determinists suggest that given X, Y will occur. If you fail this practice test 3 times, you will fail the final examination (for example)
- Probabilists look to develop a likelihood that a learner conforms to one persona or another to inform their predictions; that each activity a user performs makes them more or less likely to conform to a particular set of characteristics (for example, the characteristics that a successful learner generally displays).
These outcomes are less certain but probably more flexible and appropriate beyond obvious circumstances.
Initially you may chose to deploy predictions behind the scenes, for your own internal purposes or potentially to make administrations aware of learners on whom they might like to focus some attention.
But once you’ve developed a bit more confidence in your trends and predictions, your insight is now deep enough to intervene more automatically.
At your most proactive, you can set up ‘Interventions’ to try and alter a course before the outcome is realised.
That is to say, if the patterns you see in a log indicate a future trend that you don’t like the look of, you can take action to prevent that action from occurring in the first place.
This is the aim of predictive analytics projects like our work with Jisc, who are seeking to help teachers and tutors make interventions in their classrooms before the reporting tools show that someone is falling behind.
Our Log’s Legacy: A New Generation
Living through WW2 as a member of the RAF’s Bomber Command was no easy feat. For every 100 airmen that flew, just 27 survived a tour of operations unscathed.
74 years after his Great Grandparents shared an RAF training exercise as part of their work, my first son was born.
It is in their honour (and a testament to the keeping of logs) that I am very proud to introduce Harry Desmond Betts to the world..!
To find out how you can look at your training data differently and start utilising the power of logs, reach out to us to get a demo of Learning Locker and a 14 day free trial.