Data, like air, is everywhere. Like the adage, “water, water, everywhere, but not a drop to drink,” the story of data is often the same. We may have mountains of data, but none of it is being used or is even in a usable form. The importance of using data to make decisions is not lost on anyone, especially in the world of learning – and even more so in the world of digital learning, where the proliferation of digital tools in implementing learning solutions leads to data riches that were unavailable not that long ago. Given the current push to move learning and development solutions to an online world, and the exponential increase in data that has come along with it, how can we begin to use the data better?
The Importance of Data-Driven Decision Making in Learning Experiences
Before we step into important topics such as collecting the correct data and using it to generate insights, we must first touch upon why it is imperative to use data in learning. As mentioned previously, the large amount of data is one reason to use data to drive decisions that impact learning experiences. The abundance of data notwithstanding, though, there are other reasons that using data to drive learning insights is a good idea.
When decisions are backed by data, the chances of success are much higher, and learning programs are no exception. Learning programs designed, released, and revised based on data are proven to be more successful. Another reason for data-driven decision making is that it often presents a logical approach to decision-making in L&D, which can sometimes fall victim to “fanciful whims” of stakeholders, who decide to make changes or implement new learning interventions on little more than a whim. Learning data, when deliberately collected and analyzed, can generate actionable insights about learners, learning experiences, learning interventions, and the learning departments in an organization.
How to Collect Important Data About Learners and the Learning Context
An LMS, LCMS, LXP, or other platform variant that hosts learners can generate a large volume of data about learners and the context in which this data is being generated. For example, an LMS can capture the modules that a learner has access to, the ones that they’ve completed, and the ones that they’ve signed up for but have not completed. In addition, data can show the modules that take longest to complete, the questions that are answered incorrectly the most, the learners who are more involved in upskilling, and more.
One important consideration in gathering learner data is to plan for this early in the design stage rather than as an afterthought. For example, when analyzing a skill or knowledge gap that may require a learning solution, one of the key considerations is the desired level of performance and how that performance will be measured. Similarly, knowing how the learning experience will be evaluated can provide key inputs that must be gathered during the learning process. For example, is learner satisfaction an important criterion? If so, a survey can be planned at the beginning and end of a learning experience.
Additional data about a learner’s activities can be captured by using an eLearning specification, such as xAPI. For example, do your learners spend more time on some modules or activities than the others? Do they just click through some modules without spending any time on them? All of this data can be used to generate useful insights. An advantage of using xAPI is that the performance in training can be compared with on-the-job performance. These results can be measured against each other to find deviations, which can provide valuable insights into your learning initiatives.
How to Use the Collected Data to Generate Useful Insights
The abundance of data generated by learning platforms can be extremely meaningful or completely meaningless depending on how you choose to use the available data. Data by itself can sometimes be revealing. For example, in one organization, despite a week-long campaign to drive employees to use a new LMS, employees still referred to documents stored on SharePoint. The traffic headed toward the document repository on SharePoint was much higher than the traffic on the LMS. This raw data indicates that employees preferred getting information on SharePoint versus using the LMS, despite the fact that the level and quality of information was the same in both places.
However, this data by itself cannot answer questions about why employees preferred the older system over the new system, which supposedly had a lot more features and was easier to use. This additional information can be collected by probing employees with specific questions in a survey sent a week after implementing the new system. Alternatively, you could dive deeper into the data and analyze which segment of users is using the LMS more versus the others. Maybe some teams are driving user adoption of the new system while others aren’t, despite everyone being provided with the same directions.
Some insights are more obvious than others. For example, if the raw data shows that most of your learning content is being accessed by users on mobile devices, it’s easy to make a case for developing more mobile-learning, or mLearning, content. Or, if the data indicates that your learners have a clear preference for more interactive content such as games and simulations or video content over a PPT-based format, you can invest more in creating games and videos.
How to Elaborate on These Insights and Make Key Decisions
Once you have analyzed the data and generated insights, you can take specific actions that are backed by the data. Continuing with the previous example, let’s say the data revealed that the implementation of the new LMS was successful in two teams whose managers prioritized the rollout. On digging deeper, you identified that these managers were involved in the LMS pilot and had bought into the benefits of the new LMS. This key insight can help you plan future pilots.
Consider another example. You notice that on one online courses, learners are able to complete the module in 10 minutes despite a course projected seat time of 1.5 hours. And most of these learners are able to answer all of the questions on the final quiz correctly. There may be several reasons for this behavior. Maybe the course content is not engaging enough and learners are just clicking through; in this case, you can redesign the course to make it more engaging. Or maybe learners are only enrolling in the course to easily gain development units; in this case, you can reevaluate if the course is worth the earned development units. Or it could be that the content is so easy that a course was not required at all; the content could have been shared as an article or job aid instead. Each of these decisions adds value for the learner, the learning experience, and the learning and development team.
The Ethics of Data Collection and User Privacy
One of the key foundations of ethical data collection is informing learners about the data that is collected and how it will be used. If the data will be transferred to a third part for analysis or generating insights, learners need to know about this beforehand. Any personally identifiable information that is collected should always remain secure. Unless a learner permits it and unless absolutely required, information should not be made available to parties without explicit learner consent.
