skip to main |
skip to sidebar
How important is it to evaluate and improvise on you work? As instructional designers, we surely do evaluate and review a lot of work done by SMEs, graphic designers, content developers, and courseware engineers to add value to the training and learnings that we develop; however, how often do we try to evaluate the trainings that we develop? When was the last time you evaluated the training that you developed? Lost. Are you?
Evaluating training/learning is the last phase (E - evaluate) of the instructional design model called ADDIE. Often considered as a cost- and time-consuming process, evaluation is frequently neglected. Let’s look at some statistics to get real. Only 3% of the trainings delivered in U.S. companies are evaluated. This small figure speaks louder considering that U.S. companies spend more than 300 billion dollars in training every year. It is amazing that the claim to deliver these trainings is that they will generate or save business revenue and such a claim is easily let loose by scrapping off the evaluation phase in the instructional design model.
Trainings are always seen under the cost column in an organization’s budget spreadsheet. Unless and until, evaluation is done, this cost cannot be justified. Increasingly, senior management is asking to justify the training costs and such justifications cannot be made at the spur of the moment.
To justify the business cost of the training (which includes the idle time of the employees gone in training and the time and effort spent on training development, delivery, and maintenance), it is imperative to evaluate how business results get impacted by trainings. In addition to supporting the business need for trainings, evaluating trainings helps identify which trainings are effective from the organization’s perspective and what features can be improved in the training. It helps identify relevant and redundant trainings too.
How to Evaluate a Training?
Kirkpatrick model, devised by Donald Kirkpatrick in 1959, provides a framework for evaluating trainings. The model outlines 4 levels at which training evaluation can be carried out. Each level builds on the other --- a given level looses its meaning if carried out without the predecessor level. Time, cost, and effort to carry out the evaluation increases with each successive level. Let’s quickly look at these levels.
Level 1: Reaction: Reaction of the Learners: At this level, you assess the reaction of the learners soon after the training gets over. Get to know how the learner feels about the training. Learner’s reaction can be captured in “Happy Sheets” or “Smile Sheets,” which might contain questions such as “How do you feel about the training, Are you satisfied with the training, Was the delivery mode effective, Are you satisfied with the way trainer delivered the training, Are the facilities amiable for this training, Do you think it was an effective training?” and so on. You can give learners more room for reflecting on training by asking them broader questions on what they think could have been better and if there were any gaps in the training module they attended. This level is the most basic level at which any training can be evaluated. Although, in my opinion, this level does not help in gauging the effectiveness of a training, it surely does tell whether the training was utterly ill-designed and/or ill-delivered to take it off the training map.
Level 2: Learning: Change in Learning: This level aims at evaluating whether there has been an increase in skills, attitudes, and knowledge of the learner post training. You can make such an evaluation by conducting pre- and post-training tests. This will, however, not be a measure of how the learner will perform on the job (OTJ). It only gives an indication if the learner has learned something (skill, attitude, and/or knowledge) to remember after the training. Because this level is above Level 1 evaluation, it requires more effort and cost as compared to the level 1 evaluation.
Almost all training programs, whether technical or soft skills, can be easily evaluated at Levels 1 and 2 vis-à-vis Levels 3 and 4, which are better suited for technical skills. Also, when compared to Levels 3 and 4, training effectiveness measurements at Levels 1 and 2 require less effort, cost, and time, both on the learner as well as on the organization’s part.
Level 3: Behavior: Behavior that Learner Exhibits OTJ: At this level, you need to assess whether the learner has taken the learning a step ahead of remembering (Level 2), i.e., whether the learner has taken the learning to the job. Such an evaluation can be made, 3 to 6 months after delivering the training, by using Behavioral Sheets in which inputs can be taken from the customers, peers, and supervisors of the learner. Behavioral Sheets can contain questions such as “Has the leaner been able to demonstrate XYZ skill in his work/interaction? Is the learning reflecting consistently in the learner’s behavior? Is there any particular behavior change that the learner is exhibiting in past 3-6 months? What is it? Is the learner able to help others with the skill set that he or she has recently acquired after training? Is the earner aware of the behavioral change he or she is demonstrating?” and so on. In case you are unable to get the appropriate response on the Behavioral Sheets or other such templates, try gathering this data from interviews and/or 360-degree feedback.
One problem that I see with Kirkpatrick’s model Level 3 is that following it you can measure effectiveness of a technical training more successfully than a soft skills training, however, it is equally important to understand the effect of behavioral change post technical training as is post a soft skills training. So, how do we evaluate effectiveness of a soft skills training w.r.t. behavioral change --- the only change that we look out for after a soft skills training? Thoughts invited.
Level 4: Results: Business Results: Fourth level of evaluation assesses how far the training has been effective to contribute to the increase in the business revenue of the organization. It would involve assessing how training has impacted the organizational performance measures such as percentages, turnover, retention, wastage, and so on. It might need to involve intervention of business intelligence systems wherein performance (learner) measurement systems can be integrated into business (management) data and the data computed for the impact that the learner’s improved performance has on the business bottomline. It is not easy to make evaluations at this level because the business results are a lot dependent on the external factors and evaluating the impact of the training on business results might get clouded. Howsoever tough evaluating results at this level be, this is the only level that actually matters to the higher management and is capable to justify the training costs.
Please note that Levels 3 and 4 require inputs from the line managers --- learner’s inputs might not be adequate. These are the levels that actually determine the effectiveness of the training and whether it is beneficial for the business cause or not.
My Take
If I was to come up with a quick action plan to determine training/learning effectiveness, I would consider the following for sure:
- See how learners fare: How learners fare in the in-line exercises (of big modules) and assessments (of all modules) is a fair indication of how effective the training has been to the learner.
- Ask for their experience: The learner can be asked to share the learning experience elaborately. Learner becomes the customer here and can judge our product (the learning) in a better way. Many a times, looking at the learning from second pair of eyes helps to identify the training gaps. Once you are aware of the gaps (this part evaluates the effectiveness of the training), you can fill them up to deliver a more wholesome training (this part is the impact of evaluating the training).
- Monitor their work at regular intervals: Ask the learners as well as their peers, customers, and managers for their feedback on the skill imparted by the training at regular intervals, say 0.5, 1, 2, 3, 6 months. Ask them what is lacking in the skill that they learned in the training, how comfortable they are w.r.t. demonstrating that skill under different work conditions, and so on. This will help you evaluate how effective the training has been from learner’s perspective.I am still a bit confused as to how to evaluate training from business results perspective. To me it appears that filtering out the external factors is too big a challenge and requires careful administration of the impact of learner’s performance improvement on the business bottomline. Can an LMS do that? I am very doubtful. Suggestions, thoughts invited!
I personally feel that any training that does not contribute positively to the business results is futile, from organization’s point of view; and therefore, all trainings should be evaluated, at one level or the other, to determine their worth of delivery to the learners, by the organization.
Web 3.0 marks the third generation of Internet that is supposed to coincide with the third decade of Web browsing and usage (2010-2020). After Web 1.0 (Internet prior to 2000) and Web 2.0 (2000-2010), the expectations from the Web and its possibilities are growing. Researchers, professionals, and entrepreneurs are looking at Web to give more, including better search results. In today’s world of Web 2.0, if you want to plan a vacation for 4 days with 3 friends within a budget of 60K, you will probably run some Google searches on hotels, resorts, eating joints, tourists places, travel agents; read reviews on almost most of them; and then, choose a place that fits your budget and your idea of a vacation with friends. In the whole process, you will most likely browse not less than, say a dozen or two dozen Web sites. Now, think that you key in “Vacation options for three people within a budget of 60,000 INR, in India, for 4 days.” and your search engine gives you planned packages that include exactly where all you can go within the stipulated budget. It means getting a planned package as meticulously done as would be by a travel agent. That is not it. Imagine the more you use this Web browser to carry out your searches, the more the browser knows about your choices, and the less specific you need to be with your search, meaning that even a simple search such as “Where do I go for a vacation with my friends” can yield satisfactory results. You think that is a dream. Then, let me share that the time to live that dream is about to come. Yes, Web 3.0 works on such standards that allow Web to think like you do and make intelligent decisions.
Web 3.0 will enable browsers to read, understand, analyze, and deliver results as humans do. This is the basis of a Semantic Web---a concept that many take synonymous to Web 3.0. Web 3.0, however, encompasses much more than Semantic Web vision (a vision that contemplates that machines will read information on Web like humans do). Let’s see what all Web 3.0 will be capable of?
Web 3.0 will be able to carry out natural language searches. Today, when you key in phrases like “Indian Lion”, the Internet does not recognize difference between the searches as “Lions in India” or “Indian Lion” and throws up all kinds of results. Web 3.0 will understand the language as we humans understand it and deliver results in that manner, specific to what we are asking.
In addition, the 3.0 version is postulated to be mediacentric. It means that no longer you need to depend on wordy keywords to execute a search. You can post media, such as images, to get the desired search results. For example, you can post the image of a flower and get the related results on where you can get, buy, or learn more about that particular flower. Awesome! Isn’t it? An extension of this search is also being tried out with the audio files.
Adding to its claim, Web 3.0 will be pervasive in nature---it will be everywhere. These days, reach of Web has extended from desktops and laptops to mobile phones and handhelds. Web 3.0; however, plans to take another step forward and you can make Web a part of your watches, jewelry, window panes, and bathroom mirrors. Imagine your boring toothbrush routine turning into news time as you could turn your Web-connected bathroom mirror to a Web screen. This is not all. Many researchers are working on enabling the Web to control opening and closing of your lobby windows depending on the temperature and weather at your location.
Another significant development that is expected to happen in Web 3.0 environment is development of more sophisticated 3D applications. A similar example that can be experienced these days is the Second Life, an environment in which the users can live a virtual life in the form of any Avtar that the user wishes to choose. The user can communicate/interact with other members in the virtual community. This, with Web 3.0, can be taken to another level, wherein you can move in your neighborhood or an unseen foreign city, look at the location of other households or shops, and decide where you want to roam around or think of buying land. Say, your virtual Avtar can roam within a radius of 10-kilometes around Jaipur airport where the land is on sale, read the details of the plots up for sale, and decide where you want to invest your money.
Too futuristic, as it may sound, however, you will begin to see the changes soon. Many companies like Ojos and Polar Rose are working on developing the mediacentric version of Web. HP, Yahoo, and Radar Networks are adopting the Semantic Web standards. Microsoft and Google are researching on 3D. In addition, IBM and Oracle are using and developing structures and technologies that closely resemble the Web 3.0.
How Will Web 3.0 Impact E-learning?
Web 3.0 promises to be a Web of data that will read your response, search the Internet for possible options, and then intelligently evaluate the best result for you to throw up in your search.
I would like to think aloud here. Imagine that E-learning companies offer their training modules through Web, with no third party involvement. Suppose an E-learning company posts, on the Web, the overviews, costs, and time conditions of professional courses on different skill sets. So, a person who searches for the courses that he or she needs to undertake to reach a particular position in the industry, a short write-up about the appropriate course might appear, including link to take the course. This way a learner can buy the course directly from an e-learning organization removing all third parties involved.
Another point that I would like to make is that e-Learning consultants might find Web 3.0 as a competitor as well as a great help to their profession. Which part will weigh more, I cannot say. When Web 3.0 comes into picture, anyone can run a search reading “Best strategy for time management (soft skill) course for professionals with 0 to 1 year of experience” or “Problem solving course for professionals with 0 to 1 year of experience” will turn up results for best strategy or rather the course itself. Think about it. Is that possible? I think it is although I am not sure how soon. With such results, however, consultants might find their job a bit replaceable.
Imagine if a Program Manager needs to buy an LMS, they can run the search and see which one suits their needs and requirements the best, without needing to browse through a dozen odd sites.
With intelligent searches, students will be able to search for courses, colleges, and universities that will best fit their priorities. Web will be able to help them make better career decisions.
The information is all out there however it is still in the unstructured form. With Web 3.0, the information will get structured and we can expect intelligent computing wherein computers will (almost) think like humans to make decisions on financial planning, retirement planning for a couple, education planning for a kid considering the financial needs of the family, market research that big organizations want to carry out. It is nebulous to say if artificial intelligence will be more of a cause or an effect of Web 3.0, however, one thing that I am sure of is that this new version of Web, once in place, will ease many tasks, give better computing results, and save our time. It will however demand a lot of research into hardware, software, infrastructure, platforms, and so on, before coming into the big picture.
Accessibility refers to making the products and services accessible to all, including people with disabilities (PwD) and a lot has been talked about it as a social, philanthropic effort, almost so much so that it sounds like a corporate social responsibility (CSR) initiative. Do u really think that big companies, such as IBM, Microsoft, and Cisco, are continuously improving their products, services, and processes w.r.t. accessibility to accommodate only a social cause? Well, I do not think so. Addressing social needs might be one reason…one small reason I would say, however, there is more to such an initiative.
Big companies are claiming to make the outputs of their businesses, such as products (phones, laptops, computerized kiosks), documentation, and services (learning solutions, Web sites), accessible or available almost equally to all, be it a professional, or an older person, or a PwD. And, the reason given for such a move, most of the times, is to provide equal opportunities to all. That, in my opinion, is only partly true.
It Means Business
An important reason for implementing accessibility is the big money involved. As far as the statistics goes, 6% of world population is suffering from disabilities, and around 19% of the U.S. population contributes to this number. Now, these people are customers with special needs who need accessible products and services to make the transactions that other people make. Also, PwD are all the more dependent on Internet and technologically advanced solutions to make their day-to-day operations a bit smoother. Technology can assist them, like other non-disabled people, right from paying off their bills sitting at home to controlling their businesses from a location of their choice. However, PwD need these services more and the reason is obvious. Therefore, if companies are implementing accessibility in their products and services, they are very well capturing a new customer set (PwD) for their products that others did not even consider.
In addition to the customers that companies can capture and satisfy with their accessible solutions, companies can also hire and retain employees who are good at work however did not get hired because the company lacked accessible products and services. Apart from PwD, some people face disabilities because of age—many old people develop certain disabilities with age and if the companies do not retain them as employees just because of their recently acquired disability, companies lose on the expertise that these people have earned through their experience. And this population of aged people is above and beyond the 6% (PwD, worldwide) and 19% (PwD, U.S.) statistics.
Now, think of the revenue that a company can earn by making new customers, catering to them, and retaining diverse employees. And then, philanthropy just becomes one of the peripheral advantages that emerge out of this business need called accessibility.
So, when IBM introduces shortcuts/special buttons on this Think Pad (for people with locomotor disability), or Cisco launches accessible phones (that converts speech to text) for hearing-impaired people, or Microsoft releases Windows Speech Recognition in its Windows Vista (to enable people with locomotor disability to interact with heir PCs by their voice), I think there is more than a social cause to it---it all translates to wider customer base and therefore more business.
Competition Demands It
Another important reason of implementing accessibility is that in these times of severe competition, you cannot and rather would not like to be left behind. All big organizations are involved in addressing needs of PwD, be it visual, auditory, speech, locomotor, neurological, or cognitive disabilities. IBM and Microsoft are actively involved in implementing accessibility in their products since early 1980s. And now Cisco is also joining the bandwagon by having an Accessibility Unit for the organization that takes care of accessibility needs for the organization on whole. When the top-notch players are competing in providing accessible solutions, other companies are bound to follow the suit.
Clients Demand It
Many companies, such as Cisco, have U.S. federal government and other U.S. state governments, as their clients. For these companies, with such clients, it becomes a stated requirement to make their deliverables accessible because they need to comply with Section 508 of the Rehabilitation Act (in U.S.). This act requires all products and services to be accessible to all in the public. With more states and clients asking for accessible solutions, be it in learning or product domain or otherwise, the demand of accessible solutions will rise and an organization can only be ignorant to not proactively learn more about it.
Pleasing Customer
Making the products and services accessible to PwD makes things easier for the other lot too, i.e., people who do not have disabilities. If an organization like Microsoft allows the flexibility to use the IE7 browser in a way that best suits customer’s needs and preferences, it benefits the PwD and others alike. It becomes a comfort for all to be able to customize text, style it, and color it; use keyboard shortcuts; use the zoom in and zoom out functionality for the Web content, and so on. This makes a customer of your product happy and the more customers you please; the more you make your business grow. After all, customer is God.
Accessibility has already gained a lot of momentum within big companies and the medium sized companies as well are catching up with it. The management in the latter should consider proposing accessibility as a part of the solutioning for the client. Because if your product is good and the client intends to use it for long, it might not be very far that the client comes back to you saying he wants to make it accessible and then turning back the wheel would be a tedious process. So, treating accessibility as a solution that a business can offer to make better products and services for PwD and otherwise is a wise thing to do---a wiser business proposal than a philanthropic effort.
Instructional design is all about delivering information to a known learner in the most effective manner. It is about managing learning by creating effective instructions. Instructional designing is a field that leverages the human art of teaching/facilitating learning by studying the difference between the current (present) and ultimate (desired) skill/knowledge set of a learner. It revolves around bridging the difference between the two. Simple as it might sound and so it is; however, to say this, you need to understand it. As I understand it, instructional deigning is keeping yourself in the shoe of a learner and (1) imagining how you would learn best---by what means, by which content, by which delivery medium, and so on, and (2) developing and delivering content that fulfils that imagination.
Like any other field of science, instructional design owes its present state of understanding to various theories, models, and scientists (mostly psychologists and behaviorists). Apart from talking about the most significant contributions that all of them have made to instructional designing, this blog will present and discuss the new and emerging changes in this field.
