Engineering a Blended Learning Solution for a Globally Dispersed Team of Auditors
A Case Study in Strategy, Scaling, and Global Reach
Embedded Files
The Challenge
The Challenge
The assurance practice of a mid-size public accounting firm wanted to expand its business of auditing client employee benefit plans (EBPs) while fulfilling continuing professional education (CPE) requirements for its professionals.
EBPs are programs offered by employers that provide extra perks to their employees (beyond regular wages), such as health insurance, retirement savings, or paid time off. EBP audits are instrumental in protecting employee retirement savings from errors, mismanagement, and other risks.
The firm's strategic business requirement created a significant training challenge:
Prepare a team of globally-dispersed professionals, many of whom are new to EBP auditing, to perform this highly technical, compliance-critical, deadline-driven procedure. The training simultaneously must fulfill continuing professional education (CPE) requirements as set forth by the National Association of State Boards of Accountancy (NASBA) and the American Institute of Certified Public Accountants (AICPA).
The Business Goals
The Business Goals
Increase and upskill the number of assurance professionals capable of conducting EBP audits to provide additional value for existing clients.
Generate new business development opportunities in this practice area.
Fulfill NASBA CPE and AICPA EBP training requirements.
My Roles
My Roles
Instructional Designer
Project Manager
Producer
Moderator
Analysis
Analysis
I led the analysis phase in partnership with the EBP Practice Leader, identifying knowledge gaps across global teams. I conducted my analysis of the challenge, which revealed several key factors that informed our instructional strategy.
Audience: The team consisted of new auditors across two different countries (U.S. and India) with varying levels of experience. Some were brand new to the profession, while others had 1-2 years of general auditing experience but had not yet conducted a specialized EBP audit.
The Constraints: The subject matter was dense and complex, covering specific regulations like the Employee Retirement Income Security Act of 1974 (ERISA), knowledge of the various types of EBPs, and technical accounting skills such as determining population sample size along with a wide variety of statistical testing. The content required both foundational knowledge and opportunities for practical application. Budget and resource limitations precluded flying in the assurance team and instructors from across the U.S. and India to a central training location, which necessitated a scalable and remote delivery model.
Solution and Instructional Strategy: To deliver this solution, I designed a blended learning solution consisting of:
Self-Paced e-Learning for building foundational knowledge, allowing learners to master core concepts at their own pace several weeks ahead of the VILT/ILT portion of the blended learning solution. Completing this e-learning would allow the learners to dedicate more time during the VILT/ILT portion to practice hands-on activities and complete case studies.
Virtual Instructor-Led Training (VILT) to reach the team working across the U.S. in different time zones, focusing on interactive debriefs, Q&A, and peer-to-peer learning activities (case studies) leveraging the breakout rooms feature in Zoom.
Instructor-Led Training (ILT) delivered in India by senior-level auditors working locally, providing hands-on application and personalized support during normal (IST) business hours. Since we had experienced auditors working in the offices who were experienced in teaching, a smaller class size, and enough training space, we ultimately decided to hold the training in-person for our colleagues in the India offices.
Design
Design
With the instructional strategy formulated, I continued my partnership with the stakeholders to set about the task of designing and developing the course.
To guide the design phase, I first wrote the performance objectives for the training, which detailed what learners needed to be capable of performing after successfully completing the course. These performance objectives were supported by enabling objectives that defined the specific in-course knowledge and activities necessary to help the learners achieve their final performance goals. Coupled together, these objectives provided the blueprint for developing the course.
Bloom's Taxonomy and the appropriate Bloom's action verbs were used to write these objectives ensuring the course would have measurable outcomes. Since this was a foundational level course, the taxonomy of the course ranged from knowledge and comprehension to analysis and evaluation.
The screenshot above maps the performance objectives (left column: what the learners will be able to do after completing the course) with the enabling objectives (right column: what specific in-course knowledge and activities will be provided to help learners achieve the performance objectives).
With the performance and enabling objectives established, I applied backwards design by focusing on activities that mirrored what the learners would be doing in the field after completing the course.
The activities provided an opportunity for the learners to practice conducting a variety of participant transaction tests used during EBP audits. EBP participant transaction tests are checks the auditors perform on items such as contributions, withdrawals, and distributions in an EBP to make sure each employee's account activity is accurate and follows regulations.
I focused our efforts on ensuring there was enough time built in for the instructors to debrief the activities with the learners, provide the correct solutions, and answer questions. The activity and the debrief were a critical part of the instructional strategy since it allowed learners to immediately apply what they've learned alongside their peers under the guidance of experienced instructors.
The above is a sample screenshot from a part of the design document I created for the VILT/ILT course. This section outlines the activity (case study) on eligibility testing and distributions and the key takeaways of the activity.
Once the activities were determined, the subject matter experts (SMEs) and I were prepared to fully develop the course content necessary for learners to meet the performance objectives.
Once I had the objectives, outline, and design document completed, I was able to identify the foundational concepts and knowledge I would use to create the e-Learning portion of the course as "pre-work" for the VILT/ILT portion. By covering this content in the e-Learning, it freed up additional time in the VILT/ILT course for learners to engage in hands-on activities and learn alongside their peers.
As a result of my work during the analysis and design phases, I effectively transitioned to the development phase where I had three courses being developed simultaneously:
EBP Basics
(U.S. and India)
(Click to expand)
Description: This interactive self-study course will introduce professionals new to auditing employee benefit plans (EBPs) to the basics of EBPs, familiarizing them with standard audit terminology related to EBPs, orienting them with the Firm’s workpapers used to conduct EBP audits, and introducing them to the software systems used during EBP audits. This self-study is a prerequisite for the EBP Fundamentals course.
Audience: Teams in both the U.S. and India with 0-1 years of experience auditing EBPs
Modality: e-Learning (Asynchronous)
Time: 1 hour
Credits: 1 NASBA CPE Credit
Field of Study: Auditing, EBP
Prerequisites: None
EBP Fundamentals
(U.S.)
(Click to expand)
Description: This interactive virtual instructor-led course will prepare professionals new to auditing employee benefit plans (EBPs) on identifying the unique aspects of EBPs during their audits, determining sampling populations and sample sizes for statistical testing, and conducting a variety of participant transaction tests to ensure EBPs are meeting compliance requirements and fundamental fiscal responsibilities.
Audience: Teams in the U.S. with 0-1 years of experience auditing EBPs
Modality: Virtual Instructor-Led Training (VILT) (Synchronous)
Time: 6 hours
Credits: 6 NASBA CPE Credits
Field of Study: Auditing, EBP
Prerequisites: Successful completion of the EBP Basics course
EBP Fundamentals
(India)
(Click to expand)
Description: This interactive instructor-led course will prepare professionals new to auditing employee benefit plans (EBPs) on identifying the unique aspects of EBPs during their audits, determining sampling populations and sample sizes for statistical testing, preparing the System and Organization Controls (SOC) 1 report, and conducting a variety of participant transaction tests to ensure EBPs are meeting compliance requirements and fundamental fiscal responsibilities.
Audience: Teams in India with 0-1 years of experience auditing EBPs
Modality: Instructor-Led Training (ILT) (Synchronous)
Time: 7 hours
Credits: 7 NASBA CPE Credits
Field of Study: Auditing, EBP
Prerequisites: Successful completion of the EBP Basics course
Development
Development
Partnering with the stakeholders and SMEs, I applied the 4-MAT Model and Gagne's 9 Events of Instruction to the development process.
Engineering the Learning: 4-MAT provided the macro-design and larger structure for the course, while Gagne provided the micro-design filling in the finer, detailed learning objects within the 4-MAT structure.
(4-MAT) Why:
e-learning, VILT, and ILT: The "why" is where I connect learners with the content. I ensured each module and concept clearly communicated why this information was important for the learners and what's in it for them (WIIFM).
Gain Attention (Gagne Event 1):
I preferred to use a bit of theater and storytelling for the opening. For example: "You are responsible for protecting people's retirements. At times, you will be the only thing between the quality of somebody's life after retirement and the myriad of risks that can take that away from them. You do important work and will likely save many families from financial ruin, though you will never know exactly how many you've saved."
VILT, ILT: Icebreakers were used in these modalities to grab the learners early and put them into their breakout rooms/groups to break down the barriers to peer-learning early. Applying gamification, I built a Jeopardy Board using concepts from the e-Learning to get the learners working together, recall core concepts, and gain their attention.
The screenshots above capture the Jeopardy Board I created to use as an icebreaker while also testing knowledge from the e-Learning module. The student group would choose a category (column) and a clue with an assigned point value. The clue would display, each student group would come up with an answer, and "buzz" in. The correct answer is displayed and points assigned when a group correctly answers.
Inform Learners of the Objectives (Gagne Event 2): The performance and enabling objectives were summarized here to directly inform the learners what exactly they should be able to do after completing the course and what they'll do during the course to get there.
The screenshots above from the e-Learning course displays the course learning (performance) objectives at the start of the course. As the learner progresses through the course, module learning (enabling) objectives broadcast what will be covered in the module followed by a visual progression marker showing the learner which performance objectives they've achieved at the end of each module.
The screenshots above from the VILT/ILT courses display the course learning (performance) objectives at the start. As learners move through the course, the module (enabling) objectives broadcast what will be covered in the module.
(4-MAT) What:
e-Learning, VILT, ILT: I procured information from the IRS website, internal firm resources, and developed content from scratch leveraging SME knowledge. I also leveraged Co-Pilot to develop outlines, overviews, and content which was reviewed for technical accuracy by the SMEs.
Recall Prior Learning or Experience (Gagne Event 3): I used short introductory activities or overhead/polling questions to set up and introduce topics such as "Why are EBP Financial Statement Audits so important?" or "How do you think we use the SOC 1 Report?", etc. to stimulate knowledge recall and prior relevant experience so I could begin connecting learners to the content.
The screenshots above capture a matching activity in the e-Learning course stimulating recall of content as the learner moves through the course. These are interactive knowledge checks designed to help the learner recall the different types of EBPs.
The screenshots above capture some of the overhead questions used in the VILT/ILT course to introduce new topics, get the learners thinking and talking, and to connect the content to any prior knowledge or experience.
Present the Content (Gagne Event 4): This is where the chunking of content, writing, and flow come together to make the content digestible and present it methodically. I strive to chunk and structure the content as much as possible so learners do not get overwhelmed (re: cognitive load theory) and are easily able to follow the flow of the course.
The screenshots above capture the table of contents shown in the e-Learning course as well as the design document demonstrating alignment of course material to the learning objectives. The e-Learning course content was chunked where core concepts and terminology were introduced first as the rest of the content built on those core concepts. Ultimately, the e-Learning course itself was positioned as the first building block of the EBP curriculum.
The screenshots above capture the timed agenda for the VILT/ILT and how the course content was chunked. The total time learners would spend listening to lectures and completing activities is also calculated confirming the majority of the time was dedicated for learners to complete activities and case studies so they could apply what they were learning.
Provide Learning Guidance (Gagne Event 5): I strive to leave "air time" in the VILT/ILT course agenda to allow time for questions from the learners. The Q&A time is a direct method to help guide the learning. In addition to "air time", overhead/polling questions and short activities allow "just-in-time" teaching if the audience is having difficulty grasping a concept after being given a chance to apply it. In the e-Learning course, evaluative feedback for both correct and incorrect answers were provided for review questions.
The screenshots above capture one of the review questions in the e-Learning demonstrating the evaluative feedback provided to the learners for the correct and incorrect answers.
The screenshots above capture a flip chart activity used in the ILT course to allow the learners immediate practice and application of concepts with real-time feedback from the instructor. In the VILT course, the screenshot shows a polling question as one example of providing a learning opportunity with a quick knowledge check and evaluative feedback from the instructor in real-time.
(4-MAT) How:
e-Learning, VILT, ILT: SMEs developed mock workpapers for the participant transaction testing activities in the VILT/ILT course to mirror real-time work the auditors would be performing during their EBP audits. These mock workpapers formed the basis of the case studies the learners would complete with their peers. The e-Learning course mainly used matching activities and review questions to assess the learner's progress formatively. e-Learning courses can accommodate peer-to-peer learning via online discussion forums where learners respond to a prompt and provide replies to responses from at least 2 of their peers. In this particular case, the e-Learning was crafted primarily as a self-study compliant with NASBA Quality Assurance Standards (QAS).
Perform Activity/Case Study (Gagne Event 6): The hands-on activities and case studies are the "star of the show" where learners are provided the opportunity to engage in peer-to-peer learning (re: breakout rooms/groups) and apply what they've learned in a safe and open learning environment.
The screenshots above are from the VILT/ILT courses setting up the case study by providing the learners with the purpose of the case study and the instructions. Learners in the VILT course were placed into breakout rooms in groups of 5-6 learners. In the ILT course, learners were grouped in teams of 3-4 learners. In both modalities, the learners were given 40 minutes to work together on the case study. This mirrored actual field work. Instructors in the VILT would periodically visit the breakout rooms to answer questions and provide guidance, but not to do the work for the learners. The same approach was used with the ILT instructors as they walked the room and answered questions/provided guidance.
The screenshots above are from the workbook provided to the learners. The mock workpapers developed by the SMEs were inserted into each tab of the workbook and formed the basis of the case study the learners had to complete. The learners, in their groups, would use the workbooks to record their work and their answers. Groups were instructed to choose a "spokesperson" to provide the answers when called upon during the debrief.
Debrief/Feedback (Gagne Event 7): Time was set aside specifically for the instructors and the learners to debrief the case studies, go over their findings, ask/answer questions, and ultimately provide the correct solutions and key takeaways. The interaction between learners, their peers, and the instructors during the debrief is where "the magic happens" in the learning process as immediate and specific feedback is provided with an opportunity for the learners to ask follow-up questions.
The screenshots above show the amount of time and the aim of the debrief session, which immediately followed the case study. The answer key was also provided to the learners providing the correct answers as part of the debrief.
(4-MAT) What-If:
e-Learning, VILT, ILT: The e-learning course built the foundation for learners to progress to the EBP Fundamentals course. Its ‘What-If’ activity ensured core concepts carried forward into the VILT/ILT session, where learners would deepen their knowledge. To meet NASBA QAS and reinforce retention, a course exam assessed performance and supported knowledge transfer. In the VILT/ILT course, the ‘What-If’ focused on applying new skills in the field. Debriefs after each case study served as performance checks and reinforced learning. Providing answer keys, participant materials, additional resources, and a senior auditor contact list gave learners reference tools for future stretch assignments.
Assess Performance and Enhance Knowledge Transfer/Retention (Gagne Events 8 and 9): The e-Learning course had a course exam to assess performance, ensure retention of core concepts, and prepare learners to move into the larger, more intensive VILT/ILT sessions. The VILT/ILT sessions provided case studies and debriefs on those case studies to assess the learner's performance and to enhance knowledge transfer as well as retention through peer-to-peer learning and immediate feedback.
The screenshots show a resource list at the end of VILT/ILT providing learners with additional reference materials they can use when in the field.
The screenshots also show the course exam at the end of the e-Learning course used to measure performance, enhance retention of material, and meet NASBA QAS requirements for providing CPE credit.
Implement
Implement
With the courses fully developed, it was time to deploy my 3-Step Quality Assurance Protocol to ensure technical accuracy of the content, sound instructional design, adequate preparation of the instructors, and readiness for delivery.
Stage 1: Technical Review
The EBP Practice Leader reviewed all three courses for technical accuracy and signed off on them as the Technical Reviewer. The Director of L&D also reviewed the courses and signed off as the Instructional Design Reviewer to ensure the design was sound and aligned with our best practices. This stage achieved two goals:
The content and design of all three courses were sound, and
Review of the courses by a CPA-licensed professional was documented in accordance with NASBA standards.
Stage 2: Instructor Preparation
Once the courses were reviewed and finalized, I held train-the-trainer (T3) sessions with the instructors (many of whom were also SMEs on the project). To prepare the instructors, I created a Leader's Guide for each course, provided them to the instructors, and held virtual T3 sessions with the instructor teams in the U.S. and India. The T3 sessions covered platform skills, reviewed the content and flow of the course, and demonstrated how the activities would be administered.
The screenshots above from the T3 prepare the instructors by describing the importance of their role and the platform skills used for presenting content virtually and in-person.
The screenshots above from the T3 prepare the instructors by providing best practices and tips for preparation and to have a relaxed approach on the day of the training. In addition, the correlation between course delivery and course design is shown to the instructors to emphasize the importance of delivery and how even the best designed courses may not be as impactful if the delivery is not adequate.
Stage 3: Delivery Readiness
The final stage ensured smooth rollout by addressing logistics such as securing the training space, ensuring internet connectivity, uploading participant materials to the LMS, working with Operations/Events/IT/HR on communications, technology, logistics, etc. A final check on all live delivery assets was also completed at this stage (for the VILT, it was ensuring polling questions were loaded into Zoom, presenter links and invitations were sent, etc.). The Operations and IT teams in India worked with me to help secure the training space and set it up for the instructors and students.
The e-Learning course followed a different implementation rollout than the VILT/ILT courses described above. Implementing the self-study was also a 3-Step Quality Assurance Protocol.
Alpha Review
The course was built using Storyline 360. It was published in Review 360 and the EBP Practice Leader reviewed and signed off on its technical content to close the Alpha Review.
Beta Review
The course (still in Review 360) was then sent to my peers in L&D for design review and to test the interactions in the course for operability. Feedback was provided and implemented. The Director of L&D signed off on its design and technical operability to close the Beta Review.
Gold Review
The course was published as a SCORM file and tested in the SCORM Cloud. This review focused purely on the technology. Once it passed verification in the SCORM Cloud, the course was uploaded to our LMS and tested by my peers in L&D to ensure the course was behaving as expected before closing the Gold Review and assigning it to the target population in the LMS.
Producer and Moderator
During the implementation phase once it was time to deliver the VILT, I shifted my role from instructional designer to producer and moderator.
As the producer, I acted as the "wizard behind the curtain" managing the Zoom platform during the VILT session. I conducted the technical setup and managed the Zoom platform, launched/closed polls, assisted participants with technical issues, kept track of the time, monitored the chat and the Q&A, setup and managed the breakout rooms, shared my screen and drove the slides so the instructors focused solely on presenting the content and leading classroom discussions.
As the moderator, I acted as the bridge between the learners and the instructors as appropriate by kicking off the session, welcoming the learners, covering the housekeeping and NASBA requirements, providing instructions on the breakout rooms and case studies, closing the session, and overall being the "glue" that held the session together socially.
Evaluate
Evaluate
The courses were successfully delivered. Immediately upon completion, I engaged Kirkpatrick's First Level.
This step consisted of course evaluation surveys launched during the VILT/ILT course wrap up in an effort to crowd source fresh feedback from the learners. The same course evaluation launched from the e-Learning as soon as learners completed and exited the course.
The screenshots above show a part of the course evaluation using the Likert Scale. The questions on the evaluation were specific to comply with NASBA CPE requirements. Opportunities for open ended feedback and comments were also provided.
Working with our L&D operations group, a 60-day "transfer of learning" survey was deployed to the learners to measure retention of information (Kirkpatrick's Second Level).
I initiated discussions with stakeholders to establish long-term metrics for further evaluation (Kirkpatrick's Third and Fourth Levels).
Proposals currently under review include:
Level 3:
Establishing key performance indicators (KPIs) - specifically the number of audits completed by the learners after the training, the complexity of the audit completed by the learners, and the amount of time it took them to complete the audit
Interviews with managers to ascertain their observations and feedback of the learners performance on the job
Level 4:
Establishing metrics upfront before the next iteration of the course
Compare pre- and post-training data, namely to track the number of EBP audits before the training vs. after the training and how those numbers are aligning with strategic business goals
The Impact
The Impact
14 hours of accredited training delivered
130 auditors trained
1,820 NASBA CPE credits awarded
100% NASBA and AICPA regulatory compliance
Google Sites
Report abuse