This is a guest post co-authored by Asem Akhtar from AWS and Vimal Sharma, Patricia Yi, Kevin McAllister and Jennifer Kearney from NBME (National Board of Medical Examiners).
NBME (National Board of Medical Examiners) advances healthcare quality for the public by offering high-quality assessments and learning tools for doctors and other health professionals, such as student self-assessments that help people gauge their level of preparedness for licensing exams.
Our data and analytics team delivers learnings to external customers and internal staff. We were using a variety of business intelligence (BI) tools, but we had some limitations, such as frequent staff time for tasks like upgrades and administrative activities and trouble ingesting large files with many columns and rows.
We were already modernizing the data and analytics solution delivery from Oracle’s data warehouse to AWS data and analytics services. As part of that, we decided to consolidate our BI tools and find a better, more robust, and cost-effective solution.
In this post, we share how NBME used Amazon QuickSight, a unified, hyperscale BI service, to provide meaningful data and analytics for a variety of users.
Evaluating self-service BI software options
We evaluated several highly-rated software products: MicroStrategy, Qlik, QuickSight, and Tableau. We were looking for a solution that would fit neatly in the AWS ecosystem, support embedding, offer generative BI capabilities (like answers to natural language queries), and would be cost-effective for a large user population.
We also wanted to provide our student test-takers with a tool that would give them a deeper understanding of their results. We already provided an interactive reporting service, but it offered only limited details and no cross-exam functionality.
Security was another big factor. Our huge external user population—including about 90,000 students a year—is continually activating and deactivating accounts, which has led to maintenance challenges. We needed a way to manage external access without adding a single sign-on (SSO) solution.
With all the capabilities we wanted and pay-as-you-go pricing, QuickSight was the clear winner.
Implementation: From QuickSight adoption to 30,000 users in under a year
We began our QuickSight journey by creating an external-facing dashboard that provides medical students with a comprehensive view of their performance and clear indications for how to improve—we call it INSIGHTSSM. We started building this value-added product in Q2 of 2022, had a proof of concept going that used static data from various sources by the next quarter, and completed the end-to-end solution by Q1 2023, which went live the next quarter.
In this use case, data is pushed from various sources, including third-party, on-premises, and cloud, to Amazon Simple Storage Service (Amazon S3). It’s transformed using AWS Lambda and AWS Glue, and orchestrated via AWS Step Functions to land in our warehouse, Amazon Redshift. We use Amazon CloudFront to access secure documents on Amazon S3 and Amazon CloudWatch to log activity. NBME classifies Score Reports as protected information. We need to ensure secured documents can be accessed securely.
The following diagram illustrates the solution architecture and workflow.
Student test-takers access QuickSight via an anonymous embedding process with tags (we use Active Directory SSO for our internal registered QuickSight users). Anonymous embedding removed the heavy burden of maintaining user accounts by securely providing data to a moving target of tens of thousands of users who are constantly activating and deactivating their accounts.
User experience
Learners launch INSIGHTS from MyNBMESM, which is built in Salesforce. MyNBME is a portal where learners can order new assessments in addition to accessing score reports and review information on previously completed assessments, as illustrated below.
After launching INSIGHTS, the Question Details tab provides feedback into the questions the learner saw in the exam. For Self-Assessment exams, learners can review the questions they completed to find the objectives, correct answers, and rationales for the answers via live links in the data table that open this specific exam content in a separate pop-up window. This level of information makes it easier to prepare for their higher stakes assessments. Interactive filters and controls allow users to further drill down to specific content and topics as needed.
Users can also see patterns and trends if they’ve taken multiple versions of the same exam, including their performance comparisons (up or down) in certain categories in the Results Comparison tab. This view also contains filters to assist learners with tracking their progress over time across multiple content areas.
Results and benefits
INSIGHTS is a significant improvement in terms of the data analysis experience for students with over 80% of voluntary survey responses collected since September 2023 rating the overall experience as “good” or better and over 63% finding the dashboard to be “somewhat” or “much better” at helping them “identify areas requiring further study” when compared to the feedback options previously provided. It’s also faster—the new solution allows us to make the data available via QuickSight within our 2-hour SLA. The time between taking an exam and getting access to the data reduced from 4 hours to less than 2 hours.
QuickSight fits seamlessly into our AWS stack, making it straightforward to implement. The AWS team helped us bridge the learning gap, which made for an on-time and on-budget implementation.
We served over 30,000 users in the first 4 months of this INSIGHTS dashboard going live for learners who purchased our exam products—and we expect to serve up to 90,000 users a year.
What’s next
We will be expanding the use of INSIGHTS to include additional assessment types from new data sources in Q1 of 2024. We also have several initiatives to support internal functions in proof of concept and development stages.
We’re looking forward to using more amazing features from QuickSight, like generative BI, on our internal dashboards to reduce duplicative dashboards that are user-specific and reduce the time to deliver data and analytics.
Get started with QuickSight
QuickSight is an important part of our modern scalable data and analytics architecture. It connects to our AWS infrastructure and combines data from on-premises and third-party sources into seamless and robust user experiences. The pay-as-you-go pricing and ability to securely provide data to users via an anonymous embedding process empowers us to keep building and enhancing our data products. To learn more, visit Amazon QuickSight.
About the authors
Vimal Sharma is the Manager of Data Engineering and Analytics at NBME. He delivers data and analytics products to internal and external customers of NBME and co-leads data strategy and analytics modernization.
Patricia Yi is a Senior Data Engineer at NBME with over 14 years in data warehouse and business intelligence solutions.
Kevin McAllister is a Product Manager at NBME with more than 18 years of experience in educational assessment and services. He is working closely with internal staff and listening to NBME customers to improve and enhance the INSIGHTS experience.
Jennifer Kearney is the Technical Product Owner working with the Data and Analytics scrum teams at NBME.
Asem Akhtar is Sr. Solutions Architect with Amazon QuickSight at AWS. As an experienced leader with over 24 years of professional experience in business intelligence and analytics, he has guided fortune 500 and public sector customers in delivering state-of-the-art data warehousing and visualization solutions. Prior to AWS, he spent several years as a MicroStrategy SME.
This is a companion discussion topic for the original entry at https://aws.amazon.com/blogs/business-intelligence/nbme-launches-insights-for-their-medical-examination-data-using-amazon-quicksight/