Unlocking Healthcare Insights: Harnessing AI & Advanced Analytics to Decode 30+ Billion Rows of EHR Data

The COVID-19 pandemic brought unprecedented challenges to global healthcare systems, raising urgent questions about disease management, patient care, and public health strategies. Amid the crisis, there was a growing demand from biomedical research organizations, pharmaceutical companies, and government agencies for comprehensive COVID-19-related data. The goal was clear: leverage all possible data to better understand and combat the pandemic. However, a significant challenge arose—the siloed nature of healthcare data.
Healthcare organizations traditionally stored data in separate repositories, making it difficult for researchers to perform cross-sectional analytics across various data types such as Electronic Health Records (EHRs), clinical trials, medical images, and genomics. Recognizing the need for a centralized and collaborative resource, Quantum Insights, in partnership with the United States’ National Institute of Health, spearheaded the creation of a cloud-based COVID-19 data repository and analytics platform called the National COVID Cohort Collaborative (N3C).

Building the N3C: A Collaborative Approach to Data Integration

Quantum Insights utilized Palantir Foundry and AWS GovCloud to develop a cutting-edge analytics platform designed for researchers to analyze diverse COVID-19 data. The platform enabled over 80 hospital systems and research organizations to ingest and harmonize their data using the OMOP (Observational Medical Outcomes Partnership) common data model. By standardizing the data format, Quantum Insights ensured consistency, allowing researchers to perform large-scale analytics across multiple data sets within the N3C environment, also known as the “Enclave.” With the platform established, researchers could access tools and resources directly through Palantir’s analytics capabilities. To further streamline the research process, Quantum Insights facilitated the creation of 30+ specialized research groups, each focusing on specific aspects of COVID-19. These groups collectively answered over 600 research questions, providing critical insights into the virus’s transmission, impact, and treatment.

Tackling Multi-Modal Data with Privacy-Preserving Linkage

One of the key breakthroughs was the development of a Privacy Preserving Record Linkage (PPRL) system, which used cryptographic hashes to link patient data across multiple repositories while protecting privacy. These hashes, generated from Personally Identifiable Information (PII), allowed researchers to connect records without exposing underlying data. A trusted “honest broker” matched the hashes, ensuring that integrated datasets could be accessed without violating privacy regulations or governance rules, such as informed consent requirements.

The Role of AI and Advanced Analytics in N3C

At the heart of the N3C platform lies a robust infrastructure for AI-powered analytics and advanced data processing. By leveraging AI algorithms and machine learning models, the platform processed over 30 billion rows of EHR data. From tracking patient outcomes to predicting the efficacy of treatments, AI has transformed how researchers interpret complex datasets, offering invaluable insights for both immediate crisis management and long-term healthcare planning.

Redefining Healthcare Research Collaboration

Quantum Insights’ work with N3C has redefined biomedical research by integrating AI, advanced analytics, and collaborative governance. The platform securely links diverse data sets and provides AI-driven insights, equipping researchers to tackle future pandemics and advance scientific understanding. As a model for data collaboration and innovation, N3C highlights AI’s transformative role in healthcare research. These capabilities are not only essential for addressing the current pandemic but also set the stage for future applications in rare disease research and pandemic preparedness. Looking ahead, Quantum Insights plans to expand this framework to address rare diseases and chronic conditions, continuing to drive innovation and impact global health outcomes.