data_ingestion_adaptors

Data Ingestion Adaptor

A Data Ingestion Adaptor is a specialized software module developed for each ingestion pipeline within the IUDX-Novo platform. It is a key part of the Resource Server operating in Data Ingestion Mode.

The primary purpose of a Data Ingestion Adaptor is to maintain a continuous, secure, and reliable data flow between a Data Provider and the IUDX platform.

Core Functions

In addition to establishing the ingestion pipeline, Data Ingestion Adaptors perform the following functions:

• Securely access data from the Data Provider’s systems

• Remove Personally Identifiable Information (PII) where applicable, in accordance with privacy and governance policies

• Detect and handle duplicate data during the initial preprocessing stage

• Transform incoming data to align with the IUDX platform’s data model and schemas

These preprocessing steps ensure that ingested data is standardized, privacy-compliant, and ready for downstream access and analytics.

Deployment and Customization

• Each ingestion adaptor is tailored to the specific ingestion pipeline it supports

• Adaptors are hosted within the IUDX environment and managed as part of the platform infrastructure

• Custom development is typically required for each Data Provider and dataset type

Ingestion Modes Supported

A Data Ingestion Adaptor may operate in one of the following ways:

API-Based Data Pull

• The adaptor pulls data from APIs exposed by the Data Provider

• Suitable for providers offering REST or streaming APIs

• Enables scheduled or near-real-time ingestion

Push-Based Data Streaming

• The Data Provider pushes data directly into the IUDX message broker

• Uses the AMQPS streaming protocol for secure, reliable delivery

• Suitable for high-frequency or event-driven data streams

Role in the Platform

• Enables scalable and reliable data onboarding

• Enforces privacy and data quality controls at ingress

• Bridges Data Provider systems with the IUDX data plane

---