Core DX Technology Stack

Overview

This section describes the Open Source Software (OSS) stack used in the development and deployment of the Data Exchange (DX) platform.

The stack consists entirely of open-source components, used in their original form, that collectively support data ingestion, processing, discovery, access, and distribution. Each layer of the stack delivers a specific functional capability and contributes to the platform’s scalability, reliability, security, and operational efficiency.

img.png

Figure 7: Core DX Technology Stack ⚠️ Diagram to be updated / added

Functional Layers and Technologies

The DX technology stack is organized into the following functional layers.

API Server Layer

Used for building high-performance, event-driven APIs:

  • Vert.x

  • Node.js

  • OpenAPI

These technologies enable scalable, non-blocking API services with well-defined contracts.

Data Broker Layer

Supports asynchronous communication and event-driven architectures:

  • RabbitMQ

RabbitMQ acts as the message broker for decoupled service interactions and streaming workflows.

Data Ingestion Layer

Used for data ingestion, transformation, and streaming:

  • Python

  • Apache Flink

  • Logstash

This layer enables both batch and streaming ingestion pipelines with transformation and preprocessing capabilities.

Development and Testing Tools

Support application development, scripting, and testing:

  • Java

  • JavaScript

  • Bash

  • JMeter

  • Postman

These tools are used for service development, automation, performance testing, and API validation.

Data Store Layer

Provides persistent storage, indexing, caching, and audit logging:

  • Elasticsearch – Search and indexing

  • PostgreSQL – Relational data management

  • Redis – In-memory caching

  • ImmuDB – Immutable audit logging

Together, these systems support diverse data storage and access requirements.

API Gateway and Security Layer

Implements gateway functionality, authentication, and secure communication:

  • NGINX

  • Keycloak

  • SSL / TLS

This layer enforces security, authentication, authorization, and traffic management policies.

Deployment, Scaling, and Clustering

Enables containerization, orchestration, and scalable deployment:

  • Hazelcast

  • Docker

  • Kubernetes

  • Rancher

  • Zookeeper

These technologies support cloud-native deployment, clustering, and horizontal scalability.

Logging and System Metrics

Used for observability and operational monitoring:

  • Micrometer

  • Loki

  • Prometheus

This stack provides metrics collection, log aggregation, and runtime visibility.

User Interface Layer

Used for building platform user interfaces:

  • Angular

  • React

These frameworks support modern, responsive UI development.

System Monitoring and Visualization

Provides dashboards and visualization for metrics and logs:

  • Grafana

  • Kibana

These tools enable operators to monitor system performance and analyze operational data.

Summary

The Core DX technology stack is designed to:

  • Support cloud-native, scalable architectures

  • Ensure security and compliance

  • Enable efficient data ingestion and distribution

  • Provide strong observability and operational control

This makes the DX platform well-suited for modern, large-scale data exchange deployments.

PreviousAnalytics and Sandbox Technology StackNextDeployment Architecture

Last updated