Host-Based Pipeline Deployment for the Debezium Platform
Extending change data capture to bare metal and cloud VMs through zero-touch host discovery, concurrent scheduling, and Quarkus CDI selection.
Provisioning Pipeline Flow
Edit ~/.ssh/config
Engineer declares target host IP, port, identity file, and SSH parameters.
Host Discovered
Java NIO WatchService detects file modification with debounce and triggers reconciliation.
Ansible Provisions
Isolated thread pool triggers idempotent Ansible playbook, verifies systemd daemon, and pre-pulls ~500MB image.
Host Marked READY
Health checks pass, bearer tokens exchange, and conductor marks pipeline node READY in 10m 43s.
Context & Engineering Challenge
Debezium is the leading open source distributed platform for Change Data Capture (CDC). Previously, Debezium Server pipelines could be orchestrated exclusively through its Kubernetes operator. This architectural constraint excluded engineering organizations operating high-throughput databases on bare-metal clusters, non-containerized private clouds, or dedicated virtual machines.
The host-based deployment path had to ship from the exact same compiled binary image as Kubernetes mode, requiring zero breaking configuration changes for existing enterprise users. Remote Ansible deployment tasks could not block conductor threads, and concurrent host provisioning required strict deadlock prevention.
System Architecture
Architecture Decisions & Trade-offs
Quarkus CDI Selection (@LookupIfProperty)
Leveraged Quarkus @LookupIfProperty to select deployment orchestrators at build/runtime without polluting the core engine with conditional runtime branches. This enables a unified container image with dead-code elimination.
Zero-Dependency OpenSSH Config Parser
Implemented a custom OpenSSH config parser fully covering the ssh_config(5) specification backed by 24 unit tests, paired with a Java NIO WatchService with debounce and scheduled reconciliation.
Deadlock-Safe Scheduling with Sorted Locks
Eliminated database deadlocks and port collisions in high-concurrency environments by implementing sorted PESSIMISTIC_WRITE locks combined with atomic file operations and Command-pattern Ansible runners.
Asynchronous Host Agent Daemon
Architected a lightweight Quarkus daemon deployed via systemd exposing 6 REST endpoints with bearer-token authorization. Endpoints return 202 Accepted immediately so slow multi-gigabyte remote deployments never block central conductor threads.