Revolutionise the Data Lake through Kafka

When an organization establishes a data lake that serves as its source of truth, relying on batch jobs to collect data may prove inadequate resulting in suboptimal decision-making driven by outdated information. This session will dive into the basics of integrating event streaming into the Delta Lake, with Kafka at the center of it all.

As we shift from on-premises to cloud-based infrastructure, obtaining near real-time data in a scalable and dependable manner becomes imperative. This presents a unique opportunity to explore creative solutions that can endure the test of time. Join us as we explore the intricacies of incorporating data from various sources, such as SAP, into the Delta Lake, with Apache Kafka serving as the cornerstone of this process.

Through this solution, you will discover how to combine various components of AWS, and Confluent using several use cases. The process starts with the architecture of a real-time replication, followed by an overview of the connectors setup on Kafka Connect clusters in AWS Elastic Containers service (ECS). Then go together through the process of extracting data from various sources e.g. SAP tables with delta-enabled capabilities and/or Website data into Kafka Topics. Next, the Kafka Connector performs pivotal transformations, reaching the final step of piecing together the data in the Delta Lake. This way, the data can be used for real-time analysis and recommendations in client-facing applications.

During this session, we'll explore the advantages of emphasizing automation to streamline the implementation process using Infrastructure as Code - Terraform, and the overview of integrating event streaming into the Delta Lake.