Building Data Pipelines with Apache Kafka

Basic Java 8 or Scala knowledge is preferable. Please install Docker and Docker Compose if you want to run examples locally.

Apache Kafka is a distributed streaming platform. It is de facto a standard for building data pipelines and it solves a lot of different use-cases around data processing: it can be used as a message queue, distributed log, stream processor, etc.

We'll start with some theory behind data pipelines in general, then continue with fundamental concepts behind Kafka. We'll also discover important components like Kafka Streams and Kafka Connect.

• Data pipelines 101: ingestion, storage, processing
• Kafka fundamentals: topics, partitions, brokers, replication, etc.
• Producer and Consumer APIs
• Kafka Streams as a processing layer
• Kafka Connect for integrating with external systems
• Kafka best practices and tuning