Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars

Prerequisites:

Participants should have:

• Practical experience with Python programming, including functions, modules and basic object-oriented concepts.
• Basic to intermediate experience working with Pandas and tabular data-processing workflows.
• Basic familiarity with PySpark and Spark DataFrames, including reading data, transformations, actions, joins and aggregations.
• A general understanding of SQL and data-processing concepts, including filtering, grouping and joining datasets.
• Basic familiarity with Docker and Kubernetes concepts, such as containers, images and pods. No Kubernetes administration experience is required.
• Basic understanding of common data formats such as CSV, JSON and Parquet.

Participants do not need to be Kubernetes administrators or infrastructure specialists. The course focuses on how data engineers, developers and data scientists can understand and optimise their Spark workloads running on Kubernetes from the application and configuration perspective.

Target Audience

This course is designed for professionals who develop, maintain or optimise data-processing and machine-learning workloads using Python and Spark in cloud or containerised environments.

It is particularly suitable for:

• Data Engineers working with PySpark, distributed data processing and ETL pipelines.
• Data Scientists processing large datasets or running machine-learning workloads with Spark, Pandas or Polars.
• Python Developers working with data-intensive applications and looking to improve memory efficiency and processing performance.
• Machine Learning Engineers managing data preparation and model-training workloads on shared Kubernetes or cloud environments.
• Analytics Engineers working with large datasets and seeking to improve data-processing efficiency.
• DevOps, Platform and Cloud Engineers who support Spark workloads on Kubernetes and need to understand how application-level configuration affects resource usage and performance.
• Technical Leads and Solution Architects involved in designing or optimising modern data-processing platforms.

Overview:

This three-day practical course focuses on building and optimising efficient data-processing workloads using PySpark, Pandas and Polars in Kubernetes-based environments.

Participants will develop a practical understanding of how Spark applications execute on Kubernetes and how application-level configuration decisions influence performance, scalability, resource consumption and cost. The course covers key optimisation areas including executor sizing, memory allocation, dynamic allocation, partitioning strategies, shuffle behaviour, small-file problems and efficient Parquet processing.

The course also addresses common challenges when working with Pandas, including memory limitations and out-of-memory failures, and introduces Polars as a high-performance alternative for selected data-processing workloads. Through hands-on exercises, participants will diagnose performance and memory issues, compare different configuration strategies and apply optimisation techniques to realistic ETL and machine learning scenarios.

The emphasis throughout the course is on practical decision-making: understanding how to identify bottlenecks, select the appropriate tool, configure Spark efficiently and balance performance with infrastructure resource consumption and cost.

Course Outline:

Module 1: Big Data & Spark Foundations

• Overview of the Big Data ecosystem and the role of Spark in modern data platforms
• Understanding Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG and execution planning
• Differences between RDD and DataFrame APIs and when to use each approach
• Creating and configuring SparkSession and understanding application configuration fundamentals

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats: CSV, JSON, Parquet and Delta
• Working with PySpark DataFrames: transformations, actions, column expressions, filtering, joins and aggregations
• Implementing advanced operations such as window functions, handling timestamps and working with nested data
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Understanding how Docker images and Kubernetes pods relate to Spark driver and executor processes — a conceptual view for cluster consumers, not administrators
• Understanding what happens when a Spark job runs on Kubernetes: pod scheduling, resource requests and limits, and how session settings map onto them
• Understanding what you control through Spark session parameters versus what the platform team controls through cluster provisioning — and why this boundary matters for tuning
• Hands-on exercise: Inspect a running Spark-on-Kubernetes job in the sandbox cluster and identify the driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Understanding executor count versus executor size: memory-versus-cores trade-offs and how to reason about them
• Driver and executor memory allocation, memory overhead, and how these translate into pod resources
• Dynamic allocation: how it behaves on Kubernetes, when it saves resources and when it does not
• Hands-on exercise: Run the same job under different executor and core configurations and compare runtime and resource usage

Module 5: Scaling Behaviour & Cost Optimisation

• Understanding how adding or removing nodes changes job behaviour, completion time and resource consumption
• Comparing many small executors versus fewer large ones: performance and cost trade-offs
• Shuffle behaviour and shuffle partition tuning, including estimating the cost impact of configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes and observe the effect on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Understanding the small-files problem: why sources split across many 1–5 MB Parquet files degrade performance and distort partitioning
• Repartitioning and coalescing strategies
• Controlling partition sizes on read and on write
• Writing Parquet efficiently to avoid recreating the small-files problem downstream
• Hands-on exercise: Load a dataset made up of many small Parquet partitions, apply different repartitioning strategies and compare performance before and after optimisation

Module 7: Pandas Memory Management & Failure Diagnosis

• Understanding the root causes of out-of-memory errors in Pandas and recognising the symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Avoiding memory blow-ups when writing large datasets to CSV
• Using chunked processing and dtype optimisation for constrained environments
• Hands-on exercise: Reproduce a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation and memory behaviour
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Rewrite a Pandas-heavy transformation in Polars and compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning and memory-management principles across a realistic ETL pipeline
• Understanding optimisation considerations specific to machine-learning workloads running on the same cluster
• Applying a practical tuning workflow to diagnose cost and performance issues systematically
• Hands-on exercise: Complete an end-to-end mini-project combining data loading, transformation and a simple model-training step, with participants tuning the Spark configuration themselves