Practical Testing Strategies for Databricks: A Software Engineer’s Journey into Data Engineering

Data Data Bricks Data Engineering Kafka PySpark python
By Isoeb Laghidze
February 2, 2026
  • Ioseb Laghidze, a Solution Consultant at Sahaj Software, shared practical strategies for implementing disciplined and reliable testing practices in Databricks-based data engineering projects.
  • Drawing from proven software engineering techniques, he demonstrated how data engineers can validate their pipelines more effectively by designing for testability from the ground up.
  • The talk covered modularizing PySpark transformations for effective unit testing, setting up containerized local environments using PySpark, Delta, and Kafka, designing integration tests that simulate real-world data flows, incorporating testing strategies into CI/CD pipelines, and building confidence in production-grade data pipelines.
  • Ioseb brought his background as a signal processing software engineer and experience in platform engineering and data engineering to help clients solve complex engineering challenges using technologies like Python, TypeScript, and C++.
Generated using GPT-4o-mini.
img

More Videos of our talks

rounded-arrow
What Happens As You Code with AI? Beyond Vibe Coding
rounded-arrow
rounded-arrow
Agentic AI 101
rounded-arrow
rounded-arrow
Autoscaling on Autopilot: Let Kubernetes Do the Heavy Lifting
rounded-arrow