Databricks Architect

Job Description

Develop and maintain ETL/ELT pipelines using Databricks (PySpark, SQL, Notebooks). Design end‑to‑end Lakehouse architecture including ingestion, transformation, modeling, and consumption. Define and implement Medallion Architecture (Bronze/Silver/Gold) using Delta Lake. Build high-performance ETL/ELT pipelines using PySpark, Spark SQL, Delta Lake. Architect and optimize batch and streaming data pipelines. Perform advanced Spark optimization, cluster sizing, and cost management. Implement CI/CD, version control, and DevOps best practices within Databricks. Databricks Lakehouse Platform PySpark & Spark SQL Delta Lake, Structured Streaming Unity Catalog & governance frameworks Strong experience with a major cloud platform (Azure/AWS/GCP). Strong understanding of: Distributed computing & Spark internals Data warehousing concepts Data modeling (dimensional, normalized, Lakehouse modeling) ETL/ELT architecture Experience in CI/CD pipelines, Git, orchestration tools. Experience in leading large‑scale data platform implementations.

Required Skills