Senior Data Engineer - Databricks

Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of large-scale data. Develop and optimize data processing logic using PySpark on Databricks (Apache Spark) . Implement ETL/ELT pipelines integrating data from multiple structured and semi-structured sources. Contribute to the design and implementation of lakehouse architectures (Delta Lake, Medallion architecture). Ensure data quality, reliability, performance, and observability across pipelines. Optimize Spark jobs through partitioning, caching, and performance tuning techniques. Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data solutions. Implement best practices in CI/CD, version control, and pipeline automation . Support the evolution of modern data platforms and analytics capabilities . Work with orchestration tools (ADF, Airflow, or similar) to manage workflows. Communicate technical solutions and trade-offs effectively to both technical and non-technical stakeholders.