Senior Data Engineer

What is this position about?

We are looking for a Senior Data Engineer to join a high-impact Customer Insights product engagement for a global QSR client. This role is hands-on-keyboard and focused on building enterprise-grade data pipelines that power a unified Analytics ID, modular data marts, and a scalable feature store. The ideal candidate brings deep expertise in Databricks, Spark (PySpark), SQL, and large-scale identity resolution pipelines, and thrives in complex, production-ready data environments.

Job Description

• Design and build production-grade data pipelines in Databricks using Spark/PySpark and SQL.

• Develop and maintain an Analytics ID stitching pipeline using deterministic and probabilistic matching techniques across multiple customer data sources.

• Build and manage modular data marts (Identity, Behavior, Demographics) with independent refresh cadences.

• Implement and maintain a scalable feature store supporting downstream analytics and data science use cases.

• Own the end-to-end data lifecycle: ingestion, transformation, validation, deployment, monitoring, and optimization.

• Develop data quality frameworks including schema drift detection, anomaly monitoring, match-rate validation, and automated deduplication audits.

• Implement CI/CD processes for multi-environment promotion (dev/staging/prod) in Databricks environments.

• Coordinate orchestration workflows and manage dependencies using Databricks Workflows or similar tools.

• Collaborate closely with Data Architects and Client stakeholders to translate business rules into scalable technical solutions.

• Produce comprehensive technical documentation including data contracts, lineage maps, architecture diagrams, and operational runbooks.