Data Engineer Manager
Job Overview
We are seeking a Data Engineer Manager to contribute to our next level of growth and expansion.
Design, develop, and maintain scalable data pipelines and modeling workflows within Databricks, focusing on data processing, scoring automation, and performance optimization.
Implement and execute model scoring workflows in Databricks via Lakehouse Federation, including daily refresh automation and notebook orchestration.
Build, optimize, and monitor ETL/ELT pipelines integrating Python, and cloud-based data ecosystems.
Develop and maintain .NET pipelines to transfer model scores from Databricks to external systems via Azure EventHub.
Create and manage Databricks asset bundles required to promote solutions to higher environments.
Collaborate with cross-functional teams to ensure data governance, reliability, and performance across environments.
Document architectural and technical decisions, including design assumptions, implementation rationale, and scalability considerations.
Troubleshoot issues, optimize performance, and ensure data workflow stability and accuracy.