Data Engineer

We are seeking Data Engineer with strong expertise in Azure Databricks. This role will focus on building, supporting, and administering scalable, high-performance data pipelines that power real-time and batch analytics for trading, risk, and operational use cases. The ideal candidate will have a deep background in data bricks data engineering, administration, capital markets data, and thrive in an Agile, fast-paced environment.

Key Responsibilities:

• Design, develop, and maintain robust data pipelines using Azure Databricks, Confluent, DLT, Spark pipleline, and Delta Lake to support trading and market data workflows.
• Self-study the existing pipeline and enhance existing data pipelines, ensuring continuity, scalability, and performance improvements.
• Production pipeline support services, including job monitoring, incident resolution, and performance tuning in production environments.
• Administer Databricks workspaces, unity catalog, including cluster configuration, job scheduling, access control, and workspace optimization.
• Build and maintain CI/CD pipelines using GitLab, enabling automated testing, deployment, and versioning of data engineering code.
• Follow and enforce best practices in code management, including modular design, code reviews, and documentation using GitLab workflows.
• Collaborate with fellow team members, business analysts, and data architect to understand data requirements and deliver high-quality solutions.
• Build reusable components and frameworks to accelerate development and ensure consistency across data platforms.
• Actively participate in Agile ceremonies (e.g., sprint planning, stand-ups, retrospectives) and contribute to continuous improvement of team processes.