Data Architect
Job Overview
The Role:
We are looking for hands-on Principal Data-Architect for a challenging and fun filled work of building a data architecture that is future proof for one of the customers in the financial domain.
Responsibilities:
Own design and maintenance of all aspects of data solutions including modeling, developing, technical documentation, data diagrams and data dictionaries.
Provide expertise in the development of standards, architectural governance, design patterns, and practices, evaluate best applicable solutions for different use cases
Determines and develops architectural approaches and solutions, conducts business reviews, documents current systems, and develops recommendations
Lead the data strategy and own the vision and roadmap of data products
Work with stakeholders to ensure that data related business requirements for protecting sensitive data are clearly defined, communicated, and well understood and considered as part of operational prioritization and planning
Develop, maintain, and optimize data infrastructure using Delta Lake, MLflow, and Databricks SQL to enhance data management, processing, and analytics.
Utilize Snowflake’s features such as data sharing, zero-copy cloning, and automatic scaling to optimize data storage, accessibility, and performance. Ensure effective management of both semi-structured and structured data within Snowflake’s architecture.
Implement and manage data storage solutions using Amazon S3, perform data warehousing with Amazon Redshift.
Design and implement data integration workflows using AWS Glue to orchestrate and automate data movement and transformation.
Design and implement scalable data pipelines using tools like Apache Kafka or Apache Airflow to facilitate real-time data processing and batch data workflows.
Apply advanced analytics techniques, including predictive modeling and data mining, to uncover insights and drive data-driven decision-making.
Requirements:
Overall experience of 15+ years
Data architecture, data platform, data warehouse related experience of 10+ years
Hands on experience with snowflake - 4+ years experience, Data bricks 4+ years
Between snowflake and Data bricks (5+ years experience)
Proficiency in features such as Delta Lake, MLflow, and Databricks SQL. Experience in managing Spark clusters and implementing machine learning workflows.
Solid experience in emerging and traditional data stack components such as: batch and real time data ingestion, ETL, ELT, orchestration tools, on-prem and cloud DW, Python, structured, semi and unstructured databases
Knowledge of features like Snowflake’s data sharing, zero-copy cloning, and automatic scaling. Experience in working with Snowflake’s architecture for semi-structured and structured data.
Experience with services like Amazon S3, Amazon Redshift, and AWS Glue.
Proficiency in tools such as Apache NiFi, Talend, Informatica, or Microsoft SQL Server Integration Services (SSIS).
Experience in designing and implementing data pipelines using tools like Apache Kafka or Apache Airflow.
Ability to perform data profiling, data quality assessments, and performance tuning.
Experience in comparing and evaluating different data technologies based on criteria like performance, scalability, and cost.
Skills in applying advanced analytics techniques, including predictive modeling and data mining.
Expert with industry standard data practices, data strategies and data concepts
Demonstrated experience in architecting/re-architecting complex data systems and data models.
Demonstrated experience in overall system design, including database selection and solutioning.
Nice-to-Have Skills:
Experience with data governance tools such as Collibra or Alation.
Knowledge of data quality frameworks and standards, such as Data Quality Dimensions (completeness, consistency, etc.).
Familiarity with tools like Apache Beam or Luigi for managing complex data workflows.
Awareness of emerging data technologies such as data mesh, data fabric, and real-time data processing frameworks.