Data Engineer-AWS, Python
Full-timeJob Overview
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Job Summary
For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.
Work Location: Noida, Sec 62
Working days per week: 5
Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Job Summary
For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.
Work Location: Noida, Sec 62
Working days per week: 5
Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week
Responsibilities:
- Job Monitoring: Monitor approximately 2,300 scheduled jobs (daily, weekly, monthly) to ensure timely and successful execution. Execute on-demand jobs as required by the business.
- Troubleshooting & RCA: Troubleshoot job failures, perform detailed root cause analysis (RCA), and provide clear documentation for all findings.
- Issue Resolution: Address and resolve bugs and data-related issues reported by the business team.
- Data Integrity: Verify source file placement in designated directories to maintain data integrity. Reload Change Data Capture (CDC) tables when structural changes occur in source systems.
- Data Synchronization: Help manage synchronization between external databases (including Teradata write-backs) and AWS Glue tables.
- Development: Assist in developing new solutions, enhancements, and bug fixes using AWS Glue, Lambda, and Step Functions.
- Stakeholder Support: Answer questions from the business and support User Acceptance Testing (UAT) inquiries.
- Decision Making: Make timely decisions to resolve issues, execute tasks efficiently, and escalate complex problems to senior or lead engineers as needed, all while maintaining agreed-upon SLAs.
- Documentation: Participate in knowledge transfer to internal resources and assist in creating necessary documentation and runbooks.
Qualifications:
- Experience: 3-5 years of relevant experience.
- Technical Skills: In-depth knowledge of AWS Glue, AWS Lambda, and AWS Step Functions.
- Core Concepts: A deep understanding of ETL processes and data warehouse design.
- Problem-Solving: Proven ability to troubleshoot data pipelines and perform root cause analysis (RCA).
- Should have worked on Python scripting
- Experience: Hands-on experience with Glue, Lambda, and Step Function development.
- Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
- Preferred Skills: Familiarity with Change Data Capture (CDC) processes, Teradata, and integration with SAS & RMJ events.