Assistant Lead Engineer - ETL (Software Development)

Position Overview

Our Software development team is currently hiring a Lead Engineer for an AWS cloud migration project. The role requires the potential candidate to be involved in designing, developing, and optimizing applications for the cloud environment. Extensive experience in ETL tools, data warehousing concepts, and AWS Cloud services will be crucial in ensuring the success of our transformative project.

Role & Responsibilities

• Design, develop, and implement integration solutions for applications migrating to the AWS Cloud platform.
• Design and develop ETL processes to extract data from various sources (e.g., databases, flat files, APIs), transform as per business requirements, and load it into data warehouse or other data storage systems.
• Integrate data from multiple heterogeneous sources and ensure it is transformed and loaded correctly into the target system, maintaining data quality and integrity.
• Write complex SQL queries and use transform logic to clean, aggregate, and transform data as per business requirements.
• Ensure ETL processes are optimized for performance, scalability, and reliability.
• Implement data validation rules and perform data quality checks to ensure accuracy and completeness of data.
• Troubleshoot and resolve issues related to data integration, loading failures, and performance bottlenecks in the ETL process.
• Document ETL workflows, processes, logic, and prepare reports as needed.
• Prepare detailed migration plans, timelines, and risk mitigation strategies.
• Collaborate with cross-functional teams to understand business requirements and implement effective integration strategies.
• Work closely with the migration team to execute the migration of applications to the AWS Cloud.
• Conduct thorough testing to ensure functionality, performance, and data accuracy.
• Provide documentation for troubleshooting and support purposes.

Requirements

Must have:

• Degree in Computer Science, Computer Engineering, or equivalent.
• Minimum 12 years of experience in IT and at least 6 years of experience in using Informatica PowerCenter tools for data integration and ETL processes.
• Knowledge of Informatica Designer, Workflow Manager, and Repository Manager.
• Experience with creating and managing ETL mappings, sessions, and workflows.
• Proficiency in designing and developing data transformations.
• Strong SQL skills for database querying and manipulation.
• Understanding of relational database concepts and performance tuning, and familiarity with data warehousing concepts, star schema, snowflake schema, and data marts.
• Ability to optimize ETL processes for performance and efficiency.

Good to have:

• Understanding of AWS Glue components like Crawlers, Jobs, Triggers, and Data Catalog.
• Experience in creating and managing ETL jobs using AWS Glue Studio.
• Proficiency in Python, especially with PySpark, for writing ETL scripts in AWS Glue.
• Familiarity with other AWS services like S3, Redshift, RDS, and Lambda for integration with AWS Glue.
• Understanding of IAM roles and permissions for secure access to AWS services.
• Skills in planning and executing data migration from Informatica to AWS Glue.
• Ability to translate Informatica workflows and mappings into AWS Glue jobs.
• Experience with AWS CodePipeline or similar tools for automating deployment and integration processes.