Software Data Engineer

Job Overview

We are looking for a Data Engineer to join our team and play a critical role in designing, building, and maintaining robust data systems and pipelines that empower our organization to make data-driven decisions. You will work closely with cross-functional teams to ensure our data architecture is optimized for scalability, reliability, and performance. If you are passionate about data infrastructure, automation, and enabling insights through seamless data flow, we want to hear from you!

Responsibilities

• Design, build, and optimize scalable data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
• Develop and maintain data storage solutions such as data warehouses and data lakes.
• Ensure data quality and integrity through validation, cleansing, and error handling.
• Collaborate with data analysts, architects, and software engineers to understand data requirements and deliver relevant datasets.
• Implement data security measures and access controls to safeguard sensitive information.
• Automate and optimize data workflows to improve scalability and efficiency.
• Monitor and troubleshoot data infrastructure for performance and reliability issues.
• Stay updated with industry trends and emerging data engineering technologies.
• Document data processes, pipelines, and best practices for knowledge sharing.
• Participate in data governance initiatives to ensure compliance with regulatory requirements.

Requirements

• Proficiency in programming languages such as Python, JavaScript, etc.
• Strong knowledge of SQL and NoSQL databases (e.g., PostgreSQL, Cassandra).
• Familiarity with big data tools like Hadoop, Spark, etc.
• Experience with ETL tools and workflow automation (e.g., Airflow, Luigi, Azkaban).
• Understanding of stream processing systems (e.g., Apache Storm, Spark Streaming).
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Solid grasp of data modeling, database design, and performance tuning.
• Excellent analytical and problem-solving abilities.
• Strong communication and collaboration skills to work in a dynamic environment.

Preferred Skills (Nice to Have)

• Familiarity with data governance frameworks and regulatory compliance.
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Knowledge of event-based systems and queue management (e.g., RabbitMQ, Kafka Streams).