puestos de Data Engineer en United States

Roles and responsibilities

The client is seeking an exceptional Data Platform Engineer to join the team of AI specialists in the UAE. As a Data Platform Engineer, you will be responsible for designing, implementing, and optimizing data pipelines and infrastructure to support our cutting-edge AI systems. You will collaborate closely with the multidisciplinary team to ensure the efficient collection, storage, processing, and analysis of large-scale data, enabling us to unlock valuable insights and drive innovation across various domains.

Responsibilities:

1. Pipeline Development: Design, develop, and maintain scalable and reliable data pipelines to ingest, transform, and load diverse datasets from various sources, including structured and unstructured data, streaming data, and real-time feeds.
2. Data Integration: Implement robust data integration processes to seamlessly integrate data from different sources, ensuring consistency, reliability, and data quality.
3. Data Storage: Design and optimize data storage solutions, including relational databases, NoSQL databases, data lakes, and cloud storage services, to efficiently store and manage large volumes of data.
4. Performance Optimization: Optimize data processing and query performance to enhance systems scalability, reliability, and efficiency, leveraging techniques such as indexing, partitioning, caching, and parallel processing.
5. Data Governance: Implement data governance frameworks to ensure data security, privacy, integrity, and compliance with regulatory requirements, including data encryption, access controls, and auditing.
6. Monitoring and Maintenance: Monitor data pipelines and infrastructure components, proactively identify and address issues, and perform routine maintenance tasks to ensure system stability and reliability.
7. Collaboration: Collaborate closely with cross-functional teams, including data scientists, architects, and domain experts, to understand requirements, gather insights, and deliver integrated solutions.
8. Documentation: Create comprehensive documentation, including technical specifications, data flow diagrams, and operational procedures, to facilitate understanding, collaboration, and knowledge sharing.

Qualifications:

1. Proven experience as a Data Platform Engineer, with a track record of designing and implementing complex data pipelines and infrastructure solutions.
2. Expertise in data modeling, ETL (Extract, Transform, Load) processes, and data warehousing concepts, with proficiency in SQL and scripting languages (e.g., Python).
3. Strong hands-on experience with Data Bricks.
4. Excellent analytical, problem-solving, and communication skills, with the ability to translate complex technical concepts into clear and actionable insights.
5. Proven ability to work effectively in a fast-paced, collaborative environment, with a passion for innovation and continuous learning.
6. Can explain in detail how you have designed, built, and integrated data platforms.
7. You also need to be able to explain in detail and show how many integration points you have worked on, and your work with Platforms.

Desired candidate profile

1. Data Engineering: Strong knowledge of data engineering principles and practices, including ETL (Extract, Transform, Load) processes.
2. Programming Languages: Proficiency in programming languages such as Python, Java, or Scala, used for data manipulation and processing.
3. Database Management: Experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
4. Big Data Technologies: Familiarity with big data tools and frameworks, such as Apache Hadoop, Spark, or Kafka.
5. Cloud Platforms: Experience with cloud services (e.g., AWS, Google Cloud, Azure) for data storage and processing.
6. Data Warehousing: Understanding of data warehousing concepts and experience with platforms like Snowflake, Redshift, or Google BigQuery.
7. Data Modeling: Skills in designing data models that efficiently organize and structure data for analytics.
8. Data Governance: Knowledge of data governance and data quality best practices to ensure data integrity and compliance.
9. APIs and Microservices: Familiarity with building and integrating APIs to facilitate data access and sharing.
10. Collaboration: Strong communication and collaboration skills to work with data scientists, analysts, and other stakeholders.