Data Engineer

Big Data Developer (3 – 10 Years)

Responsibilities:

1. Good understanding of concurrent software systems and building them in a way that is scalable, maintainable, and robust.
2. Experience in designing application solutions in the Hadoop ecosystem.
3. Deep understanding of the concepts in Hive, HDFS, Yarn, Spark, Spark SQL, Scala, and PySpark.
4. HDFS file formats and their use cases (e.g., Parquet, ORC, Sequence, etc.).
5. Good knowledge in data warehousing systems.
6. Experienced in any scripting language (SHELL, PYTHON).
7. HortonWorks distribution and understanding of SQL engines (Tez, MR).
8. Java/RestServices/Maven experience is a value addition.
9. Control-M development. Monitoring resource utilization using Grafana Tool.
10. Good skillset to create automation scripts in Jenkins & knowledge in working on automating builds, test frameworks, app configuration, etc.
11. Experience in implementing scalable applications with fully automated deployment and control using Bitbucket, Jenkins, ADO, etc.

Skillset:

1. Mandatory: Big Data – Hive, HDFS, Spark, Scala, PySpark.
2. Good to have: Schedulers (Control-M), ETL Tool – Dataiku, Unix/Shell scripting, Knowledge of Integration services of FileIT/MQ and others, CI/CD tools – Jenkins, Jira, ADO DevOps – tools suite, Oracle.