Big data engineer

Job Title: Big Data Developer

Responsibilities:

1. Good understanding of concurrent software systems and building them in a way that is scalable, maintainable, and robust.
2. Experience in designing application solutions in the Hadoop ecosystem.
3. Deep understanding of the concepts in Hive, HDFS, Yarn, Spark, Spark SQL, Scala, and PySpark.
4. HDFS file formats and their use cases (e.g., Parquet, ORC, Sequence, etc.).
5. Good knowledge in data warehousing systems.
6. Experienced in any scripting language (SHELL, PYTHON).
7. HortonWorks distribution and understanding of SQL engines (Tez, MR).
8. Java/Rest Services/Maven experience is a value addition.
9. Control-M development and monitoring resource utilization using Grafana Tool.
10. Good skill set to create automation scripts in Jenkins and knowledge in working on automating builds, test frameworks, app configuration, etc.
11. Experience in implementing scalable applications with fully automated deployment and control using Bitbucket, Jenkins, ADO, etc.

Skillset:

1. Mandatory: Big Data – Hive, HDFS, Spark, Scala, PySpark.
2. Good to have: Schedulers (Control-M), ETL Tool – Dataiku, Unix/Shell scripting, Knowledge of Integration services of FileIT/MQ and others, CI/CD tools – Jenkins, Jira, ADO DevOps tools suite, Oracle.