Senior Data Engineer

Stream Hatchet is a leading data analytics company in the esports and gaming streaming industries. We leverage data from social platforms like Twitter, Instagram, TikTok, and live streaming platforms to provide actionable insights for businesses focused on go-to-market strategies, content initiatives, and sponsorship measurement.

As a Senior Data Engineer, you will play a key role in the migration of our current data ingestion and aggregation system from a Ruby-on-Rails monolith to a more scalable, cloud-native architecture. You will work closely with our engineering team to design and implement efficient data pipelines that can scale to ingest, process, and aggregate social media metrics from millions of creators daily.

If you thrive in fast-paced environments, enjoy solving complex technical challenges, and have a passion for building scalable cloud architectures, this is the role for you!

Responsibilities

• Lead the migration from a Ruby-on-Rails system to a Python-based, cloud-native architecture.
• Design, build, and maintain scalable data ingestion and aggregation pipelines using cloud services.
• Optimize existing data workflows to improve reliability, scalability, and performance.
• Collaborate with cross-functional teams to ensure seamless integration of new data sources and the delivery of data insights.
• Ensure data quality, accuracy, and reliability across the entire ingestion and aggregation process.
• Evaluate and implement best practices for monitoring, alerting, and maintaining production-level data systems.

Requirements

• BS degree in Computer Science, Engineering, or a related technical field.
• 3+ years of experience in data engineering, including data pipeline development, aggregation, and ingestion.
• Proficiency in Python and experience with cloud-native development using AWS services.
• Proficiency with SQL databases and query optimization. Familiarity with NoSQL databases is a plus.
• Understanding of data architecture patterns, ETL processes, and distributed data systems.
• Familiarity with Ruby on Rails is a plus.
• Experience with Elasticsearch for building search and analytics solutions.
• Excellent communication skills in English.