Data Engineer

Experience and Skills Our data engineers do these kinds of things:

Pull, clean, augment and master data coming from a variety of public and private sources. This goes way beyond simple ETL - we’re preparing data sets that will be used to power machine learning algorithms, not just pulling data out of a database to display in a graph.

Work within a Spark/Scala environment from notebooks through mature pipelines. Our data engineers are equally comfortable writing Scala code to normalize names or pulling together disparate notebook functions into a reusable class architecture.

Explore new ways to extract signal from data, including novel algorithms and approaches. Our most challenging problems within data engineering require a good understanding of both computer science and math (especially linear algebra and probability) Build out scalable data pipelines on top of AWS infrastructure. Understanding Terraform/Ansible is a plus here, but experience with the AWS stack is necessary (especially Elastic Beanstalk and EMR)

Above all, the data engineering we do at H1 is code-centric, not tool-centric. ETL tools like Talend, Informatica, and the like are all great, but our use case is much more about data analysis than it is about moving and joining data together. For this reason we’re looking for candidates who are comfortable in the Scala/Spark framework.