Lead Data Engineer

Job Description

As a Lead Data Engineer, you will design and maintain data platform road maps and data structures that support business and technology objectives.  Naturally inquisitive and open to the deep exploration of underlying data, finding actionable insights, and working with functional competencies to drive identified actions.  You also enjoy working both freely and as part of a team and have the confidence to influence and communicate with stakeholders at all levels, and to work in a fast-paced complex environment with conflicting priorities.

About the role:

Reporting into the delivery leader, you will deliver consumable, contemporary, and immediate data content to support and drive business decisions. The key focus of the role is to deliver a custom solution to support various business critical requirements. You will be involved in all aspects of data engineering from delivery planning, estimating and analysis, all the way through to data architecture and pipeline design, delivery, and production implementation.  From the beginning you will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise, and hybrid client technology landscapes.

Description of Role:

We are looking for 5+ years of experience in data engineering in a customer or business facing capacity and experience in the following:

• Ability to understand and articulate requirements to technical and non-technical audiences
• Stakeholder management and communication skills, including prioritizing, problem solving and interpersonal relationship building
• Strong experience in SDLC delivery, including waterfall, hybrid, and Agile methodologies
• Experience of implementing and delivering data solutions and pipelines on AWS Cloud Platform.
• A strong understanding of data modelling, data structures, databases, and ETL processes
• An in-depth understanding of large-scale data sets, including both structured and unstructured data
• Knowledge and experience of delivering CI/CD and DevOps capabilities in a data environment
• Develop new inbound data source ingestions required within the multi-tiered data platform to support analytics and marketing automation solutions
• Supports data pipelines – Builds the required dimensions, rules, segments, and aggregates
• Support all database operations: performance monitoring, pipeline ingestion, maintenance, etc.
• Monitor platform health - data loads, extracts, failures, performance tuning
• Create/modify data structures/pipelines
• Leveraging capabilities of Databricks Lakehouse functionality as needed to build Common/Conformed layers within the data lake
• Application of software best practices into consideration. E.g. infrastructure cost, security, scalability, feature extensibility, ease of maintenance, etc.
• Develop, document, and test software and environment setup to ensure that the outcome meets the needs of end-users and achieves business goals

Qualification

The following technical skills are required:

• Tech Stack: Databricks, Python, SQL, Spark, etc.
• Designing and building the Data Lakes or Lakehouses  
• Extensive experience in ETL and audience segmentation
• Developing sustainable, scalable, and adaptable data pipelines
• Attention to detail in design, documentation, and test coverage of delivered tasks
• 4+ years of experience with designing and developing Data Pipelines for Data Ingestion or Transformation using cloud technologies
• 4+ years of experience developing applications with Monitoring, Build Tools, Version Control, Unit Test, TDD, Change Management to support DevOps
• 4+ years of experience with Databricks implementations
• 4+ years in the design and development of data solutions utilizing Snowflake
• 3+ years of experience in the following Big Data frameworks: File Format (Parquet, etc.), Resource Management, Distributed Processing

The following soft skills are required:

• Strong written and verbal communication skills, team player
• Strong analytical and leadership skills
• Strong business acumen, interpersonal skills, and ability to work with a team and independently

The following skills are nice to have:

• Streaming using Spark, Flink, Kafka or other methods
• Spark programming (ex., PySpark or Scala)

Additional Information

About Epsilon

Epsilon is a global advertising and marketing technology company positioned at the center of Publicis Groupe. Epsilon accelerates clients’ ability to harness the power of their first-party data to activate campaigns across channels and devices, with a