Senior Data Engineer

The Senior Data Engineer is responsible for designing, building, and maintaining the data pipelines, transformation layers, and data models that power the enterprise lakehouse. This role is a technical anchor on the data engineering team, delivering robust ELT/ETL solutions and serving as a mentor to junior engineers.  

KEY RESPONSIBILITIES  

▸  Design and implement scalable batch and streaming data pipelines using Apache Spark, Kafka, and Flink  

▸  Build and maintain the Bronze/Silver/Gold medallion architecture within the lakehouse (Delta Lake / Iceberg)  

▸  Develop and optimize complex SQL and PySpark transformations for large-scale datasets  

▸  Integrate structured, semi-structured, and unstructured data sources into the lakehouse  

▸  Collaborate with data architects to evolve the physical and logical data models  

▸  Implement data quality checks and monitoring using Great Expectations or dbt tests  

▸  Write Infrastructure-as-Code for pipeline environments (Terraform, Helm)  

▸  Participate in code reviews and enforce engineering standards and best practices  

▸  Troubleshoot pipeline failures, performance bottlenecks, and data incidents   

▸  Mentor junior and mid-level data engineers and contribute to internal knowledge sharing  

REQUIRED QUALIFICATIONS  

▸  6+ years of data engineering experience with a track record of enterprise-scale delivery  

▸  Expert proficiency in Python and SQL; PySpark experience required  

▸  Hands-on experience with Apache Spark, Delta Lake, or Apache Iceberg  

▸  Experience with orchestration tools: Apache Airflow, Prefect, or Dagster  

▸  Strong knowledge of cloud data services: AWS Glue, Azure Data Factory, GCP Dataflow  

▸  Proficiency with version control (Git), CI/CD pipelines, and containerization (Docker/Kubernetes)  

▸  Experience with dbt (data build tool) for transformation layer management  

▸  Bachelor's degree in Computer Science, Engineering, or related technical field