Senior Cloud Data Engineer (Databricks)

We are seeking a Senior Cloud Data Engineer (Databricks) to join our growing Data Engineering team. In this role, you will play a critical role in designing, developing, and maintaining scalable data pipelines using Azure Databricks and the broader Azure cloud platform. You will leverage your expertise in Azure Databricks, Azure Data Factory (ADF), and data engineering principles to build efficient and reliable solutions for ingesting, transforming, and delivering data for our data analytics needs.

Responsibilities:

• Design and implement data pipelines using Azure Databricks and Apache Spark for large-scale data processing and transformation.

• Utilize Azure Data Factory (ADF) to orchestrate data flows between various data sources and sinks.

• Develop and maintain data pipelines using SQL and Python scripting languages.

• Collaborate with data analysts and data scientists to understand data requirements and design solutions.

• Implement data quality checks and ensure data accuracy throughout the pipeline.

• Optimize data pipelines for performance and scalability.

• Monitor and troubleshoot data pipelines to identify and resolve issues.

• Stay up-to-date on the latest advancements in Azure Databricks, Azure ADF, and data engineering best practices.

Required Skills:

• 3-5 years of experience in data engineering with a focus on big data technologies.

• Proven experience with Azure Databricks and Apache Spark for data processing and analytics.

• Familiarity with Azure cloud services, particularly Azure Data Factory (ADF) and Azure Storage.

• Strong proficiency in SQL and Python scripting languages.

• Experience with data warehousing and data modeling concepts.

• Excellent understanding of data quality principles and techniques.

• Strong problem-solving and analytical skills.

• Ability to work effectively in a fast-paced, collaborative environment.

Bonus Points:

• Experience with cloud infrastructure best practices for data engineering.

• Experience with containerization technologies like Docker and Kubernetes.

• Experience with data lineage tools for data pipeline traceability.

• Knowledge of version control systems like Git.