Sr. Data Engineer

Sr. Data Engineer

 

Pay: Competitive

 

Shift: Monday - Friday 8am - 5pm

 

JOB SUMMARY: 

The ideal candidate will possess strong background on Data Engineering development technologies & experience with cloud-based resources. The candidate must possess excellent written and verbal communication skills with the ability to collaborate effectively with domain experts and technical experts in the team. This role will be responsible for both team collaboration as well as project initiative ownership and leadership. This role is for contributing to both technical expertise as well as process leadership.

 

An Azure Databricks Engineer designs, builds, and optimizes scalable data pipelines and lakehouse solutions within the Microsoft Azure ecosystem.

 

Core Responsibilities

• Data Engineering & ETL: Develop and maintain batch/streaming data pipelines using Azure Databricks, Azure Data Factory and PowerBI
• Azure Integration: Integrate Databricks with (ADLS), Power-BI, ADO, Key Vault
• Performance Optimization: Optimize Spark jobs, cluster configurations, Serverless Policy & Cost Optimization
• Data Governance & Security: Implement Unity Catalog for access control, data lineage, and security compliance.
• DevOps & Automation: Build CI/CD pipelines for Databricks notebooks and workflows using Azure DevOPS & Terraform.
• Collaboration: Work with data Engineers and analysts to enable AI/ML workloads

 

Required Qualifications

• Education: Bachelor’s degree in computer science, Engineering, or a related field.
• Technical Skills: Proficiency in Python (PySpark), SQL.
• Cloud Experience: Strong hands-on experience with Microsoft Azure services.
• CI/CD: Azure DevOPS
• Databricks Expertise: Experience with Delta Lake, Spark SQL, and Databricks Jobs.

  

SUPERVISORY RESPONSIBILITIES: 

• No direct reports but responsible for project ownership which may have reports.
• Responsible for mentoring of Jr. resources.

 

ESSENTIAL FUNCTIONS:

Responsibilities:

 

• Understand the business data needs, both current and future to help deliver and guide a data strategy

• Set and ensure proper adherence to strategic and tactical patterns
• Set technical direction for obtaining and implementing technology
• Help evaluate current and future staffing needs

• Drive the creation of new standards and best practices as technology evolves; communicate and drive adoption across the company

• Drive creation of new data capabilities and offerings
• Develop and maintain batch/streaming data pipelines using Azure Databricks, Azure Data Factory and PowerBI
• Integrate Databricks with (ADLS), Power-BI, ADO, Key Vault
• Optimize Spark jobs, cluster configurations, Serverless Policy & Cost Optimization
• Implement Unity Catalog for access control, data lineage, and security compliance.
• Build CI/CD pipelines for Databricks notebooks and workflows using Azure DevOPS & Terraform.
• Work with data Engineers and analysts to enable AI/ML workloads

  

*Performs Additional Responsibilities As Assigned*

 

QUALIFICATIONS: 

 

Education and/or experience: 

 

• Bachelor’s Degree in a technical field (statistics, mathematics, science, accounting, finance) or related field