Databricks Data Engineer (Remote)

Position Summary: 

Join the Vallen team as a Senior Data Engineer. We are seeking to fill this role with an individual with 5–7 years of experience in data engineering and a strong background in Databricks. This role is ideal for someone passionate about building scalable data solutions, enabling analytics, and driving innovation in a modern data platform.

 

Essential Duties and Responsibilities:

• Design and develop scalable data pipelines using PySpark, Spark SQL, and Python within the Databricks environment.
• Build modular, version-controlled transformation workflows using DBT (Data Build Tool).
• Package and deploy Databricks workflows and notebooks using Databricks Asset Bundles (DABS) for CI/CD and environment management.
• Integrate Databricks workflows with Azure DevOps for automated testing, deployment, and version control.
• Develop and maintain robust CI/CD pipelines for data engineering workflows using Git, Azure DevOps, and DABS.
• Implement and optimize dimensional data models, ELT/ETL processes, and performance tuning in cloud data platforms.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-impact data solutions.
• Implement logging, alerting, and monitoring for data pipelines using tools like Databricks Jobs, MLflow, or Azure Monitor.

Qualifications

Job Qualifications: 

• 5–7 years of experience as a Data Engineer, with at least 3 years working in Databricks.
• Strong proficiency in Python, PySpark, and Spark SQL.
• Hands-on experience with DBT in a cloud data platform.
• Experience with DABS for workflow packaging and deployment.
• Proven expertise in Azure DevOps, Git, and CI/CD pipeline development.
• Solid understanding of data modeling, ETL/ELT, and performance optimization.
• Experience implementing monitoring and observability for data pipelines.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Experience in Azure cloud ecosystem.
• Familiarity with MLflow and other observability tools.
• Exposure to data governance and metadata management practices.

Work Environment & Physical Demands:

Reasonable accommodations will be evaluated and may be implemented to enable individuals with disabilities to perform essential functions of this position.

• This job operates in a professional office environment and routinely uses standard office equipment such as computers, phones, photocopiers, filing cabinets, etc.
• The work environment is generally favorable. Lighting and temperature are adequate and there are no hazardous or unpleasant conditions caused by noise, dust, etc.
• Long periods of time working on a computer and performing repetitive key-boarding activities.
• Long periods of time performing work over the phone and participating in group conference calls.
• Occasional evening and weekend work may be required as job duties demand and may include little to no advanced notice.
• Specific vision abilities required by this job include close vision, distance vision, color vision, peripheral vision, depth perception and ability to adjust focus.
• Frequently required to sit and/or stand.