Senior Data Engineer

Position Summary

The Senior Data Engineer will work with our data management team. This role focuses on leveraging advanced technologies such as Databricks, Delta Lake, FiveTran, and Google Cloud Platform (GCP) to enhance our data infrastructure and drive innovative data solutions. Utilizing experience with Databricks administration, Unity Catalog, and JAR file management to optimize our data processes and support advanced analytics initiatives.

Essential Functions and Duties

• Design and implement robust data pipelines using Databricks and integrate these with Delta Lake for efficient data storage and management. 
• Manage end-to-end data workflows from ingestion to insights, utilizing FiveTran for seamless data integration and GCP for scalable cloud solutions.
• Administer Databricks environments, ensuring optimal configuration, security, and performance across all data operations.
• Utilize Unity Catalog to manage data governance across all Databricks workspaces, ensuring compliance with data privacy and security policies.
• Develop and maintain scalable and efficient data models and architecture, supporting the strategic goals of the organization.
• Collaborate with data scientists and analysts to deploy machine learning models and complex analytical projects.
• Monitor, troubleshoot, and optimize data systems, proposing and implementing improvements to enhance performance.
• Document all data procedures and systems, ensuring clear and accurate records are maintained for compliance and operational efficiency.
• Stay updated with the latest innovations in data engineering and introduce new tools and techniques to keep the data infrastructure at the forefront
• Always represent the Company in a professional manner and appearance.
• Understand and internalize the Company’s purpose,
• Display loyalty to the Company and its organizational values.
• Display enthusiasm and dedication to learning how to be more effective on the job and share knowledge with others.
• Work effectively with co-workers, internal and external customers and others by sharing ideas in a constructive and positive manner; listen to and objectively consider ideas and suggestions from others; keep commitments; keep others informed of work progress, timetables, and issues; address problems and issues constructively to find mutually acceptable and practical business solutions; address others by name, title, or other respectful identifier, and respect the diversity of our work force in actions, words, and deeds.
• Comply with the policies and procedures stated in the Injury and Illness Prevention Program by always working in a safe manner and immediately reporting any injury, safety hazard, or program violation. 
• Ensure conduct is consistent with all Compliance Program Policies and procedures when engaging in any activity on behalf of the company. Immediately report any concerns or violations.
• Other duties as assigned.

Education, Knowledge, Skills, and Experience

Required Education:

• Bachelor’s degree in computer science, engineering, information technology or related field or four (4) years of relevant experience in lieu of degree. 
• Master’s degree in related field a plus.

Required Knowledge:

• General knowledge of advanced concepts, practices and procedures related to design and implementation of robust data pipelines
• Proven expertise in Databricks administration and managing large-scale data environments.
• Strong experience with SQL, Python, and other scripting languages commonly used in data engineering.
• Familiarity with Fivetran or similar data integration tools and understanding of ETL processes
• Previous experience managing end-to-end data workflows