Data Engineer

We’re a cognitive science company on a mission to optimize the measurement of brain health to advance the development of new medicines and to enable earlier clinical insights in healthcare.

 

That’s why we’re seeking a Data Engineer responsible for building and maintaining data pipelines in Azure Databricks to enable Cogstate’s data strategy. The candidate will have a central critical role for establishing and maintaining the Azure Databricks infrastructure, provide best practices and recommendations for using these tools, and will working closely with data scientists, data managers, clinical scientists, and other business stakeholders as a point of contact for the data platform infrastructure.

Core Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory, and integrate data from various sources and systems into a coherent data ecosystem.
• Utilize Azure Databricks for complex data transformation and processing tasks, ensuring high performance and alignment with business requirements.
• Manage Azure Functions for efficient data ingestion and processing, optimizing for performance and cost.
• Implement and maintain data storage solutions in Azure Data Lake Storage Gen2 (ADLS2), applying best practices for data lifecycle management, security, and access control.
• Collaborate with cross-functional teams to define data requirements, deliver comprehensive data solutions, and support data-driven decision-making processes.
• Employ GitHub for source control management, adhering to best practices for code reviews, branching, and versioning.
• Facilitate CI/CD pipelines using GitHub Actions to automate testing, integration, and deployment processes, enhancing productivity and ensuring high-quality deployments.
• Monitor, troubleshoot, and optimize data systems and pipelines for performance, reliability, and scalability.
• Stay abreast of industry trends and advancements in cloud and data engineering technologies, continuously seeking opportunities for innovation and improvement.
• Maintain Data Governance infrastructure to ensure appropriate data access rights and restrictions for internal and external users.

Qualifications

• BS/BA in Computer Science, Data Science, or a related field or relevant experience
• 5+ years’ experience as a data engineer.
• Must have 2+ years experience in implementing data engineering solutions with Apache Spark and Databricks.
• Must have deep expertise in PySpark for data process and to write data pipelines and data processing layers. Experience with other technologies such as Python, Scala, Hadoop, Hive, and Spark are nice to have.
• Direct experience with relational databases like SQL Server. Excellent SQL experience for writing complex SQL transformations.
• Strong knowledge on Databricks configuration, troubleshooting and performance tuning.
• Experience with development tools for CI/CD, unit and integration testing, automation and orchestration, including GitHub, Azure Data Factory and Azure functions.
• Familiarity with Agile workflow methodologies, demonstrating an ability to work effectively in a dynamic, iterative development environment and manage tasks with varying priorities.

What’s In It For You

• Remote Work Practices: Cogstate is a virtual first company. Cogstate employees can work from anywhere where Cogstate is registered to business within the United States, Australia, or the United Kingdom!
• Generous Paid Time-off: Cogstate employees receive 20 days of vacation leave, 10 days of personal leave and 10 paid public holidays.
• 401(k) Matching: As you invest in yourself and your future, Cogstate invests in you too: we match up to3% of your yearly salary in Cogstate’s 401k program.
• Competitive Salary: We offer competitive base salaries plus additional earning opportunities based on the position.
• Health, Dental & Vision Coverage: We've invested in comprehensive health & dental insurance options with competitive company contributions to help when you need it most. We also offer free vision insurance for all full-time employees.
• Short-Term & Long Term Disability Life Insurance: 100% employer sponsored
• Pre-Tax Benefits: Healthcare and Dependent Care Flexible Spending Accounts
• Learning & Development Opportunities: Cogstate offers a robust learning program from mentorships to assistance with programs to improve knowledge or obtain certifications in applicable areas of interest.

 

Wage Range$125,000—$140,000 USD

Our Culture
We bring our whole selves to work every day. We’re courageous and we deliver together. We’re passionate individuals who e