Software Engineer — Compute Platform

We’re on a mission to democratize AI by building the definitive AI data development platform. The AI landscape has gone through incredible change between 2016, when Snorkel started as a research project in the Stanford AI Lab, to the generative AI breakthroughs of today. But one thing has remained constant: the data you use to build AI is the key to achieving differentiation, high performance, and production-ready systems. We work with some of the world’s largest organizations to empower scientists, engineers, financial experts, product creators, journalists, and more to build custom AI with their data faster than ever before. Excited to help us redefine how AI is built? Apply to be the newest Snorkeler!

As a Compute Platform Engineer, you will play a pivotal role in building the backbone of SnorkelFlow's infrastructure. Your focus will be on orchestration, MLOps, SDK maintainability, and other tooling development that will serve our AI and Data teams. Whether you’re an entry-level engineer eager to dive into the complexities of compute engineering or a senior engineer ready to lead impactful projects, this role provides opportunities to work on cutting-edge AI infrastructure and scale SnorkelFlow’s capabilities.

Main Responsibilities

For All Levels:

Orchestration and Infrastructure Development:

• Design, implement, and maintain orchestration tools for workflows using Ray and Prefect.
• Build and manage infrastructure for scalable data connectors to integrate with systems like S3, Snowflake, and Databricks.
• Optimize compute resource utilization for AI pipelines and ensure reliable, fault-tolerant execution of tasks.

MLOps and SDK Development:

• Develop and maintain robust CI/CD pipelines to support seamless model deployment and orchestration workflows.
• Enhance the SnorkelFlow SDK to provide user-friendly access to compute layer functionalities.
• Collaborate with AI Platform and Data Platform teams to ensure SDK usability and extensibility.

Backend Development:

• Build backend services to support compute operations, including job scheduling, resource allocation, and API integrations.
• Partner with the Application team to design APIs that enable seamless orchestration and workflow management.

Observability and Performance Optimization:

• Implement monitoring tools and dashboards to track the performance and health of compute resources.
• Define metrics and logging strategies to optimize orchestration pipelines and SDK efficiency.

For Senior Engineers:

• Lead the design and development of core components of the compute platform, including orchestration workflows and connectors.
• Mentor and guide junior engineers, fostering growth within the team.
• Identify and drive improvements in infrastructure scalability and reliability.
• Collaborate with cross-functional teams to align the compute platform with overall product goals.

Required Qualifications

Entry-Level Engineers:

• Bachelor’s degree in Computer Science, Software Engineering, or a related field.
• 1-2 years of experience in backend development or infrastructure engineering.
• Proficiency in Python and familiarity with frameworks like FastAPI or Flask.
• Basic understanding of orchestration tools and MLOps practices.
• Strong problem-solving skills and eagerness to learn and grow in the compute engineering domain.

Senior Engineers:

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.
• 4-6 years of experience in backend or infrastructure engineering, including experience with orchestration tools (e.g., Prefect, Ray, Airflow).
• Proven expertise in Python, with experience in building scalable APIs and SDKs.
• Strong knowledge of MLOps practices, including CI/CD, Kubernetes, and model lifecycle management.
• Experience with infrastructure scaling and integration of data connectors.
• Proven ability to lead technical projects and mentor junior team members.

Preferred Qualifications

• Experience with Prefect or similar for orchestration and distributed computing.
• Experience with developing microservices pipelines and infrastructure
• Familiarity with vector databases and data storage solutions.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Experience building data connectors for systems like Snowflake, Databricks, or S3.
• Familiarity with observability tools like Prometheus, Grafana, DataDog or OpenTelemetry.