SWEAP Engineer for Training AI Data

Project/Opportunity Details

A SWEAP task is a real-world GitHub problem packaged with a testing environment and solution (i.e., the golden patch). This environment and solution are used to train an agent to solve the problem and verify the solution in the testing environment. A contributor’s job is to verify the testing environment based on testing logs, categorize the problem by specificity and knowledge areas, and write additional notes to help an agent understand what a good solution looks like based on the code edited in “the golden patch” and notes on GitHub.

Accepted Locations

We accept applicants from the US, Canada, and most countries in LATAM and Europe. We are also accepting candidates from some countries in Africa and Asia. For the complete list of accepted locations, click here. This work is 100% remote.

Loom Video

We have created an Loom video that we highly recommend you watch! Check it out here: Loom Video

Overview

Join our expert annotation team to create training data for the world's most advanced AI models. No previous AI experience is necessary. You'll get your foot in the door with one of the most prominent players in the AI/LLM space today. We're seeking contributors with professional software engineering experience on production repositories and experience building and maintaining large-scale coding repositories. Projects typically include discrete, highly variable problems that involve engaging with these models as they learn to code. We currently have 200+ roles open!

What Will I Be Doing?

• Verifying the testing environment based on testing logs.

• Categorizing problems by specificity and knowledge areas.

• Writing additional notes to help an agent understand what a good solution looks like based on the code edited in “the golden patch” and notes on GitHub.

• Evaluating the quality of AI-generated code, including human-readable summaries of your rationale.

• Solving coding problems and writing functional and efficient code in various programming languages.

• Writing robust test cases to confirm code works efficiently and effectively.

• Creating instructions to help others and reviewing code before it goes into the model.

• Engaging in a variety of projects, from evaluating code snippets to developing full applications using chatbots.

Pay Rates

Compensation rates average at $30/hr and can go up to $50+/hr. Expectations are 15+ hours per week; however, there is no upper limit. You can work as much as you want and will be paid weekly per hour of work done on the platform.

Contract Length

This is a long-term contract with no end date. We expect to have work for the next 2 years. You can end the contract at any time, but we hope you will commit to 12 months of work.

Flexible Schedules

Developers can set their own hours. Ideal candidates will be interested in spending 40 hours a week. You will be assigned to projects, so strong performers will adapt to the urgency of projects and stay engaged, but we are incredibly flexible on working hours. You can take a 3-hour lunch with no problem. Instead of tracking your hours, you are paid according to time spent on the platform, calculated in the coding exercises.

Interview Process

• Apply using this Ashby form.

• If you seem like a good fit, we'll send an async RLHF code review that will take 35 minutes and must be finished within 72 hours of us sending it.

• You’ll receive credentials to the RLHF platform. We’re doing regular calls to answer any further questions about onboarding, as well as providing a support team at your disposal.

• You’ll perform a simulated production-level task (RLHF task) on the platform. This will be the final stage, which will ultimately determine your leveling and which project you’ll be assigned. Successful completion of this process provides you with an opportunity to work on projects as they become available.

Application Requirements

• Valid identification document from your current country of residence is required

• LinkedIn profile with 5+ connections is mandatory

Tech Stack Priorities

The current priority for this team is contributors w