Lead Big Data Engineer

The Lead Big Data Engineer, working independently, will develop, test, debug and document software components commensurate with their experience as well as direct development staff in support of a software engineering effort.

We are seeking a Big Data Engineer to lead the design, develop, and maintain scalable data pipelines and infrastructure to support our market regulation technology applications. You will play a key role in modernizing existing data pipelines, enabling downstream users to experiment with data using no-code/low-code features.You will collaborate closely with product managers, data scientists, and engineers to deliver high-performance systems that power market surveillance and regulatory workflows.

Essential Job Functions:

• Design and modernize scalable, modular data pipelines from legacy SQL-based workflows.

• Build efficient and reliable ETL processes using Apache Spark and cloud-native tools on AWS.

• Create configurable pipelines that accelerate experimentation and feature selection for data science teams.

• Optimize pipeline performance, reduce storage costs, and improve compute efficiency.

• Ensure compliance with governance standards while maintaining data integrity and lineage.

• Integrate seamlessly with enterprise data platforms, lakes, and cloud-based storage systems.

• Implement and maintain CI/CD pipelines using Jenkins or similar tools.

• Write robust automated tests, including unit, integration, and end-to-end coverage.

• Collaborate within Agile teams, participating in sprint planning, reviews, and retrospectives.

Nice to have:

• Experience in large-scale data engineering or data infrastructure roles.

• Proficient in Python or Scala for building and maintaining data pipelines.

• Deep expertise with Apache Spark, Hive, and distributed computing frameworks.

• Strong knowledge of SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, DynamoDB).

• Experience with feature stores (e.g., Feast, Tecton, SageMaker Feature Store).

• Hands-on with AWS services for compute, storage, and orchestration.

• Experience implementing CI/CD practices and managing production deployments.

• Familiarity with Apache Airflow and workflow orchestration tools.

• Experience with machine learning feature engineering and model serving frameworks.

• Comfortable with automated testing and working in test-driven environments.

• Proven ability to thrive in Agile/Scrum teams and contribute to iterative development cycles.

Education/Experience Requirements:

• Bachelor’s degree in Computer Science, Information Systems or related discipline with at least seven (7) years of related experience, or equivalent training and/or work experience; Master’s degree and past Financial Services industry experience preferred.
• Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
• Past experience with developing enterprise quality solutions in an iterative or Agile environment.
• Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
• Strong written and verbal technical communication skills.
• Demonstrated ability to develop effective working relationships that improved the quality of work products.
• Should be well organized, thorough, and able to handle competing priorities.
• Ability to maintain focus and develop proficiency in new skills rapidly.
• Ability to work in a fast paced environment.
• Experience with object oriented programming languages such as Java, Scala or Python.

Work Conditions:

• Ability to work in a hybrid work environment

For work that is performed in CA, CO, HI, MN, VT, IL, Jersey City, NJ, New York City, NY, MA, WA, MD, Washington, DC, NJ State and NY State, please refer to the chart below for the salary range for the corresponding location. In addition to location, actual compensation is based on various factors, including but not limited to, the candidate’s skill set, level of experience, education, and internal peer compensation comparisons.

CA: Minimum Salary $128,000, Maximum Salary $242,600

CO/HI/MN/VT*: Minimum Salary $111,400, Maximum Salary $202,100

IL*: Minimum Salary $122,800, Maximum Salary $222,400
Jersey City, NJ/NYC, NY: Minimum Salary $133,700, Maximum Salary $242,600

MA/WA: Minimum Salary $111,400, Maximum Salary $23