Senior Big Data Engineer

<p><u><b>Posting Type</b></u></p><p>Hybrid</p><p></p><p></p><p><u><b>Job Overview</b></u></p>Here at Relativity we prioritize flexibility and work-life harmony. Our Hybrid work environment provides options tailored to your role and location, aiming to enhance engagement, connectivity, and productivity.<br/> <br/>Join us to experience a culture of collaboration and innovation, where connecting in-person adds value to our collective growth. Let's work together!<br/><br/>Join our team as we innovate the future of data platform architecture, enabling massive scaling and data processing for ML and Gen AI projects. You'll be at the forefront of processing vast unstructured data, building high-throughput APIs, and supporting distributed compute frameworks for seamless model deployment. Ready to dive into the heart of cutting-edge tech?<p></p><p></p><p><u><b>Job Description and Requirements </b></u></p><p><b><span>Your role in action</span></b><span> </span></p><ul><li><p><span><span>Build our next-generation data platform tooling and services to support the ingestion and processing of billions of documents at scale.</span></span><span> </span></p></li></ul><ul><li><p><span><span>Improve and extend our Spark based distributed data processing pipeline.</span></span><span> </span></p></li></ul><ul><li><p><span><span>Improve and extend our Rust based distributed query engine used to request </span><span>large amounts</span><span> of document data.</span></span><span> </span></p></li></ul><ul><li><p><span><span>Create tools to automate and </span><span>optimize</span><span> processes across disciplines</span></span><span> </span></p></li></ul><ul><li><p><span><span>Actively </span><span>participate</span><span> in the on-call schedule to investigate and fix production issues related to our data processing pipeline or query engine.</span></span><span> </span></p></li></ul><ul><li><p><span><span>Participate in code reviews for projects written by your team</span></span><span> </span></p></li></ul><ul><li><p><span><span>Focus on quality through comprehensive unit and integration testing</span></span><span> </span></p></li></ul><p><span> </span></p><p><b><span>Your Skills</span></b><span> </span></p><ul><li><p><span><span>4+ years of software development experience in writing performant, commercial-grade </span><span>systems</span><span> and applications </span></span><span> </span></p></li></ul><ul><li><p><span><span>Experience with monitoring and troubleshooting production environments</span></span><span> </span></p></li></ul><ul><li><p><span><span>Proficiency</span><span> in programming languages used in high volume data processing and applications like Java or Scala and Python</span></span><span> </span></p></li></ul><ul><li><p><span><span>Experience building data pipelines with distributed </span><span>compute</span><span> frameworks like Hadoop. Spark, or </span><span>Dask</span></span><span> </span></p></li></ul><ul><li><p><span><span>Knowledge of Linux/Unix systems, Docker/Kubernetes and CI/CD including scripting in Python or other scripting languages to automate build and deployment processes</span></span><span> </span></p></li></ul><ul><li><p><span><span>Knowledge of professional software engineering practices &amp; software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations</span></span><span> </span></p></li></ul><ul><li><p><span><span>Leverages best practices and past experiences to mentor and improve the productivity of the team</span></span><span> </span></p></li></ul><p><span> </span></p><p><b><span>We’d</span><span> particularly love it if you have:</span></b><span> </span></p><p><span><span>Deep experience building </span><span>and debugging distributed data pipelines</span></span><span> </span></p><p><span><span>Experience with columnar databases and storage formats like Delta Lake and Parquet</span></span><span> </span></p><p><span><span>Experience deploying and managing services on Kubernetes</span></span><span> </span></p><p><span><span>Experience </span><span>building with Rust</span></span><span> </span></p><p><span> </span></p><p><u><span><span>If you </span><span>don’t</span><span> meet 100% of the above qualifications, you should still seriously consider applying. </span></span><span> </span></u></p><p></p><p><span><span>Relativity is a diverse workplace with different skills and life experiences—and we love and celebrate those differences. We believe that employees are happiest when </span><span>they're</span><span> empowered to be their full, authentic selves, </span><span>regardless</span><span> how you </span><span>identify</span><span>.</span></span><span> </span></p><p><span> </span></p><p><u><span>Benefit Highlights:</span></u><span> </span></p><p><span><span>Comprehensive health plan</span></span><span> </span></p><p><span><span>Flexible work arrangements</span></span><span> </span></p><p><span><span>Two, w