Data Engineer II, PySpark/Databricks

This is a remote position.

Ad Hoc is a technology company that empowers organizations to deliver scalable, impactful digital services. Using modern, agile methods, our team creates products that meet people’s needs and transform their experience of government.

Work on things that matter

Our collaborations have shaped some of the defining moments in public-sector service delivery. We’ve helped build products that connect Veterans to tailored services, help millions access affordable health care, and support important programs like Head Start. As we work with agencies to deliver critical services, we’re also changing how the government approaches technology.

Built for a remote life

Our culture, communications, and tools are built for remote work, enabling us to bring together top talent nationwide. At Ad Hoc, remote life empowers our teams to design work environments that fit their lives and that foster flexibility and collaboration to achieve positive outcomes for our customers.

Committed to high expectations and a welcoming culture

Ad Hoc values acceptance, accountability, and humility. We aren’t heroes. We learn from our mistakes and improve the process for the next time. We build small, inclusive teams to collaborate closely with our partners to solve the right problems and deliver software that works.

The Federal Civilian business unit supports many customers spanning the federal, commercial, and nonprofit space. Our customers include NASA, the General Services Administration, Office of Personnel Management, the Library of Congress, Health & Human Services, and the FDIC. We partner with these agencies to build new capabilities, deliver products, establish data as a strategic asset for informed decision-making, modernize legacy systems, and build the digital service infrastructure necessary to scale their mission impact. This role is on a program within Health & Human Services.

Primary Responsibilities:

Data Engineer II, PySpark/Databricks serves as an emerging individual contributor within a team, expanding your leadership, guidance and mentoring skills. With the support and guidance of leadership, you will be responsible for supporting the goal of meeting scope, schedule and delivery requirements. You will interact with stakeholders and utilize influential skills to drive improvements in data engineering processes and practices. Primary expectations of a Data Engineer II, PySpark/Databricks include:

• Build and maintain PySpark data pipelines in the Databricks environment

• Optimize Spark jobs performance and resource usage, identifying and addressing bottlenecks and inefficiencies in backend systems

• Design, develop, and maintain high-quality backend software components and services, ensuring functionality, performance, and scalability

• Research and build proof of concepts in the data space

• Write clean, well-structured, and maintainable code, adhering to established coding standards and best practices

• Perform thorough code reviews, providing constructive feedback to peers and identifying potential risks or areas for improvement

• Debug and resolve defects, proactively identifying and addressing potential issues before they impact users

• Create and maintain comprehensive technical documentation

• Actively participate in Agile ceremonies, such as stand-ups, sprint planning, and retrospectives, ensuring effective communication and collaboration across the team

• Assist in the estimation, prioritization, and planning of development tasks, ensuring projects are delivered on time and within budget

• Continuously evaluate and recommend new dataframe related technologies, frameworks, and tools, helping to drive innovation and keep the team up-to-date with industry trends

• Engage in ongoing professional development to stay current with industry best practices, and share knowledge and insights with the team as appropriate

• Assist in the implementation and maintenance of security, compliance, and governance policies within the Databricks and AWS environment to ensure adherence to industry standards and regulatory requirements.

Basic Qualifications:

• Bachelor’s degree and 8 years of experience

• Strong experience with Python / Apache Spark

• Solid understanding of data modeling, ETL process, and distributed computing

• Bachelor's degree in Computer Science, Computer Engineering or related field

• Strong und