Staff Data Engineer - Databricks SME

This is a remote position.

Ad Hoc is a technology company that empowers organizations to deliver scalable, impactful digital services. Using modern, agile methods, our team creates products that meet people’s needs and transform their experience of government.

Work on things that matter

Our collaborations have shaped some of the defining moments in public-sector service delivery. We’ve helped build products that connect Veterans to tailored services, help millions access affordable health care, and support important programs like Head Start. As we work with agencies to deliver critical services, we’re also changing how the government approaches technology.

Built for a remote life

Our culture, communications, and tools are built for remote work, enabling us to bring together top talent nationwide. At Ad Hoc, remote life empowers our teams to design work environments that fit their lives and that foster flexibility and collaboration to achieve positive outcomes for our customers.

Committed to high expectations and a welcoming culture

Ad Hoc values acceptance, accountability, and humility. We aren’t heroes. We learn from our mistakes and improve the process for the next time. We build small, inclusive teams to collaborate closely with our partners to solve the right problems and deliver software that works.

The Veterans Affairs business unit helps transform the VA into a modern digital services organization where Veteran outcomes are at the center of every effort. We partner with the VA to design and deliver seamless user experiences for Veterans, their families and caregivers, and VA employees. By applying better practices in service design, product management, and technology, we enable the VA to increase the use, quality, and reliability of services and decrease the time Veterans spend waiting for outcomes.

Primary Responsibilities:

Staff Data Engineer - Databrick expert builds scalable data pipelines, optimizing workloads, and enabling advanced analytics across cloud platforms, collaborates with data scientists and engineers to transform raw data into actionable insights, ensuring high performance, data integrity, and seamless integration within the Databricks environment. The Databricks expert builds robust data lakehouse architectures and advanced analytics and machine learning workflows. This role involves collaborating with various stakeholders to understand data requirements and develop solutions. Primary expectations of a Staff Data Engineer include:

• Develop and maintain ETL/ELT pipelines using Databricks notebooks and workflows

• Optimize Spark jobs for performance, scalability, and cost-efficiency

• Design and implement data lakehouse architectures

• Integrate Databricks with external data sources, APIs, and on-prem databases

• Collaborate with data scientists, and engineers to support end-to-end data solutions

• Ensure data governance, security, and compliance using tools like Unity Catalog and RBAC

• Serves as a mentor, supporting junior level team members within the business unit and organization

Basic Qualifications:

• Bachelor’s degree in computer science, information technology, or a related field, and 8+ years of experience

• Minimum of 5 years of experience as a data analyst

• Strong knowledge of database structures, theories, principles, and practices

• 5+ years of experience in big data engineering and analytics

• Experience with cloud platforms, ML and data cataloging tools

• Understanding of data privacy and protection regulations

• Strong analytical and problem-solving skills

• Proficiency with SQL, Python, R, or other data analysis languages

• Experience with data visualization tools and reporting platforms

• Strong understanding of statistics, data modeling, and analytical techniques

• Familiarity with databases, data warehouses, and ET processes

• Ability to lead and motivate a remote team

• Excellent time management and organizational skills

• Adaptability and openness to new ideas

Preferred Qualifications:

• Prior government contracting experience

To learn more about working at Ad Hoc, please visit:https://adhocteam.us/join

Benefits:

• Company-subsidized health, dental, and vision insurance


• Flexible P