Senior Data Engineer

<p>540 is seeking a Senior Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will lead the design and evolution of Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale.</p> <p>Working with engineers, architects, cybersecurity teams, and mission stakeholders, you will translate complex requirements into secure, scalable solutions using Databricks, Apache Spark, and Delta Lake. You will define engineering standards, guide technical delivery, and mentor engineers while ensuring data quality, governance, and platform reliability.</p> <p><strong>Location</strong>: Arlington, VA<br><strong>Citizenship &amp; Clearance Requirement</strong>: Per client requirements, candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance<br><strong>Education Requirement:&nbsp;</strong>Bachelor’s degree in Computer Science, Engineering, or a related technical field preferred; equivalent combinations of education and relevant experience will be considered<br><strong>540 Internal Thrive Level: </strong>Senior<strong> </strong>Data Engineer</p> <p><strong>WHY 540?</strong></p> <p>540 is a forward-thinking company that the government turns to in order to #getshitdone. We don’t just talk about innovation – we deliver it. We break down barriers, build impactful technology, and solve mission-critical problems.</p> <p><strong>HOW YOU’LL DRIVE IMPACT</strong></p> <ul> <li>Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities</li> <li>Translate mission requirements into scalable data architectures and implementation strategies</li> <li>Define data engineering standards, reusable patterns, and best practices across engineering teams</li> <li>Architect automated ETL/ELT pipelines using Python, SQL, PySpark, Apache Spark, and Delta Lake</li> <li>Design scalable data models, schemas, data contracts, and medallion architecture patterns</li> <li>Lead the development of batch and streaming capabilities supporting operational, analytical, and AI/ML workloads</li> <li>Establish data quality, lineage, metadata, observability, and governance practices using Unity Catalog or similar technologies</li> <li>Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency</li> <li>Establish CI/CD, infrastructure-as-code, testing, monitoring, and operational practices for Databricks environments</li> <li>Lead design reviews and resolve complex issues spanning data pipelines, infrastructure, and production services</li> <li>Pa