Senior Data Engineer

Waverly Advisors’ primary goal is to serve our clients, one another, and our communities. We aren’t your typical wealth management firm. Our intense client focus is at the center of everything we do. We go far beyond just managing our clients’ investments, offering truly in-depth financial planning. We set ourselves apart by actually living and acting on our guiding principle, ‘Serve.’ It is the reason we go to work every day.   

In this role, you’ll be challenged to take on work that upholds our guiding principle and drives Waverly Advisors forward. We hope you’ll grow as a person and leader in your field and transform those around you as well.    

Waverly Advisors is building a modern data platform on Databricks. It brings together custodian, portfolio management, CRM, and planning data into one governed source of truth that BI, internal applications, and AI workloads consume. Everything is deployed as code and governed to the standards expected of a regulated financial services organization. 

We are looking to add a Senior Data Engineer to our team in our Birmingham, AL office. The role is an essential part of the Waverly Advisors team. As Senior Data Engineer, you will help design how data arrives, how it is validated, how it is modeled, and how the rest of the firm consumes it. Some sources arrive as files, others through APIs that were not designed for our convenience. You will extend the ingestion framework we have started so that onboarding a source becomes configuration and review rather than a new engineering project. 

 

Responsibilities: 

• Extend our ingestion framework across data contracts, schema validation, quarantine handling, freshness monitoring, and provenance. 

• Onboard data sources end to end, from contract authoring through production go-live and monitoring. 

• Design the master account, entity, and household model that conforms data from every source system. 

• Build the curated datasets the firm runs on, from conformed core tables through the reporting and analytics layers that BI, internal applications, and AI consume. 

• Set testing standards for data code, including unit tests, contract fixtures, and integration tests on masked data. 

• Build production data operations, including quarantine triage, reprocessing workflows, and data quality expectations. 

• Write the runbooks and decision records that let the next engineer pick up your work and let an examiner trace a number back to its source file. 

 

Required Qualifications and Experience:  

• Production-depth experience with Spark and Delta Lake, including Structured Streaming, Auto Loader, schema evolution, and idempotent pipeline design.  

• Hands-on Databricks experience, including jobs, serverless compute, and working within Unity Catalog as a data producer: schema and table design, tagging, row filters and column masks, and grants managed as code.  

• Databricks Asset Bundles or equivalent deployment tooling. Our jobs and pipelines are deployed as code through CI, not built in the workspace UI.  

• Experience building or substantially owning a framework, not just a collection of pipelines.  

• Real API integration experience, including rate limits, inconsistent pagination, expiring credentials, and vendor behavior that changes without notice.  

• Testable Python, with transform logic structured as functions and pytest as part of your normal workflow.  

• Strong SQL skills, including window functions and modeling patterns for slowly changing data.  

• Comfort inheriting an existing codebase, learning its conventions, and improving it.  

• Sound judgment about which technical decisions are easy to reverse and which ones are long-term commitments.  

 

Domain knowledge is a real advantage. If you have worked with custodial data, portfolio accounting, or trust and entity structures, tell us. If you came to engineering through the industry rather than the other w