Senior Data Engineer

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career.

Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express.

We are seeking a highly skilled and motivated Senior Databricks Engineer to join our growing data engineering team. In this role, you will lead the design, development, optimization and implementation of large-scale data processing systems using Databricks. Your deep experience in data engineering and familiarity with either AWS or GCP will be essential in building robust, scalable, and efficient data pipelines. Our team is embarking on a transformational journey to migrate current Databricks based Data platform built on AWS to GCP based Databricks. You will lead several modules related to this project while interacting and building solutions aside some of the sharpest minds at Amex.

Key Responsibilities

• Work closely with enterprise teams such as LUMI, Cloud engineering, Infosec etc. to design, build, and maintain scalable data pipelines using Apache Spark on Databricks.
• Migrate existing AWS based data platform to GCP lumi and integrate with enterprise solutions.
• Architect and optimize ETL/ELT workflows for high-volume, low-latency data processing.
• Collaborate with data scientists, analysts, and business stakeholders to deliver reliable and high-quality data solutions.
• Manage and monitor Databricks clusters and job workloads for performance and cost optimization.
• Implement best practices in data lakehouse architecture, data governance, and security.
• Work with various data storage technologies including Delta Lake, Parquet, SQL, and NoSQL databases.
• Integrate data across various cloud services and APIs.
• Mentor junior engineers and promote knowledge sharing across the team.
• Drive automation and CI/CD practices for data workflows and infrastructure deployment.
• Stay up to date with the latest trends and advancements in the data engineering ecosystem.

Required Qualifications

• 8+ years of experience in data engineering with a strong focus on distributed systems and big data technologies.
• 3+ years of hands-on experience with Databricks and Apache Spark in production environments.
• Strong experience with Delta Lake, data lakehouse architectures, and performance tuning in Databricks.
• Proven expertise in building and managing data pipelines (ETL/ELT) using Python, Scala, or SQL.
• Solid understanding of cloud-native services on at least one public cloud (AWS, or GCP).
• Experience working with tools like Airflow, DBT, or similar for orchestration and transformation.
• Familiarity with DevOps practices, CI/CD pipelines, and Infrastructure as Code (e.g., Terraform).
• Strong knowledge of data modeling, data quality, and data governance practices.
• Excellent problem-solving, communication, and collaboration skills.

Preferred Qualifications

• Databricks certifications (e.g., Databricks Certified Data Engineer Professional).
• Experience with streaming data technologies like