AVP, Senior Data Engineer

Job Description:

Role Summary/Purpose:

We are looking for an Enterprise Data Lake Principal Product Engineer who will play an integral role in the administration of our Enterprise Big Data platform. The platform will host significantly large datasets sourced from batch, real-time feeds, variety of user interactions with mobile apps and internal transactions. As Principal Product Engineer, you will be responsible for ensuring our platform availability, performance, security, maintenance - upgrades, installation, and user administration across all Data Lake environments both on-prem & cloud. We are looking for a Hadoop professional who is passionate about creating data driven systems and have experience in maintaining large complex distributed systems at scale in production.

Our Way of Working

We’re proud to offer you choice and flexibility. At Synchrony, our way of working allows you to have the option to work from home, near one of our Hubs or come into one of our offices. Occasionally you may be required to commute to our nearest office for in person engagement activities such as business or team meetings, training and culture events.

Essential Responsibilities:

• Be the SME for all platform issues / POCs for immediate complex issues and finalize the new feature which to be enabled on EDL.
• Responsible for the design and implementation of scalable & fault tolerant data applications on Big-Data & Cloud Platforms to store and process terabytes of data from upstream sources with high availability. 
• Partner with information architects, platform architects, data scientists and product management on solution requirements to design solutions.
• Resolve problems through in-depth evaluation of business processes, system processes, enterprise standards & procedures. 
• Responsible for design, development, testing oversight and production implementation using Spark, Hive, Kafka, API, Scala, Python, SQL & AWS. 
• Work with development teams to assist in code promotion across environments and deployments in production, including CMDB CI creation and updates
• Perform capacity planning and expansion activities working across Infrastructure and other Enterprise Services teams, including financial planning
• Ensure all existing data lake functionality is running as expected and we are meeting SLA’s and reporting on SLA compliance
• Drive implementation of monitoring tools to measure the performance of the environments
• Drive integration of the environments with service management tools to drive automated incident management and reporting
• Define and lead a feedback mechanism for the build teams to identify areas of improvement to enhance supportability of new functionality
• Define metrics and provide reporting on application outages on all clusters
• Provide RCA analysis to key environment stakeholders
• Work with build teams to analyze the impact of new functionality releases on the production clusters to identify potential risks and mitigate them and be part of the DevOps process
• Analyze, Design & Builds operational services which run and repairs themselves
• Works on reducing Manual, Repetitive and Non-value work
• Capacity and life cycle management – trend analysis & software concurrency
• Perform other duties and/or special projects as assigned

Qualifications/Requirements:

• Bachelor's degree with minimum 4+ years of industry experience as a Hadoop platform run lead Or in lieu of Degree, a High School Diploma/GED and 4 years of experience as a Unix/Linux Admin and 4 years as a Hadoop Platform Run Lead - in a production environment.
• 5 years of experience with production support operational processes and tools
• 5 years of experience with enterprise level Hadoop administration, support and code deployments on any major Hadoop distribution – MapR, Hortonworks, Cloudera – in a production environment.
• 4 years of experience with issue management and escalation processes, Hardware and Software Monitoring including Network performance, production installation & rolling upgrades, administering Hadoop Security components – in a Hadoop production environment.
• 4 years of experience troubleshooting and analyzing Hadoop application issues
• 4 years of experience developing of Capacity Plans, Runbooks, and other operational documentation
• 4 years of experience with design, management, implemen