Senior Data Engineer - Databricks

Senior Data Engineer - Databricks<p></p><p>Description -</p><p>The HP Ecommerce CRM enablement team is seeking for its CDP program a proven and hands-on Databricks process architect and engineering leader to drive our Databricks transformation, pipelines architecture, governance and engineering patterns.  Our CDP delivers a unified Customer data model across our global CRM operations. In this role, you will be responsible for driving our Databricks migration, designing and executing a comprehensive pipeline architecture strategy that aligns with CDP team's skillsets, business objectives and drives compliant, efficient and effective data management. </p><p> </p><p>Responsibilities </p><ul><li><p>Drive our AWS Databricks transformation from current relational DB setup. </p></li><li><p>Collaborate with our diverse teams, SQL developers/data engineers/CRM analysts, to develop, implement and maintain sustainable, high-performance, growth-ready data-processing and data integration systems </p></li><li><p>Provide best practice on CI/CD techniques with Git and Databricks </p></li><li><p>Provide best practice on Databricks cluster management. </p></li><li><p>Reviews and evaluates designs and project activities for compliance with privacy, architecture, security and quality guidelines and standards; provides tangible feedback to improve product quality and mitigate failure risk. </p></li><li><p>Design and maintain an appropriate metadata repository to enable the organization to understand its data assets. </p></li><li><p>Builds and manages relationships throughout the organization. </p></li><li><p>Drives innovation and integration of new technologies into projects and activities in the big data space. </p></li><li><p>Collaborates and communicates with project team regarding project progress and issue resolution. </p></li><li><p>Provides guidance and mentoring to less experienced staff members. </p></li></ul><p>Requirements </p><ul><li><p>Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent. </p></li><li><p>10+ years of experience in data architecture, database design, and data management. </p></li><li><p>Expert in AWS Databricks  </p></li><li><p>Expert in SQL &amp; relational database systems (PostgreSQL preferred) </p></li></ul><ul><li><p>Proficient knowledge in working with distributed data processing frameworks like Apache Spark. </p></li><li><p>Hands-on experience in Python and Pandas </p></li><li><p>Experience in overall architecture of big data systems within AWS </p></li><li><p>Experience in Global Companies a great advantage. </p></li></ul><p>Knowledge &amp; Skills </p><ul><li><p>Extensive experience with designing data engineering solutions to manage data streams in complex, distributed and parallel systems. </p></li><li><p>Excellent analytical and problem-solving skills. Ability to collaborate with and analyze source systems to drive data profiling/cataloging. Can present clear insights to colleagues to support the end use of the data. </p></li><li><p>Extensive experience with data engineering pipelines, frameworks to mine, cleanse and explore data. </p></li><li><p>Strong problem management. Initiating and monitoring actions to investigate patterns and trends to resolve problems. Determining preventative measures. </p></li><li><p>Evaluating forms and processes for database architecture testing and methodology, including writing and execution of test plans, debugging, and testing scripts and tools. </p></li><li><p>Excellent written and verbal communication skills; mastery in English and local language. </p></li></ul><p>Complexity </p><ul><li><p>Applies advanced subject matter knowledge to solve complex business issues and is regarded as a subject matter expert. </p></li><li><p>Frequently contributes to the development of new ideas and methods. </p></li><li><p>Works on complex problems where analysis of situations or data requires an indepth evaluation of multiple factors. </p></li><li><p>Acts as an expert providing direction and guidance to process improvements and establishing policies. </p></li><li><p>Exercises significant independent judgment within broadly defined policies and practices to determine best method for accomplishing work and achieving objectives. </p></li><li><p>Provide mentoring and guidance to lower level employees. </p></li></ul><p> </p><p></p><p></p><p>Job -</p>Software<p></p><p>Schedule -</p>Full time<p></p><p>Shift -</p>No shift premium (Spain)<p></p><p>Travel -</p><p></p><p>Relocation -</p><p></p><p>Equal Opportunity Employer (EEO) - </p><p></p><p>HP, Inc. provides equal employment opportunity to all employees and prospective employees, without regard to race, color, religion, sex, national origin, ancestry, citizenship, sexual orientation, age, disability, or status as a protected veteran, marital status, familial status, physical or mental disability, medical condition, pregnancy, genetic predisposition or carrier status, uniformed servi