Senior Data Engineer

Job Description

Guardian is seeking a Senior Data Engineer to join the team responsible for establishing and maintaining an optimized, scalable data pipelines to provide access to high quality, consistent data in an easy and convenient manner to all authorized end users and applications of Guardian and its subsidiaries.

As a Senior Data Engineer, you will play a vital role in our data infrastructure and analytics ecosystem. You will work closely with data architects and data scientists to build and maintain data pipelines, transform raw data into actionable insights, and contribute to the development of cutting-edge data solutions. This is an excellent hands-on opportunity to experience the latest technologies and make a significant impact on our data-driven initiatives.

You will

• Collaborate with cross-functional teams to understand data requirements and translate them into effective data pipelines.
• Provide technical leadership and mentorship to a team of Data Engineers. Oversee the design, development, and optimization of scalable and robust big data solutions and pipelines.
• Utilize extensive expertise in Databricks technologies with Python, PySpark, and SQL, and distributed data processing frameworks to design, develop, and implement highly performant and fault-tolerant ETL/ELT pipelines for large-scale data ingestion, processing, and transformation.
• Partner with multiple management teams to ensure appropriate integration of functions to meet goals. Identify and define necessary system enhancements to deploy new products and process improvements. Conduct in-depth analysis of complex business processes and system processes to recommend innovative solutions.
• Shape opportunities and create execution approaches throughout the lifecycle of client engagements
• Ensure all deliverables are of high quality by setting development standards, adhering to the standards, and participating in code reviews
• Assist in data modeling, schema design, and database optimization.
• Monitor and troubleshoot data pipelines to ensure data accuracy, reliability, and performance.
  Participate in code reviews, testing, and documentation efforts to maintain high-quality code and data documentation.
• Stay updated on industry trends and best practices in data engineering.
• Establish and maintain the quality, supportability, and performance of all processes.
• Continue to look for innovative and next generation solutions for solving data challenges.
• Establish real time data pipelines using AWS services like SQS / SNS and OpenSearch.

You are

• A self-starter, self-motivated team player and have excellent communication skills, enabling you to engage with all levels of stakeholders who is fueled by collaboration, able to transform conceptual designs into reliable, scalable processes that meet or exceed customer needs.
• An individual who works closely with IT and business leaders across the enterprise to drive and embed the Enterprise Data Strategy into transformational roadmaps ensuring key principles of Data Governance, Data Management, Data Analytics and Architecture are implemented.

You have

• 15+ Years of IT experience, with 8+ years of experience as a Data Architect, data engineer or other data professional
• Experience with Databricks and ETL tools
• Knowledge of Mainframe and VSAM data structures
• Knowledge and hands on with Jenkins pipelines and CI/CD
• Expert Relational database concepts and experience across multiple database technologies (e.g., SQL, NoSQL, Oracle, Hadoop, Postgres) environments.
• Hands on experience in data engineering desired with extensive knowledge of SQL and PySpark, on Databricks Platform.
• Hands on experience designing, building, and optimizing data pipelines, architecture for large scale data warehousing projects.
• Experience with cloud data services such as Redshift, Relational database services (RDS), Snowflake and Databricks.
• Knowledge of AWS Data services is a big plus
• Demonstrated expertise with data sourcing, profiling, data mapping and data analysis.
• Strong SQL, PL/SQL experience and skilled in creating and optimizing queries and data structures.
• Knowledge of ETL concepts, preferably Informatica or Syncsort
• Experience with one or more SQL-on-Hadoop technology (HIVE, Presto, Spark SQL) and Hadoop ecosystem.
• Ability to design cloud and on Prem data solutions and provide guidance across storage, big data platform services, Hadoop ecosystem, vendor SaaS integrations, RDBMS and NoSQL databases is a plus.
• Excellent written and verbal communication skills and creative problem-solving skills.
• Ability to adapt to changing business priorities and experien