AWS/Databricks Software Engineer

Job Title: AWS/Databricks Software Engineer
Job location: New York NY.

Job Description:

• Design and implement Azure cloud-based Data Warehousing and Governance architecture with Lakehouse paradigm Integrating technical functionality ensuring data accessibility accuracy and security.
• Architect the Unity Catalog to provide centralized access control auditing lineage and data discovery capabilities across Databricks workspaces.
• Define and organize data assets structured and unstructured within the Unity Catalog.
• Enable data analysts and etl engineers to discover and classify data notebooks dashboards and files across clouds and platforms.
• Implement a single permission model for data and AI assets Define access policies at a granular level rows columns features to ensure secure and consistent access management across workspaces and platforms.
• Leverage Delta Sharing to enable easy data sharing across regions and platforms.
• Ensure that data and AI assets can be securely shared with minimal replication maintaining a unified experience for users.
• Monitoring and Observability utilize AI to automate monitoring diagnose errors and maintain data and quality.
• Set up alerts for personally identifiable information PII detection and operational intelligence.
• Work closely with data scientists’ analysts and engineers to promote adoption of the Unity Catalog.
• Provide training and documentation to ensure effective usage and compliance with governance policies.

Skills

• Designed data warehouse and data lake solutions along with data processing Pipeline using PySpark using Databricks.
• Performed Data Modelling on Databricks Delta Table for transactional and analytical need Designed and developed pipelines to load data to Data Lake Databricks Platform.
• Proficiency including its components like Databricks SQL Delta Live Tables Databricks Repos and Task Orchestration.
• Deep understanding of data governance principles especially related to data cataloging access control lineage and metadata management.
• Strong SQL skills for querying and managing data Ability to design and optimize data models for structured and unstructured data.
• Understand how to manage compute resources including clusters and workspaces.
• Ability to adapt to changes and emerging trends in data engineering and governance Involved in hands on development and configuration of Unity Catalog.

Life at Capgemini:

Capgemini supports all aspects of your well-being throughout the changing stages of your life and career. For eligible employees, we offer:

• Flexible work
• Healthcare including dental, vision, mental health, and well-being programs.
• Financial well-being programs such as 401(k) and Employee Share Ownership Plan
• Paid time off and paid holidays
• Paid parental leave.
• Family building benefits like adoption assistance, surrogacy, and cryopreservation
• Social well-being benefits like subsidized back-up child/elder care and tutoring
• Mentoring, coaching, and learning programs.
• Employee Resource Groups.
• Disaster Relief.

About Capgemini:
Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world, while creating tangible impact for enterprises and society. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, its clients to unlock the value of technology to address the entire breadth of their business needs trust Capgemini. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, cloud and data, and combined with its deep industry expertise and partner ecosystem. The Group reported 2023 global revenues of €22.5 billion.

Get the future you want | www.capgemini.com

Disclaimer:
Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.

This is a general description of the Duties, Responsibilities and Qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed. Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini wil