Senior Associate - DevOps Engineer

 

Location Designation: Hybrid - 3 days per week 

 

 

Role Overview:

We are looking for a skilled Sr. DevOps Engineer to join our Artificial Intelligence and Data (AI&D) group, focusing on deploying and managing configurations for our integration platform using cloud-native infrastructure. The role involves managing and optimizing microservices hosted in the AWS ecosystem, leveraging AWS various services. The ideal candidate will have a strong background in DevOps practices, cloud infrastructure, and integration technologies, with a passion for automation using AI/ML frameworks, scalability, and reliability.  

This role requires ability to consistently communicate across multiple channels (Teams, emails, meetings, conversations), proactively identifying blockers and escalating as needed, organizational skills to support the team in maximizing its performance and ensuring successful delivery of projects 

 

What You’ll Do:

Infrastructure Provisioning and Management

• Provision and manage AWS resources such as EC2, EKS, Redshift, RDS, DMS, SFTP, Lambda, Glue, and Step Functions and much more
• Ensure consistent and scalable infrastructure setup using tools like Terraform for infrastructure as code (IaC) management

CI/CD Pipeline Development and Implementation

• Develop, implement, and maintain comprehensive CI/CD pipelines for Data Lake and Data Warehouse deployments, leveraging tools such as Jenkins, Harness, Ansible, and GitHub to ensure seamless integration and automation.
• Design and deploy CI/CD pipelines for database scripts (DDL, DML) using Liquibase. Establish Liquibase instances and enforce quality checks through rigorous policy implementation.
• Utilize bash and Python scripting to automate a variety of tasks, enhancing efficiency and reducing manual intervention.
• Manage build artifacts and versioning of builds using repositories like JFrog or Nexus, ensuring organized and traceable build histories.
• Integrate automated testing, security scanning (CheckMarx), and compliance checks into the CI/CD pipeline to ensure robust security, quality, and regulatory adherence throughout the development lifecycle.

Data Streaming and Integration: 

• Configure and manage data streaming and integration using change data capture (CDC) and AWS Manage Streaming(Msk) for real-time data processing. 
• Ensure seamless integration between on-premises systems, cloud environments, and third-party services. 

Monitoring and Observability

• Set up and manage monitoring and alerting solutions NewRelic and AWS CloudWatch.
• Route all system logs to Sumo Logic for centralized logging.
• Ensure observability into deployment processes.
• Monitor and manage EKS clusters, ensuring high availability, fault tolerance, and performance optimization. 

Deployment and Configuration Management: 

• Ensure smooth and automated deployments across development, testing, and production environments, minimizing downtime and manual intervention. 
• Deploy and manage Data & Analytics environments, ensuring they are optimized for performance, scalability, and security. 
• Automate the deployment and configuration of services in AWS using tools like Jenkins, Harness, Terraform, Ansible, and AWS CloudFormation. 
• Manage the configuration and deployment of various AWS various services to support integration and microservices architectures. 

Cost Optimization

• Cloud Cost Management: Set up CloudWatch alarms and alerts to optimize cloud costs.

Security and Compliance: 

• Implement and manage security best practices for AWS infrastructure, including IAM roles, security groups, encryption, system manager and secrets management. 
• Ensure compliance with organizational policies and industry standards for data protection, auditing, and logging. 

 

Collaboration and Support: 

• Work closely with development, operations,