Senior Data Engineer (Hybrid; 80-100%; m/f/x/d)

Are you and experienced Senior Data Engineer ready to contribute in one of the most fast-growing digital insurance platform in the market? If yes - then we have an outstanding opportunity for YOU! Buckle up and read on.

 

About the Role

 

As Senior Data Engineer helps us to maintain and improve our existing data ingestion into the data lake and building up a data mesh in Tangram.

 

Your key responsibilities involve:

• Infrastructure Development and Optimization: Design, build, and maintain the infrastructure and baseline services required for implementing a data mesh architecture. Optimize existing infrastructure to ensure optimal performance, reliability, and cost-efficiency.

• Data Architecture Transition: Lead the transition from classical data lake architecture to a data mesh, ensuring a smooth migration and minimizing data loss or downtime. Establish domain-driven decentralized data architectures, creating a robust and scalable data ecosystem.

• Real-time Data Streaming Management: Design, implement, and manage real-time data streaming pipelines using Apache Kafka or other relevant technologies. Ensure the reliability, efficiency, and accuracy of real-time data pipelines.

• Backend Development: Work closely with backend developers to ensure the seamless integration of data services with Java and SpringBoot applications. Ensure data availability and consistency across microservices and the broader application ecosystem.

• Database Management: Administer and optimize the performance of AWS RDS/PostgreSQL, AWS DocumentDB/MongoDB databases. Ensure data integrity, availability, and security across all database systems.

• CI/CD Pipeline Management: Enhance existing CI/CD pipelines to accommodate new data services, ensuring a smooth deployment and management process.

• Data Governance and Compliance: Establish and maintain data governance practices ensuring data quality, data lineage, and compliance with GDPR or other regional data protection laws.

• Team Collaboration and Mentoring: Collaborate with cross-functional teams to drive the successful execution of data projects. Mentor junior data engineers and other team members, promoting a culture of continuous learning and improvement.

• Performance Monitoring and Troubleshooting: Monitor system performance, identify issues, and implement necessary optimizations to ensure optimal performance. Debug and troubleshoot data-related issues, ensuring the stability and reliability of the data infrastructure.

• Emerging Technology Exploration: Stay updated with the latest industry trends and technologies, evaluating and recommending new tools and technologies that can enhance the platform's capabilities.

• Documentation and Knowledge Sharing: Document system architectures, data models, and processes, ensuring that knowledge is shared across the organization. Facilitate training and knowledge sharing sessions to champion a deeper understanding of data infrastructure among the team.

 

Our Technology Stack:

 

Our cloud-native, multi-tenant SaaS platform operates on AWS, embodying a robust and scalable architecture. Here are the key technologies we use:

 

• Backend: Utilizing Kotlin, Java, and SpringBoot, orchestrated via Docker and Kubernetes. Real-time data streaming is managed through Apache Kafka.

• Database Management: We employ MongoDB and PostgreSQL for a balanced approach to NoSQL and relational data handling.

• Frontend: Our user interfaces are built with ReactJS and TypeScript, ensuring a responsive and intuitive user experience.

• Continuous Delivery: Our pipeline is streamlined with GitLab, facilitating a consistent and reliable delivery process.

• Monitoring and Logging: Employing a suite comprising Prometheus, ElasticSearch, Kibana, Grafana and AWS X-Ray for comprehensive system monitoring and log management.

• Data Lake: Built on AWS S3 and AWS Aurora, with AWS Lambda and DTB handling most data transformations, ensuring a well-structured and manageable data ecosystem.

• Reporting: Tableau serves as our primary tool for data visualization and reporting, aiding in data-driven decision-making.

• Data Mesh Initiative: We are in the preliminary stages of implementing a Data Mesh architecture to enhance data discoverability and decentralization.

• AI & Real-time Analytics Exploration: Initiating investigations into AI and real-time analytics to potentially enhance our platform's capabilities.

 

About the Team

 

We are a diverse, international team of highly motivated individuals with a strong team spirit. We believe i