Jobs and Careers
NE
Associate Partner-Kafka/Flink SME
Ness Digital EngineeringNew Jersey, New Jersey, United States, United Statesfull_timeVerifiedPosted 15 Apr 2025
About the role
Position Overview: The Associate Partner – Kafka/Flink Subject Matter Expert (SME) will be responsible for leading and overseeing the design, implementation, and optimization of Kafka and Flink-based systems within the organization. This leadership role requires deep technical expertise and the ability to guide the strategy, architecture, and execution of complex streaming data solutions. The SME will collaborate with cross-functional teams to deliver high-performance, scalable, and reliable data processing pipelines and analytics platforms.Key Responsibilities:
- Leadership and Strategy:
- Lead the strategic direction for implementing Kafka and Flink-based solutions in the organization.
- Provide thought leadership on industry best practices, design patterns, and emerging trends in streaming data systems.
- Collaborate with business and technical stakeholders to align data streaming strategies with organizational goals.
- Technical Expertise:
- Design and architect complex data streaming systems using Apache Kafka and Apache Flink, ensuring scalability, performance, and reliability.
- Develop and implement solutions to optimize data flow, reduce latency, and enhance real-time data processing capabilities.
- Troubleshoot and resolve complex technical issues related to Kafka and Flink environments.
- Guide the technical teams in implementing best practices for stream processing, data modeling, and fault tolerance.
- Solution Design and Development:
- Lead the design, development, and deployment of Kafka and Flink pipelines for data integration, real-time analytics, and event-driven architectures.
- Collaborate with engineering teams to implement data pipelines that meet business requirements and data governance standards.
- Ensure that solutions are aligned with data security, privacy, and compliance requirements.
- Collaboration and Mentorship:
- Act as a mentor and guide for engineers and developers, providing expertise and fostering a culture of knowledge sharing.
- Collaborate with data scientists, analysts, and architects to design and optimize data processing workflows.
- Conduct training sessions and workshops to build internal capabilities in Kafka, Flink, and stream processing technologies.
- Performance and Optimization:
- Monitor and optimize the performance of Kafka and Flink clusters, ensuring high availability, fault tolerance, and scalability.
- Work closely with DevOps teams to automate deployment pipelines, ensuring smooth production rollouts and continuous integration/continuous delivery (CI/CD).
- Develop and enforce best practices around monitoring, logging, and alerting for Kafka/Flink-based systems.
- Continuous Improvement:
- Continuously assess the effectiveness of existing streaming data architectures and make recommendations for improvements.
- Stay current with advancements in the streaming data ecosystem and evaluate new tools and technologies to enhance capabilities.
- Stakeholder Communication:
- Present technical recommendations and solutions to senior leadership, ensuring alignment with broader business goals.
- Lead regular meetings with internal teams, including IT, data engineering, and product teams, to drive progress on Kafka and Flink initiatives.
- Bachelor's or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 8+ years of experience in data engineering, software engineering, or a related technical field, with at least 4+ years specializing in Kafka and Flink.
- Strong expertise in designing, building, and managing distributed data streaming systems using Apache Kafka and Apache Flink.
- Extensive experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and containerization technologies (e.g., Kubernetes, Docker).
- Deep understanding of stream processing, event-driven architectures, and messaging systems.
- Proven track record in leading complex projects, particularly in high-performance, real-time data processing environments.
- Proficiency in Java, Scala, Python, or similar programming languages commonly used in the Kafka/Flink ecosystem.
- Strong understanding of data modeling
Apply for this role
Generate a tailored application kit with a matched cover letter, interview prep, and CV highlights — in under 60 seconds.
Apply Now →Generate Application KitFree account required — sign up in 30s