Senior Associate - Security Data Lake Engineer

Location Designation: Hybrid - 3 days per quarter 

As part of Technology, you'll have the opportunity to contribute to groundbreaking initiatives that shape New York Life's digital landscape. Leverage cutting-edge technologies like Generative AI to increase productivity, streamline processes, and create seamless experiences for clients, agents, and employees. Your expertise fuels innovation, agility, and growth — driving the company's success.

Role Overview
New York Life is seeking a Security Data Lake Engineer to design, implement, and optimize the enterprise-scale security data lake and advanced analytics capabilities. This role will serve as a technical specialist responsible for building scalable data pipelines, enabling data science-driven use cases, and ensuring the long-term health and cost-effectiveness of the data lake.

The Security Data Lake Engineer will collaborate with SOC Analysts, Threat Intelligence, Data Scientists, and other Cybersecurity teams to unlock new detection and analytics capabilities, including AI-driven anomaly detection, semantic search, and retrieval-augmented generation (RAG). This role is ideal for an engineer who thrives at the intersection of big data, cloud services, and applied data science, with a focus on enabling cybersecurity outcomes.

What You’ll Do:
Data Lake Engineering & Stewardship
- Architect, build, and manage data pipelines across AWS services (S3, Kinesis, Lambda), APIs, and Cribl
- Ensure data lake performance and usability through indexing, query optimization, and aggregation pipeline management for 3.5TB+ daily volumes.
- Act as the dedicated owner to maintain data quality and prevent “data swamp” degradation.
- Establish a Data Lake Center of Excellence (C4E) to empower decentralized analytics development while maintaining governance and standards.
Security Analytics & Data Science Enablement
- Collaborate with security teams to define, develop, and deploy data-driven use cases.
- Build ML pipelines for anomaly detection, vector search, semantic search, and other advanced security analytics.
- Integrate and optimize Elasticsearch, vector databases, and KNN/percolator search capabilities.
- Develop and maintain Jupyter Notebook workflows for data wrangling, modeling, and analysis.
Innovation & AI Integration
- Apply machine learning and natural language processing techniques, including retrieval-augmented generation (RAG).
- Partner with AI adoption initiatives to ensure secure and effective use of enterprise data for AI/ML.
- Stay current with emerging technologies and techniques in data science, big data, and cloud analytics.
Collaboration & Knowledge Retention
- Work closely with SOC Engineers, Threat Hunters, and data platform teams to integrate log sources and telemetry into the data lake.
- Provide guidance and mentorship across Cyber teams on building analytics use cases.
- Serve as the in-house expert to reduce reliance on external consulting spend, ensuring institutional knowledge is retained within NYL.

What You’ll Bring:
- 5+ years of hands-on experience in data engineering, big data platforms, or applied data science.
- Proficiency with AWS services (S3, Kinesis, Lambda, Bedrock) and enterprise-scale analytics design.
- Strong programming skills in Python and/or Scala; experience with Jupyter Notebooks.
- Experience with indexing, query optimization, and data pipeline optimization at scale (3.5TB+ daily).
- Advanced knowledge of Elasticsearch including advanced search methods (KNN vector search, percolator reverse search), and performance tuning, platform maintenance, etc.
- Experience applying ML techniques for anomaly detection, semantic search, embeddings, and NLP.
Bonus Qualifications
- Familiarity with Cribl for data routing and transformation.
- Advanced degree in mathematics, computer science, or related empirical sciences.
- Previous experience in cybersecurity environments or willingness to learn security context quickly.
Education & Certifications
- Bachelor’s degree in Computer Science, Data Science, Cybersecurity, or related field; advanced degree preferred.
- Certifications in AWS Data Engineering, Machine Learning, or Big Data technologies a plus.

​

Pay Transparency

Salary Range: $121,000-$172,500 

Overtime eligible: Exempt 

Discretionary bonus eligible: Yes 

Sales