Senior Data Engineer

<div class="content-intro"><p>SpyCloud is on a mission to make the internet a safer place by disrupting the criminal underground. SpyCloud’s solutions thwart cyberattacks and protect more than 4 billion accounts worldwide. Cybersecurity is an exciting, evolving space, and being at the forefront of the fight to disrupt cybercrime makes SpyCloud a special place to work. If you’re driven to align your career with a fantastic mission, look no further!</p></div><p>As a Senior Data Engineer at SpyCloud, you will lead the architectural transformation of our data systems, working closely with Product Management, SpyCloud Labs, and Engineering teams. Your focus will be on modernizing our data platform utilizing Databricks and serving customers via purpose-built data stores (RDS, Elasticsearch, DynamoDB…etc), This is a hands-on, strategic leadership role shaping the future of data at SpyCloud.</p> <p><strong>What You'll Do:</strong></p> <ul> <li>Redesign and modernize data architecture for scalability, performance, and AI-readiness</li> <li>Use databricks/spark to prepare, transform and serve data to customers</li> <li>Build and optimize real-time and batch data pipelines using AWS tools (e.g, Lambda, S3, DynamoDB, ECS, EKS, Terraform)</li> <li>Design systems to prepare and serve high-quality data for ML models and LLM applications</li> <li>Optimize cost and performance of daily data ingestion, transformation, and storage workflows</li> <li>Work with Product and Engineering teams to ensure data solutions meet product requirements</li> <li>Evaluate and implement modern tooling for data processing, ML data workflows, and observability</li> <li>Guide and mentor data engineers, championing best practices and technical excellence</li> <li>Ensure operational data quality, security, and reliability across systems</li> <li>Lead long-term data architecture planning aligned with company and AI-driven goals</li> </ul> <p><strong>Requirements:</strong></p> <ul> <li>8+ years of experience in data engineering or closely related disciplines</li> <li>2+ years in a technical leadership or senior engineering role</li> <li>Experience building and operating Lakehouse, Databricks, or large-scale distributed data lake architectures in AWS</li> <li>Knowledge of distributed systems as it pertains to data storage and computing</li> <li>Experience building and managing large Elasticsearch/Opensearch data stores</li> <li>Hands on experience building ETL scripts</li> <li>Hands-on experience preparing and managing data for machine learning and LLM workflows</li> <li>Strong development skills in Python, especially in building data-intensive systems</li> <li