Staff Software Engineer, Data Incrementalization

TRM is on a mission to build a safer financial system for billions of people. We deliver a blockchain intelligence data platform to financial institutions, crypto companies, and governments to fight cryptocurrency fraud and financial crime. We consider our business — and our profit — as a way to move towards our mission sustainably and at scale. 

The Data Platform team sits at the core of TRM's products, working with data scientists, engineers, and product managers to build scalable and reliable data infrastructure. As a Senior or Staff Engineer, you will focus on data incrementalization — the process of efficiently processing, updating, and synchronizing massive datasets as new information becomes available. This approach ensures that TRM's blockchain intelligence remains timely, accurate, and actionable at petabyte-scale. You will design and build internal tools that empower Data Scientists and Machine Learning Engineers to transform raw blockchain data into real-time blockchain intelligence, driving innovation and impact across TRM's product offerings.

The impact you’ll have here:

• Design and build our Cloud Data Warehouse with a focus on incremental updates to improve cost efficiency and scalability.
• Research innovative methods to incrementally optimize data processing, storage, and retrieval to support efficient data analytics and insights.
• Develop and maintain ETL pipelines that transform and incrementally process petabytes of structured and unstructured data to enable data-driven decision-making.
• Collaborate with cross-functional teams to design and implement new data models and tools focused on accelerating innovation through incremental updates.
• Continuously monitor and optimize the Data Platform's performance, focusing on enhancing cost efficiency, scalability, and reliability.

What we’re looking for:

• Bachelor's degree (or equivalent) in Computer Science or a related field.
• 5+ years of experience in building distributed system architecture, with a particular focus on incremental updates from inception to production.
• Strong programming skills in Python and SQL.
• Deep technical expertise in advanced data structures and algorithms for incremental updating of data stores (e.g., Graphs, Trees, Hash Maps).
• Comprehensive knowledge across all facets of data engineering, including:
  • Implementing and managing incremental updates in data stores like BigQuery, Snowflake, RedShift, Athena, Hive, and Postgres.
  • Orchestrating data pipelines and workflows focused on incremental processing using tools such as Airflow, DBT, Luigi, Azkaban, and Storm.
  • Developing and optimizing data processing technologies and streaming workflows for incremental updates (e.g., Spark, Kafka, Flink).
  • Deploying and monitoring scalable, incremental update systems in public cloud environments (e.g., Docker, Terraform, Kubernetes, Datadog).
  • Expertise in loading, querying, and transforming large datasets with a focus on efficiency and incremental growth.

About the Team:

• The Data Platform team is the funnel between all of TRM's data world and product world. We care about all layers of stack including petabyte of data stores, pipelines, and processes.
• We have quite a big scope as a the team with new and exciting projects every quarter. As a result, we collaborate across the board with most teams at TRM.
• We believe in async communication and are also not afraid to jump on a quick huddle if that helps to move things faster. We are both scrappy when the situation demands and also process-oriented when we need to achieve our OKRs.
• We are always looking for people who can elevate the quality our tech and our execution. If you enjoy a remote-first and async friendly environment to achieve efficacy and efficiency at petabyte scale, our team could be a great pick for you!
• Team members are based in the US across almost all timezones! Our on-call tends to be in EST/PST shift, whatever suits you the best.
• We do try to reserve some overlap in the day for meetings. Our north star - no IC spends more than 3-4 hours/week in meetings.

Learn about TRM Speed in this position:

• Build scalable engines to optimize routine scaling and maintenance tasks like create self-serve automation for creating new pgbouncer, scaling disks, scaling/updating of clusters, etc.
• Enable tasks to be faster next time and reducing dependency on a single person.
• Identify ways to compress timelines using 80/20 principle. For instance, what does it take to be operational in a new environment? Identify the must have