Senior Software Engineer – Data Pipelines and Services

DTN is a global leader providing insights and analytics to our customers to feed, fuel, and protect the world.  We help people make critical business decisions that impact the agriculture, oil and gas, trading, and weather industries.

It’s a difference you’ve likely felt without even knowing it. From the food on your plate to the gas in your car to the last flight you took, chances are a DTN customer made it possible. And that’s just for starters. That’s why we say, “When our customers prosper, we all win.”

Job Description:

DTN is looking for an experienced Senior Software Engineer to join a close-knit, high-energy development team within the Eco Data organization. If you're passionate about building custom data processing pipelines at scale, working with complex geospatial datasets, and solving data transformation challenges for time-sensitive weather content, we want to hear from you! In this role, you will help design, build, and optimize data models and services that convert raw weather data into fit-for-purpose formats, supporting time-series applications. If you have a deep understanding of distributed systems, cloud-native architectures, and data engineering, this role could be a great fit.
 

Responsibilities:

• Join an international team, actively participate in daily stand-ups and scrum ceremonies, and engage with peers on Slack.

• Gain a thorough understanding of the team’s backend data services and API products.

• Use open-source software and cloud infrastructure to plan, create, test, and optimize custom data transformation pipelines that convert weather model outputs into high-performance formats.

• Collaborate with stakeholders to gather product and technical requirements, providing clarity when needed.

• End-to-End Ownership: Take charge of designing, building, and optimizing data pipelines that span from raw data ingestion to delivery, ensuring high performance, scalability, and maintainability.

• Lead asynchronous code reviews to give and receive constructive feedback.

• Create comprehensive documentation to share knowledge and support operational readiness.

• Assist with incident response and provide technical guidance during production issues.
   

Required Experience:

• Deep expertise in Python and data transformation tools, including Apache Arrow, Pandas, NumPy, and Polars.

• Experience with scientific data formats like Zarr or Parquet

• Hands-on experience building distributed, cloud-native data pipelines using AWS services (e.g., EC2, S3, SQS, SNS, EKS).

• Proficient in designing, working with, and optimizing relational databases and queries (e.g., PostgreSQL, Alembic).

• Proven experience working with event-driven architectures and scaling cloud-based systems.

• Strong expertise in building custom ETL pipelines and transforming large, time-sensitive data.

• Proficient in using data partitioning and batch processing strategies, along with efficient data storage formats, to handle complex transformations at scale.

• Experience with container-based applications, particularly with Kubernetes.

• Proficient in Infrastructure-as-Code and CI/CD practices (e.g., Terraform and GitLab).

• Experience with parallel processing, utilizing multiple cores of a machine to accelerate data processing, and asynchronous coding/concurrency techniques to maximize efficiency and throughput.

• Ability to communicate complex technical concepts in simple terms, making them understandable for a wider team.
   

Nice-to-Have Experience:

• Experience with geospatial data and time-series data models, particularly for applications involving weather data and high-volume datasets.

• Familiarity with industry-specific scientific data formats like GRIB and NetCDF.

• Experience designing, building, and supporting REST APIs using frameworks like FastAPI or Typescript/Express.

• Additional expertise in a language built for concurrency, such as Node.js.

• Experience with performance optimization and observability tools like Datadog in distributed systems.

• Experience working in the Palantir platform

This position is remote-eligible in most U.S. states.  We offer a comprehensive benefits package including flexible working hours, unlimited paid time-off, insurance (medical, dental, vision, and life), company-matching 401(k), and more. 
 

The targeted hiring base pay range for this position is between $101,250 and $148,500.