Senior Data Engineer

<p></p> <p> </p> <p>About us</p> <p>At Auctane, we are united by a passion to help sellers — wherever they are, however they operate — fulfil the promises they make to consumers. The Auctane mission is to fuel commerce through exceptional delivery. We make it possible for businesses to meet the ever-rising expectations of their customers, and we make the world smaller and more accessible to consumers everywhere. Auctane brands enable hundreds of thousands of merchants to annually deliver billions of products — over $200 billion worth — to customers around the globe. And Auctane is just getting started.</p> <p>Auctane is a team of shipping and software experts with a passion for helping merchants move their ideas, dreams and innovations around the globe. The Auctane family includes ShipStation, ShipWorks, ShipEngine, ShippingEasy, Stamps, Endicia, Metapack, Shipsi, GlobalPost, and Packlink. Our partners include Amazon, UPS, USPS, eBay, BigCommerce, Shopify, WooCommerce, and Walmart.</p> <p>Our values</p> <p></p> <p>About the team</p> <p>The Auctane Engineering Tech and Data team does one thing first - provide our customers with world class production software. This team ensures that our customers experience a fast and reliable system and provides our engineering teams with the tools to deploy, monitor, and maintain our production systems.</p> <p>About the role</p> <p><strong>Data is key to Auctane’s strategy.</strong> We work at scale, pace and with the latest architecture patterns and tech. <strong>We process thousands of events per second and our massive dataset keeps growing at a staggering pace</strong>. We keep improving our data platform and data engineering stack to accommodate growth, enable novel solutions and provide the best service to our customers.</p> <p>We have a flat and open engineering culture where data &amp; evidence beats opinion and hierarchy, backed by honest and frank discussions. <strong>We passionately believe in forming autonomous, cross functional teams who are empowered to deliver our ambitious strategy</strong>. Energy and passion for our business and customers is a part of the Auctane culture – and we love working with like-minded people.</p> <p>What you'll be doing?</p> <ul> <li>Contribute to the design, build and operational management of our data lake and analytics solution on top of proven AWS data technologies like S3, Athena, Lambda, Kinesis and ECS </li> <li>Use state of the art technologies like Airflow, dbt and Spark to process data and get our dataset just right</li> <li>Develop frameworks and solutions that enable us to acquire, process, monitor and extract value from our massive dataset</li> <li>Support the Data Analysts and Data Scientists with automation, tooling, data pipelines and data engineering expertise, enabling them to deliver complex analytics products and machine learning solutions</li> <li>Deliver highly reliable software and data pipelines using Software Engineering best practices like automation, version control, continuous integration/continuous delivery, testing, security, etc.</li> <li>Define, implement and enforce automated data security and data governance best practices within the solutions designed</li> <li>Mentor more junior colleagues and receive mentorship by more senior colleagues</li> </ul> <p>What are we looking for?</p> <ul> <li>A Software Engineering background</li> <li>Experience developing and supporting robust, automated and reliable data pipelines in Python and SQL</li> <li>Experience with data processing frameworks like Pandas, Polars, dbt or Spark</li> <li>Experience with streaming data processing</li> <li>AWS, Azure or Google Cloud experience</li> <li>Continuous integration/delivery environment experience with a passion for automation</li> <li>Knowledge of a data orchestration solution like Airflow, Oozie, Luigi or Prefect</li> <li>Knowledge of both relational and non-relational database design and internals</li> <li>Knowledge of how to design distributed systems and the trade-offs involved</li> <li>Experience working with software engineering best practices for development, including source control systems, automated deployment pipelines like Jenkins and devops tools like Terraform</li> </ul> <p>What will make you stand out?</p> <ul> <li>Production experience working with very large datasets</li> <li>Knowledge and direct experience of using business intelligence and analytics tools (Tableau, Looker, Power BI, etc.)</li> <li>Practical understanding of PII related issues and other considerations regarding data security</li> <li>Experience with big data cloud technologies like EMR, Athena (Trino), Glue, Big Query, Dataproc, Dataflow.</li> <li>MLOps know-how</li> <li>A desire to constantly challenge the norm</li> <li>Fast-learner with a keen interest in evaluating and adopting new technologies: Apache Iceberg, Dagster, dbt, GreatExpectations, Prefect, Snowflake, etc.</li> <li>Willing to attend conferences, webinars and meet-ups and