Data Engineer

In a world where trades are happening faster than ever to answer our needs, where sustainability is not just an option anymore, clarity and trust in the information we trade with are a must. <br/>So, say hello to Kpler! We exist to facilitate sustainable and efficient trade to meet the changing needs of our world. To do so we've built a Data-as-a-Service solution that does exactly that across the lifecycle of a trade.<br/>Our solution aggregates data from hundreds of sources including radar and satellite imagery as well as logistics, governmental, shipping databases and more. Intelligently connecting the dots across fragmented information landscapes, we bring to our clients a unique, real-time understanding of the trades happening all over the world, by giving them access to live information about the movement of cargos, the availability of vessels as well as commodity storage.<br/>To support this endeavor, we have teams in more than 9 countries and 11 key locations (Brussels, Paris, London, Athens, Vienna, Rostock, Hyères, Dubai, Singapore, Houston and New York). With individuals of various backgrounds, diverse skills, and international experiences, being global &amp; inclusive is in our DNA!<br/><br/>Data Engineers play a crucial role in creating the architecture and systems needed to collect, store, transform, and access data. They design, build and maintain large-scale processing systems, data pipelines and other infrastructure.<br/>Involvement is expected in architecture, product line and relevant data science projects, and in the development and deployment of applications serving both the Kpler and MarineTraffic platforms and the organization’s internal clients. In this role, you will be part of our new Compliance product line, contributing to our mission of delivering a revolutionary vessel compliance and risk management solution. We aim to provide maritime industry stakeholders with unparalleled safety, regulatory compliance, and operational efficiency by harnessing advanced technologies like machine learning, and predictive analytics. Your involvement will bring transparency to sanctioned entities and dark vessel activities, while our innovative approach redefines industry standards for a safer and more sustainable maritime ecosystem. <h3>Responsibilities</h3> <ul> <li>A Data Engineer is expected to be involved in all stages of the data lifecycle. The following is an indicative list:</li><li>Develop and deploy containerised applications within AWS, utilizing tools like Python, Docker, Kubernetes and Airflow</li><li>Design and deploy robust database structures and procedures using technologies such as PostgreSQL and Elastic, ensuring data integrity and optimal performance</li><li>Consume and write APIs to ingest and share data across various crews boosting collaboration and overall enablement</li><li>Work with Event Streaming. Use Kafka to facilitate seamless event handling and CI/CD practices to enhance deployment efficiency</li><li>Automate and produce internal tools that boost efficiency and productivity in data-related workflows</li><li>Utilize IaC tech like Terraform and CloudFormation to build and own AWS infrastructure through the principles of Infrastructure as Code</li><li>Implement logging, monitoring, and alerting mechanisms that guarantee the reliability and resilience of production-grade data pipelines</li><li>Collaborate and work with engineers, product and data science teams as part of the same crew</li><li>Contribute, execute and lead complex data pipelines design, brainstorming sessions, and peer reviews</li><li>Keep up-to-date with the latest tech in data lakes &amp; warehousing, processing frameworks and data analytics. Research and implement new tools and technologies to improve the data infrastructure.</li><br/></ul> <h3>Accountabilities</h3> <ul> <li>A Data Engineer is fully accountable for the quality of developed pipelines and code ensuring their continuous support &amp; maintenance thereafter. Additionally:</li><li>Active participation in team ceremonies and code reviews</li><li>Adherence to the data team's common best practices and development paradigms and contribution toward continuous improvement</li><li>Delivery of well-documented projects and effective communication of results</li><li>Strong sense of ownership and a growth mindset</li></ul> <h3>Requirements</h3> <ul> <li>Required</li><li>STEM background</li><li>2+ years of experience in a similar position</li><li>Strong Python skills (pandas, numpy, sqlalchemy, pyodbc, pep8 standards)</li><li>Good SQL &amp; database development skills (indexing, execution plans &amp; query efficiency and performance)</li><li>Expertise in versioning and Git best practices</li><li>Experience with logging, alerting &amp; monitoring best practices</li><li>Experience with consuming and writing REST APIs</li><li>Knowledge of Data Engineering concepts, tools, architectures, and practices (streams, distributed computing frameworks, shell scrip