Senior Data Engineer

<p>Advance your career with Sunscrapers, a leading force in software development, now expanding its presence in a data-centric environment. Join us in our mission to help clients grow and innovate through a comprehensive tech stack and robust data-related projects. Enjoy a competitive compensation package that reflects your skills and expertise while working in a company that values ambition, technical excellence, trust-based partnerships, and actively supports contributions to R&amp;D initiatives.<br/></p><p><strong>The project:</strong></p><p></p><p>We are carrying out the project for our client, an American private equity and investment management fund - listed on the Forbes 500 list - based in New York.</p><p>We support them in the area of the infrastructure and data platform, and <strong>very recently we also build and experiment with Gen AI applications.</strong> The client operates very widely in the world of finance, loans, investments and real estate.</p><p>As a Senior Data Engineer you’ll design and implement core systems that enable data science and data visualization at companies that implement data-driven decision processes to create a competitive advantage. </p><p>You’ll build data platform for data and business teams, including internal tooling, data pipeline orchestrator, data warehouses and more, using:</p><p></p><p><strong>Technologies</strong>: Python, Terraform, SQL, Pandas, Shell scripts</p><p><strong>Tools</strong>: git, Docker, Snowflake, Pinecone, Neo4j, Jenkins, Jupyter Notebook, OpenAI API, Apache Airflow / Astronomer, Kubernetes, Artifactory, Windows with WSL, Linux, Gitlab</p><p><strong>AWS</strong>: EC2, ELB, IAM, RDS, Route53, S3, and more</p><p><strong>Best Practice</strong>s: Continuous Integration, Code Reviews</p><p></p><p>The ideal candidate will be well organized, eager to constantly improve and learn, driven and, most of all - a team player!</p><p></p><p><strong>Your responsibilities will include:</strong></p><ul> <li>Developing PoCs using latest technologies, experimenting with third party integrations</li> <li>Delivering production grade applications once PoCs are validated</li> <li>Creating solutions that enable data scientists and business analysts to be self-sufficient as much as possible.</li> <li>Finding new ways how to leverage Gen AI applications and underlying vector and graph data storages</li> <li>Designing datasets and schemes for consistency and easy access</li> <li>Contributing data technology stacks including data warehouses and ETL pipelines</li> <li>Building data flows for fetching, aggregation and data modeling using batch and streaming pipelines</li> <li>Documenting design decisions before implementation</li> </ul><p><strong>Requirements</strong></p><p>What's important for us?</p><ul> <li>At least 5+ years of professional experience in data-related role</li> <li>Undergraduate or graduate degree in Computer Science, Engineering, Mathematics, or similar</li> <li>Expertise in Python and SQL languages</li> <li>Experience with data warehouses (Snowflake)</li> <li>Experience with different types of database technologies (RDBMS, vector, graphs, document based, etc.)</li> <li>Expertise in AWS stack and services</li> <li>Proficiency in using Docker</li> <li>Experience with infrastructure-as-code tools, like Terraform</li> <li>Great analytical skills and attention to detail - asking questions and proactively searching for answers</li> <li>Excellent command in spoken and written English, at least C1</li> <li>Creative problem-solving skills</li> <li>Excellent technical documentation and writing skills</li> <li>Ability to work with both Windows and Unix-like operating systems as the primary work environments</li> </ul><p></p><p><strong>You will score extra points for:</strong></p><ul> <li>Experience with integrating LLMs (OpenAI but also others, maybe open source)</li> <li>Understanding of LLMs fine tuning, embedding and vector semantic searching</li> <li>Experience with Pinecone or Neo4j</li> </ul><ul> <li>Familiarity with data visualization in Python using either Matplotlib, Seaborn or Bokeh</li> <li>Proficiency in statistics and machine learning, as well as Python libraries like Pandas, NumPy, matplotlib, seaborn, scikit-learn, etc</li> <li>Experience in building ETL processes and data pipelines with platforms like Airflow or Luigi</li> <li>Knowledge of any Python web framework, like Django or Flask with SQLAlchemy</li> <li>Experience in operating within a secure networking environment, like a corporate proxy</li> <li>Experience in working with repository manager, for example Jfrog Artifactory</li> </ul><p><strong>Benefits</strong></p><p><strong>What do we offer?</strong></p><ul> <li>Working alongside a talented team of software engineers who are changing the image of Poland abroad</li> <li>Culture of teamwork, professional development and knowledge sharing (<a href="https://www.youtube.com/user/sunscraperscom" rel="nofollow noreferrer noopener">https://www.you