Data Engineer (Senior-Lead)

Company Description

Are you ready to shape the future technological landscape in Europe?

We are dedicated to responsible digitalisation, building innovative, inclusive solutions that drive meaningful impact. With over 8,900 professionals across Europe, we help EU institutions, public and private organisations thrive in a rapidly evolving digital world. Are you ready to shape the future through technology? Your journey starts here. Take a look at some of our impactful projects here: https://netcompany.com/cases/

Job Description

Ever wondered what it feels like to be a Data Engineer (Senior-Lead) in Netcompany ?

Joining us as a Data Engineer (Senior-Lead), you will have strong experience working with relational databases, SQL, big data tools such as PySpark and Spark SQL, and orchestration tools like Apache Airflow. This role also requires experience in using Notebooks (e.g., Jupyter, Databricks) for development and collaboration. Depending on your experience level, you may contribute to or lead data engineering projects, mentor team members, and drive key architectural decisions.  

As a Data Engineer (Senior-Lead) you will:

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes to support business requirements. 
• Write and maintain complex SQL queries to extract, transform, and load data into various systems. 
• Utilize PySpark and Spark SQL in Notebooks (e.g., Jupyter, Databricks) to process and analyze large datasets efficiently in distributed computing environments. 
• Use of orchestration tools such as Apache Ariflow to schedule, monitor, and optimize data workflows and pipeline execution. 
• Collaborate with business analysts, data architects, and other stakeholders to ensure seamless data integration across systems. 
• Ensure the quality, accuracy, and consistency of data by implementing data validation and monitoring mechanisms. 
• Troubleshoot and resolve issues related to data processing performance, data integrity, and system scalability. 
• Contribute to the development of data models, architectures, and workflows to support analytics and reporting needs. 
• Lead data architecture and design decisions, mentor junior team members, and drive best practices within the data engineering team.  Take ownership of large-scale data engineering projects and provide strategic leadership in data initiatives.

Qualifications

What would make you a fit for the role: 

• 5+ years of experience in data engineering, including leadership and strategic decision-making responsibilities.
• Experience working in cloud platforms (e.g. MS Azure, AWS, GCP etc.).  
• Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field. 
• Excellent knowledge of relational databases, such as Oracle, IBM DB2, SQL Server, PostgreSQL etc., and proficiency in SQL. 
• Strong experience with PySpark and Spark SQL for processing and querying large datasets in distributed computing environments. 
• Proficiency in using Notebooks (e.g., Jupyter, Databricks) for developing and testing data engineering workflows. 
• Experience with Apache Airflow or other orchestration tools (e.g., Prefect, Luigi) for managing and scheduling data workflows. 
• Experience with designing, developing, and optimizing ETL/ELT processes. 
• Knowledge of data modeling principles and best practices. 
• Familiarity with cloud platforms and tools (e.g., MS Azure, AWS, GCP) for data storage and processing is a plus but required for Lead-Level candidates. 
• Excellent problem-solving skills and attention to detail. 
• Strong communication skills and the ability to work collaboratively in a team environment. 

It would also be a plus if you match some of the following:

• Experience with NoSQL databases (e.g., MongoDB, Cassandra) and processing of unstructured/semi-structured data.
• Experience with scripting languages (e.g., Python, Bash) for data manipulation and automation. 
• Knowledge of data governance for the lead-level candidates. 

Additional Information

Being a part of the Netcompany team, you will be provided with: 

• The opportunity to work in a modern environment & in a hybrid working model
• A seamless onboarding experience and a buddy to support you on your first steps
• A competitive compensation & benefits package
• Health and life insurance program
• Meal and commuting allowance
• Well-being activities (on premises)
• Continuous learning opportunities using the most modern methods (unlimited access to Udemy for Business, ad-hoc trainings)
• A personalized development plan for targeted career growth