Senior Data Engineer (01267)

At Nielsen, we believe that career growth is a partnership. You ultimately own, fuel and set the journey. By joining our team of nearly 14,000 associates, you will become part of a community that will help you to succeed. We champion you because when you succeed, we do too. Embark on a new initiative, explore a fresh approach, and take license to think big, so we can all continuously improve. We enable your best to power our future. 
Nielsen Global Media collects billions of data points on what consumers watch. We study consumers in more than 100 countries, and help our clients globally identify unmet needs, develop new products and deliver them in an engaging way.  We are constantly evolving, not only in terms of where we measure, or who we measure, but in how our insights can help drive profitable growth.  DESCRIPTION Conduct one-on-one meetings with the manager and team meetings to gather the information needed for developing new software or code. Analyze methodology documents to understand client requirements. Design and develop software and related database infrastructure that will feed into the media data lake (ecosystem) through Nielsen’s data flow processes. Conduct code review with senior developers before pushing the code into production. Support analytics performed by the Data Science team by creating or modifying and then deploying and supporting software applications and data infrastructure to respond to custom analytics requests and client inquiries, and incorporate changing analytical and statistical methodologies, standards and best practices. Combine software engineering expertise with knowledge of Nielsen TV measurement tools and services. Create job flow, automate the jobs and run jobs. Implement data pipeline Directed Acyclic Graphs (DAGs) and maintenance DAGs. Configure and setup DAGs based on the data to run Spark commands in parallel and sequential. Create hooks, as well as Python, Bash, Spark and custom operators. Schedule a pipeline to run the DAG to fetch data for loading into a database. Collaborate with cross-functional Nielsen Media teams to develop and implement customized software solutions that will validate enhancements to the universe estimate methodologies. Perform unit testing using test cases and fix any bugs. Write and run test scripts for Media and Annual Estimate data. Run test scripts for data generated from team members.  Collaborate with Data Scientists by leveraging software engineering expertise to identify and address software development and data quality issues, and detect and resolve quality escapes. Perform detailed analysis of data quality issues, report to other teams on significant data changes, and rectify the problem. Respond to internal data analytics inquiries by determining how to execute the analyses with the platforms and applications available, modifying specifications as needed, providing an estimated delivery date, and accurately delivering data-driven analyses on time. Tools used: Python, SQL, Apache Spark, and Presto, AWS (S3 and EMR); Git, ORC, Parquet, AWS Lambda, Java, Databricks, Athena, Terraform, Bash, Rest API, Amazon ECR, Airflow, GitLab CICD, Gitlab Package Registry. QUALIFICATIONS Bachelor’s degree in computer science, engineering, or a related field with a technology focus (foreign equivalent degree acceptable) plus 3 years of experience in software development. This must include: Two (2) years of experience in: data engineering; (big) data workloads; including handling data quality issues and big data scaling issues; Python, SQL, Apache Spark, and Presto; AWS (S3 and EMR); Git for source code version-control; data lakes and data warehousing; column-oriented data storage formats (ORC, Parquet); and Airflow. Must also include One (1) year of experience in: AWS System Manager Parameter Store, AWS Step Functions, AWS Lambda using Java and Python; Databrick; Athena; Terraform; Bash; Rest api development; Amazon ECR and Amazon EventBridge; Airflow DAG development; AWS Code pipeline; AWS CloudFormation, AWS CodeBuild, AWS Docker Services, AWS SNS, and AWS Glue; IAM Roles Configuration; and GitLab CICD, and Gitlab Package Registry. Salary $109,283 - $155,000/year. M-F, 40 hours/week. About Nielsen: Enabling your best to power a better media future. Our comprehensive benefits package (including health & wellness plans for full-time employees, 401(k) retirement coupled with a Nielsen match, a generous paid time off policy, and if eligible, a discretionary incentive/bonus) is designed to be inclusive for all employees and families, and we take pride in ensuring that employees are rewarded holistically for the role they are doing and their performance. Nielsen is committed to hiring and retaining a diverse workforce. We are proud to be an Equal Opportunity/Affirmative Action-Employer, making decisions without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, geneti