Data Engineer

 

Job Summary

 

• Augment and maintain the existing repositories and data structures within AWS (used to process and store large amounts of data from unrelated sources)
• Experience with several formats and means for data ingestion.  Including, data types (structured, semi-structured, and unstructured), and sources (on premise, and in the cloud) using the most appropriate techniques in each case
• Continue to expand and enhance the model, utilizing best practices, in regards to the organization of data and the various relationships
• Optimize existing and future models for fast and scalable queries (while maintaining performance and related price thresholds)
• Work with the team to define, construct, and maintain self-service dashboards for the Business and Advanced Analytics teams within PowerBI
• Implement scalable and flexible, high performance data pipelines for AWS to support analytics
• Develop and maintain data maps and their relationships
• Generate associated technical documentation including follow-up reports
• Work with Data Governance to implement quality rules and data governance measures (data dictionary, metadata, traceability, ...)
• Propose improvements and actions based on provided results
• Communicate results effectively with required teams

 

 

Knowledge, Skills and Abilities:

 

• Bachelor's Degree with 6+ years of experience.
• Advanced knowledge and experience using Python, Airflow, Spark, AWS, and Snowflake
• Database architectures:  SQL, NoSQL, graph databases
• CI/CD and Orchestration:  Jira, Jenkins, Bit Bucket, Terraform, and Airflow
• Past experience with data modeling tools, ETL tools (e.g. Informatica Power Center)
• Computer languages, data query and transformation tools: AWS Athena, Jupyter notebooks, Spark, Pyspark, Python, and EMR Studio
• Algorithm analysis (for working with our Data Scientists)
• Understanding of multidimensional modeling for quantitative and fact related data storage
• OS:  Linux, and MS Windows
• Code IDE:  Microsoft Visual Code, Jupyter notebooks
• Artificial intelligence, machine learning, and deep learning are a plus