Senior Data Engineer

This is a remote position.

About the Job As a Senior Data Engineer at Aspire,  will play a key role in designing, developing, and maintaining data workflows and pipelines, ensuring the efficient and reliable movement of data across various systems.

• Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.
• Utilize Operators, macros, connections, variables, and xcom for efficient task execution.
• Conduct unit testing to ensure the robustness of Airflow workflows.
• Perform file operations and data movement between HDFS and S3 (and vice versa).
• Manage Hive external tables, update table DDL, and handle partition operations.
• Execute basic Hadoop commands and navigate/traverse HDFS.
• Work with Hive on Tez for optimized query execution.
• Utilize AWS command line tools for tasks such as listing and copying files from S3.
• Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.
• Develop and optimize data processing tasks using PySpark.
• Understand and leverage Livy for efficient Spark job execution.
• Import and export data between Microsoft SQL Server and HDFS using Sqoop.
• Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.
• Implement effective branching strategies in Git.
• Participate in the process of raising and reviewing Merge Requests in GitLab.
• Utilize SSIS for migrating data workflows, ensuring a smooth transition.
• Model data in Snowflake, differentiating between external and internal tables.
• Understand the use of views and materialized views in Snowflake.

• Bachelor’s degree in computer science, information technology, or a related field.
• Proficient in Python (both Python 2.7 and 3.6+).
• Experience in unit testing and test-driven development.
• Familiarity with Secrets Management tools and techniques.
• Knowledge of data modeling principles and practices.
• Strong understanding of Git version control and GitLab workflows.
• Experience with SSIS and Snowflake is a plus.
• Excellent problem-solving and communication skills.
• Awareness or knowledge of IT security best practices as defined by ISO/SOC or similar.

• Be part of