Data Engineer

Data Engineer

Key Responsibilities:

●       Data Architecture

○       Provide and partner with developers to create general data architecture in support of a website and mobile application.

○       Implement and enforce data security best practices across pipelines and infrastructure, including vulnerability assessment, dependency patching, and compliance with data protection standards.

○       Work with developers to ensure data quality assurance and implement data validation and cleansing processes to maintain data accuracy and consistency.

○       Work with developers to ensure proper data lineage to understand the flow and transformation of data.

○       Work with developers to implement data quality and integrity

■   Data Validation and Cleansing:  Implement robust data validation and cleansing processes to ensure data accuracy and consistency.

■   Data Profiling:  Use data profiling techniques to understand data characteristics and identify potential issues.

■   Data Metadata:  Maintain comprehensive data metadata to provide context and facilitate data discovery.

●       Data Engineering, Operations & Transformation:

○       Collaborate cross-functionally with the analytics team, architects, developers, product owners, and business stakeholders to understand data requirements and deliver scalable solutions that enable actionable insights.

○       Write clear documentation for data processes, code, and reports to ensure transparency and reproducibility.

○       Design and implement ETL (Extract, Transform, Load) processes to clean, organize, and prepare data for analysis.

○       Develop and optimize complex data transformation workflows using SQL,  Python, Estuary, dbt and other transformation tools.

○       Work with large datasets to ensure data accuracy, consistency, and integrity.

○       Level 3 support for on-call data issues related to data transformation failures or critical data alerts.

●       Data Development (PostgreSQL, Mongo, Dynamo, & MSSQL):

○       Develop and maintain MULTIPLE database platforms/products, including the design of tables, views, and indexes for optimized query performance.

○       Write and support AWS Lamb