Position Summary

The Data Engineer builds pipelines that are used to transport date from a data source to a data warehouse. These pipelines are crucial: they are what enable an organization to access and analyze its data and use the insights to make business decisions. Data pipelines transport and transform data according to established business rules or a line of exploratory analysis the business wants to undertake. The Data Engineer prepares and organizes the data that organizations have built in their databases and other formats.

A Glimpse into the Daily Routine of a Machine Learning Engineer

A day in the life of a Data Engineer states with building and delivering high quality data architectures and pipelines that support clients, business analysts, and data scientists. A Data Engineer also interfaces with other technology teams to extract, transform, and load [ETL] data from a wide variety of data sources. Effectual Data Engineers continually improve ongoing reporting and processes, as well as automate or simplify self-service for our clients. Effectual Data Engineers develop, code, and deploy scripts written in the Python programming language, as Python is the language of Data. All Data Engineers are first and foremost Software Engineers with an understanding of the SDLC process.

Essential Duties and Responsibilities

• Develop, construct, test and maintain data architectures from the data architect
• Analyze organic and raw data
• Build data systems and pipelines
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies
• Develop code and scripts for data architects, data scientists, and data quality engineers
• Data acquisition
• Identify ways to improve data reliability, efficiency, and quality
• Develop data-set processes
• Prepare data for prescriptive and predictive modeling
• Automate the data collection and analysis processes, data releasing and reporting tools
• Build algorithms and prototypes
• Develop analytical tools and programs
• Collaborate with data scientists and architects on projects/efforts

Qualifications

• Bachelor's or master's degree in computer science, Engineering or a related field
• AWS Certified Big Data - Specialty
  • Must have or be willing to obtain within two weeks of hire
• 5+ years with proven experience working as a Data Engineer, preferably in a professional services or consulting environment
• Strong proficiency in programming languages such as Python, Java, or Scala, with expertise in data processing frameworks and libraries (e.g., Spark, Hadoop, SQL, etc.)
• In-depth knowledge of database systems (relational and NoSQL), data modeling, and data warehousing concepts
• Experience with cloud-based data platforms and services (e.g., AWS, Azure, Google Cloud) including familiarity with relevant tools and technologies (e.g., S3, Redshift, BigQuery, etc.)
• Proficiency in designing and implementing ETL processes and data integration workflows using tools like Apache Airflow, Informatica, or Talend
• Familiarity with data governance practices, data quality frameworks, and data security principles
• Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions
• Excellent communication and collaborations skills, with the ability to effectively work with clients and cross-functional teams
• Self-motivated and proactive, with a passion for learning and staying updated with the latest trends and advancements in the field of data engineering
• Able to work with ambiguity and turn client wants and needs into working stories, epics which can be executed upon during a sprint. This means Data Engineers understand and know the ‘agile’ progress software delivery
• A firm understanding of the SDLC process
• An understanding of object-oriented programming
• Needs minimal direction
• AWS background
• Solution Engineer mindset

Must Have Skills

• AWS Glue
• AWS Lake Formation
• AWS Step Functions
• Amazon Redshift
• Amazon S3

Nice-to-Have Skills and Experience

• A curious nature and inquisitive attitude when approaching problems
• Have the attitude of ‘good is not good enough’ for our clients
• Snowflake or Databricks certifications and/or hands-on-keyboard experience

Company Offered Benefits

Full-time employees are eligible to participate in our employee benefit programs:

• Medical, dental, and vision health insurances,
• Short term disability, long term disability and life insurances,
• 40