Software Development/Data Engineer

This Opportunity

WSP is currently initiating a search for a Software Development/Data Engineer to join our Bridge Inspection Group based in either NYC or Valhalla, New York. Be involved in projects with our Bridge Inspection Team and be a part of a growing organization that meets our client’s objectives and solves their challenges.

Your Impact

This role is responsible for the design, deployment, development, and maintenance of data solutions.

• Participate in a variety of data-related projects and work closely with Business users, and other stakeholders to gather requirements and build data pipelines that meet the organizations’ needs.
• Deploy and support Data platforms that process and store data from structural engineering, bridge design and bridge inspection in required format.
• Contribute to software development methods, tools, and techniques and apply agreed standards and tools to achieve well-engineered outcomes.
• Design, develop, and implement data solutions to meet business requirements and data ingestion needs, facilitating accurate and timely data availability for analysis and decision-making.
• Extract, load, and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files.
• Develop APIs as needed to transform data in required format from various sources of bridge design and inspection.
• Write well-designed, efficient code that adheres to security standards.
• Monitor and troubleshoot data pipelines to identify and resolve issues promptly to minimize disruptions in data processing for on-premises and cloud environments.
• Implement data quality checks and validation processes to ensure accuracy and completeness of data.
• Write complex queries and scripts to efficiently manipulate, transform, and process raw data from bridge inspection.
• Create and execute data validation processes to ensure the reliability and consistency of incoming data. 
• Build processes to monitor data quality.
• Continuously optimize data pipelines for performance, scalability, and reliability.
• Create and maintain technical documentation.
• Contribute to building and maintaining data catalog and lineage.
• Design and develop CI/CD processes that ensure high availability and agility.
• Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability.
• Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform.
• Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities.
• Monitor and report on supplier performance, customer satisfaction, adherence to security requirements and market intelligence.
• Remain current in latest discipline-specific techniques and practices.
• Collaborate with professionals from a variety of disciplines to provide future ready solutions for clients.
• Exercise responsible and ethical decision-making regarding company funds, resources and conduct, and adhere to WSP’s Code of Conduct and related policies and procedures.
• Perform additional responsibilities as required by business needs.
   

Who You Are

Required Qualifications

• Bachelor’s Degree or equivalent in IT, Computer Science, Engineering, or related fields.
• At least 3 years of relevant post education experience.
• Strong knowledge of Big Data architectures, large data warehouses, and Data Lake solutions.
• Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture and medallion architecture.
• Proficient in cloud services including Azure Databricks.
• Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview, and Power BI.
• Experience with major database platforms including Oracle, SQL Server as well as Cloud databases and NOSQL Databases.
• Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling.
• Experience with data integration techniques for both structured and unstructured data.
• Solid programming skills in languages such as Python, Pyspark, and SQL.
• Experience with Airflow.
• Experience in DevOps, Git Repos, and CI/CD pipelines for code deployment.
• Experience deploying and administering cloud-based data solutions using infrastructure-as-code and infra-automation tools like Terraform, Ansible, etc.
• Strong knowledge of Microsoft Azure Cloud.
• Functional knowledge of Microsoft Power BI.
• Experie