Senior Software Engineer, Big Data Systems

Microsoft Advertising is an online advertising platform, where advertisers bid to display brief ads, service offers, product listings and videos to web users. Our data-enabled technology platform, encompassing Invest, Monetize, and Curate, optimize return on investment for both buyers and sellers, while maintaining a commitment to an open marketplace and empowering the open web globally.   

As a Senior Software Engineer, Big Data Systems, you will operate and support our diverse Big Data Platform. This platform consists of large Hadoop, HBase, and Kafka clusters in an all Linux environment. The platform currently ingests 300TB of new data and runs 20,000 ETL jobs every day across 14 Hadoop, HBase, and 6 Kafka Clusters. 

This growing team consists of curious, passionate, talented technologists who enjoy working on complex, large scale data stores. Our team members thrive in a learning and teaching environment. Each team member is encouraged to explore solutions and efficiencies to support, optimize, and maintain our systems. We are enthusiastic about automation and optimization.   

Your primary responsibilities will involve supporting and enhancing data stores for extremely intricate, high-availability, low-latency, business-critical real-time systems. This will entail utilizing contemporary open-source tools and architectural approaches, working closely with other highly skilled engineering teams, honing your problem-solving abilities in complex scenarios, and becoming proficient in managing large-scale data stores.  

We anticipate exponential growth in data and transactions on our platform and so we are seeking an engineer who is excited about this opportunity and would love to:  

•     Work on large-scale, distributed systems  
•     Learn new technologies, tools and applications 

Your primary responsibilities will involve coding, support, upgrades, troubleshooting, performance tuning, maintenance and automation. This will entail utilizing contemporary open-source tools and architectural approaches, working closely with other highly skilled engineering teams, honing your problem-solving abilities in complex scenarios and becoming proficient in managing large-scale data stores. 

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Responsibilities

• Support a complex Data Pipeline Platform by monitoring, maintaining, provisioning, and upgrading Hadoop, HBase, Kafka, Trino, Cassandra/ScyllaDB, IRONdb, Druid, QFS, Metrics, and ETL systems using proprietary automation tools.   
• Develop new tools to automate routine day-to-day tasks, such as security patching, software upgrades, and hardware allocation. Utilize automated system monitoring tools to verify the integrity and availability of all hardware, server resources, and critical processes.
• Code scripts to automate and optimize operational processes. 
• Troubleshoot and analyze hardware or software failures and provide solutions for recovery. Identify and resolve faults, inconsistencies, and systemic issues.   
• Collaborate with engineering team partners to resolve complex system performance issues.   
• Prepare documentation for standard operating procedures. 
• Effective with time management and ability to manage priority tasks and requests.  
• Provide production support and participate in on call rotation.  
• Embody our Culture an