Infrastructure Senior Engineer, Low Latency and E-Trading Infrastructures

About the Position

We are looking for an industry wide Senior Subject Matter Expert (SME) – Infrastructure who can provide leadership and direction for Change-The-Bank (CTB) electronic trading infrastructure initiatives covering all technologies (in this specific domain) for Compute, Accelerators, Performance Tuning and HPC. 

The role sits within Citi’s Global Markets: High Performance Architectures – Infrastructure (HPAi) group.  This team is responsible for all electronic trading infrastructure architecture across all markets globally for the firm.  It is composed of low latency Subject Matter Experts (North America, EMEA), Systems Architects (both regional and asset class aligned) and an aligned initiative management function (e.g. Technical Project Manager).   Technology is key to our business; this role will require you to drive innovation and technology adoption in a large multi-faceted and complex environment.  This ICG-Tech “Subject Matter Expert” resource will collaborate with ICG-Markets “the business”, CTI “Enterprise Infrastructure”, and CISO “Security” teams driving technical architecture and engineering of CTB and End of Vendor Support (EoVS) initiatives where we choose to innovate.

Reporting to the head of HPAi, the Compute, Accelerators, Performance Tuning, HPC SME will be responsible for ensuring best in breed Linux and surrounding technology architecture and engineering in alignment with business requirements.  This will be done in collaboration with CISO and CTI Compute resources aligned to HPAi who represent Enterprise and CISO priorities and standards.  Active collaboration with all parties is required to be successful.  The resource should be an industry thought leader, and able to not only steer the direction of Compute technologies for electronic trading but also roll their sleeves up and work as an individual contributor.  The resource will leverage the full strength of the supporting non-ICG technology organizations (dedicated and non-dedicated to HPAi).  The role is very broad requiring a resource that is not afraid to do new things and be challenged on a regular basis.

Low Latency Trading –Markets, domain specific technology is required in (solid knowledge for as many of the following as possible with focus on Compute, HPC File Systems, Interconnects, HW and related technologies: 

The Technologies:

• Compute (Linux, CoreOS, Tuning, kernel bypass technologies, core dump analysis)
• Network (R&S, cut-through, Multicast, SDN, BGP, L1-L3, PCAP analysis)
• HPC File Systems (Lustre, GPFS, Weka, DDN, Ceph)
• Interconnects (InfiniBand, Slingshot, PCIe, NVLink, UALink, UPI, RoCE)
• HW (Server, FPGA, GPU, Memory, Cache, over clocking, cooling, ASIC, Quantum, NUMA)
• DevOps (OpenShift, Kubernetes, DevOps, Prog[Python, Perl, Shell], CI/CD)
• Programming (C, C++, Java, Rust, VHDL, Verilog)
• Time Sync (PTP, NTP, GPS, White Rabbit, SyncE)
• Wave (RF, Millimeter, Microwave, DWDM, Photonics)
• Storage (I/O, NAS, SAN, Object Store, SDS)
• Database (relational, columnar, graph, NoSQL, architecture, tuning)
• Monitoring (SPANs, TAPS, OEO, Packet Brokers, Corvil, SNMP, FIX, Logs, ELK, Open Telemetry, Grafana, Promethius)
• Messaging (TibRV, Solace, Aeron, ZeroMQ, Kafka, Multicast, Market data (TCP/UDP), Tibco, Exegy/Vela)
• Load Balancers (HW, SW)
• Security (Firewalls, PKI, TLS, IDS, Optical Encryption, AD, LDAP)
• Connectivity (WAN, Lucera, cross-connects, ECX, venue, client)
• Colo / Datacenter (Power, Space, Cage, Rack, Cabinet, Design)

The Skills:

• Deep knowledge of operating system fundamentals – hyper focus on Linux (REL, CoreOS) 
• Understanding of OSI model and TCP and UDP network protocols
• Experience with Python and data libraries
• Fluency with the Unix command line and shell scripting
• Practical experience with modern Linux systems and systems programming concepts like C, IPC mechanisms, virtual memory, and process life cycle
• Strong troubleshooting skills and knowledge of profiling/debugging tools such as gdb, perf, DTrace, SystemTap
• Can understand, interpret, and improve application performance via system tuning
• Thorough understanding of modern computer architecture - modern x86 AMD and Intel cache hierarchies as an example.
• Kernel-bypass technology knowledge (such as Solarflare OpenOnload / TCPDirect / ef_vi, RoCE, DPDK)
• Prior experience in a similar Site Reliability Engineering (SRE), DevOps, distributed computing, systems engineering/administration
• Ability to context switch to work on several projects at once
• Knowledge of JIRA, SCRUM, Sprints

Details: 

• Ingress/pl