Senior AI Engineer

Job Category

We are looking for a Senior AI Developer & Cloud Architect to design, build, and own the AI-powered compliance scraping engine and cloud infrastructure layer for an internal platform monitoring up to 70,000 credit card offer pages per month. This is a hands-on, sole-builder contractor role that sits at the intersection of cloud architecture, AI engineering, and large-scale web scraping, with a clear mandate: deliver a production-grade system that detects compliance violations across issuer offer pages with high accuracy and controlled token costs.

You will do this by architecting the AWS environment from the ground up, building a containerized worker fleet that integrates Playwright rendering with Claude-powered contextual analysis, and defining clean API contracts with the internal team that owns the Laravel control panel.

This is not a managed-PaaS or prototype role. You will be accountable for end-to-end delivery — architecture, build, documentation, and knowledge transfer — owning the full scraping and AI pipeline from URL intake through compliance findings, screenshot evidence, and results delivery back to the portal.

Responsibilities

• Design and configure the production AWS environment (ECS/Fargate, SQS, API Gateway, RDS PostgreSQL, S3, IAM, CloudWatch) using infrastructure as code (Terraform or CDK).
• Build a stateless, containerized worker fleet that integrates Playwright-based page rendering, structured rule evaluation, and Claude API analysis.
• Implement token optimization strategies across the LLM pipeline — prompt engineering, context pruning, caching, model selection, and batching — with measurable cost outcomes.
• Define and document API contracts, job payload schemas, and database write patterns with the internal Laravel portal team to enable parallel development.
• Build third-party API ingestion and field-level diff-detection logic that automatically adjusts monitoring rules when product data changes.
• Handle modern web rendering challenges at scale: JavaScript-heavy SPAs, interstitials, cookie consent overlays, dynamic content, viewport switching, and full-page screenshot capture.
• Evaluate when LLM analysis is the correct tool versus a classifier or rules-based approach, and design the two-stage rule-engine-plus-AI pipeline accordingly.
• Build and maintain a unit test suite covering all modules and APIs to ensure uptime and proper functionality.
• Document every architecture decision, configuration, API contract, and operational procedure continuously — not as a final-week deliverable.
• Deliver a complete runbook and knowledge transfer to the internal team at engagement close.
• Operate independently end-to-end while coordinating closely with the internal portal team and reporting directly to the Senior Director, surfacing risks and trade-offs early.

Requirements

• Production backend Python experience, including async patterns, type hints, packaging, and testing.
• Direct production experience designing and configuring AWS ECS/Fargate, SQS, API Gateway, RDS (PostgreSQL), S3, IAM, and CloudWatch, with infrastructure as code (Terraform or CDK).
• Real shipped systems calling the Anthropic Claude API in production, with demonstrated experience in prompt design, structured output, error handling, and cost trade-offs.
• Demonstrated track record reducing token spend on production LLM workloads, with specific before/after results you can walk through.
• Working knowledge of other LLM providers sufficient to recommend cheaper or better alternatives for specific tasks.
• Production Playwright experience at scale, including headless