Senior Principal AI Agent Software Engineer
US client via staffing partner · San Jose, CA 95110 (Remote)
- Location
- San Jose, CA 95110 (Remote) · Remote
- Salary band
- Band not stated
- Type
- Contract
- Level
- Principal
- Work authorization
- Not stated
Stack
Python · LLM orchestration · LiteLLM · DSPy · PyTorch · ROCm · distributed ML training · distributed ML inference · GPU clusters · Megatron-LM · DeepSpeed · TorchTitan · vLLM · Docker · Kubernetes · Slurm · JSON Schema · CI/CD · ML performance optimization · agent frameworks
About the role
Job Title - Senior Principal AI Agent Software Engineer - Remote Location - [San Jose,CA,95110] (REMOTE) Job Type - 6 months Contract Job Overview: Our client is building agentic software to improve how teams develop, deploy, and operate machine learning workloads on GPUs. This senior contract role focuses on designing and delivering production-grade intelligent agents across the software stack: training and inference frameworks, cluster tooling, performance analysis, and operational workflows for large-scale GPU deployments. The successful candidate will lead implementation of Python [contact hidden] that combine LLM orchestration, tool integration, and rigorous evaluation to automate high-value engineering work: investigating distributed job failures, interpreting telemetry and profilers, accelerating performance analysis, and reducing manual toil in ML operations. This is a senior individual contributor engagement requiring full ownership of significant components end to end, making sound technical decisions within program direction, and delivering maintainable code with minimal supervision. Responsibilities: - Architect and implement Python agent frameworks: tool adapters, orchestration loops, structured outputs, session handling, and CLI or service interfaces suitable for production use. - Build LLM-powered workflows (LiteLLM, DSPy, or equivalent) with bounded prompts, allowlisted tools, budget caps, structured JSON outputs, and evidence-backed reporting. - Integrate agents with the GPU and ML software ecosystem: ROCm tooling, PyTorch, distributed training and inference stacks, profilers, cluster metrics, log pipelines, and debug utilities. - Own quality and eval infrastructure: fixture-driven tests, JSON Schema validation, regression suites, accuracy metrics, and CI integration before rollout. - Solve ML operations problems on training and inference clusters: multi-node failure analysis, performance regression investigation, configuration and launch issues, and operational automation. - Deliver deployment-ready artifacts: Docker / Kubernetes packaging, runbooks, documentation, and security-conscious defaults for customer-controlled environments. - Partner with framework, performance, and field teams; incorporate code review feedback and iterate based on production and pilot learnings. Required Qualifications: - Bachelor's degree or higher in Computer Science, Computer Engineering, Electrical Engineering, or a related field; equivalent practical experience accepted. - 10+ years of professional software engineering experience, including 5+ years on production Python [contact hidden] at scale. - Deep experience with distributed ML training or inference on GPU clusters: multi-node jobs, collective communication failures, log and metric correlation across ranks and nodes. - Strong hands-on PyTorch background and production familiarity with large-scale stacks (Megatron-LM, DeepSpeed, TorchTitan, vLLM, or equivalent). - Proven ML debugging and performance engineering across multiple failure modes: throughput regression, memory errors, numerical instability, misconfiguration, and profiler or trace analysis. - Demonstrated delivery of LLM agent or orchestration [contact hidden] in production or near-production settings: tool routing, structured outputs, reliability, and testability. - Track record of shipping complex software on schedule: clean code, automated tests, code review discipline, and excellent technical communication skills. - Ability to work independently, prioritize across ambiguous requirements, and align weekly with a technical lead. Preferred Qualifications: - Open-source ML development: upstream framework repos, community CI patterns, and integration with OSS tooling. - ML performance optimization: parallelism, throughput/MFU tuning, roofline analysis, and profiler-driven investigation. - ROCm and GPU cluster operations: metrics, profilers, health monitoring, Slurm or Kubernetes job environments. - Security-aware agent deployment: secret handling, egress control, and customer VPC or on-prem constraints. Working Model: - Duration: 6 months, full-time contract. - Cadence: weekly sync with technical lead; milestone-driven delivery. - Scope: implementation across agent and ML infrastructure programs; allocation may shift with program priorities at lead direction. Scope Boundaries: - Not a graphics driver, kernel, or low-level GPU driver development role. - Not a research-only role without production deliverables. - Not a junior or mid-level implementation; SDE 5 expects senior ownership and technical depth across ML [contact hidden] and agent engineering.
Recruiter contact
Just $1 to see this recruiter's name, email and phone. One-time, per role — no profile needed.
Want the full job-search package instead? Talk to us →
The recruiter's details are not public. Helen makes the introduction, checks your resume against the requirements first, and follows up on the reply.