Agentic AI Observability Senior Engineer is responsible for deploying, integrating, and operating a scaled Agentic AI observability platform across both internal and external agent frameworks. This role focuses on production-ready instrumentation and telemetry pipelines that provide end-to-end visibility across multi-step agent workflows—including planner/executor loops, tool/function calls, RAG retrieval, and memory/state—ensuring reliability, safety, performance, and cost governance at scale

• Agentic AI Observability at Scale (0%)

• Platform Deployment & Operations (Agentic AI Observability at Scale)• Deploy and run the Agentic AI observability platform across dev/uat/prod with HA, resiliency, and controlled rollouts• Implement release automation (CI/CD), canary deployments, rollback strategies, and configuration management for platform components• Own operational readiness: on-call runbooks, incident response, and production support for agent observability services
• End-to-End Agent Workflow Tracing (Planner → Tools → Retrieval → Response)• Implement distributed tracing for full agent execution graphs, including correlation across: prompts, intermediate reasoning steps (where permitted), tool calls, external APIs, retrieval queries, and final responses• Enforce consistent trace context propagation, correlation IDs, and semantic conventions across agent services• Build instrumentation patterns to represent agent flows as spans (e.g., plan span, tool span, retrieval span, memory span, response span)
• Agent Framework Integrations & Standardized Instrumentation• Deploy and maintain integrations for internal agent frameworks and external ecosystems such as Crew.ai, LangChain, Semantic Kernel, AutoGen, and custom orchestrators• Create reusable SDKs/middleware/sidecar patterns for teams to instrument agents with minimal effort• Define and implement tagging standards for: agent name/version, tool name, model provider, prompt template ID, retrieval source, tenant/app, and environment
• Agentic AI Telemetry Pipelines & AI-Specific Signals• Build scalable pipelines for agent telemetry (logs/metrics/traces) using OpenTelemetry and platform observability tooling• Capture AI-specific metrics including: token usage, cost per task, tool-call latency, retrieval latency, grounding score proxies, error rates, and agent loop iterations• Implement sampling and redaction strategies for sensitive agent payloads (prompts, responses, retrieved content) aligned to governance requirements

• Collaboration with Teams (10%)
  • Collaborate with transformation teams and business stakeholders to understand requirements and tailor AI agents to specific domains.
  • Work closely with AI platform teams to build scalable and cross-domain AI agents while ensuring end-to-end observability.
• Integration & Deployment (10%)
  • Build and maintain CI/CD pipelines for agent services and operations center components, including automated testing and deployment
  • Automate onboarding for new agent use cases (templates, scaffolding, configuration checks)
  • Drive best practices for secure, scalable, and cost-effective agent deployments
• Continuous Learning (10%)
  • Stay updated with the latest advancements in AI and machine learning technologies and integrate these into existing or new AI agents.
  • Conduct thorough testing and validation to ensure the reliability and accuracy of AI agents and solutions.

• Education: Bachelor’s or Masters in Computer Science, AI/ML, Data Science, or a related field.
• Experience: 4-8+ years of software engineering experience; 2-3+ years building and observe AI/ML or GenAI applications preferred
• Required Expertise:

• • Strong hands-on experience deploying observability solutions (Prometheus/Grafana/Elastic/Splunk/Datadog or equivalent)
  • Deep working knowledge of OpenTelemetry instrumentation and telemetry pipeline operations
  • Experience observing agentic AI systems: tool/function calls, orchestration, routing, memory/state, and RAG pipelines
  • Familiarity with Crew.ai, LangChain, Semantic Kernel, AutoGen, or similar agent frameworks
  • Experience with evaluation/quality monitoring and safe logging strategies for LLM systems
  • FinOps experience for tracking token and GPU spend, chargeback/showback, and cost anomaly detection
  • Experience implementing data governance controls for AI telemetry (PII redaction, retention, auditability)
  • Strong Kubernetes experience (AKS/EKS/GKE) including Helm, operators, ingress, and service networking
  • Strong automation skills (Python/Bash/Go) and CI/CD experience
  • Infrastructure-as-Code (Terraform/Bicep/CloudFormation)
  • Agent workflow tracing and telemetry correlation
  • Production operations and debugging distributed systems
  • Observability-as-a-platform enablement and automation
  • Strong documentation, collaboration, and stakeholder influence
  • Technical Proficiency: Implement monitoring for agent failure modes: tool-call failures, infinite loops, timeouts, hallucination risk signals, retrieval misses, and degraded response quality. Create alerts aligned to operational SLOs (availability, latency, tool reliability) and AI-specific indicators (cost spikes, loop bursts, retrieval anomalies). Support guardrail observability: policy blocks, content filtering events, and safety classifier outcomes (where applicable). Build onboarding automation (IaC, templates, CI checks) that makes observability “default-on” for all agentic services.
  • Problem-Solving: Ability to translate business challenges into technical solutions.
  • Collaboration Skills: Effective at working within cross-functional teams.
  • Agility: Flexibility to adapt to changing requirements and new technologies.
  • Communication Skills: Capable of explaining complex technical concepts to non-technical stakeholders.