QA Tester – AI Application

Location: India, Remote

Work Experience: 5 Years

Requirements:

• Strong experience in manual, functional, integration, regression, and API testing.
• Experience with API testing using Postman, REST Assured, SoapUI, Swagger, or similar tools.
• Experience testing REST APIs, WebSockets, event streams, and asynchronous workflows.
• Basic SQL knowledge for data validation.
• Understanding Databricks, Delta tables, data pipelines, or cloud-based data platforms.
• Familiarity with RAG, LLMs, embeddings, semantic search, knowledge graphs, and AI application testing.
• Ability to validate AI responses against source grounding, expected business rules, and trace evidence.
• Experience testing rule-based workflows, exception workflows, state machines, or case management systems.
• Understanding authentication, authorization, role-based access, data privacy, and audit validation.
• Experience with test management and defect tracking tools such as Jira, Azure DevOps, or similar.
• Good analytical, communication, and documentation skills.
• Experience testing applications on Azure Databricks, Unity Catalog, Mosaic AI, or MLflow.
• Exposure to EDI/X12, healthcare transactions, pharmacy, payer, provider, claims, pricing, or drug data workflows.
• Exposure to vector search, graph databases, context graphs, triple stores, or decision trace stores.
• Experience with automated API testing and Python-based test automation frameworks such as PyTest.
• Experience validating model outputs, prompt changes, agent workflows, and retrieval of quality.
• Familiarity with FDA drug data, DEA registry validation, NADAC/GPO pricing, EPCIS, VRS, or serialization workflows.
• Familiarity with CI/CD validation, smoke testing, release certification, and production sanity testing.
• Knowledge of observability tools such as Grafana, Prometheus, OpenTelemetry, Azure Monitor, or Log Analytics.
• Understanding healthcare, pharma, or regulated enterprise environments is a plus.

Qualifications: Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Job Description:

• Prepare test strategy, test scenarios, test cases, test data, and test execution reports for the Application.
• Validate end-to-end operator workflows across dashboard, exception workspace, context inspection, learning console, and local client state behavior.
• Test operator actions such as acknowledge, resolve, escalate, annotate, re-evaluate, and case state transitions.
• Validate transactional request/response APIs and live event-stream behavior for real-time updates.
• Test exception arrival, agent progress updates, queue health metrics, and re-evaluation events.
• Validate EDI/X12 parsing, exception normalization, and mapping to the internal exception model.
• Test deterministic exception handling through L1 reference data lookups and L2 policy-based routing.
• Validate L3 agentic resolution workflows, including bounded tool execution and fallback behavior.
• Test resolution tools such as drug lookup, pricing check, serial verification, DEA check, and document lookup.
• Verify Graph RAG and Document RAG results for relevance, accuracy, grounding, source traceability, and response consistency.
• Validate context retrieval from context graph, reference data, decision  trace store, policies, documentation, and regulatory guidance.
• Test decision trace generation for both deterministic and agentic decisions.
• Validate auditability, explainability, trace completeness, and decision lineage.
• Test feedback capture from operator resolutions and learning-loop workflows.
• Validate pattern mining outputs, candidate rule generation, routing optimization results, validation of sandbox results, and promotion of gate approvals.
• Perform regression testing after model, prompt, rule, routing, ingestion, retrieval, or external-source changes.
• Validate Unity Catalog permissions, data governance, access controls, and restricted data visibility.
• Conduct negative testing, boundary testing, failure scenario testing, retry testing, idempotency testing, and security validation support.
• Validate observability metrics such as queue health, exception volume, agent latency, retrieval quality, provider errors, token usage, ingestion lag, and decision trace completeness.
• Log defects with clear reproduction steps, expected results, actual results, screenshots/logs, test data references, and severity.
• Collaborate with developers, DevOps/MLOps, architects, business users, and operators for defect triage and release readiness.