Section 4: Evaluating and deploying agentic workflows

Question 1, 4.1 Evaluating agents in development and in production

A telecom company is building a plan-change agent with Agent Development Kit (ADK). The agent needs a customer's account number and the target plan before it calls its change_plan tool, but depending on the conversation it may ask for them together or one at a time. The team has no production traffic yet and found that scripted multi-turn test cases break whenever the agent asks its questions in a different order. They want multi-turn test cases that stay valid regardless of question order and that also reflect less technical customers. What should you do?

1. Record several sessions in the adk web Eval tab, save each as an eval case, and set tool_trajectory_avg_score with the EXACT match type.
2. Write .test.json files that list every expected user turn and the expected change_plan arguments, and run them with pytest in CI.
3. Add ConversationScenario eval cases with a starting prompt, conversation plan, and NOVICE persona, scored with reference-free criteria.
4. Wait for launch, sample production logs into a dataset, and score the sampled sessions with response_match_score against the logged replies.

Question 2, 4.1 Evaluating agents in development and in production

A travel company has built a booking agent, and an immutable version of it is already registered in Agent Registry. The agent has not launched, so no production traces exist. The QA analysts who own test design work only in the Google Cloud console and do not write Python or use command-line tools. They need a starting set of multi-turn test cases in which users begin booking a flight and then change their minds, and they want to review and edit the cases before running them. What should they do?

1. Ask a developer to run agents-cli eval dataset synthesize locally with an instruction, and then share the generated trace files with the QA team.
2. On the Agent Platform Evaluation page, select New evaluation > Simulate sessions, enter a generation instruction, and edit the generated table.
3. Create an online monitor that evaluates the agent's traces, and copy the lowest-scoring sessions into a dataset once enough traffic arrives.
4. Write ConversationScenario JSON files by hand, and add them with adk eval_set add_eval_case using a session input file for the agent.

Question 3, 4.1 Evaluating agents in development and in production

A team scaffolded its agent with Agents CLI and runs agents-cli eval run in Cloud Build on every merge request. The team lead has set these success criteria: the mean tool_use_quality score must be at least 0.8, and the team's custom response-length metric must score 1.0. If either criterion is missed, the build must fail automatically without anyone reading reports. Which approach meets these requirements with the LEAST effort?

1. Set both minimum scores under thresholds in eval_config.yaml, and let the build fail on the exit code of agents-cli eval run.
2. Add a build step that parses the HTML report in artifacts/grade_results and fails the build when either score is too low.
3. Run agents-cli eval compare between the last release's results file and the new results file, and fail when any score differs.
4. Run agents-cli eval analyze after each run, and fail the build whenever the command returns one or more failure clusters.

Question 4, 4.1 Evaluating agents in development and in production

An e-commerce agent returns order summaries as JSON that a downstream system parses. The team needs an automated check that each final response is valid JSON and matches a published JSON Schema. The check must give the same result every time and must not add any judge-model charges. Field order and whitespace in the output can differ between runs, even when the content is correct. Which evaluation approach should the team use in the Gen AI evaluation service?

1. Register a custom LLM metric whose prompt asks the judge model to confirm that the response is valid JSON that follows the schema.
2. Use the INSTRUCTION_FOLLOWING adaptive rubric metric, and include the schema in each prompt so that the generated rubrics check it.
3. Use the exact_match computation metric, and compare each response with a reference JSON string prepared for every test case.
4. Use a custom code metric with an evaluate function that parses the response and validates it against the schema with jsonschema.

Question 5, 4.1 Evaluating agents in development and in production

A legal-research agent answers attorneys' questions from retrieved case excerpts. The team has a golden dataset in which every row contains the question, the excerpts that the agent retrieved, the agent's answer, and an answer that an attorney wrote. Correct answers are long-form and are rarely worded the same way as the attorney's answer. The team wants Gen AI evaluation service metrics that show whether each answer stays faithful to the retrieved excerpts and whether it agrees with the attorney's answer. Which two metrics should the team use? (Choose two.)

1. GROUNDING, with the retrieved excerpts that the agent used provided in the context field of each row.
2. exact_match, with the attorney's answer provided in the reference field of each row.
3. FINAL_RESPONSE_MATCH, with the attorney's answer provided in the reference field.
4. TEXT_QUALITY, so that fluency and coherence show how closely the answer follows the excerpts.
5. SAFETY, so that answers that contradict the retrieved excerpts are flagged as policy violations.

Question 6, 4.2 Deploying and scaling production workloads

A logistics company runs a fleet of document-processing agents. Upstream systems publish shipment documents to Pub/Sub; each agent pulls messages, calls Gemini and internal tools, and writes results to BigQuery. The agents must not expose any HTTP endpoint, must scale horizontally as the message backlog grows, and the platform team has standardized on Cloud Run for container workloads. Which Cloud Run resource type should you use for these agents?

1. Cloud Run worker pools that pull from the Pub/Sub subscription and scale horizontally without serving HTTP.
2. Cloud Run services that receive Pub/Sub push deliveries, with ingress restricted to internal traffic only.
3. Cloud Run jobs triggered every five minutes by Cloud Scheduler to drain the messages waiting in the subscription.
4. Cloud Run instances running a single always-on agent loop that pulls and processes every message in sequence.

Question 7, 4.2 Deploying and scaling production workloads

A travel company's ADK agent on Agent Runtime sometimes gets stuck alternating between a flight-search tool and a re-planning step for the same request. Traces show some runs making several hundred model calls before ending, which drives up cost and keeps users waiting for minutes. Normal requests rarely need more than 20 model calls. The team wants runaway runs to stop early without changing normal behavior. What should you do first?

1. Raise max_instances so that runaway sessions do not slow down other users while they keep calling the model.
2. Set max_llm_calls in RunConfig to 0 so that ADK manages the model-call budget dynamically for each request.
3. Set max_llm_calls in RunConfig to a bound near the observed need, such as 30, so runaway runs end early.
4. Wrap the agent in a LoopAgent with max_iterations set to 20 so that each tool call counts as one iteration.

Question 8, 4.2 Deploying and scaling production workloads

A telecom company's billing-support agent on Agent Runtime passed offline evaluation at launch. Over two months, customer complaints suggest that response quality is slowly declining as new plan types and new user phrasing appear. The team wants automatic, continuous scoring of a sample of real production conversations with quality and hallucination metrics, while keeping evaluation costs capped. Cloud Trace is already enabled. What should you do?

1. Re-run the launch evalset with the ADK evaluation tooling every two months and compare the scores manually.
2. Create an online monitor for the agent with those metrics, a sampling percentage, and a max samples per run.
3. Create a log-based metric that counts responses containing the word sorry and alert when the count grows.
4. Create a Cloud Monitoring alert on request latencies for the ReasoningEngine resource and review the spikes.

Question 9, 4.2 Deploying and scaling production workloads

An ADK-based agent on Agent Runtime runs with min_instances=10 and the default container_concurrency. In a load test with 300 concurrent requests, median latency is about 4 seconds but maximum latency reaches 60 seconds, while CPU and memory use per container stay low. Most of each request's time is spent waiting on Gemini and tool calls. You want the existing instances to absorb traffic spikes. What should you do?

1. Increase container_concurrency to a multiple of 9, such as 36, and watch memory for out-of-memory errors.
2. Increase resource_limits to 8 CPUs and 16Gi of memory per container and keep container_concurrency at 9.
3. Set min_instances to 75, the maximum allowed value, so that no request ever waits for a new instance to start up.
4. Convert the agent to a synchronous implementation so that each container processes one request at a time.