Part 1 of this series made the case that AI observability is not just another dashboard problem. Once an agent is in production, the question is no longer can it respond; the question is whether it continues to respond accurately, safely, efficiently, and consistently as models evolve, prompts change, retrieval indexes drift, and real-world traffic exposes edge cases you never saw in test. That is why Microsoft Foundry’s observability story is built around three connected disciplines evaluation, monitoring, and tracing rather than traditional uptime metrics alone.
This article is the practical half of that story. It stays in the Foundry (new) experience throughout and focuses on the configuration and operational patterns that actually matter once you are moving agents toward production:
- connecting Application Insights
- enabling continuous evaluation
- reading the monitoring dashboard like an operator
- using traces to debug quality regressions
- wiring alerts into Azure Monitor
- adding quality gates to CI/CD
- enforcing guardrails and compliance
- integrating Defender for Cloud and Microsoft Purview and keeping a close eye on cost, quota, and token usage across the estate.
Portal note: Everything below assumes the Foundry (new) portal. If you are not seeing Build, Operate, and Admin in the current navigation, stop and switch to the new experience before you begin.
Start with the telemetry backbone: connect Application Insights
Every meaningful observability workflow in Foundry starts with one dependency: Azure Monitor Application Insights. It is the storage and correlation layer for the telemetry that powers monitoring, tracing, and the production-facing quality views. If you skip this step, the rest of the experience becomes partial at best and invisible at worst.
In the current portal experience, the cleanest place to verify that your project is wired correctly is Operate > Admin > Connected resources. That is where you confirm whether an Application Insights resource is already attached to the project, and where you add one if it is missing. Some agent-specific experiences also surface connection entry points elsewhere, but Connected resources is the most reliable control-plane view when you want to validate the configuration before you go any further.
In practice, the setup is straightforward. Open your project, go to Operate > Admin, select your Project, look for the tab Connected resources, and verify that an Application Insights resource is present. If not, add one there and keep it in the same region as the rest of your project resources where possible. Once that connection exists, Foundry has a place to emit and correlate the telemetry that the rest of this article depends on.
One operational detail is worth calling out early: the people who will investigate traces or query logs later need access to the monitoring resources themselves. In other words, connecting the resource is only half the job; the second half is making sure the operations or engineering team can actually read what Foundry writes there.
From connection to continuous evaluation
Part 1 described evaluation has to move from an occasional pre-release ritual to an always-on production discipline. The portal now supports that model directly: once your project is connected, you can turn on continuous evaluation so sampled production traffic is scored over time instead of only during offline test runs.
The natural entry point is the agent’s monitoring surface. Open Build > Agents, select the agent you want to operate, and then go to Monitor. This is where the platform brings together operational metrics and evaluation signals for the agent you actually care about, rather than forcing you to jump between disconnected tools. If your tenant has the built-in assistant enabled, Ask AI (Preview) can also summarize what the dashboard is showing and highlight abnormal behavior for the time range you have selected.
When you open the monitor settings, you are not just flipping a switch, you are deciding what “good” means for your workload. For a general conversational agent, the right starting point is usually coherence and relevance. For a RAG-heavy agent, groundedness and retrieval quality quickly become non-negotiable because they separate model behavior problems from retrieval problems. For safety or policy sensitive workloads, the relevant safety dimensions belong in the first wave as well, not as an afterthought once you have shipped.
The portal also forces a real-world tradeoff that Part 1 already warned about: sampling. Continuous evaluation is valuable precisely because it runs on live traffic, but it is not free. A sensible production posture is to begin with a modest sample, see how noisy or stable the results are in your environment, and then raise or lower the rate based on risk, volume, and budget.
One practical point that often gets missed: if you are enabling continuous evaluation against production traffic, the project’s managed identity must have the permissions required to execute those evaluation calls.
Read the dashboard like an operator, not a tourist
The Agent Monitoring Dashboard is easy to mistake for a status page. It is more useful than that. Used well, it becomes the place where you correlate quality, latency, failures, token consumption, and if you have the right workflows enabled, safety findings in one operational view backed by Azure Monitor data.
That matters because production AI rarely fails in a single dimension. A rise in latency might be harmless if quality stays steady and the cause is obvious. A flat latency curve can still hide a serious regression if groundedness or retrieval quality starts slipping. Token usage can expose cost drift long before a finance alert ever fires. The dashboard is valuable precisely because it puts those signals side by side so you can see whether the problem is quality-only, operations-only, or a combination of both.
In the current portal, you can reach that view from Build > Agents > Monitor for agent-centric investigation, and from Operate > Overview when you want a broader control-plane perspective across the subscription. Microsoft’s current documentation for Ask AI (Preview) explicitly points to both of those surfaces: Build > Models or Agents > Monitor for workload-specific analysis and Operate > Overview for estate-level analysis.
This is also where the most useful operator habit emerges: use metrics to spot the problem, but do not stop there. A low groundedness score or a rising error rate should trigger a trace investigation, not just a dashboard screenshot for later discussion. The dashboard is the front door, not the full diagnosis.
When a score drops, tracing tells you why
Monitoring tells you something is changing. Tracing tells you what actually happened inside a run. That distinction is what turns observability into an operational system rather than a collection of charts. Microsoft’s tracing model is built on OpenTelemetry, and the goal is not simply to log a request, it is to capture the execution path across model calls, tool invocations, retrieval steps, orchestration decisions, and the timings and tokens associated with each stage.
For Foundry-managed experiences, a good portion of that visibility is available once the telemetry backbone is connected. For custom agents and mixed stacks, the current guidance puts the emphasis on making sure the agent is registered and observable from the control plane, then sending standards-based telemetry into the same monitoring backend. Operationally, that means Foundry becomes the builder-facing surface while Azure Monitor remains the store, query, and correlation layer underneath.
The most important technical correction to many older tracing examples is the authentication and wiring model. The current guidance favors using the Application Insights connection string instead of older configurations that relied on the instrumentation key as a standalone setting, and the cleanest Python example today uses the AzureAIOpenTelemetryTracer from the LangChain integration path with that connection string explicitly provided.
For teams using other frameworks, the broader principle is the same even if the code differs: emit OpenTelemetry spans and send them to the Application Insights resource already attached to your Foundry project. That keeps traces, quality signals, and operational telemetry in the same backend instead of fragmenting them across different tools and teams.
There is one operational caution here that should never be buried in a footnote: traces can capture prompts, responses, tool arguments, and other sensitive runtime content. In enterprise environments, trace retention, RBAC, and content hygiene matter just as much as trace completeness. If your prompts or tool payloads can contain secrets or personal data, fix that before you scale tracing, not after.
Use Foundry to discover issues, and Azure Monitor to operationalize them
Foundry is an excellent place to see that something is wrong. Azure Monitor is where you should generally act on it. That distinction creates a durable operating model because it keeps AI-specific insight in the portal while letting SRE and platform teams use the same alerting, routing, and automation surfaces they already trust for the rest of their workloads.
A practical alerting strategy does not start with every possible metric. It starts with a short list of conditions that justify operator intervention. For most agents, that means some combination of:
- sustained latency increases,
- run failure spikes,
- abnormal token consumption,
- and material drops in quality or safety signals that represent user-visible risk.
Foundry’s own monitoring surfaces can help you decide which thresholds matter, but the production-grade enforcement usually belongs in Azure Monitor alerts and action groups attached to the connected Application Insights resource. That is the pattern Microsoft’s observability guidance consistently points toward: the portal helps you understand the signal, while Azure Monitor helps you route and automate the response.
CI/CD quality gates: make quality a release decision
The real test of whether observability is part of engineering culture is whether it shows up in CI/CD. If evaluation only happens after deployment, it is already too late for many regressions. The better pattern is simple: run a repeatable evaluation set during the build or release process, publish the results, and fail the promotion if the thresholds that matter to the business are missed. That is how you turn “quality” from a conversation into a gate.
Because task and action names can change over time, the most durable approach for publication is not to hard-code a potentially short-lived marketplace extension name. Instead, keep the pipeline pattern stable and call your evaluation script explicitly. That gives readers something they can implement regardless of whether they use GitHub Actions or Azure DevOps and regardless of how the surrounding wrapper evolves.
A simple GitHub Actions pattern looks like this:
The principle matters more than the wrapper: use the same evaluator definitions across development, CI, and production monitoring so a “good” score means the same thing wherever you see it. That consistency was one of the core ideas in Part 1, and it is what makes evaluation an operating system instead of a one-off test harness.
Guardrails and compliance belong in the operating loop
Once an agent is live, “responsible AI” has to move out of slideware and into operations. That is what the Compliance workspace in the control plane is for. In Part 1, the compliance story was framed around policy posture, guardrails, and remediation across the estate. In practice, that means Operate > Compliance becomes part of your operating routine, not something you visit only when audit season arrives.
The value of this workspace is not just visibility. It is that it lets you compare what your organization expects:guardrail controls, security posture, data-governance integration against what individual assets are actually doing. The policy view tells you where you are out of compliance. The asset views help you see which deployments are drifting. The remediation flow closes the loop before policy violations become real incidents.
Use Operate > Compliance to review policy posture, identify violations, compare required controls with actual deployment settings, and remediate the drift. The exact button labels can change over time; the operating motion does not.
Extend the view: Defender for Cloud and Purview
Foundry’s built-in controls are necessary, but enterprise deployments often need the larger Microsoft security and governance ecosystem around them. That is where Defender for Cloud and Microsoft Purview extend the story from application behavior into security posture, threat detection, audit, and data governance.
From an operational perspective, Defender for Cloud helps security teams see posture recommendations and AI-adjacent threats such as suspicious prompt activity or exposed configurations in the places they already work. Purview extends the governance model further by helping organizations treat prompts and responses as governed data, subject to the same kinds of audit, classification, and security workflows that apply elsewhere in the enterprise.
The right place to review those integrations in the current portal is the security and compliance experience under Operate, especially the views focused on posture and governance. Availability and exact onboarding flows can vary by tenant, role, and licensing, so what matters most for an enterprise is the operational principle: if your organization relies on Defender or Purview elsewhere, AI workloads should not become the exception.
Cost, quota, and token usage: keep them in the same conversation as quality
One of the easiest mistakes in AI operations is to discuss quality, safety, and cost as if they were separate systems. They are not. Token usage affects spend, spend influences sampling choices, and sampling choices shape what you can see about quality in production. Foundry’s Operate surfaces are useful precisely because they let you see those pressures in one operating loop rather than across unrelated tools.
The three views that matter most are still the ones Part 1 highlighted. Operate > Overview gives you the estate-level operational summary, including token and cost trends. Operate > Assets gives you the searchable inventory view where heavy consumers and unhealthy assets become obvious. Operate > Quota tells you how deployment capacity is being consumed and whether you are heading toward avoidable throttling or scaling friction.
What has to be explicit in production is the cost model around observability itself. Monitoring does not introduce an additional Foundry platform charge, but tracing still depends on the underlying Azure Monitor / Application Insights storage and retention footprint, and evaluation adds execution cost because more model work is being done. In other words, you are not only monitoring your agents, you are also operating a telemetry system. Treat it that way.
What good looks like in production
The biggest shift between Part 1 and Part 2 is this: once the features are enabled, observability becomes a habit. Good teams do not just turn on dashboards. They decide which signals matter, which thresholds justify intervention, where human review belongs, and how those decisions show up in daily operations and release engineering.
In practice, that usually means a handful of operating principles:
- Do not monitor everything at once. Start with the few signals that map directly to user harm or operational pain.
- Use the same evaluation logic across environments. Quality should not mean one thing in CI and another in production.
- Keep traces useful, not reckless. Rich traces are valuable until they start capturing content you should never have stored.
- Treat cost as a first-class signal. Token usage and trace volume are not background details; they are operational constraints.
- Revisit the configuration periodically. The platform is evolving quickly, and what was preview-only a few months ago may already be operationally relevant in your tenant now.
Conclusion
Part 1 detailed that production AI requires a different kind of observability because AI systems fail in different ways. Part 2 is where that argument becomes operational. Once you connect Application Insights, enable continuous evaluation, use the monitoring dashboard to spot change, follow traces to root cause, route the right signals into Azure Monitor alerts, and bring compliance, security, and cost into the same operating loop, you stop treating quality and safety as release-time paperwork and start treating them as runtime disciplines.
Foundry’s observability model gives builders, operators, and governance teams a shared system for answering the same question from different angles: Is this agent still doing the right thing in production, and how do we know?
If Part 1 was the architectural blueprint, Part 2 is the operational playbook. Put together, they describe the shift every serious AI team eventually has to make: from “the agent works” to “the agent stays reliable, explainable, and governable in production.”