VCF 9.1 Update Runbook: Testing, Backtracking, and Preparation Gates

Upgrading a VMware Cloud Foundation (VCF) infrastructure to version 9.1 is not a simple patch. It's a process that involves coordinating multiple layers—compute, networking, storage, identity management, observability, and applications—and that, if not run with a robust runbook, can lead to endless maintenance windows, risks of unavailability, and a sense of operational insecurity that drags on for weeks. In this article, we explore how to build an upgrade runbook that not only lists steps, but establishes readiness gates, technical validation, and clear rollback plans.

Many organizations approach VCF upgrades with a lightweight approach: they trust that Broadcom's procedures are sufficient, review the release notes, and prepare a to-do list. However, when the environment is in production and the operations team has to respond in real time, the lack of a plan with decision doors becomes a burden. What do we do if post-validation fails? Who authorizes to continue or stop? How do we ensure that restoring a backup actually works? These questions must be answered before opening the change window.

A VCF 9.1 runbook oriented towards readiness gates changes the dynamic: each phase has a person in charge, concrete evidence and a criterion for progress or stopping. This allows the team to not just execute commands, but to make informed decisions based on real data. The experience accumulated in digital transformation projects shows that the most serious failures do not occur due to an isolated technical error, but due to the failure to identify a hidden dependency – a DNS record that was not updated, an expired license, an expired certificate, a Firewall that blocks traffic between management services.

Hidden heading for structure only

For a runbook to be truly effective, it must cover seven fundamental stages, from source validation to Day-N delivery. Each stage should include its own set of checks and a rollback mechanism that is not generic, but specific to the stage we are in. We can't pretend that 'restore from backup' will do the job: recovering a virtualized domain controller is not the same as restoring an entire ESXi cluster. A well-defined fallback decreases panic and accelerates the return to a known state.

In this context, companies that have already adopted DevOps methodologies and reliability engineering (SRE) practices are better prepared. However, many organizations still lack the maturity to handle an upgrade of this magnitude. This is where collaboration with specialized technology partners makes all the difference. For example, at Q2BSTUDIO we work with companies that need not only the infrastructure, but also the advice and automation to make their upgrade processes predictable. Our experience in custom applications has taught us that customization not only applies to business software, but also to operations playbooks. A generic runbook serves as a template, but each environment has its own peculiarities: integrations with legacy systems, regulatory compliance requirements, dependencies on critical applications.

Hidden heading for structure only

The first stage of preparation should be the validation of the source of the update. It's not just about downloading the right ISO; it must be verified that the upgrade path proposed by Broadcom is compatible with the exact version of each component. Interop matrices, release notes, and known KBs should be collected and attached to the change log. This prevents the computer from relying on its memory or outdated documentation. In addition, it is essential to use planning tools such as the VCF Upgrade Planning Tool, which helps identify potential conflicts before they become live issues.

The second door is the current state inventory. It's not enough to have a list of IP addresses and software versions. External dependencies must be documented: DNS, IPAM, NTP, PKI, identity providers, backup platforms, logging systems and monitoring tools. Each of these can be a point of failure. For example, if the VCF 9.1 license server requires a specific name resolution and the corresponding DNS record has not been updated, the SDDC administrator update will silently fail. An inventory validated by the owners of each service significantly reduces the risks of surprises.

Hidden heading for structure only

The third gate, and perhaps the most critical, is the preparation of VCF Operations (formerly Aria Operations). It is often assumed that observability is reviewed at the end, when in fact VCF Operations is a central part of the sequencing model. The exact version, patch level, compatibility with the new VCF 9.1 engine, and the health of adapters and collections should be validated before touching the core of the platform. If monitoring fails during the upgrade, the team loses visibility and cannot detect issues in time. It is advisable to define observability acceptance criteria: dashboards that must continue to work, critical alerts that must be triggered, logs that must reach the event center. No improvisations.

The fourth gate encompasses SDDC Manager and VCF management services: IP address plans, network connectivity management, DNS, security, and licensing. This is where many projects run into network locks or firewall policies that were not contemplated. The runbook should include actual connectivity tests, not just a ping. AWS and Azure cloud services offer automation models that can be applied to programmatically validate these requirements, reducing manual load and increasing the reliability of checks. At Q2BSTUDIO we integrate these practices so that our clients can run automatic pre-checks before any major changes.

Hidden heading for structure only

The fifth gate is the upgrade of the core: NSX, vCenter, ESXi and NSX Edge. Each of these components should be treated as a separate work package, with its own backup validation, temporary IP requirements, capacity plan, and recovery testing. A common mistake is to upgrade ESXi without verifying the health of the vSAN storage or ensuring that the hosts' maintenance mode can be completed without impacting workloads. Post-validation doesn't end when the host comes back online; you need to verify that the network traffic, storage connectivity, and visibility from VCF Operations are correct.

The sixth gate is validation across three layers: platform, operations, and workloads. The platform should display healthy states in SDDC Manager and in the individual components. The operations layer must ensure that alerts, dashboards, and policies continue to work. And the application layer must pass smoke tests with representative workloads. This is where the need to have AI for companies and AI agents that automate part of these validations comes into play, detecting anomalies in logs or performance metrics that a human would overlook. At Q2BSTUDIO we develop artificial intelligence solutions that help monitor the status of complex infrastructures and generate automatic acceptance reports.

Hidden heading for structure only

The seventh gate is the fallback plan. We cannot have a single 'undo'. For each phase, it is necessary to define what it means to stop and how to restore the previous state. Sometimes, fallback involves restoring an entire virtual machine; other times, simply stabilize the environment, collect logs, and open a support case. It's crucial to set clear shutdown conditions: for example, if the VCF Operations upgrade fails and we can't guarantee observability, everything should be stopped. If critical application validation fails, it should not be continued. The culture of 'closing the window even if there are issues' is dangerous; It's better to stop, investigate, and reprogram.

Finally, the Day-N delivery should not be an add-on, but a formal checklist with assignees: cleaning up VCF Operations, updating licenses, transitioning identities, updating documentation, and planning the next windows for workload domains. Many teams celebrate the management domain upgrade and forget that workloads still need their own upgrade. A full runbook includes that continuation plan.

Hidden heading for structure only

To provide intelligence to this entire process, companies can benefit from business intelligence tools that allow them to visualize the status of the runbook in real time and make decisions based on data. Business intelligence services such as Power BI can be integrated with orchestration platforms to generate progress dashboards, crash alerts, and post-mortem reports. At Q2BSTUDIO we've helped organizations build these dashboards, connecting data from multiple sources (VCF, vSphere, NSX, backup systems) to provide a unified view of the health of the environment during the upgrade.

We cannot forget cybersecurity. Each update opens a window of risk: temporary credentials, elevated access, services momentarily unprotected. A runbook should include security controls, such as post-update password rotation, patch signature verification, and ensuring that no untracked emergency accounts have been created. Cybersecurity solutions offer frameworks that can be applied to these processes, and at Q2BSTUDIO we collaborate with security teams to design post-upgrade hardening plans.

Hidden heading for structure only

In short, an upgrade to VCF 9.1 is much more than running a wizard. It requires discipline, automation, and a runbook that controls the process, not describes it. Staging gates, layered validations, and specific rollback plans turn a high-risk operation into a manageable procedure. Assistive technology – from bespoke apps to AI agents – can make the difference between an incident Friday and an incident-free Monday. At Q2BSTUDIO we help companies design these runbooks, integrate automation tools, and build the business intelligence capabilities needed to make the upgrade predictable and secure. Because in the end, the goal is not only to upload the version, but to leave the platform in a higher operational state than it was before it started.