Why Have a Test Environment

As part of our best practices Kelverion always recommends having a test automation platform.

There are several reasons explained below why Kelverion recommends customers have a test environment.  We know it takes a little bit of time and resources, but it truly can be a safe zone that allows you to test and grow your IT automation processes.  Let’s explore some of the most common reasons.

Evolution of Automation

Often customers have the intention of automating a specific process.  For example, they decide they want to automatically create incident tickets in their service desk for alerts on servers.  They design it from the very beginning of the process and get it working like clockwork.  Then they see it working so well, but the technicians are doing the same process to resolve the issues every time.  So, they want to add in some diagnostics and maybe even some remediation.  Making changes like this on a live production system could result in duplicate incident tickets for the same alert, no tickets created for multiple alerts, SLAs missed because of a missing step…all these things could be avoided if they could test everything in a test environment first.

Upgrades and Migrations

Some of our most recent professional services projects for customers have been upgrading from an Orchestrator 2016/2019 environment to an Orchestrator 2022 environment.  This means going from a 32-bit version of Orchestrator to a 64-bit version…and all the integration packs also must be upgraded.  This means each piece must be thoroughly tested before moving forward in a production environment.  Imagine you’ve upgraded to Orchestrator 2022 on your production environment, and you did not realize that applying the latest Update Rollup to Orchestrator 2022 will break it.  Now how do you go back?  After all, this is your production environment.  In this situation, if it were a test environment, you could easily wipe everything out and start over – no production is lost, and a new lesson is learned about preparing for this type of upgrade. 

Early Bug Detection

We have seen instances when the runbooks are designed correctly, but there is an issue with the API that had not been previously detected.  Imagine you have a runbook designed to delete Active Directory User accounts and their Exchange Mailboxes when an employee has left the company.  You never thought to test what happens when you try to delete an Exchange Mailbox for an employee if their Active Directory account has already been deleted.  Then much to your surprise, when the runbook queries that Exchange Mailbox, it does not return an error or a message stating ‘None found’…instead it decides to return the first 10,000 Exchange Mailboxes.  This has never been tested because you were not even thinking about any type of bulk mailbox deletion option.  So, the runbook deletes what was returned by the query which just happens to be 10,000 mailboxes.  Wouldn’t it be easier to find this out in a test environment rather than a production environment?

Pay Now or Pay More Later

One of the most common reasons we hear from customers as to why they do not have a test environment in place is cost.  But if we take the previous example of accidentally deleting 10,000 Exchange Mailboxes, how much time and money would it take for your IT staff to restore and/or recreate 10,000 Exchange Mailboxes?  Is that worth being able to tell the CIO, “I saved us $5,000 by not having a test environment”?  Probably not the response they are looking for.

In our experience, we have seen this result in production data being deleted or in one case, a faulty test runbook ending up in a runaway memory leak that took the whole automation system offline.  Wouldn’t it be better if you could tell the CIO, we just saved the company from a huge outage because we found this issue in our test environment!

At this point, it might be easier to think about the reasons you justify a good backup system.  In both instances, you want the ability to easily recover from anything you are implementing with minimal-to-no down time in your production environment.  Having a test environment gives you the ability to test, identify vulnerabilities, and recover while your production is still moving along without interruption.

It’s Never Too Late

What happens when you come up with another process you want to automate?  Maybe now you want to create incident tickets in your service desk when someone requests a virtual machine.  Do you want to risk taking down a perfectly good working system to test something new?  It would be much better to use a test environment – see if there is any impact on the overall system and make sure everything is working as expected before exporting it to your production system.

Ultimately, as customers explore more and more opportunities for automation, it will grow and evolve into other areas to facilitate many of your more routine IT responsibilities and tasks.  This will continue to free up your IT staff to focus on trends occurring in your environment and identify more root causes of these issues rather than just staying at the low-level focus of closing multiple incident tickets.  To plan for this level of success and future growth with a more stable IT environment, you really do need a test environment. 

If you would like any further information on building a test environment for your automation project, please get in touch

.

Further Reading

If you found this blog helpful, here are a selection of other articles that might interest you:

Cloud or On-Premise Automation?

Frequently Asked IT Automation Questions

Evolve Employee Onboarding with Automation