Introduction

As an ASP Today subscriber, you are no doubt familiar with an Application Service Provider (ASP) type of environment. I am using the term narrowly to describe businesses that provide paid access to web applications. There are many advantages to the ASP model compared to the packaged software development; one of them is a possibility for rapid project delivery. New features and enhancements can be deployed to the production environment as soon as they are developed and tested. After that, they are immediately available to the users. Same is true for bug fixes. However this often causes companies to cut corners and deploy code that has not been unit tested, regression tested, stress tested, or tested for deployment. The assumption is, "if it breaks, we will just deploy a hotfix". Unfortunately, this strategy doesn’t work all the time and may result in unscheduled downtime, loss of data, or both. Since ASPs usually have service level agreements with their partners, such unscheduled downtime can be costly.

Of course, most ASPs use version control software such as Visual Source Safe (VSS) and even build management tools, but the tools themselves do not guarantee trouble free deployment if they are not "plugged" into a process. Change management is a combination of complex workflows, and the extended team that implements them – consisting of developers, quality assurance (QA) testers, operation engineers, and database administrators (DBA) – can be compared to an orchestra. Everyone needs to be competent and know their tasks, but they don’t play well together without a conductor. Extending this analogy, we can say that project manager is the conductor, and the process – sheet music.

The goal of this article is not to describe theoretical aspects of change management process, but rather to provide development managers and release administrators some practical advice on organizing efficient change management process. I will review a branching technique for parallel development and show how deployment can be streamlined using pre-production environments. Finally, I will discuss a few deployment guidelines.

Change Management Challenges

I will begin by reviewing specific change management issues and challenges that are typical in an ASP environment. Subsequent sections will describe how these issues can be addressed.

Parallel Development

Most organizations have at least two development teams: one responsible for the maintenance of the production codebase (bug fixes, content changes), and another working on new features. It is hardly unusual on large systems to have three or more development teams each working on their own timeline. The problem is that these teams often share common code and database structures and are thus depend on each other’s changes. Even with two teams sharing a single source control repository and a single development environment, the situation can quickly get out of hand: developers will spend more time troubleshooting than programming, and production deployments will start to fail.

Deployment Problems

The famous programmer’s mantra "it works on my machine" is often laughed at, but deployment issues are widespread in the industry. It could be a missing stored procedure, wrong version of a DLL, absence of third party components, insufficient privileges – a lot of things can go wrong during deployment of a typical distributed web application.

Some of the problems are caused by inherent differences between environments, unaccounted for in the deployment documentation. Consider, for example, a database table that contains thousands of records in development, but hundreds of thousands in production. If a programmer used an Int16 variable to hold an identity column value, the code will not fail until it is pushed to the live environment.

Many issues arise because deployment packages were built using incorrect version of the source code.

Insufficient Testing

While deployment issues are relatively easy to troubleshoot and fix, insufficient testing can lead to much more serious production problems. In a traditional development model, each new release should ideally undergo the following testing routines:

• Functional testing – makes sure the application matches its specifications


• Regression testing – makes sure the existing functionality has not been broken


• Stress testing – makes sure the application can handle the planned workload


• Usability testing – makes sure the application’s user interface is clear and user-friendly

In reality, all of the above tests are rarely performed, either because there aren’t enough QA resources (of course, that’s a project management issue outside of scope of this article) or because there is no suitable test environment. For example, regression testing performed on a single web server in a QA environment may not reveal all issues that can surface in the production web farm.

Consider the common situation where business analysts have to run their test cases in the development environment. These tests are constantly disrupted by code updates, application restarts, and corrupt data. A lot of bugs can be left uncovered and surface at a later stage in the production environment because of this.

Version Control for Parallel Development

I will illustrate version control for parallel development using Visual Source Safe (VSS). VSS is a source control system bundled with Microsoft development environments and therefore the most widely used even though specialized third party software (such as open source Subversion or Serena ChangeMan) have much richer functionality and built-in support for parallel development. Let’s consider an imaginary distributed application for the healthcare industry. The initial release of the system includes two web applications: ERDashboard and PatientBilling which in turn rely on certain components in the application and database tiers. The VSS database screenshot in Figure 1shows all the projects included in the first release.

Figure 1. Initial release of the application

Every time a new build is ready to be deployed for testing, I create a new label which incorporates all the changes made since the last build. Eventually, one of the builds will be approved for deployment to production, for example, Build 4 as shown in Figure 2. Thus, Release 1.0 becomes the production codebase. Several development scenarios can happen at this point.

Figure 2. Label history of Release 1.0

Production Bug

When users report an error in production, it is logged and assigned to a developer to be fixed. Let’s say a developer found a bug in the source code of Page1.aspx.cs in the PatientBilling web application. After the bug is fixed, the build manager can prepare the next build and deploy it. This is very straightforward.

Enhancement to an Existing Project

Let’s assume that our development team received new requirements for the PatientBilling web application. These enhancements will require several weeks of development and testing time. Should programmers use the code under Release 1.0 to make changes and additions? VSS allows this approach, but it will make fixing production bugs difficult. Consider the previous scenario, what if the file Page1.aspx.cs has already been modified with new enhancements? Clearly, this file cannot be included into any build that goes to production until all PatientBilling enhancements are tested and approved. As a result, the production bug fix will be delayed.

Therefore, instead of making enhancements under Release 1.0, it is best to create a brand new project folder (let’s call it Release 1.5) and copy the necessary projects there. Figure 3 shows the new VSS structure: Release 1.0 remains the primary codebase for production support while Release 1.5 becomes the codebase for new development.

Figure 3. VSS structure after share/branching code for the 1.5 release

One important implication for the support programmers is that in addition to fixing bugs in Release 1.0, they also have to apply the same fixes to source code in Release 1.5 (assuming relevant files have been copied there). Indeed, if we only apply the bug fixes to the original Page1.aspx.cs, the bug will be re-introduced when Release 1.5 is deployed. In some cases, source code can change so much between releases that the bug fix is no longer applicable. Usually, however, changes can easily be merged using SourceSafe->Merge Branches command.

Note: In order to merge changes from one branch to another, highlight the target file and select SourceSafe -> Merge Branches command. VSS will display a list of projects that contain different versions of the file. Choose the correct one and click the Merge button. If the merge creates conflicts, VSS checks out the file to the working folder. You can then modify the file and resolve any conflicts before checking the file back into VSS.

There are two possible approaches to branching releases. One is to branch the entire codebase. Alternatively, you can branch just the projects that require modification. The first option is probably more straightforward, but it creates extra work for support programmers since they have to merge their changes to files in all of the new branches. Another reason I prefer the second option is because it provides some visibility into the amount of code affected by a new release. Regardless of the chosen approach, Visual Studio projects should be copied as a whole (I then use the change bindings command from Visual Studio to make sure I’m working with the new source code location). In the case of database scripts or asp files, it is perfectly all right to copy individual files. Always use the share/branch option in VSS, which copies the file and its version history.

Share/branching can be done using the SourceSafe -> Share menu option (as shown in Figure 4). Make sure to check the Branch after share box. Alternatively, drag the folder icon to its desired destination, with right mouse button and choose share/branch from the context menu which pops up when the button is depressed.

Figure 4. Making sure to check the Branch after share box

Switch to the New Production Codebase

Although the process I described above can result in extra work for support programmers, it allows you to completely decouple production support from new development. Release 1.5 can be delayed, merged with 1.6 or cancelled altogether – this doesn’t impact your capability to fix production issues as soon as possible. When enhancements to the PatientBilling application are fully tested and deployed to production, The VSS project Release 1.5 becomes the current production codebase (see Figure 5). At this point I would normally share/branch the entire codebase from Release 1.0 to Release 1.5 (this operation will not overwrite projects that were copied there earlier) and restrict access to Release 1.0. Since share/branching preserves individual file histories, older releases can be archived and deleted from the VSS database. This is important because as the system evolves, whole codebases get replicated over and over again, resulting in a large VSS database size, poor performance, and potential data corruption.

Figure 5. Release 1.5 -now the current production codebase

Adding a New Project

Another typical scenario is creating a completely new project. This can be another web application, a Windows service, middle tier component, etc. Choosing a proper location for a new project’s source code in VSS is important. In my example, Release 1.5 now contains the current production codebase, so I will need to create a new release folder (e.g. Release 2.0).

In the real world, release folders should not be created arbitrarily, they must be derived from the product roadmap and release schedule. Development teams are subsequently notified which release folder to use for their new code projects. If a project is created in an earlier than necessary release folder, there is a risk that code will be deployed prematurely. In addition, source code will have to be share/branched each time the system is switched to a new production codebase (e.g., if project was created in 1.5 instead of 2.0, it will travel from 1.5 to 1.6, then to 1.7, 1.8, 1.9 and finally to 2.0). Not only is this annoying for developers, but also leaves a possibility that someone will forget to update server bindings and end up making changes in the wrong release folder.

Pre-Production Environments

Now that the production codebase is safely separated from the codebase for new projects and enhancements, I will now review the different deployment paths associated with them. A deployment path defines where the code is deployed. Later in this article I will discuss deployment methodology, i.e. how the code can be deployed.

Note: Environment is a combination of physical/virtual servers where an application is deployed. An environment may consist of a single server or have separate web and application farms. All environments should have the same baseline: operating system, 3rd party components, web and database server software.

Bug Fixes

In order to test a production bug fix, you need an environment that is running the same codebase as production. Ideally, that environment should also contain a copy of the production databases, but this isn’t always practical due to information security concerns. Once the fix is validated (usually by a business analyst), it is ready to be deployed to another environment where QA engineers can verify the fix and perform regression testing. Naturally, the QA environment should also be running the production codebase. On a side note, the QA environment should mirror the production environment as much as possible, whether it is hardware, architecture, database size, or the use of SSL. Setting up such environments might be expensive, but it is the only way to avoid unpleasant surprises after deployment to production. The production codebase should be re-applied periodically in order to minimize impact of patches that didn’t make it into production.

In my experience a majority of bug fixes do not require a lot of the developer’s time. Therefore, it is possible to deploy service releases as often as once a week. Figure 6 illustrates this simple deployment path.

Figure 6. Deployment path for bug fixes

In practice however, a development environment is less than the ideal place for the business analyst to test bug fixes. Several developers could be using it at the same time to unit test their changes, so the environment is simply not stable enough. It makes sense to introduce a special purpose pre-production environment, called PP0 ("0" simply indicates that it has the same codebase as production, as in Production Plus 0). Another big benefit is that developers can test their deployment scripts while deploying to PP0. Under the old scheme, deployment to QA was the first time deployment scripts were used.

A valid question would be, why not test the fix in QA only? The main reason is that developers normally have very limited access to servers in a QA environment. While this ensures the integrity of the environment, it becomes very difficult for them to troubleshoot. PP0, on the other hand, should be fully accessible. Figure 7 shows this environment.

Figure 7. Deployment path for bug fixes using the PP0 environment

New Development

Earlier I made a case for separating the production codebase from new code. The development environments for these two codebases should be kept separate for essentially the same reasons. If programmers use a single development environment, they end up overwriting each other’s files and reconfiguring processes all the time. You probably know how frustrating it is to troubleshoot some problem for an hour only to discover that someone else has rolled back your changes. On the hand, separate development environments allow both teams to concentrate on their direct tasks.

New functionality usually requires much more functional testing than bug fixes. Not only are there more test cases, but they are also more complicated and take longer to execute. The development environment, as I mentioned earlier, is too unreliable: servers aren’t stable because of all the development activity and the databases contains a lot of corrupt data or test data. Although it is possible to ask programmers not to update the servers while tests are being run, why would anybody want to waste valuable man-hours? The easier solution is to establish a new pre-production environment, PP1 ("1" indicates that the servers are one version ahead of production, as in Production Plus 1). Figure 8 shows the deployment diagram for enhancements and new projects.

Figure 8. Deployment path for enhancements using the PP1 environment

Combined Deployment Paths

Figure 9 illustrates collaboration between two release processes. The blue arrows follow the deployment steps of a service release, the black arrows show the deployment of new functionality. Note the black lines connecting QA with DEV0 and PP0. Since the production support environments must always reflect the production codebase, it’s easy to see that they should be updated after a successful push of a new release to production.

On the other hand, a successful push of a service release does not require the immediate update of DEV1 and PP1. The reason is simple, all bug fixes that are included into a service release have already been propagated to the source code of the new release (see the section Version Control for Parallel Development). Of course, release managers can choose to deploy service releases to those environments anyway (for consistency). This will revert the codebase back to production and the development teams will have to re-publish their new code.

Figure 9. Combined deployment paths

Secrets of Efficient Deployment

I use the term "deployment" quite often in this article. It is a process of installing code and data on servers within an environment. The deployment process can be lengthy, tricky and error-prone, mainly because it is a collaboration of development, DBA, and operations teams. Developers have the first-hand knowledge about the deployment artifacts, but they rarely have security privileges on servers and databases outside of the development environment. DBA’s and operations, on the other hand, have permissions and are usually in charge of performing deployments, but they rarely have a good understanding of the deployment artifacts and their relationships. How can deployments be made robust, repeatable and stress-free? Below are some guidelines (in no particular order) that can be used to streamline the process in an ASP environment.

Document the Release

This recommendation may sound obvious, but we’ve all seen releases with incomplete or outdated documentation. Release documentation ideally contains the following information:

• Release identification (build number, date, and time)


• List of included bug fixes


• List of included enhancements


• List of deployment artifacts


• Special deployment instructions


• Special post-launch testing instructions


• Special rollback instructions

Special instructions are only needed if standard instructions are not sufficient.

Standardize File System Structure

More often than not, servers in various environments have slightly different folder structures. For example, a development web server may have an ASP.NET application located in C:InetpubwwwrootERDashboard, while the production servers may have it stored in D:CompanyNameWebApplicationsERDashboard. The difference is insignificant from the point of view of operating the website, but when code is being deployed to multiple servers it’s easy to make a mistake and copy files to the wrong location. This layout makes it more challenging to automate deployment using scripts. The best practice would be to have the same file system structure across your server base, at a minimum within the same environment.

Create Deployment Scripts

Requests for deployment usually contain a list of steps that an operations engineer needs to perform, for instance:

• Stop Windows service


• Create directory


• Copy files to new directory


• Register COM objects


• Register .NET assemblies in the GAC


• Create message queues

The longer the list, the better the chance for human error (keep in mind, production deployments are normally performed after hours when engineers can be tired). One way to address this problem is to have a setup project that will generate an installer package. In practice, developers seldom opt for this approach, probably thinking that internal deployment is not worth the effort.

An alternative to an installer package is a deployment script in the form of a command (.BAT) and/or VBScript (.VBS) files. The combination of VBScript and WMI is a very powerful mechanism that can be used to automate practically any deployment task. Because scripts are interpreted, they can be easily tailored to accommodate different steps of a specific deployment. Armed with a solid deployment script, the only thing an operations engineer needs to do is check the output for any error messages. Of course special logic and scripting is required to automatically rollback the release if required.

Prepare Robust SQL Scripts

Database objects are usually deployed using a two-phase approach: restoring a backup and then applying SQL scripts. Sometimes DBAs create the actual script using a list of artifacts provided by developers, other times it is the programmer’s responsibility. There are some specifics related to SQL scripts that are worth mentioning. For example, I always use DROP PROCEDURE / CREATE PROCEDURE combination instead of ALTER PROCEDURE, because this updates the timestamp of the object. Sequence is important, too, a script that creates a new table should precede the script for the stored procedure that uses the table.

Also, scripts should be repeatable. If it generates errors during execution, a DBA will reapply the script after correcting the errors. Therefore, instead of writing:

INSERT INTO dbo.MyTable(SomeID, SomeName) VALUES (5,  ~Riyad Mammadov’)

I would write:

IF(FROM dbo.MyTable WHERE SomeID = 5">EXISTS)

BEGIN
   INSERT INTO dbo.MyTable(SomeID, SomeName) VALUES (5, ‘Riyad Mammadov’)
   PRINT ‘Added new value’

END

ELSE
	PRINT  ~Value already exists!’

The second SQL script can be safely executed over and over again.

Tracking Deployment Artifacts

All our efforts to automate deployment using scripts will be pointless if the binaries are compiled using incorrect versions of individual source code files. Therefore, it is imperative to properly notify the build manager of the exact file names and their VSS versions. This is a simple rule, but with dozens of projects and hundreds of files in your Visual Studio solution it’s easy to make a mistake.

Ultimately, individual programmers are responsible for submitting the list, either on a simple paper form, or using more sophisticated build management software that links to a source code repository. VSS, for example, can generate a list of files modified with a date range by a specific user. This is done using the Project History Options dialog (shown in Figure 10) which can be invoked directly from Visual Studio by clicking the History button on the Source Control toolbar.

Figure 10. The Project History Options dialog box

Using a Separate VSS Database for Deployment

Even companies that have fairly sophisticated version and build control systems sometimes rely on a primitive file server directory structure to exchange binaries between development and operations. A share on the file server, such as \FileServerBuilds, is used to store release code. Drawbacks to this approach are evident: it’s easy to misplace a file, and it’s impossible to ensure validity of the build. For example, this approach makes it entirely possible to deploy a build to QA and then modify (accidentally or deliberately) a DLL inside the folder. As a result, deployment to Production will contain a new untested version of the file, and the entire change management process will become meaningless.

Fortunately, there is a simple yet powerful solution: a separate VSS database for deployment. This database should be used to store deployment artifacts such as binary code, static web content (HTML, images), and deployment scripts, including SQL. Figure 11 shows a sample project structure. It makes sense to restrict access to the new VSS, only build managers, operations engineers and DBAs should be able to work with it. This is how VSS version tracking features can ensure that deployment artifacts have not been "tampered with" between releases.

Figure 11. Deployment VSS database

When a build has been tested in the PP0 or PP1 environments and is declared ready for QA, the build manager creates a new label in the deployment VSS. The operations engineer retrieves the deployment artifacts straight from the VSS using a specific label name. All subsequent changes to deployment artifacts (e.g. to address an issue found during regression testing) will become part of the next label. Ultimately, one of the labels will be approved for deployment to production.

Conclusion

In this article I have reviewed three major change management problems that affect to a varying degree all Application Service Providers – version conflicts between production support and new development projects, insufficient testing, and deployment issues. These problems create significant risk of data loss and unscheduled downtime in the production environment.

I also demonstrated how the risk can be mitigated using a combination of process, hardware and tools. Parallel development can be addressed by clearly separating current production codebase from future release code. Separate development and pre-production environments for both codebases create the infrastructure to run additional tests and safely deploy compiled binaries.

Finally, I showed a few techniques to streamline the deployment process, most important of them is the introduction of a dedicated deployment VSS database.