Building a Deployment Pipeline with MSDeploy: Part 4 – Server Configuration

So far we’ve been looking at the development experience of a deployment pipeline, but before we go any further we need to configure where the site will be deployed to. For this, we’ll need an instance of Windows Server 2008 or higher.

Remote deployment model

MSDeploy supports three methods of remote deployment:

  1. Web Management Service (WMSvc) handler
    This is the preferred method for IIS 7+ / Windows Server 2008+ and supports non-administrator deployments. It piggybacks on the WMSvc (ie. remote IIS management) by registering a custom handler (http://server:8172/msdeploy.axd). Making use of WMSvc also means that this method is not available to Windows client versions (including 7 and 8), as WMSvc is not available on those platforms.
     At the command line, this method is specified on the dest provider as “,computerName=http://server:8172/msdeploy.axd?site=iis-site-name”
  2. Web Deploy Agent Service
    This is the only choice for IIS6 / Server 2003, is not installed by default, and requires the deployment user to be an administrator. At the comment line, this method is specified as “,computerName=server
  3. “On Demand” Agent Service (temp agent)
    This choice installs the agent service temporarily for a single deployment. Useful when you have administrator credentials but aren’t able to remotely install the handler or agent service. At the command line, this method is specified as “,computerName=server,tempAgent=true”

This series will focus on the WMSvc handler method. Any reference to non-administrator users and subsequent feature delegation is only relevant to this method, since the others require administrator deployment users.

For more information on the various methods of remote deployment, see Using Web Deploy Remotely on TechNet.

Selective deployment

All three methods of remote connection will only transfer content that has deemed to be changed (last modified date by default, checksum as an option). This is achieved by first performing a metadata exchange of the deployment providers, and then streaming the data that needs to be modified.

As an aside, the protocol between the MSDeploy client (API or exe) and the remote service is undocumented and subsequent to change.

Installing MSDeploy

The official instructions for installing MSDeploy on a server can be found here: http://www.iis.net/learn/install/installing-publishing-technologies/installing-and-configuring-web-deploy

However, there are several things to watch out for during the installation process:

  • Ignore the screenshots – while they aren’t wrong per se, they are misleading. Just install “Web Deploy V3″ (or whatever the latest version is).
  • After installation is complete, make sure the WDeployConfigWriter and WDeployAdmin users are marked as “Password never expires”. Your local security policy may select otherwise, which will cause deployments to stop working when the passwords (which are never supplied to you) expire.

Configuring a deployment user

Assuming you have your IIS site already configured, we need to create a non-adminstrator user that can publish to it. We’ll call that user “SERVER\deploy_user”, but the naming (and whether it’s a domain or local user) is up to you. In fact, you can even create an IIS user (which is not a Windows user), but I won’t be covering that here.

Once your user is created, we need to grant it permission to deploy the site. Right click on your website and select “Configure for Web Deploy Publishing…” from the “Deploy” sub menu. This does two things:

  1. Grants the user full control to the file system of that website
  2. Adds the user to the access list for that website

Select your deployment user and click “Setup”, ignoring the other fields. This process will create a “.PublishSettings” file on the desktop, but as this relates to previous incarnations of the deployment technology it can (and should) be safely deleted.

Feature Delegation

A fair question to ask at this point is: if the deployment user is not an adminsitrator, how are they able to perform additional tasks like creating virtual directories?

The answer is called feature delegation. The specifics won’t be discussed until a future post, but if you access “Management Service Delegation” from the server level features in IIS you’ll see that individual MSDeploy providers are configured to either use the “current” (ie. deploying) user, WDeployConfigWriter (who can write to IIS’s config), WDeployAdmin (who is in the administrators group) or a custom user.

Delegation rules can be further configured to prevent certain users from using specific providers, or alternatively raise the privileges for certain providers when used for a particular site. This allows authorization to be fine tuned without granting the deploying user administrative privileges.

Posted in Deployment | Tagged , , | Leave a comment

Demystifying MSDeploy skip rules

Skip rules are a feature of MSDeploy that allow specific parts of a synchronisation to be, you guessed it, skipped. While a conceptually simple concept, there are subtleties in its options that have resulted in “bug reports” from a number of people (myself included) that were actually just misunderstandings of its design.

The aim of this post is to demystify skip rules in hope of easing frustration and preventing accidental deletes in production environments.

Anatomy of a skip rule

The command line syntax of a skip rule is:

-skip:attribute1=value1[,attribute2=value2[.. ,attributeN=valueN]]

Attributes act as a filter to figure out what to skip. They can all be specified in a single skip, but the same attribute cannot be specified more than once.

The main attributes are:

  • objectName is the provider being skipped (eg. dirPath)
  • absolutePath is a regular expression (don’t forget to escape it!) of the provider path being skipped (eg. file system path for filePath/dirPath)
  • skipAction specified under what circumstances should the object be skipped (eg. Delete, Update)

The other, less common, attributes are:

  • xPath is an XPath expression that locates the object within the provider hierarchy
  • attributes.attributeName filters on an arbitrary provider attribute but cannot be used with skipAction
  • keyAttribute filters a regular expression against the providers “key attribute”. Since the key attribute is “path” more often that not, you’re unlikely to use it. It also cannot be used with skipAction.

Now that we know what a skip rules are, let’s take a look at how they work.

Skip rules and hierarchies

At its core, MSDeploy only supports synchronising a single provider from a source to a target, it’s just that providers actually exist in hierarchy, each with a very specific purpose. For example, dirPath contains other dirPath and filePath providers, iisApp contains a contentPath and a createApp provider, and manifest/package/archiveDir all contain abitrary providers.

During synchronisation, the metadata for source\target provider hierarchies from the source and target are loaded into memory and compared. Based on that comparison, items from the source are added, updated or deleted at the destination.

The important thing to understand with regard to hierarchies is that <strong>delete rules on a child are only processed if the parent is not being deleted</strong>. So if you skip a file but it’s containing directory doesn’t exist on the source, the directory (and thus the file) will be deleted anyway.

Skip Actions

  • Delete prevents the object being deleted from the destination (unless its parent is being deleted as mentioned above)
  • AddChild prevents the object from being added to the destination
  • Update prevents the destination object from being updated

Omitting a skipAction entirely prevents the object’s metadata from even being considered for comparison and is processed at a separate phase than skip directives with skip actions. While this may seem like skipAction=*, in reality it’s almost never what you want. Since each provider in the hierarchy is responsible for it’s own deletion, skipping a child without a skipAction will cause the parent to fail during a delete action. For example, deleting a dirPath with an omitted child filePath results in a “Directory is not empty” error because the child filePath was not there to handle it’s deletion.

Examples

Let’s see how this all fits together in a real life test. We’re going to sync these two folders:

source dest
Source Destination

Below are the results of the various operations options. I’ve added emphasis on the important bits, and strikethrough means I’m pointing out that the line is missing:

Skip attributes Result
None Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).
Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
objectName=filePath
asbolutePath=data\.bin$
Info: Object filePath (App_Data\data.bin) skipped due to skip directive ‘CommandLine
SkipDirective 1′.
Info: Deleting directory (App_Data).

Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
Error: An error was encountered when processing operation ‘Delete Directory’ on ‘App_Data’.
Error: The directory is not empty
objectName=filePath
asbolutePath=data\.bin$
skipAction=Delete
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).

Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
objectName=dirPath
asbolutePath=App_Data$
skipAction=Delete
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).

Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
objectName=filePath
asbolutePath=Index\.cshtml$
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).
Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Object filePath (Views\Home\Index.cshtml) skipped due to skip directive ‘CommandLineSkipDirective 1′.
Info: Updating file (Web.config).
objectName=filePath
asbolutePath=Index\.cshtml$
skipAction=AddChild
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).
Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
objectName=filePath
asbolutePath=Index\.cshtml$
skipAction=Update
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).
Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
objectName=filePath
asbolutePath=Web\.config$
skipAction=Update
Info: Deleting file (App_Data\data.bin).
Info: Deleting directory (App_Data).
Info: Adding directory (bin).
Info: Adding file (bin\App.dll).
Info: Adding directory (Views).
Info: Adding directory (Views\Home).
Info: Adding file (Views\Home\Index.cshtml).
Info: Updating file (Web.config).
Posted in Deployment | Tagged | 1 Comment

Configuring Fiddler to compress requests

Applying Content-Encoding headers to a request is tricky subject. While it’s part of the HTTP specification, there’s no way to determine if a server supports it ahead of time since no communication occurs before a request. Still if the server/client are controlled, request encoding can come in handy for certain scenarios. One such scenario that’s becoming more common is uploading log data from a mobile application.

If you use Fiddler to debug your HTTP, you might have spotted that there’s no way to compress the request body. Fortunately, since Fiddler is so extensible, adding support is easy.

First up, open the CustomRules script by selecting “Customize Rules…” from the Edit menu (or by pressing CTRL+R).

Now paste the following code just before the lines adding RulesStringValue entries for each browser (around Line 50 of an uncustomised script):

// Request GZip encoding
public static RulesOption("Apply GZIP Encoding to Request")
var m_gzipRequest: boolean = false;

Next, paste the following code at the end of OnBeforeRequest(), right after the “if (m_AlwaysFresh && …)” block (around Line 210):

if (m_gzipRequest && oSession.requestBodyBytes != null &&
    oSession.requestBodyBytes.length > 0 && !oSession.oRequest.headers["Content-Encoding"])
{
    oSession.requestBodyBytes = Utilities.GzipCompress(oSession.requestBodyBytes);
    oSession.oRequest.headers["Content-Length"] = oSession.requestBodyBytes.Length.ToString();
    oSession.oRequest.headers["Content-Encoding"] = "gzip";
}

Now whenever you want request bodies to be compressed, check “Apply GZIP Encoding to Request” from the Rules menu. Don’t forget to turn it off again!

Posted in Debugging | Leave a comment

Building a Deployment Pipeline with MSDeploy: Part 3 – Setup and Packaging

I’m continuing my series on building a deployment pipeline. This post gets the ball rolling by creating the project and defining the first command in our deployment pipeline.

Software Requirements

To contnue with this series, you’ll need one of two things:

  • Visual Studio 2012
  • Visual Studio 2010 with the Azure SDK 1.71+ (currently 1.8)

File New Project and File Structure

Create a new MVC project called “MvcApplication”. Here’s an overview of the structure and important files (some won’t exist yet) for publishing:

  • MvcApplication.sln
  • MvcApplication/
    • Properties.xml
    • MvcApplication.wpp.targets
    • Properties/
      • PublishProfiles/
        • Package.pubxml
        • Stage.pubxml

All the bold items are special in their naming convention, but we’ll describe their purpose as we need them.

Packaging

While WPP still supports the “old way” of packaging (CreatePackageOnPublish=true), the “correct” way to do so is to define a “package” publish profile. To do that:

  1. Right click on the project
  2. Click “Publish…”
  3. Select “<New…>” from the publish profile drop down, call it “Package”
  4. Change the publish method to “Web Deploy Package”
  5. Create a new publish profile called “Package” with an action of “Package”

You’ll see that a new file, Properties\PublishProfiles\Package.pubxml, has been created. We’ll just leave that alone for now to keep things simple, but we’ll be returning here in a future post.

Now we define the first command in our build pipeline:

msbuild MvcApplication.sln /t:Build /p:DeployOnBuild=true;PublishProfile=Package;IsDesktopBuild=false

Now for some options:

  • In Visual Studio 2012, you can change the target to “Publish” and remove “DeployOnBuild” (2010 only supports limited targets for sln files)
  • PublishProfile can be a full path to a pubxml file. If not it will be found in “Properties\PublishProfiles\Package.pubxml”

You also have a few choices about where the zip is generated:

  • By default, the package will be created in bin\_PublishedWebsites\MvcApplication_Package
  • Defining “DefaultPackageOutputDir” to a directory that will contain MvcApplication.zip
  • Defining “PackageFileName” to a full path to the target zip
  • Remove “IsDesktopBuild” from the build command and it will revert to using the absolute package you entered when creating the publish profile (“DesktopBuildPackageLocation”)

What’s in the box!!?

Let’s crack open MvcApplication.zip and see what we’ve created.

Manifest

The archive.xml file contains the list of providers that will be deployed with the application.

What you’ll probably see here is an iisApp provider (deploys content and creates an IIS “application” if the directory isn’t one already), and two setAcl providers (one sets read permissions for the App Pool identity and the other for static file access – IUSR).

This file isn’t modifyable in it’s current (in-zip) state, but understanding it means you can use it to diagnose issues in the future.

Parameters

The properties.xml file contains any deploy-time arguments and how they effect the deployment. In our scenario, WPP has generated two by convention:

“IIS Web Application Name” is the name of the website (or website\virtdir) that we’re going to deploy to. If you look carefully, you’ll see that the replacement value will be used for both the content deployment and the setAcl provider.

“ApplicationServices-Web.config Connection String” is the connection string value for ApplicationServices (one parameter will be generated per <connectionStrings> entry)

Content

If you navigate into “Content\C_C” and it’s children, you’ll see that the website content has been added into the package with the full path intact. There’s also nothing you can do about this, so keep it in mind in case you write obsenities into your directory structures or something.

It’s important to note here, that the content complies with Visual Studio’s “Items to deploy” in the “Package/Publish Web” tab of project properties. By default, this includes binary output and any files with a Build Action of “Content”.

System Info

Finally, the systemInfo.xml file contains information on which .NET Framework versions and IIS components are installed on the source machine. I’m not 100% sure how this information is used, but I haven’t needed to look at this file for any diagnostics so far.

Posted in Deployment | Tagged , , | 4 Comments

Building a Deployment Pipeline with MSDeploy: Part 2 – Terminology Primer

I’m continuing my series on building a deployment pipeline. This post clears up some terminology and defines the basic concepts.

MSDeploy

Officially called Web Deploy, MSDeploy is a technology for primarily deploying web applications and their dependencies, though you could use it to deploy almost anything. Currently in it’s third version, it’s distributed as an API with an accompanying command line tool, the latter of which won’t be covered in this series.

MSDeploy defines the following concepts:

  • Providers deploy certain types of objects, be it a website, a database or a directory.
  • Verbs indicate which operation should be performed on the source and dest providers. Usually “sync”, though “delete” and “dump” can be useful.
  • Parameters define what aspects of a deployment can be changed and can perform anything from web.config modifications to changing the name of the website being deployed to.
  • Rules modify how providers deploy, typically dealing with specific scenarios. Examples include disabling deletes and preventing “harmful” deletes.
  • Link Extensions (also called Links) enable or disable whether certain objects are included with their associated providers. Examples include application pools, certificates, and files.
  • Skip Rules1 allow specific deployment objects (or only specific actions to a deployment object) to be excluded from the deployment.
  • Replace Rules1 modify aspects of any deployment object (except file contents).

1 Technically Skip and Replace are implemented as rules, but the command line syntax is different so it’s easier to consider them separate.

Web Publishing Pipeline

The Web Publishing Pipeline (hereafter be referred to as WPP) is a set of MSBuild targets that Visual Studio 2010/12 uses to integrate it’s “Publish” process with MSDeploy (among other things).

WPP’s integration with MSBuild and Visual Studio provide a number of perks:

  • Publish Profiles are MSBuild files with a pubxml extension and contain all the settings required to deploy to a specific environment.
  • MSBuild’s “convention over configuration” is brought to the packaging and deployment process

There are some downsides, though:

  • Extending the deployment process requires both an understanding of the base MSDeploy feature and how to get to it via WPP.
  • WPP doesn’t support all the features of MSDeploy (specifically certain command line options).
  • It’s primarily been designed around “One Click Publishing” via the Visual Studio IDE

Still, this deployment pipeline makes use of WPP, simply because Publish Profiles are much more convenient for managing target environments and less jarring for developers new to the technology.

Posted in Deployment | Tagged , , | Leave a comment

Building a Deployment Pipeline with MSDeploy: Part 1 – Introduction

In this series of posts, I’ll be walking through how to create a deployment pipeline using MSDeploy and the Web Publishing Pipeline.

What is a Deployment Pipeline?

A Deployment Pipeline is one of the tenants of Continuous Delivery, the practice of always being able to release new software (but not necessarily actually doing it). In a Deployment Pipeline, a single package is created at the end of the build/test job and deployed to each environment, (possibly) all the way up into production. And then we start again for the next release.

It’s important that the package be the same so that the “thing you deploy is the thing you tested”.

If you’d like you learn more, Jez Humble and David Farley quite literally wrote the book on Continuous Delivery, which I cannot recommend enough.

Series Goals

This series aims for a deployment pipeline the following attributes:

  • Code is only retrieved from source control once
  • Code is only compiled once
  • Deployment operations are performed by non-administrator users
  • Deployment user credentials are not stored in source control or CI configuration
  • Per-environment configuration can be stored in the solution or supplied at runtime

Databases

Due to the complexity of deploying database, I’ve decided to leave them out of this series. However, since it’s an important topic you can be sure it will be brought up in a future article.

Posted in Deployment | Tagged , , | Leave a comment

Declaring optional MSDeploy parameters using the Web Publishing Pipeline

Today’s WPP vs MSDeploy issue is declaring parameters that need to be ignored at deployment time.

Why would I want to declare a parameter, only to ignore it? I’m a big fan of continuous delivery, and one of it’s tenants is to only create build artifacts once and deploy them to various environments (dev, uat, staging, live), as it guarantees that what you deploy is what you tested.

MSDeploy actually provides two (subtlely different) ways of skipping a parameter. The removeParam argument can be used to remove the parameter definition for the current deployment. Alternatively, the parameter can be decorated with a <parameterValidation kind="AllowEmpty" /> element to allow parameter values of empty strings.

Unfortunately, neither of these MSDeploy features are supported by the Web Publish Pipeline.

The only workaround for removeParam is to perform your deployments using MSDeploy directly, since there’s no way to specify custom command line arguments via the MSDeploy (or VsMsDeploy) tasks.

Fortunately, there is a workaround for the parameterValidation method, which ended up being what I needed. I found the workaround after snooping through Microsoft.Web.Publishing.targets (in %programfiles%\MSBuild\VisualStudio\v11.0\Web), a wealth of obfuscated knowledge on WPP features that are not-so-documented.

It turns out that WPP will actually load a Parameters.xml file (technically $(ProjectParametersXMLFile)) from the root of the web application and merge any parameters with those declared in WPP (including Project.wpp.targets) using MSDeployDeclareParameters. For example, the following Parameters.xml declares an optional text-replace parameter:

<parameters> 
   <parameter name="ReplaceVariable" 
              description="Sample variable that allows empty values" defaultValue="">
      <parameterValidation kind="AllowEmpty" />
      <parameterEntry type="TextFile" scope="Web\.config$" match="TextToReplace" /> 
    </parameter> 
</parameters> 

You can now deploy the application while specifying an empty parameter value for ReplaceVariable

External links:

Posted in Deployment | Tagged , | 2 Comments