Richard Szalay's Blog

Sitecore App Service Warm-Up Demystified

Wed, 03 Jul 2019 05:35:16 +0000

It’s well known the Application Initialization IIS module can assist in “warming up” an application. The problem this feature has always faced (for me, at least) is that its integration with Azure is not particularly well documented, and mostly limited to occasional forum posts that indicate deeper integration with the App Service load balancer without providing the detail.

That is, until now. Microsoft was kind enough to direct me to App Service Warm-Up Demystified by Michael Candido, a Principal Software Engineering Manager on the App Service team. This blog post goes into an incredible amount of detail about how the feature works in App Service. I’m not going to try to reproduce Michael’s blog post here, and I do consider it required reading if using this feature, but I’d like to summarise the feature’s behaviour in areas that directly affect Sitecore.

tl;dr

To reduce/remove any “cold” start delays to end users:

Use deployment slots for blue/green deployments
Use Application Initialization to warm up a specific resources for scale and maintenance events
Use Advanced Application Restart to restart the application

Things to be aware of when using Application Initialization:

Make sure your warmup calls can actually reach the target Sitecore “site” using specific hostName and/or URL rewrite rules
Prevent caching of unintended absolute URLs (eg. “http://localhost”) by using relative URLs or setting the targetHostname and scheme properties on the target site
Make sure your application can warm up (including app init URLs) in under 10 minutes

Overview

Generally speaking, the Application Initialization module simply requests a number of URLs sequentially when the application pool starts. It does so in the background, without interfering with traditional requests.

Azure App Service, however, provides additional support for this feature by preventing instances from receiving traffic until after all of the initialization URLs have been loaded sequentially.

This prevention applies in the following scenarios:

Adding new instances via scale out (either manually or autoscale)
Swapping deployments slots
Instances being taken in and out for maintenance
Advanced Application Restart (in “Diagnose and Solve Problems”)

It does not prevent requests from being sent to an instance in the following scenarios:

Deployments (use deployment slots)
Scaling up instances (there’s a feature request for this)
Restarting your application via the “Restart” option (use Advanced Application Restart instead)
Applications not running in Azure App Service (on-prem, Azure IaaS, etc)

Sitecore integration

There are a number of things to be aware of when using Application Initialization with Sitecore:

Any response from an initialization URL is sufficient, even 302 (Redirect) or 500 (Error)
All app initialization requests are sent using “localhost” unless configured otherwise
All app initialization requests are sent using HTTP (ie. not HTTPS)
The instance will be torn down if the application doesn’t restart in 10 minute (30 in some circumstances)

If not planned for, these nuances can result in instances receiving traffic too early because all of the requests resulted in not-found errors or redirects without actually warming anything up.

(Once receiving traffic, external requests will time out after 230 seconds and return a 502 status code. If you’re seeing this, it’s likely that your initialization URLs aren’t executing properly.)

Let’s look at these issues and how you can work around them.

Localhost

Assuming your target site isn’t configured to match localhost, the request will likely result in a 404 (or a 302 to the Not Found page). There are a number of things to consider when fixing that:

Whether to target the site via the host header or override the site via ?sc_site
Whether to configure the target site on each initialization URL, or ‘globally’ via URL Rewrite

My personal preference is to rewrite ?sc_site, since it’s portable between environments, using either Rewrite rule if there’s only one target site, or on each warmup URL if there are multiple sites being.

However, since your requirements may necessitate any combination of these options, I’ll provide configuration samples for each.

Using `<applicationInitialization>`

<system.webServer>
  <applicationInitialization doAppInitAfterRestart="true">
    <!-- Via site name -->
    <add initializationPage="/?sc_site=veryexcellent" />

    <!-- Via host name -->
    <add initializationPage="/" hostName="veryexcellentsite.com" />
  </applicationInitialization>
</system.webServer>

Using `<rules>`

NOTE: Modifying the host using a rewrite rule requires that you add HTTP_HOST to system.webServer/rewrite/allowedServerVariables via an applicationHost.xdt transform

Remember that rules are processed in order, so this will need to be first.

<rules>
  <!-- Via site name -->
  <rule name="Change warmup site" stopProcessing="true">
    <match url="(.*)" />
    <conditions>
      <add input="{HTTP_USER_AGENT}" pattern="^IIS Application Initialization Warmup$" />
      <add input="{REMOTE_ADDR}" pattern="^127\.0\.0\.\d+$" />
    </conditions>
    <action type="Rewrite" url="{R:0}?sc_site=veryexcellent" appendQueryString="true" />
  </rule>

  <!-- Via host name -->
  <rule name="Change warmup host" stopProcessing="true">
    <match url=".*" />
    <conditions>
      <add input="{HTTP_USER_AGENT}" pattern="^IIS Application Initialization Warmup$" />
      <add input="{REMOTE_ADDR}" pattern="^127\.0\.0\.\d+$" />
    </conditions>
    <serverVariables>
      <set name="HTTP_HOST" value="veryexcellentsite.com" />
    </serverVariables>
  </rule>
</rules>

HTTP

HTTPS can’t be rewritten via a rule, but if you have an HTTP->HTTPS redirect you’ll need to inject an earlier rule that allows the warmup request to go through unsecured (the 302 won’t be followed, and will be treated as a success response).

NOTE: If you’re using a rule to apply the site/host, you can simply set stopProcessing="true" on that and this rule won’t be required.

<rule name="No HTTP->HTTPS" stopProcessing="true">
  <match url=".*" />
    <conditions>
      <add input="{HTTP_USER_AGENT}" pattern="^IIS Application Initialization Warmup$" />
      <add input="{REMOTE_ADDR}" pattern="^127\.0\.0\.\d+$" />
    </conditions>
  <action type="None" />
</rule>

Cached Renderings with Absolute Links

If you have any cached renderings that use absolute links and you use AlwaysIncludeServerUrl (either in those renderings, or globally), make sure you set the following in your site configuration so that your links aren’t cached as “http://localhost/*”:

targetHostname
scheme (if using https)

Time limit

Finally, the application has 10 minutes to complete warmup before the VM is shut down. This might be as long as 30 minutes, but as it’s not reliable it’s best to aim for 10 minutes.

Individual warmup requests do not have a timeout applied so you can use them to “wait” for certain background processes to complete, if required.

I won’t cover startup optimisation as it’s a large topic on its own, but here’s two easy things that can help:

Ensure the site is configured to use Roslyn (Microsoft.CodeDom.Providers.DotNetCompilerPlatform) as the cshtml compiler rather than the default. I’ve seen startup time drop by 3-4 minutes doing this
Precompile views (for example, using RazorGenerator.MsBuild) and registering the assembly with Sitecore. Should should drop another 30-120 seconds from the startup time.

Debugging Warm-Up Requests

Warm-up requests sent by the Application Initialization module don’t appear in the IIS logs. They will be logged by Failed Request Tracing, but there’s also no way to filter this logging specifically for warmup requests.

One strategy is to create a dummy page that you can send ‘marker’ requests to:

<applicationInitialization>
  <add initializationPage="/Init.aspx?phase=home" />
  <add initializationPage="/?sc_site=awesome" />

  <add initializationPage="/Init.aspx?phase=landing" />
  <add initializationPage="/first?sc_site=awesome" />
  <add initializationPage="/second?sc_site=awesome" />
  <add initializationPage="/third?sc_site=awesome" />
</applicationInitialization>

This allows you to filter Failed Request Tracing to this page and you can review the logs to compare timestamps. If there is little to no time between marker requests, there’s probably errors occuring for the intervening requests.

You can also identify requests made by/during warmup:

Warmup requests will have a user agent of IIS Application Initialization Warmup
Regular/external requests made while warmup is in progress will have a APP_WARMING_UP server variable set to 1

Easily debug Sitecore assemblies using dnSpy

Fri, 14 Jun 2019 07:37:16 +0000

Needing to debug into Sitecore assemblies isn’t uncommon, whether it be to track down the cause of a bug or simply understand a pipeline better. Traditionally, however, this process has been somewhat complicated by a number of factors:

The JIT generates optimized code for assemblies without PDBs
IIS ‘assembly shadow copy’ behaviour makes disabling JIT optimizations difficult
Debugging without commercial tools like dotPeek is limited to either low-level IL or basic stack information

This guide attempts to alleviate all three of these barriers with free and open source tools.

Step 1: Enable debugging for Sitecore assemblies

Before we can attach a debugger, we need to solve two problems #1 and #2 above:

As outlined elsewhere (such as Anton’s excellent post), assembly optimizations can be disabled by adding an .ini file for each relevant assembly with the following contents:

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

Assembly shadow copying can feature can be temporarily disabled by setting system.web/hostingEnvironment/@shadowCopyBinAssemblies to false

Making the changes above (and reverting them afterwards) is busy work that you’ll unlikely be wanting to do when debugging a problem, so I’ve automated the process into a simple PowerShell module (gist)

Import-Module .\Downloads\IISAssemblyDebugging.psm1
Enable-IISAssemblyDebugging c:\inetpub\wwwroot\sitecore.local

# Alternatively, you can disable optimisations for a limit subset:
Enable-IISAssemblyDebugging c:\inetpub\wwwroot\sitecore.local -Filter "Sitecore*.dll"

Step 2: Debug using dnSpy

dnSpy is an excellent, open source, standalone .NET debugger that uses ILSpy to allow debugging into decompiled assemblies.

To begin, download dnSpy and run dnSpy.exe (as Administrator, since we’ll be attaching to IIS)

Next, select “Attach to Process” from the debug menu (or press CTRL+ALT+P) and select the appropriate w3wp.exe process (the Application Pool can be found in the “Command Line” column)

Now, select “Close All” from the File menu to clear the assembly list and then “Open” to select the assemblies you wish to debug.

Finally, set breakpoints and refresh the page in your browser. All the usual debugging views (Locals, Call Stack, etc) are available in the Debug :: Windows menu.

Step 3: Disable debugging for Sitecore assemblies

Once we’re done, revert the .ini and shadow copy configuration changes we made earlier:

Disable-IISAssemblyDebugging c:\inetpub\wwwroot\sitecore.local

You will also need to recycle your app pool to release the lock on the assemblies. Here’s an example using the IISAdministration module:

(Get-IISAppPool sitecore).Recycle()

Diagnosing missing package assemblies in Helix solutions

Sun, 21 Apr 2019 00:00:00 +0000

There’s a issue I’ve seen come up again and again when publishing Helix solutions: the assemblies from a NuGet package (usually Unicorn) referenced by a Feature/Foundation module are not included when the Project is published, causing errors during deployment or at runtime. For example:

error CS0246: The type or namespace name ‘Unicorn’ could not be found (are you missing a using directive or an assembly reference?)

To understand why this happens, we need to look at how assembly resolution works during the build process. For the project being built, the ResolveAssemblyReferences target is used to collect all of the assembly, project, and nuget references. So far, so good.

The issue of missing assemblies comes into play when dealing with downstream references, like the references of referenced projects. For these, the output assemblies (not projects) are analysed for their references which are then all gathered together to be included in the building-project’s output.

This works as expected for libraries like Glass, where the assemblies names and types are used directly by the project’s code and so necessitates an assembly reference in that project’s output. Packages like Unicorn cause a problem because in most scenarios the Unicorn assemblies will be required by config files but not actually referenced in code. This precludes them from being included as assembly references and are therefore ignored when it comes to publishing the upstream project.

Here’s a crude visualisation of this process:

There are a number of ways this problem can be addressed:

Reference the package directly from the project that you are publishing
Have a project reference types from the package’s assembly (make sure they’re meaningful so they don’t get optimized away)

Helix Publishing Pipeline actually automatically applies a third technique: extend the build process and invoke ResolveAssemblyReferences on all referenced projects so that their “soft” references can be included. Below is a more general purpose port of the implementation:

<!-- Add this to your pubxml or wpp.targets to have all references from referenced projects -->
<Project>
  <PropertyGroup>

    <ResolveAssemblyReferencesDependsOn>
      $(ResolveAssemblyReferencesDependsOn);
      CollectReferencesFromProjectReferences;
    </ResolveAssemblyReferencesDependsOn>

  </PropertyGroup>

  <Target Name="CollectReferencesFromProjectReferences">

    <MSBuild BuildInParallel="$(BuildInParallel)" Projects="@(ProjectReference)" RemoveProperties="DeployOnBuild;PublishProfile" Targets="ResolveAssemblyReferences">
      <Output ItemName="_ReferencesFromProjectReferences" TaskParameter="TargetOutputs"/>
    </MSBuild>

    <ItemGroup>
      <ReferencesFromProjectReferences Include="@(_ReferencesFromProjectReferences)" Condition="'%(_ReferencesFromProjectReferences.CopyLocal)'=='true'"/>

      <Reference Include="%(ReferencesFromProjectReferences.FusionName)">
        <ResolvedFrom>%(ReferencesFromProjectReferences.ResolvedFrom)</ResolvedFrom>
        <HintPath>%(ReferencesFromProjectReferences.FullPath)</HintPath>
        <Redist>%(ReferencesFromProjectReferences.Redist)</Redist>
        <CopyLocal>true</CopyLocal>
      </Reference>
    </ItemGroup>
    
  </Target>

</Project>

Reentrancy improvements to Minions in Sitecore Commerce 9.1

Wed, 17 Apr 2019 01:37:16 +0000

tl;dr In XC91, you’ll no longer need to worry about how long your Minion takes to run. Just override Minion.Execute rather than Minion.Run and you’re good to go.

Back in March of last year, I responded to StackExchange question on the limitations of Sitecore Commerce’s Minion in terms of reentrancy. Put more plainly, in XC9.0 Minions used a “fire and forget” approach for starting each ‘invocation’ which resulted in Minions running twice in parallel when the previous invocation took longer to execute than the configured WakeupInterval.

In my answer I described two workarounds to the problem, both having a subtle impact on the behavior of Minions:

Detect reentrant calls using Interlocked.CompareExchange and skip them, which may result in an “invocation” being skipped entirely
Delay the WakeupInterval pause until after the invocation is complete, which moves the pause from the “start” of the invocation to the end

The recently released Commerce 9.1 aims to tackle this issue of reentrancy and the implementation actually takes a little from both of the approaches above, as well as handling some additional scenarios. Let’s start with an overview of how Minions are started in XC9.0:

StartEnvironmentMinionsBlock
  Minion.Start()
    loop forever
      Minion.Run (fire and forget) <-- RunMinion calls this
      WakeupInterval (blocking)

And here’s a visualisation to illustrate the problem this causes:

Minion execution flow in Sitecore Commerce 9.0

As you can see, XC 9.0 starts it’s WakeupInterval delay from the moment Minion.Run is invoked. This is actually fine… for some configurations. The problem arises when Run takes longer than WakeupInterval, and results in Run being executed twice in parallel. If a Minion was not written with parallel execution in mind, it can cause some serious problems.

I’m happy to report that XC 9.1 fixes all of the flaws in the original design. Starting with the issue described above, Minions now apply the WakeupInterval delay after the main method (now called Execute, with Run being marked as “obsolete”) completes asynchronously:

Minion execution flow in Sitecore Commerce 9.1

As you can see, it’s now impossible for Execute to be invoked twice in parallel.

The changes don’t stop there, though. In XC90 it will also possible to start a minion via the RunMinion API while it was already running, resulting in it running twice in parallel. The new model not only prevents the same Minion from running twice, but will also skip an invocation if there is another Minion running that includes any of the values from its MinionPolicy.Entities array. It also doesn’t matter how the Minion is started: calling the RunMinion API will still skip execution if the Minion is currently executing on its schedule.

Here’s a deeper look into the new flow:

StartEnvironmentMinionsBlock
  Minion.StartAsync() - called by StartEnvironmentMinionsBlock
    Warn if no WakeupInterval set
    loop forever
      await Minion.Process <-- RunMinion calls this
          Skip if policy is marked as running
          Skip if policy with any intersecting `Entities` value is running
          Mark as running
          await Minion.Execute
          Mark as not running
      await WakeupInterval

To summarise the behaviour changes:

Override Minion.Execute instead of Minion.Start
As before, handling exceptions is the responsibility of Minion.Execute
WakeupInterval now starts after Execute has completed asynchronously
Execute will be skipped if the minion is already running (either on a timer or via API)

Patching Sitecore Commerce Rules

Tue, 26 Jun 2018 11:53:16 +0000

Recently I noticed that the “Number of orders [compares to] [value]” promotion qualification wasn’t working when comparing to 0, which turned out to be because it always returns false when the order count is 0. Since this was clearly a bug that would eventually be fixed, I wanted to avoid adding a “new” rule as it would require some manual database patching when the fix did eventually come along.

After a little exploration, I found that it is indeed possible to replace the implementation of existing qualifications and benefits (conditions and actions), so I thought I’d share what I learned.

Sitecore Commerce uses Sitecore.Framework.Rules for conditions and actions, so the key is understanding how rule registration is handled in that library. Rules are configured by adding types and exclusions to the registry (IReflectionDiscoveryConfig), which is then later used by the IEntityDiscoverer to find available rules at runtime.

It turns out there are two implementation details that make it easy to swap out a rule:

Only rules names (applied via attribute) are stored in the promotion entity, not the full type name.
Exclusions to the rule regitry are applied at discovery rather being a removal of existing configuration

With these two things in mind, all we need to do is apply the same rule name to our condition via the EntityIdentifier attribute:

[EntityIdentifier("CurrentCustomerOrdersCountCondition")]
public class ZeroSupportingCurrentCustomerOrdersCountCondition : ICustomerCondition, ICondition, IMappableRuleEntity
{
    // Implementation
}

And then exclude the original type during configuration:

public void ConfigureServices(IServiceCollection services)
{
    services.Sitecore().Rules(rules =>
        rules.Registry(registry => registry
            .RegisterAssembly(Assembly.GetExecutingAssembly())
            .ExcludeType<CurrentCustomerOrdersCountCondition>() // We're replacing this rule
        )
    );
}

And that’s it. Since the display text is looked up using the same EntityIdentifier attribute, it will even appear correctly in the Commerce Business Tools without any other changes.

Continuous local deployments for Sitecore Commerce Engine

Wed, 06 Jun 2018 11:20:16 +0000

Commerce Engine’s default configuration poses some interesting problems for deployment, especially for local development workflow:

It has four roles (Ops, Shop, Authoring, and Minions) that each run as their own site/application. Each has their own “startup” configuration, but otherwise load runtime configuration from the database (put their by “Bootstrapping” the Ops role)
ASP.NET Core doesn’t support assmebly shadow copies, so the assembly files are locked while the roles are running.

In this post I’ll run through the tricks I use to simplify local development workflow without causing problems for your downstream CI/CD that can come from a custom “file copy” approach in local development.

By the end of this post we’ll have continuous local deployment triggered on build without affecting build performance, as well as providing an easy way to debug the minion role.

Publish profile

To start off, we’ll create a publish profile using the FileSystem publish method. The target will be the ops role, for reasons that will be explained in the next section, but is otherwise default configuration:

<!-- Local.pubxml -->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>FileSystem</WebPublishMethod>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <PublishFramework>net452</PublishFramework>
    <UsePowerShell>True</UsePowerShell>
    <publishUrl>C:\inetpub\wwwroot\CommerceOps_Sc9</publishUrl>
    <DeleteExistingFiles>False</DeleteExistingFiles>
  </PropertyGroup>
</Project>

The first challenge is the four role directories. Publishing four times will increase publishing time in a noticable way, especilly if the aim is to do it on each build.

While it may be tempting to consolidate everything into a single role for development, this can actually complicate your downstream deployment process because your environment policies won’t correlate. Worse, it means that your minion role can’t be isolated for debugging because it’s part of the authoring role.

The approach I have taken is to configure the four roles to all use the assemblies from the Ops role’s directory, which is done by changing aspNetCore/@processPath in Web.config to a relative path:

<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <modules runAllManagedModulesForAllRequests="false">
      <remove name="WebDAVModule" />
    </modules>
    <aspNetCore processPath="..\CommerceOps_Sc9\Sitecore.Commerce.Engine.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" requestTimeout="00:10:00" stdoutLogFile=".\logs\stdout" />
  </system.webServer>
</configuration>

This approach has two caveats to be aware of:

Each of the role directories still need a bootstrap\Global.json and config.json. I currently consider these an “initial setup task”, but I may eventually try to deploy/configure them using custom targets in the publish profile.

Correlating Sitecore.Commerce.Engine.exe processes to their role becomes more difficult. In reality, though, it will usually only be the authoring role that requires being attached to so there will only be one process to choose from.

Unlocking Assemblies

Continuous deployment is pointless if it can’t overwrite any files, and ASP.NET Core doesn’t support shadow copies. Luckily, IIS has a trick up it’s sleeve: the app_offline.html file. It’s existence shuts down the application pool and rejects all requests (with a 503 status) until the file is deleted.

In our scenario, we’ll need four separate app_offline files (one for each site), but that’s easily achievable by hooking into the “Before” and “After” events of the publish. The code below should be placed in Properties\PublishProfiles\Local.wpp.targets (or directly in Local.pubxml if you prefer):

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!-- Takes the engine offline so we can override it's assemblies -->
  <ItemGroup>
    <CommerceEngineNodes Include="C:\inetpub\wwwroot\CommerceOps_Sc9" />
    <CommerceEngineNodes Include="C:\inetpub\wwwroot\CommerceAuthoring_Sc9" />
    <CommerceEngineNodes Include="C:\inetpub\wwwroot\CommerceMinions_Sc9" />
    <CommerceEngineNodes Include="C:\inetpub\wwwroot\CommerceShops_Sc9" />
  </ItemGroup>
  
  <Target Name="BeforePublish">
    <WriteLinesToFile Lines="" File="%(CommerceEngineNodes.FullPath)\app_offline.htm" />
  </Target>

  <Target Name="AfterPublish">
    <Delete Files="@(CommerceEngineNodes -> '%(FullPath)\app_offline.htm')" />
  </Target>
</Project>

Configure continuous deployment

Now that we have the means to easily update our four applications, the next logical step is to automatically deploy the files whenever we build.

By placing the following in your engine csproj (or Directory.Build.props if using VS2017), the project will be published to your local instances whenever you build in “Debug” from within Visual Studio. The impact on build time is negligable because unlike right click :: Publish it doesn’t cause a rebuild, so it’s win-win!

<Project>
  <!-- Automatically publish to Properties/PublishProfiles/Local.pubxml on build -->
  <PropertyGroup Condition="'$(Configuration)' == 'Debug' and '$(BuildingInsideVisualStudio)' == 'true'">
    <DeployOnBuild>True</DeployOnBuild>
    <PublishProfile>Local</PublishProfile>
  </PropertyGroup>
</Project>

This trick only works for “sdk-style” projects, like those used ASP.NET Core, but there’s a similar approach for legacy web projects is outlined in helix-publishing-pipeline

Debugging

Commerce Engine’s architecture lends itself to test-driven development, and I highly recommend doing it if you’re not already. Having said that, there’s always going to be the need for interactive debugging to confirm any assumptions made about what data will be available at any given time.

For the version of config.json and Global.json that you commit into source control (ie. part of your Engine project source), I recommend setting them up to run as your Minion environment. I find myself needing to “F5” into the Minion role far more often than any other role. In fact, I tend to disable the Minion app pool in development unless there’s a reason for it to run in the background.

For authoring, I currenly resort to checking the Application event log for the entry that correlates the IIS AppPool with the process id. Awkward, but it works.

Bonus: Bootstrapping

Bootstrapping as part of continuous local deployments is unlikely to be worth the time it takes to fire up the app pool, but if you’re looking to easily bootstrap without using going through Postman I’ve created a PowerShell script that defaults all it’s parameters to local development defaults. Check it out here: https://gist.github.com/richardszalay/6f03d8898bf524d4ea26ba99333e80e6

Adding custom facetable attributes to products in Sitecore Commerce 9

Mon, 23 Apr 2018 00:00:00 +0000

In this post, I’m going to walk through the steps involved in exposing custom data on a SellableItem as a facet (as displayed on the Storefront’s category page).

Let’s start with a quick summary of how all these parts are connected, since this alone would have helped immensely when I started looking into making this customisation:

Data is added to Sellable Items via Components
Fields are exposed to Sitecore via the ConnectSellableItem view
Templates are updated to include the custom field
The custom field is added to the index
A facet item is defined on the indexed field
The facet is associated with a Commerce category

Add Component to SellableItem

This is the only step that I’m going to omit the details for, both because it involves a fair amount of boilerplate code and because there’s a fairly exhaustive “How to” available from Sitecore: How to extend Catalog system entities schema in Sitecore Experience Commerce 9.0

In case that link becomes unavailable, it involves:

Adding a block to IGetEntityViewPipeline that exposes data as “Properties” to the view and also defines “Actions” that will be handled by another pipeline when triggered
Adding a block to IDoActionPipeline that updates the component based on the action and data

There are plenty of examples of these blocks in the stock plugins, so with a little help from ILSpy (or similar) it will all become clear.

Include fields in commerce templates

The “Update Commerce Templates” button in the ribbon updates the Catalog, Category, and SellableItem templates by querying a specific view for the first available entity and then populates each item by querying the same view again for each entity. This also makes use of the IGetEntityViewPipeline, with the only real important differences being:

Child views are required (as with Master views) since they map to field sections in Sitecore
Properties should be added regardless of whether the data is available, as “Update Commerce Templates” generates the template fields from the first available SellableItem and that might not be relevant to your custom data.

For SellableItem, the view is KnownCatalogViewsPolicy.ConnectSellableItem, so we’ll just include a basic block here:

public class GetCustomDataBlock : 
    PipelineBlock<EntityView, EntityView, CommercePipelineExecutionContext>
{
    public override Task<EntityView> Run(EntityView arg, 
        CommercePipelineExecutionContext context)
    {
        var request = context.CommerceContext.GetObject<EntityViewArgument>();
        var viewsPolicy = context.GetPolicy<KnownCatalogViewsPolicy>();
        var entity = request.Entity;

        if (!(entity is SellableItem))
        {
            return Task.FromResult(arg);
        }

        var isCommerceConnectTemplateUpdate = 
            request.ViewName == viewsPolicy.ConnectSellableItem;

        if (isCommerceConnectTemplateUpdate)
        {
            var fieldSectionView = new EntityView
            {
                Name = "Fabrikam",
                DisplayName = "Fabrikam"
            };

            entityView.ChildViews.Add(fieldSectionView);

            var component = entity.HasComponent<CustomComponent>()
                ? entity.GetComponent<CustomComponent>()
                : null;

            fieldSectionView.Properties.Add(new ViewProperty
            {
                Name = "CustomData",
                DisplayName = "Custom Data",
                RawValue = (component?.CustomData ?? string.Empty),
                IsReadOnly = true,
                IsRequired = false,
                IsHidden = false
            });
        }

        return Task.FromResult(arg);
    }
}

In reality it’s likely that you’ll want to combine this block with whatever exposes these properties for the “Details” view since in many use cases the output will be the same.

Once that is deployed, simply click the “Update Commerce Templates” button and the data will be populated in the catalog items.

Include the data in the index

The faceting queries use sitecore_(db)_index, so we just need to make sure the custom field is included in those. All of the base factes are simply defined on defaultSolrIndexConfiguration, but free to be more specific if your indexing setup is more complex.

Here’s an example cloned from the built in ‘brand’ field:

<field
    fieldName="customdata"
    storageType="YES"
    indexType="UN_TOKENIZED"
    vectorType="NO"
    boost="1f"
    returnType="text"
    settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider" />

Be aware that using text/UN_TOKENIZED will split multi-word values into multiple facet values and display them in lowercase, so you may want to customise the field configuration if that doesn’t work for you. A returnType of string, for example, will prevent any processing occuring.

You may need to rebuild your index(es), at least for the Commerce tree.

To define a new facet, simply create an item in /sitecore/system/Settings/Buckets/Facets using the Facet template, and set the Field Name property to match your indexed fieldName (“customdata” in our example above).

The name of the facet item isn’t too important, but I tend to prefix it with the same “namespace” that I use else where. For example “Fabrikam Custom Data”

Finally, to display the facet on a category page edit the Category item in /Commerce/Catalogs and add the facet to the Runtime Search Facets field.

There is unfortunately no installation specific “default” value for this field, so if the facet applies globally you either need to manually change each category or update the standard values for /sitecore/templates/Commerce/Catalog/CommerceSearchSettings and remember to change it again when you perform a future Commerce upgrade.

Improving Sitecore Commerce exception logging with Demystifier

Thu, 15 Mar 2018 00:00:00 +0000

Sitecore Experience Commerce’s “Engine” microservice makes heavy use of asynchronous programming in everything from API controllers and commands to pipelines and data access. While this makes the most out of your server resources by minimising thread contention, “with great power comes terrible stack traces”.

System.Exception: Error processing block: Core.block.FindEntityInMemoryCache ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Sitecore.Commerce.Core.Caching.EntityMemoryCachingPolicy.GetCachePolicy(CommerceContext commerceContext, Type arg)
   at Sitecore.Commerce.Core.FindEntityInMemoryCacheBlock.<Run>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Pipelines.ReflectionPipelineBlockRunner.<InvokeBlock>d__2.MoveNext()
   --- End of inner exception stack trace ---
   at Sitecore.Framework.Pipelines.ReflectionPipelineBlockRunner.<InvokeBlock>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Pipelines.BasePipelineBlockRunner.<Run>d__3`1.MoveNext()

The essence of the exception is shrouded by implementation details of the async/await process, making it very difficult to undersatnd.

Ben Adams decided to tackle this problem head on a few months back and created Demystifier, which transforms known stack frame patterns into something that resembles the original code. The concept has been so popular that it has been merged into .NET Core 2.1 and will likely make it into a future release of the .NET Framework.

For the time being, though, beautiful stack trackes require a little work. Thankfully, Nicholas Blumardt has developed Serilog integration for Demystifier that reduces the integration process to a few steps.

First, we need to add the Serilog.Enrichers.Demystify package to our Sitecore.Commerce.Engine project:

Install-Package Serilog.Enrichers.Demystify -IncludePrerelease

Next we need to update the LoggerConfiguration (included in the Startup constructor in the default Commerce SDK) to use the enricher:

Log.Logger = new LoggerConfiguration()
  .ReadFrom.Configuration(this.Configuration)
  .Enrich.FromLogContext()
  .Enrich.WithDemystifiedStackTraces() // Add this line
  .Enrich.With(new ScLogEnricher())

That’s it! The next time you see an exception in your log, it will look something like this:

System.Exception: Error processing block: Core.block.FindEntityInMemoryCache ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at EntityMemoryCachingPolicy Sitecore.Commerce.Core.Caching.EntityMemoryCachingPolicy.GetCachePolicy(CommerceContext commerceContext, Type arg)
   at async Task<FindEntityArgument> Sitecore.Commerce.Core.FindEntityInMemoryCacheBlock.Run(FindEntityArgument arg, CommercePipelineExecutionContext context)
   at async Task<object> Sitecore.Framework.Pipelines.ReflectionPipelineBlockRunner.InvokeBlock(IPipelineBlock block, IPipelineExecutionContext context, object current)
   --- End of inner exception stack trace ---
   at async Task<object> Sitecore.Framework.Pipelines.ReflectionPipelineBlockRunner.InvokeBlock(IPipelineBlock block, IPipelineExecutionContext context, object current)
   at async Task<TOutput> Sitecore.Framework.Pipelines.BasePipelineBlockRunner.Run<TOutput>(string name, object input, IPipelineExecutionContext context)

Decomposing Monolothic ERP Exports using Sitecore Commerce 9 Pipelines

Tue, 20 Feb 2018 20:00:16 +0000

Exporting orders is a critical part of integrating an ecommerce website with an ERP system, and the integration often takes the form of an XML file or some other large data structure.

Sitecore’s Experience Commerce 9 finally brings the entire model into Sitecore Engine with it’s testable architecture, but generating a complete XML structure from the available data involves numerous dependencies:

Billing information is obtained via FederatedPaymentComponent on the Order
Shipping information is obtained via PhysicalFulfillmentComponent on the Order, but this can change for more complex shipping scenarios
Customer contact address information is obtained via a AddressComponent on the Customer, which is in turn dereferenced with GetCustomerCommand using the customer id extracted from the ContactComponent on the Order

Attempting to test a monolothic implementation of such a mapping would involve significant setup of both data and mocks, resulting in tests that pass but that are difficult to understand and maintain. Refactoring the mapping process into a series of services that make use of Dependency Injection would help, but isn’t particularly in line with Commerce Engine’s architecture.

This article will explore making use of Sitecore Commerce Pipelines to decompose the mapping process. Pipelines are not only composable but also individually support dependency injection, and using pipeline blocks to mutate the pipeline arguments has significant precence within the core plugins.

Let’s start with a monolothic implementation. It’s still testable due to the use of interfaces and the separation of the mapping into its own public method, but there’s a lot happening which means a lot to setup in our tests.

public class MyErpOrderExport
    : PipelineBlock<Order, Order, CommercePipelineExecutionContext>
{
    private readonly GetCustomerCommand getCustomerCommand;
    private readonly ISubmitMyErpFilePipeline submitMyErpFilePipeline

    public MyErpOrderExport(GetCustomerCommand getCustomerCommand, 
        ISubmitMyErpFilePipeline submitMyErpFilePipeline)
    {
        this.getCustomerCommand = getCustomerCommand;
        this.submitMyErpFilePipeline = submitMyErpFilePipeline;
    }

    public async Task<Order> Run(Order order, CommercePipelineExecutionContext context)
    {
        var document = await CreateOrderDocument(order);
        await fileSubmissionPipeline.Run(document, context);
        return order;
    }

    public async Task<XDocument> CreateOrderElement(
        Order order, CommercePipelineExecutionContext context)
    {
        return new XDocument(
            new XElement("order",
                await CreateCustomerElement(order, context),
                await CreateBillingElement(order),
                await CreateShippingElement(order),
                await CreateLineItemsElement(order),
            )
        );
    }

    private async Task<XElement> CreateCustomerElement(Order order, 
        CommercePipelineExecutionContext context)
    {
        var contactComponent = arg.Order.GetComponent<ContactComponent>();
        var customerId = contactComponent.CustomerId;

        var customer = await getCustomerCommand.Process(
            context.CommerceContext, customerId);

        var address = customer.GetComponent<AddressComponent>();
        var party = address.Party;

        return new XElement("customer",
            new XElement("firstName", party.FirstName),
            // ...
        );
    }

    private Task<XElement> CreateBillingElement(Order order)
    {
        // ...
    }

    private Task<XElement> CreateShippingElement(Order order)
    {
        // ...
    }

    private Task<XElement> CreateLineItemsElement(Order order)
    {
        // ...
    }
}

Now, you could argue that a few more of these methods could be made public to increase testability, and you’d be right, but it wouldn’t diminish the fact that this ends up being a big class does a lot, increasing the cognitive load required to reason about it.

Let’s look at what it might look like if we built up the XML in pipeline blocks instead:

public class MyErpOrderExport
    : PipelineBlock<Order, Order, CommercePipelineExecutionContext>
{
    private readonly IBuildErpOrderDocumentPipeline buildErpOrderDocumentPipeline;
    private readonly ISubmitMyErpFilePipeline submitMyErpFilePipeline;

    public MyErpOrderExport(IBuildErpOrderDocumentPipeline buildErpOrderDocumentPipeline,
        ISubmitMyErpFilePipeline submitMyErpFilePipeline)
    {
        this.buildErpOrderDocumentPipeline = buildErpOrderDocumentPipeline;
        this.submitMyErpFilePipeline = submitMyErpFilePipeline;
    }

    public async Task<Order> Run(Order order, CommercePipelineExecutionContext context)
    {
        var document = await CreateOrderDocument(order);

        await submitMyErpFilePipeline.Run(document, context);

        return order;
    }

    public async Task<XDocument> CreateOrderElement(Order order, CommercePipelineExecutionContext context)
    {
        var args = new BuildMyErpOrderDocumentArgument
        {
            Order = order,
            Element = new XElement("order")
        };

        await buildErpOrderDocumentPipeline.Run(args, context);

        return new XDocument(args.Element);
    }
}

Our top level orchestrator now simply moves data between two pipelines interfaces, making it very testable itself, but the real value is how much easier it is to manage each aspect of the document being built. Here’s AddBillingElementBlock:

public class AddBillingElementBlock : PipelineBlock<BuildMyErpOrderDocumentArgument, BuildMyErpOrderDocumentArgument, CommercePipelineExecutionContext>
{
    public Task<BuildMyErpOrderDocumentArgument> Run(BuildMyErpOrderDocumentArgument args, CommercePipelineExecutionContext context)
    {
        // Ain't no party like a...
        var billingParty = arg.Order.GetComponent<FederatedPaymentComponent>().BillingParty;

        arg.Element.Add(
            new XElement("billingInformation",
                new XElement("address1", billingParty.Address1),
                // ...
            )
        );

        return Task.FromResult(arg);
    }
}

Each block in the pipeline can now add its own elements to the document, and can be tested in isolation.

And finally, we compose all the blocks together in ConfigureSitecore:

services.Sitecore().Pipelines(p =>
    p.AddPipeline<IBuildMyErpOrderDocument, BuildMyErpOrderDocument>(c => c
        .Add<AddCustomerElementBlock>()
        .Add<AddBillingElementBlock>()
        .Add<AddShippingElementBlock>()
    )
);

Upgrading Buggy Bits to .NET Core - Part 2: Windows Tooling

Fri, 08 Sep 2017 09:20:16 +0000

This series attempts to update Tess Ferrandez’ Debugging Labs to be applicable to .NET Core so that they can continue to benefit developers everywhere as .NET Core becomes the norm.

In Part 1, we upgraded the codebase to ASP.NET Core. In this part, we’ll take a look at the different diagnostic tools and approaches used by the labs and how they’ve been changed in Windows 10.

Load Testing

tinyget is still available via the IIS 6.0 (!) Resource Kit Tools, but if you’re looking for a more modern Windows CLI for HTTP load testing you might be out of luck.

There are a number of popular cross platform HTTP loading tools available like JMeter, boom, and vegeta, but these require additional runtimes like Java and Go to be installed.

Luckily, the Windows Subsystem for Linux means we are no longer limited by tooling availability for Windows. If you have Bash for Windows installed, you can installing a tool like wrk using sudo apt install wrk.

wrk in particular has a similar syntax to tinyget, so aligns well with the original labs:

wrk --threads 30 --duration 10 http://localhost:12345

Monitoring Memory

Monitoring process memory hasn’t changed terribly, but in addition to using Performance Monitor you can also view memory (private working set is your best choice) in Task Manager by right clicking on the headers of the Details tab and selcting “Select columns”.

For detailed information on the GC, the situation is slightly tricker. .NET Core does not currently support performance counters, so there’s no way to passively read GC generation sizes from outside the process. The only choice you have is to attach a debugger (see below) and use SOS’s !HeapStat command.

User-mode Memory Dumps

The various approaches to creating memory dumps now have official documentation, though ADPlus is still recommended for the majority of usecases. ADPlus is largely unchanged in terms of CLI options, despite now being an executable rather than a vbs script, and still ships with the Debugging Tools for Windows. Just keep in mind that the -iis argument should be replaced with -pn dotnet.exe or -p <PID> since .NET Core runs its own web server.

For crash dumps, there is now also the option of having Windows Error Reporting generate them automatically.

Task Manager now has support for generating user dumps, which can be useful for hang scenarios. Simply right click any process in the “Processes” or “Details” tab and select “Create dump file”

Debugger

WinDBG is still the built-for-purpose application for this type of debugging, and it still ships with the Debugging Tools for Windows. However, there’s also a complete refresh of WinDBG in development that you can currently preview via the Windows Store. It contains a tonne of UI improvements, including a Local/Watch window and quick access to things like threads and the callstack.

Visual Studio .NET also has reasonable native debugging support and can still load SOS, so you might want to try that if you’d prefer to stay in your warm IDE blanky.

Regardless of your choice of debugging interface, you’ll still need to load the SOS module that ships with .NET Core. Once you’ve opened a user dump, you can load SOS by typing:

.loadby sos coreclr

(Meaning “load the ‘sos’ module from the same directory as the ‘coreclr’ module”)

Fun fact: SOS stands for Son of Strike and is named after Strike, the original debugging tools used internally by Microsoft while developing .NET which at that stage was still codenamed “Lightning”.

In the next part, we’ll see how the tools and techniques translate to debugging .NET Core on Linux.

Richard Szalay's Blog

Sitecore App Service Warm-Up Demystified

tl;dr

Overview

Sitecore integration

Localhost

Using `<applicationInitialization>`

Using `<rules>`

HTTP

Cached Renderings with Absolute Links

Time limit

Debugging Warm-Up Requests

Easily debug Sitecore assemblies using dnSpy

Step 1: Enable debugging for Sitecore assemblies

Step 2: Debug using dnSpy

Step 3: Disable debugging for Sitecore assemblies

Diagnosing missing package assemblies in Helix solutions

Reentrancy improvements to Minions in Sitecore Commerce 9.1

Patching Sitecore Commerce Rules

Continuous local deployments for Sitecore Commerce Engine

Publish profile

Unlocking Assemblies

Configure continuous deployment

Debugging

Bonus: Bootstrapping

Adding custom facetable attributes to products in Sitecore Commerce 9

Add Component to SellableItem

Include fields in commerce templates

Include the data in the index

Define a custom facet

Apply the custom facet to the category

Improving Sitecore Commerce exception logging with Demystifier

Decomposing Monolothic ERP Exports using Sitecore Commerce 9 Pipelines

Upgrading Buggy Bits to .NET Core - Part 2: Windows Tooling

Load Testing

Monitoring Memory

User-mode Memory Dumps

Debugger

Richard Szalay's Blog

Sitecore App Service Warm-Up Demystified

tl;dr

Overview

Sitecore integration

Localhost

Using <applicationInitialization>

Using <rules>

HTTP

Cached Renderings with Absolute Links

Time limit

Debugging Warm-Up Requests

Easily debug Sitecore assemblies using dnSpy

Step 1: Enable debugging for Sitecore assemblies

Step 2: Debug using dnSpy

Step 3: Disable debugging for Sitecore assemblies

Diagnosing missing package assemblies in Helix solutions

Reentrancy improvements to Minions in Sitecore Commerce 9.1

Patching Sitecore Commerce Rules

Continuous local deployments for Sitecore Commerce Engine

Publish profile

Sharing assemblies

Unlocking Assemblies

Configure continuous deployment

Debugging

Bonus: Bootstrapping

Adding custom facetable attributes to products in Sitecore Commerce 9

Add Component to SellableItem

Include fields in commerce templates

Include the data in the index

Define a custom facet

Apply the custom facet to the category

Improving Sitecore Commerce exception logging with Demystifier

Decomposing Monolothic ERP Exports using Sitecore Commerce 9 Pipelines

Upgrading Buggy Bits to .NET Core - Part 2: Windows Tooling

Load Testing

Monitoring Memory

User-mode Memory Dumps

Debugger

Using `<applicationInitialization>`

Using `<rules>`