Scale Long Running Background Services in Azure

Overview

I have been doing a lot of work on Azure lately. Most recently, in a high volume message processing application that we are building for one of our customers, I quickly got into a scenario where we needed to understand how we can scale long running services to support multiple tenants.

If you are building a SaaS based application, multi-tenacity is a key quality attribute you should never lose focus of, while the architecture continues to evolve in due course.

You typically address multi-tenacity in the following tiers:

1. In the database by partitioning the data for tenants across master tables using a tenant identifier as reference.
2. In the services tier embedding the tenant identifier in the context (or as a claim for a claims enabled service).
3. Creating tenant based landing pages for tenant specific users in the user interface layer. Sometimes in mobile apps or composite desktop applications, a two factor authentication is also used.

Problem

Each of these layers can scale fairly easily, however multi-tenacity is often difficult to address in case of background services. How do you allocate an instance to process data for a particular tenant, allowing multiple tenants to be processed simultaneously?

Solution

The issue can be handled by using a combination of Azure Queues and the Blob Leasing mechanism with the Worker Role.

The Background Service Contract

Create an abstract base class defining the contracts for operations that can run under lease (meaning no instance can enter this method other than the one that acquires the lease first), and also the contract for operations that can run without lease.

/// 
    ///     /// 
    public abstract class BackgroundServiceBase    {        /// 
        /// Gets or sets the unity container.        /// 
        ///         /// The unity container.        ///         public IUnityContainer UnityContainer { get; set; }         /// 
        /// Gets or sets the BLOB provider.        /// 
        ///         /// The BLOB provider.        ///         public IBlobProvider BlobProvider { get; set; }         /// 
        /// Gets or sets the proposed lease id.        /// 
        ///         /// The proposed lease id.        ///         public string ProposedLeaseId { get; set; } 	/// 
        /// Initializes a new instance of the  	class.        /// 
        public BackgroundServiceBase()        {            UnityContainer = UnityHelper.ConfigureUnity();            BlobProvider = UnityContainer.Resolve();        }                        /// 
        /// Executes the specified context.        /// 
        public virtual void Execute()        {            // Avoid race condition            Thread.Sleep((new Random()).Next(100, 200));            if (BlobProvider.AcquireLease(TimeSpan.FromSeconds(45), ProposedLeaseId))            {                LeasedOperations();            }            else            {                Thread.Sleep(15 * 1000);            }             NonLeasedOperations();        }         /// 
        /// Non leased operations.        /// 
        public abstract void NonLeasedOperations();                 /// 
        /// Leased operations.        /// 
        public abstract void LeasedOperations();    }

Using Unity, the blob provider is injected and is responsible for acquiring lease using the Azure Blob Container.

Implementing the Background Service

Now create a long service that inherits the background service base, and implements the Leased and Non Leased Operations.

Write code inside the leased operations method to create a queue to store the tenant information. An example is shown in the code block below.

/// 
        /// Leased operations.        /// 
        public override void LeasedOperations()        {            try            {                var tenants = UnitOfWork.Repository.Get();                foreach (var tenant in tenants)                {                    var licenseContext = LicenseManager.TenantLicenseContext(tenant.TenantId);                    if (licenseContext != null && licenseContext.Features.Contains((int)LicenseFeatures.[Name of the Feature]))                        QueueProvider.PutMessage(Constants.QueueNames.[Queue Name Here], tenant.TenantId.ToString(), TimeSpan.FromSeconds(45));                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

The function gets the tenant information, checks if the tenant has appropriate licenses for the service and then puts the tenant id in queue. Note that you don’t have to fetch tenant id’s from the database all the time, you can fetch them from let’s say a cache, since this information doesn’t change that often. Also note that the message is put in the queue for only 45 seconds, precisely the duration of the lease.

Once you have put the tenant ids in queue, read them in the NonLeasedOperations and then create the business manager instances to process data based on the tenant id.

/// 
        /// Non leased operations.        /// 
        public override void NonLeasedOperations()        {            try            {                int processedCount = 0;                while (processedCount == 0)                {                    var queueMessage = QueueProvider.ReadMessage(Constants.QueueNames.[Queue Name Here], TimeSpan.FromSeconds(30));                    if (queueMessage == null)                        return;                    QueueProvider.DeleteMessage(Constants.QueueNames.[Queue Name Here], queueMessage.MessageId, queueMessage.PopReciept);                    var businessManager = UnityContainer.Resolve(new ParameterOverrides{                    {"tenantId", int.Parse(queueMessage.Message)},                    {"container", UnityContainer}                    });                    businessManager.ProcessResults(ref processedCount);                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

There are a couple of important things to note in the Non Leased Operations method. One is how the manager instance is created allowing multiple tenants to be processed at the same time, and the processCount variable in the while loop. You will need to update this variable in the manager code to indicate that there are items to process for the tenant, and if there are none, then move to the next tenant immediately.

Now you can create this service instance in the worker role and run it. You can also run it in a schedule using schedulers like Quartz.net.