Scale Long Running Background Services in Azure


I have been doing a lot of work on Azure lately. Most recently, in a high volume message processing application that we are building for one of our customers, I quickly got into a scenario where we needed to understand how we can scale long running services to support multiple tenants.

If you are building a SaaS based application, multi-tenacity is a key quality attribute you should never lose focus of, while the architecture continues to evolve in due course.

You typically address multi-tenacity in the following tiers:

  1. In the database by partitioning the data for tenants across master tables using a tenant identifier as reference.
  2. In the services tier embedding the tenant identifier in the context (or as a claim for a claims enabled service).
  3. Creating tenant based landing pages for tenant specific users in the user interface layer. Sometimes in mobile apps or composite desktop applications, a two factor authentication is also used.


Each of these layers can scale fairly easily, however multi-tenacity is often difficult to address in case of background services. How do you allocate an instance to process data for a particular tenant, allowing multiple tenants to be processed simultaneously?


The issue can be handled by using a combination of Azure Queues and the Blob Leasing mechanism with the Worker Role.

The Background Service Contract

Create an abstract base class defining the contracts for operations that can run under lease (meaning no instance can enter this method other than the one that acquires the lease first), and also the contract for operations that can run without lease.

///     ///     ///     public abstract class BackgroundServiceBase    {        ///         /// Gets or sets the unity container.        ///         ///         /// The unity container.        ///         public IUnityContainer UnityContainer { get; set; }         ///         /// Gets or sets the BLOB provider.        ///         ///         /// The BLOB provider.        ///         public IBlobProvider BlobProvider { get; set; }         ///         /// Gets or sets the proposed lease id.        ///         ///         /// The proposed lease id.        ///         public string ProposedLeaseId { get; set; } 	///         /// Initializes a new instance of the  	class.        ///         public BackgroundServiceBase()        {            UnityContainer = UnityHelper.ConfigureUnity();            BlobProvider = UnityContainer.Resolve();        }                        ///         /// Executes the specified context.        ///         public virtual void Execute()        {            // Avoid race condition            Thread.Sleep((new Random()).Next(100, 200));            if (BlobProvider.AcquireLease(TimeSpan.FromSeconds(45), ProposedLeaseId))            {                LeasedOperations();            }            else            {                Thread.Sleep(15 * 1000);            }             NonLeasedOperations();        }         ///         /// Non leased operations.        ///         public abstract void NonLeasedOperations();                 ///         /// Leased operations.        ///         public abstract void LeasedOperations();    }

Using Unity, the blob provider is injected and is responsible for acquiring lease using the Azure Blob Container.

Implementing the Background Service

Now create a long service that inherits the background service base, and implements the Leased and Non Leased Operations.

Write code inside the leased operations method to create a queue to store the tenant information. An example is shown in the code block below.

///         /// Leased operations.        ///         public override void LeasedOperations()        {            try            {                var tenants = UnitOfWork.Repository.Get();                foreach (var tenant in tenants)                {                    var licenseContext = LicenseManager.TenantLicenseContext(tenant.TenantId);                    if (licenseContext != null && licenseContext.Features.Contains((int)LicenseFeatures.[Name of the Feature]))                        QueueProvider.PutMessage(Constants.QueueNames.[Queue Name Here], tenant.TenantId.ToString(), TimeSpan.FromSeconds(45));                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

The function gets the tenant information, checks if the tenant has appropriate licenses for the service and then puts the tenant id in queue. Note that you don’t have to fetch tenant id’s from the database all the time, you can fetch them from let’s say a cache, since this information doesn’t change that often. Also note that the message is put in the queue for only 45 seconds, precisely the duration of the lease.

Once you have put the tenant ids in queue, read them in the NonLeasedOperations and then create the business manager instances to process data based on the tenant id.

///         /// Non leased operations.        ///         public override void NonLeasedOperations()        {            try            {                int processedCount = 0;                while (processedCount == 0)                {                    var queueMessage = QueueProvider.ReadMessage(Constants.QueueNames.[Queue Name Here], TimeSpan.FromSeconds(30));                    if (queueMessage == null)                        return;                    QueueProvider.DeleteMessage(Constants.QueueNames.[Queue Name Here], queueMessage.MessageId, queueMessage.PopReciept);                    var businessManager = UnityContainer.Resolve(new ParameterOverrides{                    {"tenantId", int.Parse(queueMessage.Message)},                    {"container", UnityContainer}                    });                    businessManager.ProcessResults(ref processedCount);                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

There are a couple of important things to note in the Non Leased Operations method. One is how the manager instance is created allowing multiple tenants to be processed at the same time, and the processCount variable in the while loop. You will need to update this variable in the manager code to indicate that there are items to process for the tenant, and if there are none, then move to the next tenant immediately.

Now you can create this service instance in the worker role and run it. You can also run it in a schedule using schedulers like

Share the Post:
Share on facebook
Share on twitter
Share on linkedin


The Latest

6 Tips for Setting Up a Decentralized Exchange

6 Tips for Setting Up a Decentralized Exchange

There’s no doubt that cryptocurrency is a complex and divisive topic in the modern financial landscape. There are those who are convinced that it’s nothing more than a bubble, but both who are well-informed are able to see the ways in which cryptocurrency can help them both build their fortune

user experience with someone on their phone

5 Ways to Improve Your Customers’ User Experience

They say you can’t judge a book by its cover, but just because they say that doesn’t mean it’s true. Consider how often you choose one sort of product over another just because it appeals to you. Then think about how often you’ve stopped using something because, simply put, it

How to Manage Your Finances after Buying a Home

How to Manage Your Finances after Buying a Home

Buying a home is a milestone in the journey of life – it’s one way to invest your money and create lasting memories. Now you know everything about a home purchase, home mortgage, and what is a conventional 97 loan but do you know what’s next after you sign the

Windows Logging is one of the vital aspects of any Windows system administration. However, it is mostly overlooked until the system develops a problem. This is because logs contain important information needed to troubleshoot and resolve system issues.

The Fundamentals of Windows Logging

Windows Logging is one of the vital aspects of any Windows system administration. However, it is mostly overlooked until the system develops a problem. This is because logs contain important information needed to troubleshoot and resolve system issues. Without it, tech experts might find it difficult to track a computer’s

Interstitial Ads: Best Practices for Successful Campaigns

Interstitial Ads: Best Practices for Successful Campaigns

Interstitial Ads: Best Practices for Successful Campaigns Interstitial ads are full-screen advertisements that appear to grasp the attention of on-site prospects, creating opportunities for brands seeking effective ways to communicate their proposition of value. With such an attention-grabbing format and high-impact visuals, it’s no wonder why interstitial advertising is proving