Scale Long Running Background Services in Azure

Scale Long Running Background Services in Azure

Overview

I have been doing a lot of work on Azure lately. Most recently, in a high volume message processing application that we are building for one of our customers, I quickly got into a scenario where we needed to understand how we can scale long running services to support multiple tenants.

If you are building a SaaS based application, multi-tenacity is a key quality attribute you should never lose focus of, while the architecture continues to evolve in due course.

You typically address multi-tenacity in the following tiers:

  1. In the database by partitioning the data for tenants across master tables using a tenant identifier as reference.
  2. In the services tier embedding the tenant identifier in the context (or as a claim for a claims enabled service).
  3. Creating tenant based landing pages for tenant specific users in the user interface layer. Sometimes in mobile apps or composite desktop applications, a two factor authentication is also used.

Problem

Each of these layers can scale fairly easily, however multi-tenacity is often difficult to address in case of background services. How do you allocate an instance to process data for a particular tenant, allowing multiple tenants to be processed simultaneously?

Solution

The issue can be handled by using a combination of Azure Queues and the Blob Leasing mechanism with the Worker Role.

The Background Service Contract

Create an abstract base class defining the contracts for operations that can run under lease (meaning no instance can enter this method other than the one that acquires the lease first), and also the contract for operations that can run without lease.

///     ///     ///     public abstract class BackgroundServiceBase    {        ///         /// Gets or sets the unity container.        ///         ///         /// The unity container.        ///         public IUnityContainer UnityContainer { get; set; }         ///         /// Gets or sets the BLOB provider.        ///         ///         /// The BLOB provider.        ///         public IBlobProvider BlobProvider { get; set; }         ///         /// Gets or sets the proposed lease id.        ///         ///         /// The proposed lease id.        ///         public string ProposedLeaseId { get; set; } 	///         /// Initializes a new instance of the  	class.        ///         public BackgroundServiceBase()        {            UnityContainer = UnityHelper.ConfigureUnity();            BlobProvider = UnityContainer.Resolve();        }                        ///         /// Executes the specified context.        ///         public virtual void Execute()        {            // Avoid race condition            Thread.Sleep((new Random()).Next(100, 200));            if (BlobProvider.AcquireLease(TimeSpan.FromSeconds(45), ProposedLeaseId))            {                LeasedOperations();            }            else            {                Thread.Sleep(15 * 1000);            }             NonLeasedOperations();        }         ///         /// Non leased operations.        ///         public abstract void NonLeasedOperations();                 ///         /// Leased operations.        ///         public abstract void LeasedOperations();    }

Using Unity, the blob provider is injected and is responsible for acquiring lease using the Azure Blob Container.

Implementing the Background Service

Now create a long service that inherits the background service base, and implements the Leased and Non Leased Operations.

Write code inside the leased operations method to create a queue to store the tenant information. An example is shown in the code block below.

///         /// Leased operations.        ///         public override void LeasedOperations()        {            try            {                var tenants = UnitOfWork.Repository.Get();                foreach (var tenant in tenants)                {                    var licenseContext = LicenseManager.TenantLicenseContext(tenant.TenantId);                    if (licenseContext != null && licenseContext.Features.Contains((int)LicenseFeatures.[Name of the Feature]))                        QueueProvider.PutMessage(Constants.QueueNames.[Queue Name Here], tenant.TenantId.ToString(), TimeSpan.FromSeconds(45));                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

The function gets the tenant information, checks if the tenant has appropriate licenses for the service and then puts the tenant id in queue. Note that you don’t have to fetch tenant id’s from the database all the time, you can fetch them from let’s say a cache, since this information doesn’t change that often. Also note that the message is put in the queue for only 45 seconds, precisely the duration of the lease.

Once you have put the tenant ids in queue, read them in the NonLeasedOperations and then create the business manager instances to process data based on the tenant id.

///         /// Non leased operations.        ///         public override void NonLeasedOperations()        {            try            {                int processedCount = 0;                while (processedCount == 0)                {                    var queueMessage = QueueProvider.ReadMessage(Constants.QueueNames.[Queue Name Here], TimeSpan.FromSeconds(30));                    if (queueMessage == null)                        return;                    QueueProvider.DeleteMessage(Constants.QueueNames.[Queue Name Here], queueMessage.MessageId, queueMessage.PopReciept);                    var businessManager = UnityContainer.Resolve(new ParameterOverrides{                    {"tenantId", int.Parse(queueMessage.Message)},                    {"container", UnityContainer}                    });                    businessManager.ProcessResults(ref processedCount);                }            }            catch (Exception ex)            {                LogProvider.Error(ex);            }        }

There are a couple of important things to note in the Non Leased Operations method. One is how the manager instance is created allowing multiple tenants to be processed at the same time, and the processCount variable in the while loop. You will need to update this variable in the manager code to indicate that there are items to process for the tenant, and if there are none, then move to the next tenant immediately.

Now you can create this service instance in the worker role and run it. You can also run it in a schedule using schedulers like Quartz.net.

devx-admin

devx-admin

Share the Post:
5G Innovations

GPU-Accelerated 5G in Japan

NTT DOCOMO, a global telecommunications giant, is set to break new ground in the industry as it prepares to launch a GPU-accelerated 5G network in

AI Ethics

AI Journalism: Balancing Integrity and Innovation

An op-ed, produced using Microsoft’s Bing Chat AI software, recently appeared in the St. Louis Post-Dispatch, discussing the potential concerns surrounding the employment of artificial

Savings Extravaganza

Big Deal Days Extravaganza

The highly awaited Big Deal Days event for October 2023 is nearly here, scheduled for the 10th and 11th. Similar to the previous year, this

5G Innovations

GPU-Accelerated 5G in Japan

NTT DOCOMO, a global telecommunications giant, is set to break new ground in the industry as it prepares to launch a GPU-accelerated 5G network in Japan. This innovative approach will

AI Ethics

AI Journalism: Balancing Integrity and Innovation

An op-ed, produced using Microsoft’s Bing Chat AI software, recently appeared in the St. Louis Post-Dispatch, discussing the potential concerns surrounding the employment of artificial intelligence (AI) in journalism. These

Savings Extravaganza

Big Deal Days Extravaganza

The highly awaited Big Deal Days event for October 2023 is nearly here, scheduled for the 10th and 11th. Similar to the previous year, this autumn sale has already created

Cisco Splunk Deal

Cisco Splunk Deal Sparks Tech Acquisition Frenzy

Cisco’s recent massive purchase of Splunk, an AI-powered cybersecurity firm, for $28 billion signals a potential boost in tech deals after a year of subdued mergers and acquisitions in the

Iran Drone Expansion

Iran’s Jet-Propelled Drone Reshapes Power Balance

Iran has recently unveiled a jet-propelled variant of its Shahed series drone, marking a significant advancement in the nation’s drone technology. The new drone is poised to reshape the regional

Solar Geoengineering

Did the Overshoot Commission Shoot Down Geoengineering?

The Overshoot Commission has recently released a comprehensive report that discusses the controversial topic of Solar Geoengineering, also known as Solar Radiation Modification (SRM). The Commission’s primary objective is to

Remote Learning

Revolutionizing Remote Learning for Success

School districts are preparing to reveal a substantial technological upgrade designed to significantly improve remote learning experiences for both educators and students amid the ongoing pandemic. This major investment, which

Revolutionary SABERS Transforming

SABERS Batteries Transforming Industries

Scientists John Connell and Yi Lin from NASA’s Solid-state Architecture Batteries for Enhanced Rechargeability and Safety (SABERS) project are working on experimental solid-state battery packs that could dramatically change the

Build a Website

How Much Does It Cost to Build a Website?

Are you wondering how much it costs to build a website? The approximated cost is based on several factors, including which add-ons and platforms you choose. For example, a self-hosted

Battery Investments

Battery Startups Attract Billion-Dollar Investments

In recent times, battery startups have experienced a significant boost in investments, with three businesses obtaining over $1 billion in funding within the last month. French company Verkor amassed $2.1

Copilot Revolution

Microsoft Copilot: A Suit of AI Features

Microsoft’s latest offering, Microsoft Copilot, aims to revolutionize the way we interact with technology. By integrating various AI capabilities, this all-in-one tool provides users with an improved experience that not

AI Girlfriend Craze

AI Girlfriend Craze Threatens Relationships

The surge in virtual AI girlfriends’ popularity is playing a role in the escalating issue of loneliness among young males, and this could have serious repercussions for America’s future. A

AIOps Innovations

Senser is Changing AIOps

Senser, an AIOps platform based in Tel Aviv, has introduced its groundbreaking AI-powered observability solution to support developers and operations teams in promptly pinpointing the root causes of service disruptions

Bebop Charging Stations

Check Out The New Bebob Battery Charging Stations

Bebob has introduced new 4- and 8-channel battery charging stations primarily aimed at rental companies, providing a convenient solution for clients with a large quantity of batteries. These wall-mountable and

Malyasian Networks

Malaysia’s Dual 5G Network Growth

On Wednesday, Malaysia’s Prime Minister Anwar Ibrahim announced the country’s plan to implement a dual 5G network strategy. This move is designed to achieve a more equitable incorporation of both

Advanced Drones Race

Pentagon’s Bold Race for Advanced Drones

The Pentagon has recently unveiled its ambitious strategy to acquire thousands of sophisticated drones within the next two years. This decision comes in response to Russia’s rapid utilization of airborne

Important Updates

You Need to See the New Microsoft Updates

Microsoft has recently announced a series of new features and updates across their applications, including Outlook, Microsoft Teams, and SharePoint. These new developments are centered around improving user experience, streamlining

Price Wars

Inside Hyundai and Kia’s Price Wars

South Korean automakers Hyundai and Kia are cutting the prices on a number of their electric vehicles (EVs) in response to growing price competition within the South Korean market. Many

Solar Frenzy Surprises

Solar Subsidy in Germany Causes Frenzy

In a shocking turn of events, the German national KfW bank was forced to discontinue its home solar power subsidy program for charging electric vehicles (EVs) after just one day,

Electric Spare

Electric Cars Ditch Spare Tires for Efficiency

Ira Newlander from West Los Angeles is thinking about trading in his old Ford Explorer for a contemporary hybrid or electric vehicle. However, he has observed that the majority of

Solar Geoengineering Impacts

Unraveling Solar Geoengineering’s Hidden Impacts

As we continue to face the repercussions of climate change, scientists and experts seek innovative ways to mitigate its impacts. Solar geoengineering (SG), a technique involving the distribution of aerosols

Razer Discount

Unbelievable Razer Blade 17 Discount

On September 24, 2023, it was reported that Razer, a popular brand in the premium gaming laptop industry, is offering an exceptional deal on their Razer Blade 17 model. Typically

Innovation Ignition

New Fintech Innovation Ignites Change

The fintech sector continues to attract substantial interest, as demonstrated by a dedicated fintech stage at a recent event featuring panel discussions and informal conversations with industry professionals. The gathering,

Import Easing

Easing Import Rules for Big Tech

India has chosen to ease its proposed restrictions on imports of laptops, tablets, and other IT hardware, allowing manufacturers like Apple Inc., HP Inc., and Dell Technologies Inc. more time