Hyperscale Messaging in .NET with Amazon’s Simple Queuing Service (SQS)

Hyperscale Messaging in .NET with Amazon’s Simple Queuing Service (SQS)

essage Queuing (MQ) has long been a foundation for applications that require asynchronous and disconnected communications. Implementations of Message Queuing such as Microsoft’s MSMQ, IBM’s WebSphere MQ, TIBCO’s Rendezvous, and Progress Sonic’s SonicMQ are all mature, highly reliable, and highly scalable. Now, Amazon, in a new bid to sell its capabilities as services, has entered the fray with its Amazon Simple Queue Service (SQS).

Until now, large vendors have hawked MQ services as infrastructure products?products clients buy licenses for and install on their own hardware; but Amazon’s SQS is entirely based on a service model, with no upfront software licensing or hardware setup. Amazon SQS is a Web service in which all messages are sent through, stored on, and retrieved from Amazon’s servers. All that’s necessary is that your application understand and adhere to the SOAP, HTTP Query, or REST protocols that Amazon SQS understands. This article focuses on how to take advantage of Amazon SQS from .NET applications using SOAP.

Why Consider Amazon SQS?
There are several good reasons why your organization might want to consider Amazon SQS over another MQ solution:

  • All communication between your applications and Amazon SQS is over HTTP and so for most of you will not require poking holes in your firewalls. This makes it perfect for applications that cross organizational boundaries.
  • Amazon SQS has no upfront or ongoing hardware and software licensing costs. That’s an attractive option, particularly when startup costs for current messaging products can mount to thousands of dollars?and maintenance can be equally expensive.
  • Scalability is built-in. Amazon.com’s existing and well-tested infrastructure provides your applications flexibility to scale on demand.
  • You get the proven reliability of Amazon.com’s resilient infrastructure without making any upfront investment.

Still, before you jump, there is a downside:

  • You pay by the message volume and storage requirements to the tune of 10 cents for each 1000 messages and 20 cents per gigabyte of message storage required. So if your business requires a large number of messages or sends and receives very large messages, then you should commit to making a full cost comparison between traditional MQ software and SQS.
  • You must trust Amazon.com with your message content. The license agreement does claim that “…we will not disclose, sell, or license your SQS Content?” but nevertheless, Amazon will have access to that content.
  • Your applications must be able to tolerate the latency and vicissitudes of Internet connectivity. In other words, if you have applications with time-constrained throughput, don’t throw your MQ infrastructure away just yet. In addition, Amazon SQS does not provide any local client storage, meaning that for applications to send messages, Internet connectivity to Amazon SQS is mandatory.

Without further ado, let’s get started on the nuts and bolts of using Amazon SQS. The downloadable code that accompanies this article is a Windows Forms-based application that provides an Amazon?SQS dash board. In the remainder of this article, you’ll see the steps required to set up an application to use Amazon SQS, and some practical example functions, such as retrieving a list of your Amazon SQS Queues and creating a message on Amazon’s SQS Queue.

You will need Visual Studio 2005 (the Express Edition works fine) and the Web services Extensions (WSE) for the 3.0 version of the .NET framework installed to use Amazons’ Simple Queue Service. Finally, you’ll need the OpenSSL Tool to convert Amazon’s X-509 certificates to PKCS12?a form that works with Visual Studio.

Getting Started with SQS
First, you’ll need a regular Amazon.com account. This can be the same account that you use when shopping on Amazon for books, CDs, or other products.

Second, you need to sign up for Amazon Web services by pointing your browser to the Amazon Web services home page. This signup gives you access to all the various Amazon Web services such as Amazon Storage Service, Amazon Mechanical Turk, Amazon Elastic Cloud, Alexa Web service, Amazon Historical Pricing, and of course Amazon Simple Queue Service (Amazon SQS). Some of these services are free; unfortunately Amazon SQS is not a free service.

Third, you need to specifically sign up for the Amazon SQS service. Because this is a paid service you will need to supply a credit card so Amazon can charge for the messages you send and storage you use.

Finally, you need to get an access identifier from Amazon.com. These are the unique credentials that help Amazon identify you, when you use their various Web services. You have an option of using either of two identifier types: Access Key Identifiers or X.509 certificates. Access Key Identifiers consist of Access Key ID and Secret Access Key strings that Amazon generates. X.509 certificates are certificate files provided by Amazon that contain X.509 public and private keys. Because Amazon was not forthcoming on the subject of using Access Key Identifier identifiers with Microsoft’s C# platform, this article and the examples rely on X.509 Certificates.

You can get Amazon to generate your X.509 certificate via the “View Access Identifiers” link to “AWS Access Identifiers” page from the Amazon Web services home page.

Author’s Note: The “View Access Identifier” link appears on a pop-up menu when you hover your mouse over the “Your Web Services Account” link in the top-right-corner of the home page.

Have Amazon.com create a new X.509 certificate, and then download both the public key file (cert-xxxxxx(AWS Thumb print)xxxxxxxx.pem) and the private key file (pk-xxxxxx(AWS Thumb print)xxxxxxxx pem) to a local C:HyperScaleKeys folder.

Warning: While you can re-download the public certificate from Amazon if necessary, Amazon does not save the private key file, so store it securely. If you lose your private key file, you will need to re-generate your certificate.

Extracting the PKCS12 Certificate
Extracting the X.509 certificate from the cert-xxxxxx(AWS Thumb print)xxxxxxxx.pem file requires a bit of effort.

Microsoft Windows and the C# environment expect to get X.509 certificates in PKCS12 format. This mean you need to convert the provided cert-xxx.pem and pk-xxx.pem files to PKCS12 format. There are number of tools that can do this. While Amazon.com recommends Cygwin’s OpenSSL tool, I recommend you use the Windows-based OpenSSL tool, which you can download here.

?
Figure 1. Installing PKCS12 Certificate: Right-clicking the .p12 file you created by converting the two X-509 certificate files from Amazon to PKCS12 format launches the Windows Certificate Import Wizard.

Save and extract the openssl-0.9.7c-bin.zip file from the download to a C:OpenSSL folder. Switch to that folder, open a command prompt, and execute the following commands to create an AWSCertificate.p12 certificate file in PKCS12 format in your C:HyperScaleKeys folder.

   C:HyperscaleKeys> c:OpenSSLinopenssl.exe 

When OpenSSL launches, you’ll see an OpenSSL prompt. Enter the following command at the prompt on a single line.

   OpenSSL> pkcs12 --export       --inkey pk-xxxxxx(AWS Thumbprint)xxxx.pem       --in cert-xxxxx(AWS Thumbprint)xxxx.pem       --out AWSCertificate.p12 -nodes

Installing the Certificate
After you finish creating the AWSCertificate.p12 file discussed in the preceding section, navigate to the c:HyperScaleKeys folder, right click on the file and select Install PFX. This will start the Windows Certificate Import Wizard as shown in Figure 1. Follow the wizard’s instructions, installing the certificate using the name “AWS Customer.”

Author’s Note: Important! Remember the password you use when you install the certificate, because you will need this password when the certificate is used in your application.

Building the SQS Dashboard
To build the SQS Dashboard, create a new Windows Forms project in your C:Hyperscale folder.

You’ll need to enable WSE and create a security policy named “Hyperscale” that the project can use. You can use the “WSE Settings 3.0” Wizard available in Visual Studio as shown in the Figures below. However you can use the Wizard only to initially create the policy file; for the policy file to actually work with Amazon SQS (because of MutualCertificate10Assertion incompatibility) you will need to modify the file manually (more on this later). Unfortunately, after you modify it, you will no longer be able to use the Wizard to modify it.

To create the security file, launch the Wizard by selecting “WSE Settings 3.0?” from the right-click menu of the project item. Then follow the procedure illustrated in the figures in this article. The first two screens are simple; check the boxes as shown in Figure 2 and Figure 3, then click the “Add” button shown in Figure 3 to edit the application policy.

?
Figure 2. Enable WSE: The figure shows the first step in the WSE Wizard—enabling “Web service Enhancements” for the project.
?
Figure 3. Enabling Security: Check the “Enable Policy” checkbox to enable a security policy for the project, and click ‘Add’ to add a project-specific policy.

Enter “HyperscalePolicy” for the Policy Friendly Name as shown in Figure 4, then select the options shown in Figure 5.

?
Figure 4. Name the Policy: Name the new security policy “HyperscalePolicy” as shown in this figure.
?
Figure 5. Select Appropriate Options: Choose the “Secure a client application” option and select “Certificate” as the authentication method.

Now you need to choose client certificate that your application will use (see Figure 6). Uncheck the option “Specify the X.509 Certificate in Code” because the application relies on WSE’s default functionality to attach the client certificate. Be sure to select the “AWS Customer” certificate that you installed earlier in the “Certificate Information” section.

The Wizard will prompt you for a “Server Certificate.” Your application and WSE will not be using this information, so you can choose the same “AWS Certificate” as in Figure 6.

Finally, as shown in Figure 7, uncheck the “Enable WS-Security 1.1 Extensions” and the “Establish Secure Session” checkboxes, and select the “Sign-Only” protection order.

Your application is now set up with a WSE policy as shown in Figure 8.

?
Figure 6. Select the ‘AWS Customer’ Certificate: Uncheck the “Specify X-509 Certificate in Code” option, and select the AWS Customer certificate as shown in this figure for both the client and the server certificate.
?
Figure 7. Set Message Protection: For this application, set the protection order to “Sign Only.” The application does not use the “WS Security 1.1. Extensions” or “Secure Session” options.
?
Figure 8. Policy Applied: The figure shows the final “HyperscalePolicy”‘ application policy added to the WSE Wizard.

Overcoming the WSE061 Error
Completing the policy procedure in the WSE Settings Wizard as described in the preceding section creates a wse2policyCache.config file in the root folder of the project?but there’s a catch. Using this file (as created) with Amazon SQS will cause the application to throw a WSE exception, containing this inner exception:

   {"WSE061: The Timestamp header's  element is not valid."} 

This error occurs because of an incompatibility between Amazon’s SQS SOAP response and WSE’s Microsoft.Web.Services3.Design.MutualCertificate10Assertion.

To overcome this incompatibility you will need to swap WSE’s MutualCertificate10Assertion class with one of your own. This new assertion class must override the CreateClientInputFilter function.

Open the wse2policyCache.config file in Notepad or your favorite text editor and find the following entry:

   

Now create a project-specific HyperscaleAssertion class that extends MutualCertificate10Assertion as shown below

   using Microsoft.Web.Services3;   using Microsoft.Web.Services3.Design;      namespace Hyperscale   {      class HyperAssertion : MutualCertificate10Assertion      {         public override SoapFilter CreateClientInputFilter(            FilterCreationContext context)         {            return null;         }              }   }

Add an Amazon SQS Web Reference
Add a Web reference to Amazon SQS WSDL by pointing to http://queue.amazonaws.com/doc/2006-04-01/QueueService.wsdl as shown in Figure 9. This will create com.amazonws.queue.* proxies to access the Amazon SQS service.

?
Figure 9. Web Reference to Amazon SQS WSDL: The figure shows how to find the Amazon SQS WSDL file to add the Web reference to your project.
?
Figure 10. The Completed Dashboard: The figure shows the completed Amazon SQS dashboard form, which lets you perform common queue and message operations.

The Amazon SQS Dashboard:
You are now ready to start using Amazon SQS. Among other files the downloadable code contains the Windows Form frmHyper (the file Hyper.cs?see Figure 10) and a wrapper class for Amazon SQS specific calls named AmazonSQS.cs.

The dashboard lets you list, create, and delete queues (delete is available only when a Queue is empty), and create and delete messages.

Most of the code is self-explanatory, but I’ll go over couple of functions in the AmazonSQS.cs file. The getQueueNames() method shown below retrieves a list of queues. As shown, you first create an instance of QueueServiceWse (note the “Wse” suffix). All calls require you to construct parameters, which is this case is ListQueue. You pass this parameter to the ListQueues() function which returns the response, packaged as a ListQueueResponse instance. You then check for success and return the Queue list to the caller.

You will be prompted for a certificate password. This is the same password you supplied when you installed the certificate on Windows.

The getQueueService() method is a convenience function that creates an instance of QueueServiceWse and sets the AmazonSQS URL and the name of the WSE application policy you want to use.

   public static string[] getQueueNames()   {      com.amazonaws.queue.QueueServiceWse queueService =          getQueueService();         com.amazonaws.queue.ListQueues listQueues = new          com.amazonaws.queue.ListQueues();          com.amazonaws.queue.ListQueuesResponse listQueueResponse =          queueService.ListQueues(listQueues);         if (listQueueResponse == null ||          listQueueResponse.ResponseStatus.StatusCode != SUCCESS)         return null;         string[] queueNames = listQueueResponse.QueueUrl;      return queueNames;   }   private static com.amazonaws.queue.QueueServiceWse       getQueueService()      {      com.amazonaws.queue.QueueServiceWse queueService = new           com.amazonaws.queue.QueueServiceWse();      queueService.Url = "http://queue.amazonaws.com/";      queueService.SetPolicy("HyperscalePolicy");      return queueService;   }   

To use the sendMessage() function, you first create an instance of MessageQueueWse (note the Wse suffix again) that points to a specific queue. Then you create an instance of Message, load the message with string content, and send it using the MessageQueueWse.sendMessage() method.

The getMessageQueueWse method is another convenience function that sets the WSE policy and Queue name. Here’s the code for both methods:

   public static string sendMessage(string queueURL, string message)   {      com.amazonaws.queue.MessageQueueWse messageQueue =           getMessageQueue(queueURL); ;         com.amazonaws.queue.SendMessage sendMessage = new         com.amazonaws.queue.SendMessage();      sendMessage.MessageBody = message;         com.amazonaws.queue.SendMessageResponse response =           messageQueue.SendMessage(sendMessage);      if (response == null || response.ResponseStatus.StatusCode !=           SUCCESS)      {          return null;      }      return response.MessageId;   }         private static com.amazonaws.queue.MessageQueueWse       getMessageQueue(string queueURL)   {      com.amazonaws.queue.MessageQueueWse messageQueue = new           com.amazonaws.queue.MessageQueueWse();      messageQueue.SetPolicy("HyperscalePolicy");      messageQueue.Url = queueURL;      return messageQueue;   }   

The attached code contains a number of other useful functions that you can experiment with.

While Amazon SQS provides good basic queuing capabilities, it still does not provide capabilities such as such as journaling, acknowledgements, publish-subscribe, routing, etc that organizations have come to expect from messaging infrastructure. However, given the clout and weight Amazon carries, don’t be surprised if such features begin showing up in the future as part of Amazon SQS.

devx-admin

devx-admin

Share the Post:
Clean Energy Adoption

Inside Michigan’s Clean Energy Revolution

Democratic state legislators in Michigan continue to discuss and debate clean energy legislation in the hopes of establishing a comprehensive clean energy strategy for the

Chips Act Revolution

European Chips Act: What is it?

In response to the intensifying worldwide technology competition, Europe has unveiled the long-awaited European Chips Act. This daring legislative proposal aims to fortify Europe’s semiconductor

Revolutionized Low-Code

You Should Use Low-Code Platforms for Apps

As the demand for rapid software development increases, low-code platforms have emerged as a popular choice among developers for their ability to build applications with

Global Layoffs

Tech Layoffs Are Getting Worse Globally

Since the start of 2023, the global technology sector has experienced a significant rise in layoffs, with over 236,000 workers being let go by 1,019

Clean Energy Adoption

Inside Michigan’s Clean Energy Revolution

Democratic state legislators in Michigan continue to discuss and debate clean energy legislation in the hopes of establishing a comprehensive clean energy strategy for the state. A Senate committee meeting

Chips Act Revolution

European Chips Act: What is it?

In response to the intensifying worldwide technology competition, Europe has unveiled the long-awaited European Chips Act. This daring legislative proposal aims to fortify Europe’s semiconductor supply chain and enhance its

Revolutionized Low-Code

You Should Use Low-Code Platforms for Apps

As the demand for rapid software development increases, low-code platforms have emerged as a popular choice among developers for their ability to build applications with minimal coding. These platforms not

Cybersecurity Strategy

Five Powerful Strategies to Bolster Your Cybersecurity

In today’s increasingly digital landscape, businesses of all sizes must prioritize cyber security measures to defend against potential dangers. Cyber security professionals suggest five simple technological strategies to help companies

Global Layoffs

Tech Layoffs Are Getting Worse Globally

Since the start of 2023, the global technology sector has experienced a significant rise in layoffs, with over 236,000 workers being let go by 1,019 tech firms, as per data

Huawei Electric Dazzle

Huawei Dazzles with Electric Vehicles and Wireless Earbuds

During a prominent unveiling event, Huawei, the Chinese telecommunications powerhouse, kept quiet about its enigmatic new 5G phone and alleged cutting-edge chip development. Instead, Huawei astounded the audience by presenting

Cybersecurity Banking Revolution

Digital Banking Needs Cybersecurity

The banking, financial, and insurance (BFSI) sectors are pioneers in digital transformation, using web applications and application programming interfaces (APIs) to provide seamless services to customers around the world. Rising

FinTech Leadership

Terry Clune’s Fintech Empire

Over the past 30 years, Terry Clune has built a remarkable business empire, with CluneTech at the helm. The CEO and Founder has successfully created eight fintech firms, attracting renowned

The Role Of AI Within A Web Design Agency?

In the digital age, the role of Artificial Intelligence (AI) in web design is rapidly evolving, transitioning from a futuristic concept to practical tools used in design, coding, content writing

Generative AI Revolution

Is Generative AI the Next Internet?

The increasing demand for Generative AI models has led to a surge in its adoption across diverse sectors, with healthcare, automotive, and financial services being among the top beneficiaries. These

Microsoft Laptop

The New Surface Laptop Studio 2 Is Nuts

The Surface Laptop Studio 2 is a dynamic and robust all-in-one laptop designed for creators and professionals alike. It features a 14.4″ touchscreen and a cutting-edge design that is over

5G Innovations

GPU-Accelerated 5G in Japan

NTT DOCOMO, a global telecommunications giant, is set to break new ground in the industry as it prepares to launch a GPU-accelerated 5G network in Japan. This innovative approach will

AI Ethics

AI Journalism: Balancing Integrity and Innovation

An op-ed, produced using Microsoft’s Bing Chat AI software, recently appeared in the St. Louis Post-Dispatch, discussing the potential concerns surrounding the employment of artificial intelligence (AI) in journalism. These

Savings Extravaganza

Big Deal Days Extravaganza

The highly awaited Big Deal Days event for October 2023 is nearly here, scheduled for the 10th and 11th. Similar to the previous year, this autumn sale has already created

Cisco Splunk Deal

Cisco Splunk Deal Sparks Tech Acquisition Frenzy

Cisco’s recent massive purchase of Splunk, an AI-powered cybersecurity firm, for $28 billion signals a potential boost in tech deals after a year of subdued mergers and acquisitions in the

Iran Drone Expansion

Iran’s Jet-Propelled Drone Reshapes Power Balance

Iran has recently unveiled a jet-propelled variant of its Shahed series drone, marking a significant advancement in the nation’s drone technology. The new drone is poised to reshape the regional

Solar Geoengineering

Did the Overshoot Commission Shoot Down Geoengineering?

The Overshoot Commission has recently released a comprehensive report that discusses the controversial topic of Solar Geoengineering, also known as Solar Radiation Modification (SRM). The Commission’s primary objective is to

Remote Learning

Revolutionizing Remote Learning for Success

School districts are preparing to reveal a substantial technological upgrade designed to significantly improve remote learning experiences for both educators and students amid the ongoing pandemic. This major investment, which

Revolutionary SABERS Transforming

SABERS Batteries Transforming Industries

Scientists John Connell and Yi Lin from NASA’s Solid-state Architecture Batteries for Enhanced Rechargeability and Safety (SABERS) project are working on experimental solid-state battery packs that could dramatically change the

Build a Website

How Much Does It Cost to Build a Website?

Are you wondering how much it costs to build a website? The approximated cost is based on several factors, including which add-ons and platforms you choose. For example, a self-hosted

Battery Investments

Battery Startups Attract Billion-Dollar Investments

In recent times, battery startups have experienced a significant boost in investments, with three businesses obtaining over $1 billion in funding within the last month. French company Verkor amassed $2.1

Copilot Revolution

Microsoft Copilot: A Suit of AI Features

Microsoft’s latest offering, Microsoft Copilot, aims to revolutionize the way we interact with technology. By integrating various AI capabilities, this all-in-one tool provides users with an improved experience that not

AI Girlfriend Craze

AI Girlfriend Craze Threatens Relationships

The surge in virtual AI girlfriends’ popularity is playing a role in the escalating issue of loneliness among young males, and this could have serious repercussions for America’s future. A

AIOps Innovations

Senser is Changing AIOps

Senser, an AIOps platform based in Tel Aviv, has introduced its groundbreaking AI-powered observability solution to support developers and operations teams in promptly pinpointing the root causes of service disruptions