In order to process cleanup jobs, that will run for every 8 hours, currently we have implemented as :

1. Create scheduled Job using Azure Scheduler that puts message in storage queue when it is triggered.

2. Implement client in such a way that it will poll continuously and process whenever it receives message. Sample implementation of client is :

    while (!CancellationToken.Value.IsCancellationRequested)    
           {
   
            var message = await client.GetMessageAsync();
   
               if (message != null)
               {
                   // process the message
               }
   }
   

But the problem is we are waiting indefinetly even we know that we will get messages only after 8hours and also as per documentation, every attempt to read message from queue will incur cost.

How to optimize this in such a way that listeners will be spawned up on the fly for every configurable time instead of continuous loop?

Hi,

 A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: its used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
 Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
 
 Let us know if this helps.

Regards,
Nithin Rathnakar

Hi,

 A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: its used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
 Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
 
 Let us know if this helps.

Regards,
Nithin Rathnakar

• Proposed as answer by Nithin.RathnakarMicrosoft contingent staff, Moderator Thursday, July 23, 2015 2:40 PM
• Marked as answer by Michael CurdMicrosoft employee, Moderator Monday, July 27, 2015 10:42 PM
• Unmarked as answer by PVMRAGHUNANDAN 15 hours 50 minutes ago

Hi,

 A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: its used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
 Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
 
 Let us know if this helps.

Regards,
Nithin Rathnakar

• Proposed as answer by Nithin.RathnakarMicrosoft contingent staff, Moderator Thursday, July 23, 2015 2:40 PM
• Marked as answer by Michael CurdMicrosoft employee, Moderator Monday, July 27, 2015 10:42 PM
• Unmarked as answer by PVMRAGHUNANDAN Tuesday, July 28, 2015 3:29 PM

Hi,

 A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: its used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
 Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
 
 Let us know if this helps.

Regards,
Nithin Rathnakar

• Proposed as answer by Nithin.RathnakarMicrosoft contingent staff, Moderator Thursday, July 23, 2015 2:40 PM
• Marked as answer by Michael CurdMicrosoft employee, Moderator Monday, July 27, 2015 10:42 PM
• Unmarked as answer by PVMRAGHUNANDAN Tuesday, July 28, 2015 3:29 PM

Hi,

 A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: its used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
 Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
 
 Let us know if this helps.

Regards,
Nithin Rathnakar

• Proposed as answer by Nithin.RathnakarMicrosoft contingent staff, Moderator Thursday, July 23, 2015 2:40 PM
• Marked as answer by Michael CurdMicrosoft employee, Moderator Monday, July 27, 2015 10:42 PM
• Unmarked as answer by PVMRAGHUNANDAN Tuesday, July 28, 2015 3:29 PM

Thanks for the reply.

I am thinking of this approach.

1. Expose an REST service on one endpoint of existing service.

2. Create scheduled job in such a way that it will call the hosted REST service in configured time.

3. During handling of REST call, we will spawn up listener and continue reading for 'X' minutes.

Reason for selecting this is we know that background jobs will be triggered every 8 hours . Also doing backoff mayn't be applicable.

Any Thoughts??

Thanks

RaghuNandan

• Edited by PVMRAGHUNANDAN 14 hours 3 minutes ago

Thanks for the reply.

I am thinking of this approach.

1. Expose an REST service on one endpoint of existing service.

2. Create scheduled job in such a way that it will call the hosted REST service in configured time.

3. During handling of REST call, we will spawn up listener and continue reading for 'X' minutes.

Reason for selecting this is we know that background jobs will be triggered every 8 hours . Also doing backoff mayn't be applicable.

Any Thoughts??

Thanks

RaghuNandan

• Edited by PVMRAGHUNANDAN Tuesday, July 28, 2015 5:17 PM

Thanks for the reply.

I am thinking of this approach.

1. Expose an REST service on one endpoint of existing service.

2. Create scheduled job in such a way that it will call the hosted REST service in configured time.

3. During handling of REST call, we will spawn up listener and continue reading for 'X' minutes.

Reason for selecting this is we know that background jobs will be triggered every 8 hours . Also doing backoff mayn't be applicable.

Any Thoughts??

Thanks

RaghuNandan

• Edited by PVMRAGHUNANDAN Tuesday, July 28, 2015 5:17 PM

Thanks for the reply.

I am thinking of this approach.

1. Expose an REST service on one endpoint of existing service.

2. Create scheduled job in such a way that it will call the hosted REST service in configured time.

3. During handling of REST call, we will spawn up listener and continue reading for 'X' minutes.

Reason for selecting this is we know that background jobs will be triggered every 8 hours . Also doing backoff mayn't be applicable.

Any Thoughts??

Thanks

RaghuNandan

• Edited by PVMRAGHUNANDAN Tuesday, July 28, 2015 5:17 PM

Any suggestions on this ?