Distributed Synchronized Background Jobs ASP Core






Problem:-

In the age of containerized apps, running multiple instances, and Kubernetes, sometimes you face a situation where you want to run a background jobs in the ASP Core, but you have multiple instances of application running on Kubernetes for example, so you will end up with duplicated running background jobs equal to the number of running application instances (e.g. same background job runs for each application instance)


Solution:-

  1. Easy and straight forward : separate the background jobs into a new solution and deploy one instance of it (100% confidence of running one instance)
  2. Do a synchronization mechanism to allow only one instance runs jobs ( not 100% confidence of running one instance) 

We will talk about second solution:-
  1. We need to start jobs at different times, not at the same time, so, we can synchronize work (e.g. one job-instance start with the smallest time and the others in a different times).
  2. We need a "centralized store" to store the state of synchronization mechanism with cheap access like (Redis)
  3. We need to monitor the starting instance that take the work so, schedule work to another instance for starting in case of failure.
Let us now discuss the first point in the second solution.
  • I need a unique random number generator that generates unique numbers at same time of calls (e.g. multiple instances run at the same time and each instance call the random number generator to delay the amount of time and then check synchronization state to check if it can start work or not)
    • If we choose "Random class" to get random number, it returns the same number when calls multiple times at the same time


    • Let's try "Guid", it's generates different Guids when calls at the same time




    • We need away to convert Guid to a Number, it rarely make a collision (e.g. return same number for multiple calls at the same time).

    • Code For Convert Guid to "scaled number threshold"

  • We choose Redis Distributed cache to store Synchronization State

  • For Synchronization Mechanism
    • Redis store synchronization state





  • Synchronization Workflow: 
    • namespace "DistributedSynchronizedBackgroundJobs.BackgroundJobs"
    • class "TimedHostedBackgroundService : BackgroundService"












Comments

Post a Comment

Popular posts from this blog

Async/Await - OS Concept

Image
 There are many question about Async/Await in "dot net"? what really are? what is the issues it handle? is there any dedicated thread/mechanism monitor when a task completes, so notify thread pool to choose thread and continue execution? when to use it? to answer this question let us deep dive a little in some Operating System (OS) concepts and mechanism to handle I/O. What is I/O?:-  I/O (input/output), pronounced "eye-oh," describes any operation, program, or device that transfers data to or from a computer. Typical I/O devices are printers, network card, hard disks, keyboards, and mouses. In fact, some devices are basically input-only devices (keyboards and mouses); others are primarily output-only devices (printers); and others provide both input and output of data (hard disks, diskettes, writable CD-ROMs). Programing examples require I/O operations:- - when application query database it get data from hard disk(I/O) - when call api, it call network card(I/O) - w...
Read more

Fluent Notification Sender .Net Package

Image
  Most modern software projects have need to an essential feature of sending notifications to clients (e.g., user registration & confirmation, automated request process follow up, topic & news subscription and alerting system…).   Notifications can be in many flavors (Email, SMS, Realtime, and mobile push notification) beside send by multiple methods according to system or client preferences (e.g., sending both Email and SMS notification for specific events occurs in system). Each software project has a different requirement so, it can use a different provider vendor (e.g., send email by SMTP protocol or by SendGrid API, …etc.) to fulfil it. Sending notifications is expensive I/O operation and can affect application performance if it not coded will. Some project's requirements need to log or retry failed notifications .   So , Notification project as a component designed to deal with the above concerns. Design Specifications: Support multiple notifications metho...
Read more

Thundering Herd Problem - ASP Core Solution Architect

Image
  What is the Thundering herd problem? Lets describe it using a realistic problem to get better understanding. Suppose an exposed Weather-API that returns current temperature degree, called by web/mobile apps, and optimized for performance so, for simplicity it utilize caching capabilities so, when it is called, checks for caching and if there is "cache miss", proceed and call a third-party API and cache for later use. The problem happens when there are a multiple calls to Weather-API at the same time and there is a "cache miss" (e.g. data not found in cache) so, each request go and call third-party API and then if succeed, it will cache data.   The 3-party API is slow and not well-designed for this huge number of requests so, it will be down and typically Weather-API  goes down or out of functionality (e.g. due to coupling with third-party API and not a proper handling for exceptions). It is a real complicated problem especially when there are a multiple running ...
Read more