>"Limit the rate at which jobs are pushed into the distributed job queue. This can be achieved by implementing queue-level throttling, where only a certain number of jobs are allowed to enter the queue per second or minute."
If the execution time is the same for all jobs and the nu…
>"Limit the rate at which jobs are pushed into the distributed job queue. This can be achieved by implementing queue-level throttling, where only a certain number of jobs are allowed to enter the queue per second or minute."
If the execution time is the same for all jobs and the number of jobs exceeds the queue-level throttling limit, how do you determine which jobs should be submitted to the queue—especially if no priority is defined?
>Rate Limiting at the Worker Node Level
Do we need this ? given we already know available capacity for a single worker and based on that we could be able to decide to submit a new job or not.
If a job belongs to segment X and all workers assigned to it are busy, with no indication of when they will become free, a starvation problem could arise. Although there may be free workers in segment Y, they are unable to execute jobs from segment X.
This raises the question: is there a fallback mechanism to handle such scenarios, or should the system dynamically reassign segments based on worker availability?
Good details. great work :)
Couple of follow up questions -
>"Limit the rate at which jobs are pushed into the distributed job queue. This can be achieved by implementing queue-level throttling, where only a certain number of jobs are allowed to enter the queue per second or minute."
If the execution time is the same for all jobs and the number of jobs exceeds the queue-level throttling limit, how do you determine which jobs should be submitted to the queue—especially if no priority is defined?
>Rate Limiting at the Worker Node Level
Do we need this ? given we already know available capacity for a single worker and based on that we could be able to decide to submit a new job or not.
If a job belongs to segment X and all workers assigned to it are busy, with no indication of when they will become free, a starvation problem could arise. Although there may be free workers in segment Y, they are unable to execute jobs from segment X.
This raises the question: is there a fallback mechanism to handle such scenarios, or should the system dynamically reassign segments based on worker availability?