Questions tagged [queueing]
The queueing tag has no summary.
19 questions
1
vote
0
answers
881
views
AWS multiple SQS queues on one bucket without fanout
I am coming into a project that uses an Amazon S3 Bucket for a process.
For my application, I need to know (within a reasonable time - order of a few minutes, at most) whether a new file has beeen ...
- 161
0
votes
3
answers
158
views
Queue for tasks and notifications
I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to ...
1
vote
0
answers
129
views
How to have only one consumer handle a long running task at a given point?
I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
- 177
0
votes
0
answers
185
views
What would be the best way to queue asynchronous calls received simultaneously by a web service cluster?
I have an API where every time is called there are multiple calls to multiple webservices in the background, sometimes it takes up to 20 seconds to process the request calling up to 10 different web ...
- 109
0
votes
1
answer
112
views
If I have this condition, should I queue the listeners or the notification?
I'm trying to wrap my head around a Queue system. I have read many resources about it but I keep ghosted with this question.
If I have listeners that loops over a collection of users, to send a ...
- 109
1
vote
1
answer
600
views
Queuing emails in Laravel when using an api driver such as Mailgun?
In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing.
I have a scheduled job to send out daily email alerts to users. Currently the job drops the ...
- 321
4
votes
3
answers
1k
views
Strategy for ensuring consistency when integrating using an API?
How do I ensure consistency between two separate systems which store and maintain the same set of data, when the method of communication between them is REST APIs and webhooks? Is it possible?
I find ...
- 169
1
vote
1
answer
367
views
How do I set up short-lived queues?
This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
- 1,110
-1
votes
1
answer
121
views
.NET Sort concurrent requests into a queue
I asked this at Stack Overflow but seeing as it wasn't so much a programming issue as a theory issue it wasn't well received.
I have a program that issues a unique ascending code every time one is ...
2
votes
1
answer
389
views
How to detect end of queue in a parallelized web crawler?
Let's say I want to program a parallelized web crawler which has a shared FIFO queue (multi consumer/multi producer). The queue only contains URLs. How do I detect the end of the queue?
A worker ...
- 41
0
votes
0
answers
72
views
Synchronized computation task distribution
I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
- 147
3
votes
2
answers
2k
views
Is it worth it to use a message-queue framework in this case?
We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a ...
- 195
2
votes
1
answer
1k
views
Difference between server Jobs and Workers?
I am trying to understand the difference between server workers and server jobs. Google has not been very helpful because the keyword "jobs" is being construed as an employment term... I have yet to ...
- 131
2
votes
2
answers
931
views
Shared & exclusive locking
I need a "lock" that can be either shared or held exclusively, that will provide the following behavior for the following sequence of events:
Process A: Requests and is granted a shared lock
Process B:...
- 743
4
votes
1
answer
1k
views
Seeking Guidance on Thread Safe Scripting
I am trying to write a script that will take multiple paths to files on various servers, search them all simultaneously, and return a single list of results to a user. Initially, I was just using ...
- 43
0
votes
1
answer
292
views
What, why or when it is better to choose a queuing tool vs database task table?
I would like to know what, why or when it is better to choose a queuing tool and what, why or when it is better to choose database task table? I don't understand the difference of sending my tasks in ...
- 1,633
3
votes
0
answers
418
views
How to schedule large numbers of user-defined tasks?
Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails).
You could run a cron which SELECTs a database table for unprocessed ...
- 173
1
vote
1
answer
995
views
How to queue work orders by date and priority?
In my organization I work on orders based on date, so earliest work orders first. We also have an option for order to be marked "priority" in which case those orders are done first but not always.
I'...
- 13
13
votes
3
answers
2k
views
Are we queueing and serializing properly?
We process messages through a variety of services (one message will touch probably 9 services before it's done, each doing a specific IO-related function). Right now we have a combination of the ...
- 2,814