0

Cant figure out how to send arrays of objects

Currently I have a back-end MongoDB database with multiple collections which I have to query for matching data and send the data back to the client as soon as they are received. This is the function that I currently use for my express server:

async function retrieve_multiple(req, res) {
    let requests = []

    let search_string = req.query.search_string;
    let limit = req.query.limit
    let error = false

    let keywords = [
        'are',
        'can',
        'did',
        'does',
        'how',
        'should',
        'what',
        'when',
        'which',
        'who',
        'why'
    ]

    // Sending the initial requests
    for (var keyword of keywords) {
        requests.push( models.getArticle(keyword, search_string, limit) ) //The function I use to retrieve data from the MongoDB instance. Return type is the standard array of JSON objects
    }

    res.set('Access-Control-Allow-Origin', *)

    requests.map( async (data) => {
        let da = await data
        res.write(da)
    })

    await Promise.all(requests)
    res.send()
}

With this, I face the evident problem of not having a correctly formatted JSON object on the client side. I wanted to know as to how would it be possible to send data responses back to the client such that:

  • Receive the data in chunks
  • Have the chunk formulate a valid JSON object
  • Have the data received as soon as the promise is fulfilled

Some part of me tells me that this would not be possible if I want to receive workable JSON data on the client side in real time because for as far as I know, that is just not possible. If that is so, do you think that I would be better of doing this on the client side:

let data_object = []

async function retrieve_multiple() {
    let requests = []

    let keywords = [
        'are',
        'can',
        'did',
        'does',
        'how',
        'should',
        'what',
        'when',
        'which',
        'who',
        'why'
    ]

    // Sending the initial requests
    for (var keyword of keywords) {
        requests.push( fetch( `my.api/foo/bar?$keyword={keyword}` ) ) // Calling the fetch function client side
    }

    requests.map( async (data) => {
        let da = await data
        data_object.push( da.json() )
    })

    await Promise.all(requests)


    console.log('Done')
}

and just have the server side have a function to retrieve data from a singular collection?
Would this approach have any downsides to it or is it just as valid?
Thanks!

4
  • can you tell me what you mean by on the client side in real time if you're saying you need the UI to update based on data change then you should be able to handle that with any frontend library that claims to be reactive like React and Vue Commented Oct 6, 2022 at 17:52
  • There are a few issues with your request. For example: Say the first query returns, so you open the array and start outputting the results of the first query. You don't close the array. How will you close the array if the other queries do not return data? Furthermore, how do you determine how many queries are still pending? You need to close the array after all queries end. But then again, I see the await statement is on all promises, so you're not returning data as queries complete, but only after all queries complete. Commented Oct 7, 2022 at 3:50
  • @JoséRamírez Yes that is accounted for as the bigger main array is more like an array of arrays and my svelte app handles the 'dynamic' nature of the array as is. As far as the handling errors or empty arrays go, the proposed client side solution was just aribitrary and not permanant. In production, I would have a svelte endpoint handle all the error handeling and updating the data in a store. I just wanted to ask which solution would be better suited. Commented Oct 8, 2022 at 9:04
  • @MCGRAW what I mean by that is to just have an array be updated as soon as new data comes in. The reactivity of the UI is handled quite elegantly by svelte and I dont have to worry about that, all I care about is that the array that feeds the front end UI be updated with new data as soon as it is available Commented Oct 8, 2022 at 9:06

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.