Incrementally Append to JSON File in a For Loop

Question

Is there a way to append single JSON objects to a JSON file while in a for loop in Python. I would prefer not store all my data in one giant JSON object and dump it all at once, as I am planning on performing millions of API requests. I would like to make a single API request, dump the result into a JSON file and then move to the next API request and dump that into the same JSON file.

The below code overwrites the JSON file, I am looking for something that appends:

for url in urls:
    r = sesh.get(url)
    data = r.json()

    with open('data.json', 'w') as outfile:
        json.dump(data, outfile)

Such that:

with open('data.json') as outfile:
    data = json.load(data, outfile)

type(data)
>> dict

r.json looks something like this:

{'attribute1':1, 'attribute2':10}

Have you tried anything? It would be more helpful if you can show us where you are stuck, or a pseudo-code of what you're trying to accomplish. — ilke444
– ilke444, Commented Feb 1, 2017 at 19:05
sure. its a very simple task-- which is why i didnt include any code. — eljusticiero67
– eljusticiero67, Commented Feb 1, 2017 at 19:08

Z-Bone · Accepted Answer · 2017-02-03 08:37:10Z

4

Update

Well since I don't have access to your API I just placed some sample responses, in the format you supplied, inside an array.

import json

urls = ['{"attribute1":1, "attribute2":10}', '{"attribute1":67, "attribute2":32}', '{"attribute1":37, "attribute2":12}'];
json_arr = []

for url in urls:
    data = json.loads(url)
    json_arr.append(data)
    with open('data.json', 'w') as outfile:
        json.dump(json_arr, outfile)

Basically we keep an array and append each API response to that array. Then, we can write the accumulative JSON to a file. Also if you want to update the same JSON file on different executions of the code, you can just read the existing output file into an array, in the beginning of the code, and then carry on with my example.

Change write mode to append

Try changing this:

with open('data.json', 'w') as outfile:

To this:

with open('data.json', 'a') as outfile:

edited Feb 3, 2017 at 8:37

answered Feb 1, 2017 at 19:40

Z-Bone

1,5841 gold badge10 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

eljusticiero67 Over a year ago

thanks-- probably didnt make it clear, but i was hoping that it'd append to a json file. that just appends the json objects into a file (not as one large json object). when i try to reload the file, I get an error since its not a json object. I'll edit the question to make it clearer.

Z-Bone Over a year ago

Well I'd need to see the basic structure of the JSON to give you an exact solution

Z-Bone Over a year ago

That's not a valid JSON.. you mean they are in an array maybe? Like this: [ {'attribute1':1}, {'attribute2':10} ] Or each call, one is returned like this: {'attribute2':10}

eljusticiero67 Over a year ago

sorry, im dumb. made the edit to show what it actually looks like. thanks for your patience.

Mattwmaster58 Over a year ago

This is a pretty bad solution, it rewrites the entire file for every single append

|

ioannis.th · Accepted Answer · 2022-10-09 20:47:29Z

1

The previous answer is surprisingly close to what you need to do. So I will build upon it.

import json

json_arr = ['{"attribute1":1, "attribute2":10}', '{"attribute1":67, "attribute2":32}', '{"attribute1":37, "attribute2":12}'];

with open('data.json', 'w') as outfile:
  outfile.write('[')

for element in json_arr:
    with open('data.json', 'w') as outfile:
        json.dump(element, outfile)
        outfile.write(',')
with open('data.json', 'a') as outfile:
  outfile.write(']')

answered Oct 9, 2022 at 20:47

ioannis.th

4631 gold badge5 silver badges11 bronze badges

Comments

Vincent Stragier · Accepted Answer · 2023-09-11 20:55:46Z

I rewrote @ioannis.th's code, since there are some bugs. But yes, it is quite easy to implement. Notice the indentation works in this specific case, but I did not test it with deeper dictionaries:

import json

json_arr = [
    '{"attribute1":1, "attribute2":10}',
    '{"attribute1":67, "attribute2":32}',
    '{"attribute1":37, "attribute2":12}',
]

with open("data.json", mode="w", encoding="utf-8") as outfile:
    # Start of json_arr
    outfile.write("[\n")

    # Index of last element in json_arr
    json_arr_last = len(json_arr) - 1

    for index, element in enumerate(json_arr):
        # Add indentation
        outfile.write("    ")

        # Write element
        json.dump(element, outfile, indent=4)

        # Add comma if not last element
        if index != json_arr_last:
            outfile.write(",")

        # Add newline
        outfile.write("\n")

    # End of json_arr
    outfile.write("]\n")

Collectives™ on Stack Overflow

Incrementally Append to JSON File in a For Loop

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related