how to create 1 gb csv file with random data using python

Question

import csv
import random
from faker import Faker
from datetime import datetime


l=Faker('en_GB') 
f=open("test.csv","r")
k=csv.reader(f)

g=open("1.csv","w")
w=csv.writer(g)
w.writerow(('id','name','address','college','company','dob','age'))
for i in range(20000):

    w.writerow((i+1,l.name(),l.address(),random.choice(['psg','sona','amirta','anna university']),random.choice(['CTS','INFY','HTC']),(random.randrange(1950,1995,1),random.randrange(1,13,1),random.randrange(1,32,1)),random.choice(range(0,100))))
f.close()

when i increase the range to 10000000 terminal kills the process...... please can anyone help me . how can i generate larger csv file with random data .

What do you mean the terminal kills the process? It is automatically being killed or you are killing it? — Burhan Khalid
– Burhan Khalid, Commented Jul 17, 2015 at 11:22

paxdiablo · Accepted Answer · 2015-07-17 05:12:34Z

2

If you're using Python 2, range will create a list and may therefore have memory troubles with large input values.

If that's the case, use xrange instead. It has fixed memory requirements regardless of the input value.

The Python 3 range is more akin to the Python 2 xrange so it shouldn't be a problem there.

answered Jul 17, 2015 at 5:12

paxdiablo

889k243 gold badges1.6k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

gotam · Accepted Answer · 2015-07-18 06:29:38Z

0

import csv
import random
from faker import Faker
from datetime import datetime


l=Faker('en_GB') 
f=open("test.csv","r")
k=csv.reader(f)

g=open("1.csv","a")
w=csv.writer(g)
w.writerow(('id','name','address','college','company','dob','age'))
for i in range(1000000):

    w.writerow((i+1,l.name(),l.address(),random.choice(['psg','sona','amirta','anna university']),random.choice(['CTS','INFY','HTC']),(random.randrange(1950,1995,1),random.randrange(1,13,1),random.randrange(1,32,1)),random.choice(range(0,100))))
f.close()

append option solved my problem.

g=open("1.csv","a")

after i use this its taking time to create the csv file of 1.6 gb.with random data

answered Jul 18, 2015 at 6:29

gotam

1932 silver badges11 bronze badges

Collectives™ on Stack Overflow

how to create 1 gb csv file with random data using python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related