Insert CSV data into Bigquery table using Python

Question

I'm trying to insert csv file into bigquery using python, but I think I have missed something since the result is replace,

from google.cloud import bigquery  
from google.oauth2 import service_account
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=r"C:/Users/Pamungkas/Documents/Dump_Data/testing-353407-a3c774efeb5a.json"

client = bigquery.Client() 

table_id="testing-353407.testing_coba.Sales_Menu_COGS_Detail_Report"
file_path=r"C:\Users\Pamungkas\Downloads\Sales_Menu_COGS_Detail_Report_Jan.csv"

job_config = bigquery.LoadJobConfig(
    source_format=bigquery.SourceFormat.CSV, skip_leading_rows=1, autodetect=True,
        write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE 
)

with open(file_path, "rb") as source_file:
    job = client.load_table_from_file(source_file, table_id, job_config=job_config)
    
job.result()  # Waits for the job to complete.

table = client.get_table(table_id)  # Make an API request.
print(
    "Loaded {} rows and {} columns to {}".format(
        table.num_rows, len(table.schema), table_id
    )
)

I guess the problem is in job_config, but I still didn't get it, can anyone help me on this?

Try changing the value of your write_disposition to bigquery.WriteDisposition.WRITE_APPEND — Ricco D
– Ricco D, Commented Jun 27, 2022 at 5:57

2 revs · Accepted Answer · 2022-06-28 12:39:37Z

3

As mentioned by @RiccoD, since you're appending the data from csv into BigQuery, you'll have to change the write disposition in job config to WRITE_APPEND.

So change the job config part as:

job_config = bigquery.LoadJobConfig(
    source_format=bigquery.SourceFormat.CSV, skip_leading_rows=1, autodetect=True,
        write_disposition=bigquery.WriteDisposition.WRITE_APPEND

Collectives™ on Stack Overflow

Insert CSV data into Bigquery table using Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related