0

I was using node-posgres parameterized queries.

However, it was inserting data very slowly: 100 rows was taking 5 seconds!

Switching to pg-format and producing plain old SQL strings - even when executing them one at a time - reduced the time to insert 100 rows of the same data to 125 ms.

(If I then concatenate all 100 insert SQL strings and send it as a single query it goes down to just 25ms)

I found this question (from Dec 13, 2010 mind you) that suggests a C# PostgreSQL client had serious problems with it.

Is node-postgres the same?

That is, should we not be using parameterized queries with node-postgres, or am I just doing something stupid?

  • pg v8.11.1
  • node v18.14

Thanks!

Here's the basic slow code if you're interested:

const insertSQL = `
    INSERT INTO "mytable"
    (  "column1", "column2", "column3",
        "column4", "column5", "column6",
        "column7", "column8", "column9"
    ) VALUES ( $1, $2, $3, $4, $5, $6, $7, $8, $9 );
`;

await pgClient.query("BEGIN");
for( let i=0; i<100; i+=1 ) {
    const test_data = test_datum[i];
    const values = [
            test_data.column1, test_data.column2, test_data.column3,
            test_data.column4, test_data.column5, test_data.column6,
            test_data.column7, test_data.column8, test_data.column9
        ];
    await pgClient.query(insertSQL, values);
}
await pgClient.query("COMMIT");

And the pg-format code that inserts 100 records in 125ms:

await pgClient.query("BEGIN");
for( let i=0; i<100; i+=1 ) {
    const test_data = test_datum[i];
    const insertSql = pgformat(`
            INSERT INTO "mytable"
            (  "column1", "column2", "column3",
                "column4", "column5", "column6",
                "column7", "column8", "column9"
            ) VALUES (%L,  %L,  %L, %L,  %L,  %L, %L,  %L,  %L );
        `, test_data.column1, test_data.column2, test_data.column3,
            test_data.column4, test_data.column5, test_data.column6,
            test_data.column7, test_data.column8, test_data.column9 );
    await pgClient.query(insertSQL);
}
await pgClient.query("COMMIT");

7
  • Your so-called 125ms code wouldn't run at all due to a syntax error in the SQL - there's an errant comma after the last %L. Please be accurate when posting code. Commented Jul 3, 2023 at 22:21
  • What are the data types of test_data.columnX variables and the data types of the corresponding columns? As an aside, I would delete all quotes from your SQL as they are unnecessary code noise. Commented Jul 3, 2023 at 22:23
  • This is just sample SQL, to give you an idea. Unfortunately I can't post the real code. Commented Jul 3, 2023 at 23:21
  • Some of the real column names require quotes, but agreed, I could have simplified the sample code further. The (real) columns are mostly strings, a few numbers, and a few dates. I have code similar to this for about 10 different tables, and each of them are dramatically slower with the parameterized queries than with the SQL string formatted using pg-format. Commented Jul 3, 2023 at 23:38
  • The important point I'm trying to uncover is the parameter types should agree with the column types to avoid type conversion overhead. Although the performance hit you're experiencing does seem beyond what conversion would cause, it's worth eliminating that aspect. Commented Jul 3, 2023 at 23:50

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.