Populating n level PostgreSQL table in python

Question

Im having trouble thinking through how to populate an n level table(in my case its 3 levels), Im using python to fetch the data from the query but I'm not sure how to approach populating my new table resources since it references itself. Any feedback on the approach will be much appreciated!

After running the following query in my python file I get the following table

SELECT distinct c.table_catalog AS "Database", c.table_schema AS "Schema", c.table_name AS "Table"
FROM information_schema.columns c
WHERE c.table_schema != 'information_schema' AND c.table_schema != 'pg_catalog' AND c.table_schema != 'pg_internal' AND c.table_schema not like '% %'
ORDER BY c.table_schema, c.table_name;


Database  Schema            Table
____________________________________
dev       BigBangTheory     SomeTable1
dev       BigBangTheory     SomeTable2
dev       BigBangTheory     SomeTable3
dev       Walle             AnotherTable100
dev       Walle             AnotherTable200
dev       StarWars          SpaceTablexxx
dev       StarWars          SpaceTableyyy
stage     BigBangTheory     SomeTable1
stage     BigBangTheory     SomeTable2
stage     BigBangTheory     SomeTable3
stage     Walle             AnotherTable100
stage     Walle             AnotherTable200
stage     StarWars          SpaceTablexxx
stage     StarWars          SpaceTableyyy

And I have another table that I want to populate using the above results. The table I want to populate looks like so:

CREATE TABLE IF NOT EXISTS resources
(
"id" SERIAL NOT NULL PRIMARY KEY,
"type" varchar(100) NOT NULL,             
"name" varchar(100) NOT NULL,    
"parent" int,
FOREIGN KEY (parent) REFERENCES resources (id)
);

so my goal is for table resources to look like this:

id      type         name                   parent
____________________________________________________
1       database     dev                    NULL
2       schema       BigBangTheory          1
3       table        SomeTable1             2
4       table        SomeTable2             2
5       table        SomeTable3             2
6       schema       Walle                  1
7       table        AnotherTable100        6
8       table        AnotherTable200        6
9       schema       StarWars               1
10      table        SpaceTablexxx          9
11      table        SpaceTableyyy          9

12      database     stage                  NULL
13      schema       BigBangTheory          12
14      table        SomeTable1             13
15      table        SomeTable2             13
16      table        SomeTable3             13
17      schema       Walle                  12
18      table        AnotherTable100        17
19      table        AnotherTable200        17
20      schema       StarWars               12
21      table        SpaceTablexxx          20
22      table        SpaceTableyyy          20

Thank you in advance! All feedback is appreciated <3

GMB · Accepted Answer · 2020-06-17 20:00:35Z

As a starter: you can get the information you want directly from information_schema.tables rather than information_schema.columns (it has just one rows per table, so need for distinct).

Then: in Postgres, you can do what you want in a single query, using cascading common table expressions with the returning clause to insert statements in Postgres. can

The logic is to insert the top objects first (the databases) and return the generated serials, then insert the schemas (using the database serials), and finally the tables.

with 
    info as (
        select c.table_catalog, c.table_schema, c.table_name
        from information_schema.tables
        where 
            c.table_schema not in ('information_schema', 'pg_catalog', 'pg_internal')
            and c.table_schema not like '% %'
    ),
    dbs as (
        insert into resources (type, name)
        select distinct 'database', table_catalog 
        from info
        returning id, name
    ),
    schemas as (
        insert into resources(type, name, parent)
        select distinct 'schema', i.table_schema, d.id
        from info i
        inner join dbs d on d.name = i.table_catalog
        returning id, name, parent
    )
insert into resources(type, name, parent)
select 'table', table_name, s.id
from info i
inner join schemas s on s.name = i.table_schema
inner join dbs d on d.id = s.parent and d.name = i.table_catalog

Note that the last insert joins on both schemas and dbs; this is meant to properly handle "homonym" tables that live in different schemas.

Here is a demo (I used a table to mimic the results of your initial query).

Collectives™ on Stack Overflow

Populating n level PostgreSQL table in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related