Kafka source connector for postgres - Day0 load

Question

I am looking for a Kafka source connector for Day0 load from Postgres to Kafka.

Came across Debezium postgres connector.

Docker image,

debezium/connect:1.4

docker run -it --rm --name postgres-connect -p 8083:8083 -e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses debezium/connect:1.4

How to pass the postgres host details and kafka sasl config?

Any help would be appreciated.

For SASL config: please see answer stackoverflow.com/a/64355810/7109598 — Iskuskov Alexander
– Iskuskov Alexander, Commented Jan 19, 2021 at 9:57
@IskuskovAlexander I am new to kafka connect. Can you tell me how to pass the SASL config file and postgress config to the docker container? — user1578872
– user1578872, Commented Jan 21, 2021 at 19:07

Iskuskov Alexander · Accepted Answer · 2021-01-24 16:06:00Z

2

+50

1. SASL configuration

1.1. In common case you need to add the following properties to your connect-distributed.properties:

sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

producer.sasl.mechanism=PLAIN
producer.security.protocol=SASL_PLAINTEXT
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_PLAINTEXT
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

Source: StackOverflow answer "ACL configuration in Kafka connect is not working"

Reference: Kafka Connect Security docs

1.2. For debezium/connect docker image you can try to pass SASL config directly via environment variables (using these transformation steps):

docker run -it --rm --name postgres-connect -p 8083:8083 \
  -e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 \
  -e CONFIG_STORAGE_TOPIC=my_connect_configs \
  -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
  -e STATUS_STORAGE_TOPIC=my_connect_statuses \
  -e CONNECT_SASL_MECHANISM=PLAIN \
  -e CONNECT_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
  -e CONNECT_PRODUCER_SASL_MECHANISM=PLAIN \
  -e CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
  -e CONNECT_CONSUMER_SASL_MECHANISM=PLAIN \
  -e CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
debezium/connect:1.4

2. PostgreSQL host configaration

Host details should be passed via Kafka Connect REST API using connector config:

curl -i -X PUT -H "Content-Type:application/json" \
    http://localhost:8083/connectors/debezium_postgres_source/config \
    -d '{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "source-db",
    "database.port": "5432",
    "database.user": "postgresusersource",
    "database.password": "postgrespw",
    "database.dbname" : "sourcedb",
    "database.server.name": "dbserver1"
}'

edited Jan 24, 2021 at 16:06

answered Jan 24, 2021 at 15:59

Iskuskov Alexander

4,4453 gold badges28 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1578872 Over a year ago

Calling REST API for Postgres config --> Do we need to call this whenever the docker container gets restarted for some reason?

Iskuskov Alexander Over a year ago

All metadata of connector stores in Kafka. So, if you haven't lose this info, I don't have to call this method again. Also you can check status of connector via REST API (curl -XGET localhost:8083/connectors/debezium_postgres_source/status)

Collectives™ on Stack Overflow

Kafka source connector for postgres - Day0 load

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related