Parsing string with escape characters into JSON and to Variable in Bash [duplicate]

Question

The Problem:

I am trying to fetch credentials from AWS Secrets Manager in my terminal, however the Keys and Values I want needs to be in JSON, however they come with a lot of escape characters due to quotes.

The Scenario:

After I fire the aws secretsmanager get-secret-value --secret-id snowflake-access-uat command, I get the credentials as below:

{
    "ARN": "arn:aws:secretsmanager:ap-regionnm-1:111111111111:secret:my-secret",
    "Name": "snowflake-access-uat",
    "VersionId": "dont-care",
    "SecretString": "{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": 1627104812.142
}

However, I am interested in Secret String only, for which I fire aws secretsmanager get-secret-value --secret-id snowflake-programmatic-access-uat | jq '.SecretString' command and receive this:

"{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}"

But since it has multiple escape characters, I am unable to leverage it with jq tree. I tried to get from this link for reference but I'm unable to make it work. Besides, I need the Keys and Values to be variables in my bash session.

NOTE: I cannot use any third party tools, since I need to automate this on CodeBuild (Run time fresh instance will be selected)

You're already showing and telling us that jq is enabled, so we're in a good place re: "third-party tools". — Charles Duffy
– Charles Duffy, Commented Oct 14, 2021 at 13:10
While the linked duplicate's title asks only about double quotes, requesting raw output (as the accepted answer instructs) removes the other unwanted escaping as well. — Charles Duffy
– Charles Duffy, Commented Oct 14, 2021 at 13:14
@CharlesDuffy Yes Charles, CodeBuild recently added Ubuntu series, so ya it does have jq but I am not well versed if how can I use Python in writing the buildspec.yml with it. — T3J45
– T3J45, Commented Oct 14, 2021 at 13:15
So, the thing to remember about YAML is that all JSON is valid YAML — Charles Duffy
– Charles Duffy, Commented Oct 14, 2021 at 13:15
...so you can write a JSON file with the same data, name it buildspec.yml, and you're set. — Charles Duffy
– Charles Duffy, Commented Oct 14, 2021 at 13:16

2 revs · Accepted Answer · 2021-10-14 13:17:39Z

3

The escape characters are there because you don't use -r with jq '.SecretString'. Change it to jq -r '.SecretString' and your output will instead be:

{"sf-user":"USER_123_ADMIN","sf-password":"FooBaarPassword","sf-db":"MY_SPL_DB","wh_name":"JOB_EXECUTOR","sf-role":"JOB_EXECUTOR_ROLE","sf-account":"icy-party"}

...which, being valid JSON, you can feed back into jq -r to retrieve individual fields.

SecretStringJson=$(... | jq -r '.SecretString')
### one jq call per field isn't the most efficient possible way but it's easy
sfUser=$(jq -r '.["sf-user"]' <<<"$SecretStringJson")
sfDb=$(jq -r '.["sf-db"]' <<<"$SecretStringJson")
# ...etc

Collectives™ on Stack Overflow

Parsing string with escape characters into JSON and to Variable in Bash [duplicate]

The Problem:

The Scenario:

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

The Problem:

The Scenario:

1 Answer 1

Comments

Linked

Related