Assign YAML array from input to key using `yq`

Question

I'm trying to take a YAML style array I get from an AWS command, and assign it to a key while updating my own YAML.

This represents what I have right now:

yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= "'"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')"'"' -i route53.yml

This is how route53.yml looks like before I run the command:

HostedZones:
- CallerReference: abc-123
  Id: /hostedzone/ABC123
  Name: domain.name.com.
  ResourceRecordSetCount: 5

and this is how route53.yml looks like after:

HostedZones:
- CallerReference: abc-123
  Id: /hostedzone/ABC123
  Name: domain.name.com.
  ResourceRecordSetCount: 5
  ResourceRecordSets: |-
    - Name: a.domain.name.com.
      ResourceRecords:
        - Value: some.value.com
      TTL: 300
      Type: CNAME
    - Name: b.domain.name.com.
      ResourceRecords:
        - Value: some.value.com
      TTL: 300
      Type: CNAME
    - Name: c.domain.name.com.
      ResourceRecords:
        - Value: some.value.com
      TTL: 300
      Type: CNAME
    - Name: d.domain.name.com.
      ResourceRecords:
        - Value: some.value.com
      TTL: 300
      Type: CNAME
    - Name: e.domain.name.com.
      ResourceRecords:
        - Value: some.value.com
      TTL: 300
      Type: CNAME

As you can see there's a |- right after the key, and it seems like it is treated as multiline string instead of an array of maps. How can I avoid it and assign the array as a YAML array? When I tried manually assigning an array in the style of ['a', 'b', 'c'] and the update works as it should, adding a YAML array under the key, how can I achieve it with the output of the aws command?

@Cyrus Because a popular version of yq is just a wrapper around jq, so it uses the same syntax (and just calls jq after some preprocessing). — larsks
– larsks, Commented May 22, 2022 at 14:47
You shouldn't inject into the code the data you want to append. Rather use appropriate means offered by yq. As you mentioned to be using kislyuk/yq (which is the "popular version of yq [which] is just a wrapper around jq"), this would be using options like --arg or --argjson or --argfile (see how to invoke jq) — pmf
– pmf, Commented May 22, 2022 at 15:04

larsks · Accepted Answer · 2022-05-22 15:19:59Z

2

The reason why yq is interpreting the output of the $(...) expression as a multiline strine is because you have quoted it; your yq expression, simplified, looks like:

yq 'ResourceRecordSets |= "some string here"'

The quotes mean "this is a string, not a structure", so that's what you get. You could try dropping the quotes, like this:

yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= '"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')" route53.yml

That might work, but it is fragile. A better solution is to have yq parse the output of the subexpression as a separate document, and then merge it as a structured document rather than a big string. Like this:

yq \
  '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= input.ResourceRecordSets' \
  route53.yml \
  <(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml)

This takes advantage of the fact that you can provide jq (and hence yq) multiple files on the command line, and then refer to the input variable (described in the IO section of the jq manual). The above command line is structured like this:

yq <expression> <file1> <file2>

Where <file1> is route53.yml, and <file2> is a bash process substitution.

This solution simplifies issues around quoting and formatting.

(You'll note I dropped your use of -i here; that seems to throw yq for a loop, and it's easy to output to a temporary file and then rename it.)

edited May 22, 2022 at 15:19

answered May 22, 2022 at 15:12

larsks

318k50 gold badges474 silver badges482 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

pmf Over a year ago

Using input.ResourceRecordSets directly would eliminate the need for invoking yq twice.

larsks Over a year ago

Good point! Updating the answer to match.

argamanza · Accepted Answer · 2022-05-23 08:30:38Z

0

Eventually I solved it using a combination of larsks answer about the unnecessary quotes, and using jq instead of yq (no matter what I did, using solely yq wouldn't work) to assign the array to a separate variable before editing the YAML document:

record_sets=$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" | jq '.[]')
yq '(.HostedZones[] | select(.Id=="ABC123")).ResourceRecordSets |= '"$record_sets"'' -i route53.yml

answered May 23, 2022 at 8:30

argamanza

1,1423 gold badges14 silver badges38 bronze badges

Collectives™ on Stack Overflow

Assign YAML array from input to key using `yq`

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related