set the group variable in azure devops pipeline using bash script

Question

#!/bin/bash

# Read Docker Compose file
compose_file="docker-compose.yml"
images=($(grep -E '^ *image:' $compose_file | awk '{print $2}' | cut -d ':' -f 1))

# Initialize group variable
all_variables=()

for image in "${images[@]}"; do
    # Store image name in pipeline variable
    variable_name="image_${image//[^a-zA-Z0-9_]/_}"
    # Set individual variable
    # echo "Setting pipeline variable for $variable_name: $image"
    echo "##vso[task.setvariable variable=$variable_name]$image"
    # Concatenate variable assignment to group  variable with newline
    group _variable+="$variable_name: $image"$'\n'
done
echo "$all_variables"
# Set group variable in Azure Pipelines
# echo "Setting group  variable in Azure Pipelines"
echo -e "##vso[task.setvariable variable=allVariables;isOutput=true]$all_variables"
echo "allVariables:$allVariables"

but the (all variables)group variable are not set in the pipeline so please give me any suggestion what ever i am do is right the process is right like this thing is possible

i am expecting like this

variables:
  allvariables: |
    image_postgres: postgres
    image_mongo: mongo
    image_quay_io_keycloak_keycloak: quay.io/keycloak/keycloak
    image_discovery_server: discovery-server
    image_api_gateway: api-gateway
    image_auth_service: auth-service

but this variable is dynamically

I'm not sure if multiline variables are supported. Have you considered generating a file and publishing it as an artifact, in case it's used in other jobs/stages? — Rui Jarimba
– Rui Jarimba, Commented May 14, 2024 at 13:20
I'm not familiar with Azure devops pipelines, but when you assign a variable in a shell, it's a shell variable -- it only exists in that particular shell process. Using export makes it into an environment variable, which'll be inherited by subprocesses of that shell, but there's no way for a shell (or any other program) to create variables in its parent process. — Gordon Davisson
– Gordon Davisson, Commented May 14, 2024 at 17:35
Tangentially, your useless use of grep | awk | cut can be refactored to awk '/^ *image:/ {split($2, f, /:/); print f[1]}' "$compose_file" — tripleee
– tripleee, Commented May 15, 2024 at 8:51

Alvin Zhao · Accepted Answer · 2024-05-15 07:53:54Z

0

According to this document on User-defined multi-line variables,

Azure DevOps supports multi-line variables but there are a few limitations.

Downstream components such as pipeline tasks might not handle the variable values correctly.

Azure DevOps won't alter user-defined variable values. Variable values need to be formatted correctly before being passed as multi-line variables. When formatting your variable, avoid special characters, don't use restricted names, and make sure you use a line ending format that works for the operating system of your agent.

Multi-line variables behave differently depending on the operating system. To avoid this, make sure that you format multi-line variables correctly for the target operating system.

Azure DevOps never alters variable values, even if you provide unsupported formatting.

Per the requirement to generate $multi_line_variable in script so as to use the output variable value in downstream tasks, you may consider formatting the value of $multi_line_variable as a BASE64 one-line string and decode the string in downstream script. Here is a sample for your reference.

steps:
- bash: |
    compose_file="docker-compose.yml"
    images=($(grep -E '^ *image:' $compose_file | awk '{print $2}' | cut -d ':' -f 1))

    # Initialize multi-line variable
    multi_line_variable=""
    for image in "${images[@]}"; do
        # Store image name in pipeline variable
        variable_name="image_${image//[^a-zA-Z0-9_]/_}"
        multi_line_variable+="$variable_name: $image"$'\n'
    done
    echo -e "multi_line_variable is:\n$multi_line_variable"
    
    one_line_variable=$(echo -n "$multi_line_variable" | base64 -w 0)
    echo one_line_variable is $one_line_variable
    
    echo "##vso[task.setvariable variable=allVariables;isOutput=true]$one_line_variable"
  name: OutputVar
- bash: |
    echo OutputVar.allVariables $(OutputVar.allVariables)
    echo -n '$(OutputVar.allVariables)' | base64 --decode

edited May 15, 2024 at 7:53

answered May 15, 2024 at 7:46

Alvin Zhao

6,6831 gold badge7 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Sʜɩvʌŋsʜ Tʜʌĸʋʀ Over a year ago

in that if i am run bash file inside of pipeline so how i am write the script in side of bash file and how i am write script to get group variable in pipeline

Sʜɩvʌŋsʜ Tʜʌĸʋʀ Over a year ago

- task: Bash@3 inputs: filePath: '$(System.DefaultWorkingDirectory)/bash_files/extract_docker_images.sh' name: OutputVar - bash: | echo OutputVar.allVariables $(OutputVar.allVariables) echo -n '$(OutputVar.allVariables)' | base64 --decode displayName: 'Print OutputVar' if i am using like that so i am facing error

Sʜɩvʌŋsʜ Tʜʌĸʋʀ Over a year ago

like that /root/myagent/_work/_temp/d506689f-6ac2-4435-a017-82f987e8a34f.sh: line 1: OutputVar.allVariables: command not found OutputVar.allVariables base64: invalid input ##[debug]Exit code 1 received from tool '/usr/bin/bash' ##[debug]STDIO streams have closed for tool '/usr/bin/bash' ##[error]Bash exited with code '1'.

Alvin Zhao Over a year ago

Hi there, were you using the same script as mine? Have you tried my inline script? Thx.

Alvin Zhao Over a year ago

Hi @SʜɩvʌŋsʜTʜʌĸʋʀ, Can you please let us know the latest status of the issue and does the workaround to encode and decode the one-line BASE64 output string resolve the issue? Thx.

Collectives™ on Stack Overflow

set the group variable in azure devops pipeline using bash script

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related