Creating Executable Tokens in JSON Files

When building CI/CD pipelines it is often the case that you'd like to "tokenize" a configuration file, so that the values in the file can be calculated during the build/release execution process. This can be for all kinds of reasons, such as ingesting environment variables or by basing values on the outputs of other CLI tools, such as terraform (e.g. reading IDs of recently created infrastructure). Rather than separating the key/value "tokens" from the "token calculation" logic, I wanted a way to embed Bash scripts directly into my JSON file and then have interpret the values into an output file. For example, the JSON file might look like:

[
    {
        "Key": "verbatim",
        "Value": "just a string"
    },
    {
        "Key": "environment-variable",
        "Value": "$HOME"
    },
    {
        "Key": "shell-script",
        "Value": "$(dotnet --version)"
    }
]

With the invokable tokens defined, the below script can be run against the JSON file in order to parse and execute the tokens:

#!/bin/bash

jsonFile=$1
tempFile=$2

echo "[" > "$tempFile"

itemCount=$(jq '. | length' $jsonFile)
currentIndex=0

# Read and process the JSON file line by line.
jq -c '.[]' $jsonFile | while read -r obj; do
    key=$(echo "$obj" | jq -r '.Key')
    value=$(echo "$obj" | jq -r '.Value')

    # Check if value needs command execution
    if [[ "$value" == \$\(* ]]; then
        # Remove $() for command execution
        command=$(echo "$value" | sed 's/^\$\((.*)\)$/\1/')
        newValue=$(eval $command)
    elif [[ "$value" == \$* ]]; then
        # It's an environment variable
        varName=${value:1} # Remove leading $
        newValue=${!varName}
    else
        # Plain text, no change needed
        newValue="$value"
    fi

    # Update the JSON object with the new value
    updatedObj=$(echo "$obj" | jq --arg newValue "$newValue" '.Value = $newValue')

    # Append the updated object to the temp file
    echo "$updatedObj" >> "$tempFile"

    # Add a comma except for the last item
    ((currentIndex++))
    if [[ $currentIndex -lt $itemCount ]]; then
        echo ',' >> "$tempFile"
    fi
done

echo "]" >> "$tempFile"

Running the command looks like

./json_exec.sh example.json example.out.json

And the output then looks as follows:

[
{
  "Key": "verbatim",
  "Value": "just a string"
}
,
{
  "Key": "environment-variable",
  "Value": "/home/craig"
}
,
{
  "Key": "shell-script",
  "Value": "8.0.202"
}
]