Trying to use kubernetes metadata as log group name. Cloud watch plugin

I am trying to set the log_group_name setting in the cloudwatch plugin to be the k8s labels app name.

My fluent.conf

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"
@include kubernetes.conf
@include conf.d/*.conf

<match **>
@type cloudwatch_logs
@id out_cloudwatch_logs
log_group_name ${record["kubernetes"]["labels"]["app"]}
auto_create_stream true
use_tag_as_stream true
retention_in_days 14
json_handler yajl # To avoid UndefinedConversionError
log_rejected_request "#{ENV['LOG_REJECTED_REQUEST']}" # Log rejected request for missing parts
</match>

<filter kubernetes.var.log.containers.**.log>
@type kubernetes_metadata
</filter>

<match **>
@type stdout
</match>

Error

2021-12-11 00:07:26 +0000 [warn]: #0 [out_cloudwatch_logs] failed to flush the buffer. retry_times=5 next_retry_time=2021-12-11 00:07:42 +0000 chunk=“5d2d39b5d365d5a5e6b35829f7862753” error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error=“1 validation error detected: Value ‘${record[“kubernetes”][“labels”][“app”]}’ at ‘logGroupNamePrefix’ failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+”

It seemed like it was possible from this post