Cannot remove environment variables via cf push

[bbtong]

Summary

Removing environment variables from a deployed CF app doesn't work. A cf delete and repush is necessary to remove outdated environemnt variables entirely.

We spoke to some teams and it seems to be a known issue with the platform, and have been encouraged to file a bug report here.

Expected Result

Given this Concourse task that uses the "cf" concourse resource:

# logstash2datadog
  - name: deploy-logstash2datadog
    build_logs_to_retain: 25
    serial: true
    plan:
    - get: logstash2datadog
      trigger: true
    - put: deploy-to-jynx-pcf-production
      tags: [sf-vsphere]
      params:
        manifest: logstash2datadog/manifest.yml
        path: logstash2datadog
        environment_variables:
          DATADOG_API_KEY: key
          KIBANA_PASSWORD: password 							# We remove this
          KIBANA_USERNAME: username 							# We remove this
          KIBANA_HOSTS:    host 								# We remove this
          KIBANA_DEPLOYMENT_NAMES: url							# We remove this
          GRAYLOG_PASSWORDS: password
          GRAYLOG_USERNAMES: username
          GRAYLOG_HOSTS: host
          GRAYLOG_DEPLOYMENT_NAMES: names

If we remove all KIBANA_ instances in this scenario above then we push this to a CF-deployed app, we expect the environment variables related to KIBANA to be removed in the deployment too.

Thus, a query for environment variables we expect would return:

...
DATADOG_API_KEY: key
GRAYLOG_PASSWORDS: password
GRAYLOG_USERNAMES: username
GRAYLOG_HOSTS: host
GRAYLOG_DEPLOYMENT_NAMES: names
...

Actual Result

Unlike the expected results above where our deleted environment variables are likewise removed in the CF-deployed app, it seems that Diego forever keeps old environment variables somewhere in memory / database and will not remove them without a cf delete and repush. Possibly, you could overwrite the environment variable values with empty strings; however, this does not equal a proper removal of the variables still.

Our actual result from an environment variable query looks like:

...
DATADOG_API_KEY: key
KIBANA_PASSWORD: password 
KIBANA_USERNAME: username 
KIBANA_HOSTS:    host
KIBANA_DEPLOYMENT_NAMES: url
GRAYLOG_PASSWORDS: password
GRAYLOG_USERNAMES: username
GRAYLOG_HOSTS: host
GRAYLOG_DEPLOYMENT_NAMES: names
...

Context

The affected app is running on PCF 2.5.2 on vSphere.

Steps to Reproduce

• Push an CF app with some environment variables
• Remove an env var and re-push the app
• Notice that the env var you removed is still in the output from cf env $APP_NAME

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/166770957

The labels on this github issue will be updated when the story is started.

[sunjayBhatia]

@bbtong can you ssh onto one of the diego* vms and use cfdot to give us the desired LRP before and after you remove environment variables/repush the app

cfdot desired-lrps | grep <app_guid>

if the environment variables sections are the same, that means CC is the one that is keeping these environment variables around and sending it to the BBS, not really something the diego component can control

[sunjayBhatia]

@bbtong have you had a chance to check with the capi team about this issue or get the requested info?

[bbtong]

@sunjayBhatia Hey, sorry for the radio silence. We plan to investigate this further later today (PST) and relay the information here.

[rowanjacobs]

@sunjayBhatia We updated the configuration for our CF concourse resource to add two new env variables:

  job deploy-updog has changed:
  name: deploy-updog
  serial: true
  build_logs_to_retain: 25
  plan:
  - in_parallel:
      steps:
      - get: toolsmiths-scripts-updog
        trigger: true
        tags:
        - sf-vsphere
  - put: deploy-to-jynx-pcf-production
    params:
      environment_variables:
        CONFIG_FILE: config.yml
        DATADOG_API_KEY: ad7bcbcd5e6a6ba942ac813cf87fdbf9
        DEFAULT_HTTP_TIMEOUT: "300"
        DEFAULT_INTERVAL: "5"
+       TEST_VAR: test
+       TEST_VAR2: test2
      manifest: toolsmiths-scripts-updog/updog/manifest.yml
      path: toolsmiths-scripts-updog/updog
    tags:
    - sf-vsphere

Here is the result of running that command before removing those variables:

cfdot desired-lrps | grep ${app_guid} | jq '.action.codependent.actions[0].run.env'
[
  {
    "name": "PORT",
    "value": "8080"
  },
  {
    "name": "VCAP_APP_PORT",
    "value": "8080"
  },
  {
    "name": "VCAP_APP_HOST",
    "value": "0.0.0.0"
  },
  {
    "name": "CONFIG_FILE",
    "value": "config.yml"
  },
  {
    "name": "DATADOG_API_KEY",
    "value": "<redacted>"
  },
  {
    "name": "DEFAULT_HTTP_TIMEOUT",
    "value": "300"
  },
  {
    "name": "DEFAULT_INTERVAL",
    "value": "5"
  },
  {
    "name": "TEST_VAR",
    "value": "test"
  },
  {
    "name": "TEST_VAR2",
    "value": "test2"
  },
  {
    "name": "VCAP_APPLICATION",
    "value": "{\"cf_api\":\"https://api.jynx.cf-app.com\",\"limits\":{\"fds\":16384,\"mem\":128,\"disk\":1024},\"application_name\":\"updog\",\"application_uris\":[],\"name\":\"updog\",\"space_name\":\"production\",\"space_id\":\"b8da446b-7696-4378-bf1b-92355c8248ee\",\"uris\":[],\"application_id\":\"03fa06e7-da73-4e4b-9620-9eacdacbbb38\",\"version\":\"61adfdf3-0d27-434b-91b1-1d61227febd6\",\"application_version\":\"61adfdf3-0d27-434b-91b1-1d61227febd6\"}"
  },
  {
    "name": "MEMORY_LIMIT",
    "value": "128m"
  },
  {
    "name": "VCAP_SERVICES",
    "value": "{\"user-provided\":[{\n  \"label\": \"user-provided\",\n  \"name\": \"updog-papertrail\",\n  \"tags\": [\n\n  ],\n  \"instance_name\": \"updog-papertrail\",\n  \"binding_name\": null,\n  \"credentials\": {\n\n  },\n  \"syslog_drain_url\": \"syslog://logs7.papertrailapp.com:54888\",\n  \"volume_mounts\": [\n\n  ]\n}]}"
  },
  {
    "name": "VCAP_PLATFORM_OPTIONS",
    "value": "{\"credhub-uri\":\"https://credhub.service.cf.internal:8844\"}"
  }
]

We got the exact same set of keys in '.action.codependent.actions[0].run.env' after removing the variables from our CF Concourse resource config and re-running the job to deploy the app.

[emalm]

What's the state of the environment variables on the underlying CF app, as reported by cf env APP? Regardless of what the CF Concourse resource does, that should be the source of truth for what gets sent to Diego.

[rowanjacobs]

 → cf env updog
Getting env variables for app updog in org toolsmiths / space production as admin...
OK

...

User-Provided:
CONFIG_FILE: config.yml
DATADOG_API_KEY: <redacted>
DEFAULT_HTTP_TIMEOUT: 300
DEFAULT_INTERVAL: 5
TEST_VAR: test
TEST_VAR2: test2

[rowanjacobs]

After some further conversation with @emalm this doesn't seem to be a Diego problem, since Diego is doing exactly what the cloud controller is telling it to. I also don't think it's a problem with the CF Concourse resource; we get the same behavior if we run cf push from our local workstations with a manifest missing TEST_VAR and TEST_VAR2 but having all the other variables. There just doesn't seem to be a way to declaratively remove variables (although cf unset-env works fine).

[emalm]

/cc @cloudfoundry/cf-capi @cloudfoundry/cf-cli re removing app env vars declaratively

[sunjayBhatia]

Closing this since this should be a concern of @cloudfoundry/cf-capi and @cloudfoundry/cf-cli