-
Notifications
You must be signed in to change notification settings - Fork 367
Description
CAPI pulls container metrics and quotas from Log Cache, which it then filters and transforms client-side:
cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb
Lines 38 to 44 in 8b261d7
| final_envelopes. | |
| select { |e| has_container_metrics_fields?(e) && logcache_filter.call(e) }. | |
| uniq { |e| e.gauge.metrics.keys << e.instance_id }. | |
| sort_by(&:instance_id). | |
| chunk(&:instance_id). | |
| map { |envelopes_by_instance| convert_to_traffic_controller_envelope(source_guid, envelopes_by_instance) } | |
| end |
One of the filters drops any envelope that doesn't contain a CPU, Memory, or Disk metric:
cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb
Lines 61 to 69 in 8b261d7
| def has_container_metrics_fields?(envelope) | |
| # rubocop seems to think that there is a 'key?' method | |
| # on envelope.gauge.metrics - but it does not | |
| # rubocop:disable Style/PreferredHashMethods | |
| envelope.gauge.metrics.has_key?('cpu') || | |
| envelope.gauge.metrics.has_key?('memory') || | |
| envelope.gauge.metrics.has_key?('disk') | |
| # rubocop:enable Style/PreferredHashMethods | |
| end |
This works when syslog ingress is turned off and Log Cache receives traffic from RLP. Here's an example of the relevant envelope:
<Loggregator::V2::Envelope: timestamp: 1644342692373209354, source_id: "c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"a3601fdf-136b-4ca7-80b7-907f41dfb03c", "job"=>"diego-cell", "source_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "index"=>"6deca28a-1c64-4d4c-8423-4b4644233f1e", "app_name"=>"dora", "process_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "origin"=>"rep", "app_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "process_instance_id"=>"923adbb1-c837-4c72-5541-1afc", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"d4025a57-c1e2-483e-91fd-b0615e6d3ea0", "ip"=>"10.0.1.11", "process_type"=>"web", "instance_id"=>"0"}, gauge: <Loggregator::V2::Gauge: metrics: {"memory"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 39194499.0>, "memory_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>, "cpu"=><Loggregator::V2::GaugeValue: unit: "percentage", value: 0.25252665460350726>, "disk"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 94240768.0>, "disk_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>
However, when syslog ingress is turned on, that envelope is split up into multiple envelopes:
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "instance_id"=>"0", "job"=>"diego -cell", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-5815" , "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "process_type"=>"web", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me trics: {"dik_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "process_type"=>"web", "instance_ id"=>"0", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "job"=>"diego-cell", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-5815", "organization_name"=>"o", "d eployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me trics: {"memory_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "process_type"=>"web", "instance_ id"=>"0", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-581 5", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "job"=>"diego-cell", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me trics: {"disk"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 94240768.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "source_id"=>"b1fb2c23-7592-4872- 9eaf-26be31f9d120", "process_type"=>"web", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance _id"=>"54c49de2-731f-4546-713c-5815", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "job"=>"diego-cell", "instance_id"=>"0"}, gauge: <Loggregator::V2::Gauge: me trics: {"memory"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 38201530.0>}>>
In this case, the envelopes that contain Disk Quota or Memory Quota are filtered out because they don't contain a CPU, Memory, or Disk metric. This leads to the quotas being permanently represented as 0:
$ cf app dora | grep '^#0'
#0 running 2022-02-08T00:54:12Z 0.3% 36.4M of 0 89.9M of 0