Skip to content

App quotas not retrieved when Log Cache has syslog ingress turned on #2669

@ctlong

Description

@ctlong

CAPI pulls container metrics and quotas from Log Cache, which it then filters and transforms client-side:

final_envelopes.
select { |e| has_container_metrics_fields?(e) && logcache_filter.call(e) }.
uniq { |e| e.gauge.metrics.keys << e.instance_id }.
sort_by(&:instance_id).
chunk(&:instance_id).
map { |envelopes_by_instance| convert_to_traffic_controller_envelope(source_guid, envelopes_by_instance) }
end

One of the filters drops any envelope that doesn't contain a CPU, Memory, or Disk metric:

def has_container_metrics_fields?(envelope)
# rubocop seems to think that there is a 'key?' method
# on envelope.gauge.metrics - but it does not
# rubocop:disable Style/PreferredHashMethods
envelope.gauge.metrics.has_key?('cpu') ||
envelope.gauge.metrics.has_key?('memory') ||
envelope.gauge.metrics.has_key?('disk')
# rubocop:enable Style/PreferredHashMethods
end

This works when syslog ingress is turned off and Log Cache receives traffic from RLP. Here's an example of the relevant envelope:

<Loggregator::V2::Envelope: timestamp: 1644342692373209354, source_id: "c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"a3601fdf-136b-4ca7-80b7-907f41dfb03c", "job"=>"diego-cell", "source_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "index"=>"6deca28a-1c64-4d4c-8423-4b4644233f1e", "app_name"=>"dora", "process_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "origin"=>"rep", "app_id"=>"c6d7db86-1c88-4e18-a7b5-4d8fcc654a77", "process_instance_id"=>"923adbb1-c837-4c72-5541-1afc", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"d4025a57-c1e2-483e-91fd-b0615e6d3ea0", "ip"=>"10.0.1.11", "process_type"=>"web", "instance_id"=>"0"}, gauge: <Loggregator::V2::Gauge: metrics: {"memory"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 39194499.0>, "memory_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>, "cpu"=><Loggregator::V2::GaugeValue: unit: "percentage", value: 0.25252665460350726>, "disk"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 94240768.0>, "disk_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>

However, when syslog ingress is turned on, that envelope is split up into multiple envelopes:

<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "instance_id"=>"0", "job"=>"diego    -cell", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-5815"    , "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "process_type"=>"web", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me    trics: {"dik_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "process_type"=>"web", "instance_    id"=>"0", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "job"=>"diego-cell", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-5815", "organization_name"=>"o", "d    eployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me    trics: {"memory_quota"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 1073741824.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "process_type"=>"web", "instance_    id"=>"0", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "source_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance_id"=>"54c49de2-731f-4546-713c-581    5", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "job"=>"diego-cell", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120"}, gauge: <Loggregator::V2::Gauge: me    trics: {"disk"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 94240768.0>}>>,
<Loggregator::V2::Envelope: timestamp: 1644284584602912000, source_id: "b1fb2c23-7592-4872-9eaf-26be31f9d120", instance_id: "0", deprecated_tags: {}, tags: {"organization_id"=>"212cf8c0-a90e-48d1-b07b-302ea9ba0629", "source_id"=>"b1fb2c23-7592-4872-    9eaf-26be31f9d120", "process_type"=>"web", "index"=>"24489b63-e3e4-4a10-9280-0899a9a91c9a", "app_name"=>"dora", "process_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "origin"=>"rep", "app_id"=>"b1fb2c23-7592-4872-9eaf-26be31f9d120", "process_instance    _id"=>"54c49de2-731f-4546-713c-5815", "organization_name"=>"o", "deployment"=>"cf", "space_name"=>"s", "space_id"=>"ae9fc79a-992c-47cd-b94b-3f74d47a8dcf", "ip"=>"10.0.1.11", "job"=>"diego-cell", "instance_id"=>"0"}, gauge: <Loggregator::V2::Gauge: me    trics: {"memory"=><Loggregator::V2::GaugeValue: unit: "bytes", value: 38201530.0>}>>

In this case, the envelopes that contain Disk Quota or Memory Quota are filtered out because they don't contain a CPU, Memory, or Disk metric. This leads to the quotas being permanently represented as 0:

$ cf app dora | grep '^#0'
#0   running   2022-02-08T00:54:12Z   0.3%   36.4M of 0   89.9M of 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions