Skip to content

Update Readme #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 8, 2022
Merged

Update Readme #57

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
233007f
Update Readme
mkocher Mar 7, 2022
f86b051
update scaling guidance
Mar 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 45 additions & 68 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,91 +1,68 @@
Log Cache Release
=================

Log Cache Release is a [bosh](https://github.com/cloudfoundry/bosh) release
for [Log Cache](https://code.cloudfoundry.org/log-cache). It provides
an [in memory caching layer](https://docs.google.com/document/d/1yhfl0EB_MkHkh4JdRZXGeQMx_BDMCuB-SpPuSrD3VOU/edit#) as a replacement for `cf logs --recent` and container metrics retrieval.
## What is Log Cache?

### Deploying Log Cache
Log Cache provides an in memory caching layer for logs and metrics which are emitted by
applications and system components in Cloud Foundry.

Log Cache can be deployed within
[Cloud Foundry](https://github.com/cloudfoundry/cf-deployment).
Log Cache is deployed as a group of nodes which communicate between themselves
over GRPC. Source IDs (such as the CF application ID or a unique string for a
system component) are hashed to determine which Log Cache
instance will cache the data. Envelopes from a source ID can be sent to any log cache instance and
are then forwarded to the assigned node. Similarly queries for Log Cache can be
sent to any node and the query will be forwarded to the appropriate node which
is caching the data for the requested source ID.

Log Cache relies on Loggregator and reads data from the Reverse Log Proxy.
## How does Log Cache fit into Cloud Foundry?

#### Cloud Config
Log Cache is included by default in
[Cloud Foundry's cf-deployment](https://github.com/cloudfoundry/cf-deployment).

Every bosh deployment requires a [cloud
config](https://bosh.io/docs/cloud-config.html). The Log Cache deployment
manifest assumes the CF-Deployment cloud config has been uploaded.
By default Log Cache receives data from syslog agents which is an add on which runs on all instance groups by default. Log Cache can also be configured
to read from the Reverse Log Proxy instead, though this option is not recommended because of firehose scalability limits.

#### Creating and Uploading Release
Log Cache is queried by Cloud Controller for app instance metrics such as CPU usage and memory when retrieving details for applications and
by the cf cli directly to retrieve recent logs. It can also be queried using the Log Cache CLI plugin to retrieve system component metrics.

The first step in deploying Log Cache is to create a release. Final releases
are preferable, however during the development process dev releases are
useful.

The following commands will create a dev release and upload it to an
environment named `lite`.
## How do I configure it?

```
bosh create-release --force
bosh -e lite upload-release --rebase
```
### Scaling

#### Cloud Foundry
Numerous variables affect the retention of Log Cache:
Number of instances - Increasing adds more storage space, allows higher throughput and reduces contention between sources
Max Per Source ID - Increasing allows a higher max storage allowance, but may decrease the storage of less noisy apps on the same node
Memory per instance - Increasing allows more storage in general, but any given instance may not be able to take advantage of that increase due to max per source id
Memory limit - Increasing memory limit allows for more storage, but may cause out of memory errors and crashing if set too high for the total throughput of the system
Larger CPUs - Increasing the CPU budget per instance should allow higher throughput

Log Cache deployed within Cloud Foundry reads from the Loggregator system and
registers with the [GoRouter](https://github.com/cloudfoundry/gorouter) at
`log-cache.<system-domain>` (e.g. for bosh-lite `log-cache.bosh-lite.com`).
Log Cache is known to exceed memory limits under high throughput/stress. If you see your log-cache reaching higher memory
then you have set, you might want to scale your log-cache up. Either solely in terms of CPU per instance, or more instances.

As of `cf-deployment` version 3.x, Log Cache is included by default in CF.
You can monitor the performance of log cache per source id (app or platform component) using the Log Cache CLI. The command `cf log-meta` allows viewing
the amount of logs and metrics as well as the period of time for those logs and metrics for each source on the system. This can be used in conjuction with scaling
to target your use cases. For simple pushes, a low retention period may be adequate. For running analysis on metrics for debugging and scaling, higher retention
periods may be desired; although one should remember all logs and metrics will always be lost upon crashes or re-deploys of log-cache.

The following commands will deploy Log Cache in CF.
### Reliability

```
bosh update-runtime-config \
~/workspace/bosh-deployment/runtime-configs/dns.yml
bosh update-cloud-config \
~/workspace/cf-deployment/iaas-support/bosh-lite/cloud-config.yml
bosh \
--environment lite \
--deployment cf \
deploy ~/workspace/cf-deployment/cf-deployment.yml \
--ops-file ~/workspace/cf-deployment/operations/bosh-lite.yml \
--ops-file ~/workspace/cf-deployment/operations/use-compiled-releases.yml \
-v system_domain=bosh-lite.com
```
Log Cache is an in memory cache and as such will drop envelopes when it restarts. Users should not expect 100% availability of
logs in Log Cache and should plan accordingly. For example, Cloud Controller can function without Log Cache though users are
informed that Log Cache is unavailable.

##### Log Cache UAA Client
By Default, Log Cache uses the `doppler` client included with `cf-deployment`.
## How do I use it?

If you would like to use a custom client, it requires the `uaa.resource` authority:
```
<custom_client_id>:
authorities: uaa.resource
override: true
authorized-grant-types: client_credentials
secret: <custom_client_secret>
```
### From the `cf` CLI

### Operating Log Cache
Application developers using Cloud Foundry will use Log Cache automatically. Build logs while running
`cf push` are streamed through Log Cache. Application logs when running `cf logs` are retrieved from Log Cache.
Application metrics when running `cf app APP_NAME` are retrieved from Log Cache.

#### Reliability SLO
Log cache depends on Loggregator and is expected to offer slightly lower reliability.
This is primarily due to the ephemeral nature of the cache. Loss will occur during a
deployment. Outside of deployments a 99% reliability can be expected.

#### Cache Duration & Scaling
Log cache is horizontally scalable and we recommend scaling based on the formula below.
We have set a service level objective of 15 minutes with this scaling recommendation.
```
Log Cache Nodes = Envelopes Per Second / 10,000
```

Note - this is intentionally designed to match the scaling of the Log Router used in the
Loggregator system for [colocation in cf-deployment][cf-deployment-ops] - that said more
recent testing with this colocation strategy has not met these SLOs. If targeting these
SLOs is critical to your foundation we recommend using a log-cache instance group.
### Using the Log Cache CLI plugin
To query Log Cache directly users or operators can install the [Log Cache CLI plugin](https://github.com/cloudfoundry/log-cache-cli)
by running `cf install-plugin -r CF-Community "log-cache"` which provides additional commands in the CLI for querying logs and metrics
stored in log cache. This is useful for querying system component metrics which are not exposed otherwise. See the CLI plugin README for details.

### Log Cache API
Documentation about the internals of Log Cache and its API can be found [here](https://github.com/cloudfoundry/log-cache-release/blob/develop/src/README.md)
Documentation about the internals of Log Cache and its API can be found [here](https://github.com/cloudfoundry/log-cache-release/blob/main/src/README.md)