feat(enrichment tables): add expired items output to memory enrichment table

[esensar]

Summary

Adds a new output to memory enrichment table source that exports items as they are expired out of this cache.

Vector configuration

enrichment_tables:
  memory_table:
    type: memory
    ttl: 10
    flush_interval: 5
    inputs: ["cache_generator"]
    source_config:
       source_key: "memory_table_source"
       export_expired_items: true

sources:
  demo_logs_test:
    type: "demo_logs"
    format: "json"

transforms:
  demo_logs_processor:
    type: "remap"
    inputs: ["demo_logs_test"]
    source: |
      . = parse_json!(.message)
      user_id = get!(., path: ["user-identifier"])

      # Look for existing value in the table, using "user-identifier" as key
      existing, err = get_enrichment_table_record("memory_table", { "key": user_id })

      if err == null {
        . = existing.value
        .source = "cache"
      } else {
        .referer = parse_url!(.referer)
        .referer.host = encode_punycode!(.referer.host)
        .source = "transform"
      }

  cache_generator:
    type: "remap"
    inputs: ["demo_logs_processor"]
    source: |
      existing, err = get_enrichment_table_record("memory_table", { "key": get!(., path: ["user-identifier"]) })
      if err != null {
        data = .
        . = set!(value: {}, path: [get!(data, path: ["user-identifier"])], data: data)
      } else {
        . = {}
      }

# We can observe that after some time data will be dumped to console from the cache
sinks:
  console:
    inputs: ["memory_table_source.expired"]
    target: "stdout"
    type: "console"
    encoding:
      codec: "json"

How did you test this PR?

Ran the above configuration and observed that items start appearing after ~10 seconds in the output - the items that expire in the memory enrichment table.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related (this can be used to achieve the behavior requested in this item): Dedupe optional feature: cache and emit last match instead of first #23784

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

---

Sponsored by Quad9

[esensar]

Sorry about that, I haven't run checks after that quick fix. I am running them right now and it seems they are all passing.