Enhance lifetime monitoring in EventListener

[thargy]

Background and Motivation

Further to the discussion at dotnet/diagnostics, EventCounter and EventSource both implement IDisposable after which the associated resources are released. This allows for 'instanced' event counters with a lifetime that does not match the associated EventSource; and event sources that have a shorter lifetime than the associated process.

However, the EventListener only detects one of these four life-time markers. Specifically, it only reports the creation of an EventSource. Ideally, it would be notified for all four, that is the creation and disposal of EventSources and, similarly, of EventCounters, to allow the listener, inheritors, and consumers, to clean-up any resources associated with short-lived counters and/or sources. Without such functionality, a long-running event listener could eventually run out of memory as the monitored process creates and destroys counters/sources dynamically.

Proposed API

The current API is at:

runtime/src/libraries/System.Diagnostics.Tracing/ref/System.Diagnostics.Tracing.cs

Lines 111 to 124 in 6072e4d

	public abstract partial class EventListener : System.IDisposable
	{
	protected EventListener() { }
	public event System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs>? EventSourceCreated { add { } remove { } }
	public event System.EventHandler<System.Diagnostics.Tracing.EventWrittenEventArgs>? EventWritten { add { } remove { } }
	public void DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) { }
	public virtual void Dispose() { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level) { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword) { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string?>? arguments) { }
	protected static int EventSourceIndex(System.Diagnostics.Tracing.EventSource eventSource) { throw null; }
	protected internal virtual void OnEventSourceCreated(System.Diagnostics.Tracing.EventSource eventSource) { }
	protected internal virtual void OnEventWritten(System.Diagnostics.Tracing.EventWrittenEventArgs eventData) { }
	}

Here is the proposed diff.

  System.Diagnostics.Tracing.cs | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/System.Diagnostics.Tracing.cs b/System.Diagnostics.Tracing.cs
index 5af41cf..82a73be 100644
--- a/System.Diagnostics.Tracing.cs
+++ b/System.Diagnostics.Tracing.cs
@@ -112,6 +112,10 @@ namespace System.Diagnostics.Tracing
     {
         protected EventListener() { }
         public event System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs>? EventSourceCreated { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventSourceDisposedEventArgs>? EventSourceDisposed { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterCreatedEventArgs>? EventCounterCreated { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterDisposedEventArgs>? EventCounterDisposed { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterUpdatedEventArgs>? EventCounterUpdated { add { } remove { } }
         public event System.EventHandler<System.Diagnostics.Tracing.EventWrittenEventArgs>? EventWritten { add { } remove { } }
         public void DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) { }
         public virtual void Dispose() { }
@@ -120,6 +124,10 @@ namespace System.Diagnostics.Tracing
         public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string?>? arguments) { }
         protected static int EventSourceIndex(System.Diagnostics.Tracing.EventSource eventSource) { throw null; }
         protected internal virtual void OnEventSourceCreated(System.Diagnostics.Tracing.EventSource eventSource) { }
+        protected internal virtual void OnEventSourceDisposed(System.Diagnostics.Tracing.EventSource eventSource) { }
+        protected internal virtual void OnEventCounterCreated(System.Diagnostics.Tracing.EventCounter eventCounter) { }
+        protected internal virtual void OnEventCounterDisposed(System.Diagnostics.Tracing.EventCounter eventCounter) { }
+        protected internal virtual void OnEventCounterUpdate(System.Diagnostics.Tracing.EventCounterUpdatedEventArgs eventData) { }
         protected internal virtual void OnEventWritten(System.Diagnostics.Tracing.EventWrittenEventArgs eventData) { }
     }
     [System.FlagsAttribute]
@@ -227,6 +235,21 @@ namespace System.Diagnostics.Tracing
         public EventSourceCreatedEventArgs() { }
         public System.Diagnostics.Tracing.EventSource? EventSource { get { throw null; } }
     }
+    public partial class EventSourceDisposedEventArgs : System.EventArgs
+    {
+        public EventSourceDisposedEventArgs() { }
+        public System.Diagnostics.Tracing.EventSource? EventSource { get { throw null; } }
+    }
+    public partial class EventCounterCreatedEventArgs : System.EventArgs
+    {
+        public EventCounterCreatedEventArgs() { }
+        public System.Diagnostics.Tracing.EventCounter? EventSource { get { throw null; } }
+    }
+    public partial class EventCounterDisposedEventArgs : System.EventArgs
+    {
+        public EventCounterDisposedEventArgs() { }
+        public System.Diagnostics.Tracing.EventCounter? EventSource { get { throw null; } }
+    }
     public partial class EventSourceException : System.Exception
     {
         public EventSourceException() { }
@@ -282,6 +305,13 @@ namespace System.Diagnostics.Tracing
         public System.DateTime TimeStamp { get { throw null; } }
         public byte Version { get { throw null; } }
     }
+    public partial class EventCounterUpdatedEventArgs : System.EventArgs
+    {
+        internal EventCounterUpdatedEventArgs() { }
+        public System.Diagnostics.Tracing.EventWrittenEventArgs EventWrittenEventArgs { get { throw null; } }
+        public System.Diagnostics.Tracing.EventCounter EventCounter { get { throw null; } }
+        public double Metric { get { throw null; } }
+    }
     [System.AttributeUsageAttribute(System.AttributeTargets.Method)]
     public sealed partial class NonEventAttribute : System.Attribute
     {

Which adds 3 new events (and associated handlers) to inform the inheritor or consumer of the disposal of an EventSource, and the creation, or disposal, of an EventCounter. Further, it adds a new EventCounterUpdated event which is called whenever and EventWritten event is called, that is logically associated with an EventCounter, allowing for much easier consumption.

NOTE My proposal for EventCounterUpdatedEventArgs is a starting point for discussion, it could easily be expanded to provide more useful functionality, or direct access through to the underlying EventWrittenEventArgs; it is also an addition to correspond more readily to the life-time events for counters. As discussed below, it is not technically a requirement to add the EventCounterUpdated event and this class, but it seems like the opportune time to add such an event to reduce the boilerplate implementations currently required for handling counters and to reflect more accurately the EventSource API.

Usage Examples

Instead of the example found in https://docs.microsoft.com/en-us/dotnet/core/diagnostics/event-counters, the following would be possible, which exposes a simple dictionary of the currently active counters and their latest metric (NOTE This is for demonstration purposes, it hasn't been compiled obviously, and is not concerned regarding thread-safety):

public class SimpleEventListener : EventListener
{
    private readonly int _intervalSec;
    private readonly Dictionary<EventCounter, double> _counters = new Dictionary<EventCounter, double>();

    public IReadonlyDictionary<EventCounter, double> Counters => _counters;

    public SimpleEventListener(int intervalSec = 1) =>
        _intervalSec = intervalSec <= 0
            ? throw new ArgumentException("Interval must be at least 1 second.", nameof(intervalSec))
            : intervalSec;

    protected override void OnEventSourceCreated(EventSource source)
    {
        if (!source.Name.Equals("System.Runtime"))
        {
            return;
        }

        EnableEvents(source, EventLevel.Verbose, EventKeywords.All, new Dictionary<string, string>()
        {
            ["EventCounterIntervalSec"] = _intervalSec.ToString()
        });
    }

    protected override void OnEventCounterCreated(EventCounter counter)
    {
        _counters[counter] = 0D;
    }

    protected override void OnEventCounterUpdated(EventCounterUpdatedEventArgs eventArgs)
    {
        _counters[eventArgs.EventCounter] = eventArgs.Metric;
    }

    protected override void OnEventCounterDisposed(EventCounter counter)
    {
        _counters.Remove(counter);
    }
}

Alternative Designs

• Listeners could record the last time an event was received from a specific counter and infer that it was disposed after a suitable timeout. It would be a significantly inferior solution and would not allow accurate tracking of the lifetime of sources and counters.

• The current API doesn't actually expose EventCounters, but, instead, provides a breakdown of information via EventWrittenEventArgs, the proposed API addresses this by having a new EventCounterUpdated event, which is a subset of the EventWritten event. However, an alternative might be to provide similar data as found in EventWrittenEventArgs to the EventCounter lifetime events and not add the new EventCounterUpdated event.

Risks

I am not currently aware of the information currently passed through the inter-process pipes, as such implementing these new events is likely to increase the bandwidth consumed. However, that is likely insignificant compared to the bandwidth used by the regular event updates themselves. Further, there will be an increase in the number of events raised, though this too would be likely insignificant in comparison to the frequency of the EventWritten event.

Care needs to be taken that monitored producers may not yet send notification of the 3 new lifetime events for the listener to consume. This matters most for anyone expecting the EventCounterCreated event and anticipating being able to perform setup actions prior to receiving events from the specific counter. To mitigate this the event should be fired immediately prior to an EventWritten (and EventCounterUpdated - see above) event.

Simiarly, the two disposal events may not be triggered. This can happen because the monitored app has not yet been updated to a version supporting the proposed API; but can also happen because the process is terminated prematurely, or the objects are not correctly disposed. This is likely to be less problematic to code, however, it too could be mitigated by optionally supporting a timeout mechanism that will signal the relevant xxxDisposed event(s) when a counter or source has not been updated in a particular period. The above suggestion about triggering the relevant xxxCreated event(s), automatically, for an unexpected EventWritten would mitigate premature disposal.

Tagging @sywhang and @josalem as requested.

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

Further to the discussion at dotnet/diagnostics, EventCounter and EventSource both implement IDisposable after which the associated resources are released. This allows for 'instanced' event counters with a lifetime that does not match the associated EventSource; and event sources that have a shorter lifetime than the associated process.

However, the EventListener only detects one of these four life-time markers. Specifically, it only reports the creation of an EventSource. Ideally, it would be notified for all four, that is the creation and disposal of EventSources and, similarly, of EventCounters, to allow the listener, inheritors, and consumers, to clean-up any resources associated with short-lived counters and/or sources. Without such functionality, a long-running event listener could eventually run out of memory as the monitored process creates and destroys counters/sources dynamically.

Proposed API

The current API is at:

runtime/src/libraries/System.Diagnostics.Tracing/ref/System.Diagnostics.Tracing.cs

Lines 111 to 124 in 6072e4d

	public abstract partial class EventListener : System.IDisposable
	{
	protected EventListener() { }
	public event System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs>? EventSourceCreated { add { } remove { } }
	public event System.EventHandler<System.Diagnostics.Tracing.EventWrittenEventArgs>? EventWritten { add { } remove { } }
	public void DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) { }
	public virtual void Dispose() { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level) { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword) { }
	public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string?>? arguments) { }
	protected static int EventSourceIndex(System.Diagnostics.Tracing.EventSource eventSource) { throw null; }
	protected internal virtual void OnEventSourceCreated(System.Diagnostics.Tracing.EventSource eventSource) { }
	protected internal virtual void OnEventWritten(System.Diagnostics.Tracing.EventWrittenEventArgs eventData) { }
	}

Here is the proposed diff.

  System.Diagnostics.Tracing.cs | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/System.Diagnostics.Tracing.cs b/System.Diagnostics.Tracing.cs
index 5af41cf..82a73be 100644
--- a/System.Diagnostics.Tracing.cs
+++ b/System.Diagnostics.Tracing.cs
@@ -112,6 +112,10 @@ namespace System.Diagnostics.Tracing
     {
         protected EventListener() { }
         public event System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs>? EventSourceCreated { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventSourceDisposedEventArgs>? EventSourceDisposed { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterCreatedEventArgs>? EventCounterCreated { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterDisposedEventArgs>? EventCounterDisposed { add { } remove { } }
+        public event System.EventHandler<System.Diagnostics.Tracing.EventCounterUpdatedEventArgs>? EventCounterUpdated { add { } remove { } }
         public event System.EventHandler<System.Diagnostics.Tracing.EventWrittenEventArgs>? EventWritten { add { } remove { } }
         public void DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) { }
         public virtual void Dispose() { }
@@ -120,6 +124,10 @@ namespace System.Diagnostics.Tracing
         public void EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string?>? arguments) { }
         protected static int EventSourceIndex(System.Diagnostics.Tracing.EventSource eventSource) { throw null; }
         protected internal virtual void OnEventSourceCreated(System.Diagnostics.Tracing.EventSource eventSource) { }
+        protected internal virtual void OnEventSourceDisposed(System.Diagnostics.Tracing.EventSource eventSource) { }
+        protected internal virtual void OnEventCounterCreated(System.Diagnostics.Tracing.EventCounter eventCounter) { }
+        protected internal virtual void OnEventCounterDisposed(System.Diagnostics.Tracing.EventCounter eventCounter) { }
+        protected internal virtual void OnEventCounterUpdate(System.Diagnostics.Tracing.EventCounterUpdatedEventArgs eventData) { }
         protected internal virtual void OnEventWritten(System.Diagnostics.Tracing.EventWrittenEventArgs eventData) { }
     }
     [System.FlagsAttribute]
@@ -227,6 +235,21 @@ namespace System.Diagnostics.Tracing
         public EventSourceCreatedEventArgs() { }
         public System.Diagnostics.Tracing.EventSource? EventSource { get { throw null; } }
     }
+    public partial class EventSourceDisposedEventArgs : System.EventArgs
+    {
+        public EventSourceDisposedEventArgs() { }
+        public System.Diagnostics.Tracing.EventSource? EventSource { get { throw null; } }
+    }
+    public partial class EventCounterCreatedEventArgs : System.EventArgs
+    {
+        public EventCounterCreatedEventArgs() { }
+        public System.Diagnostics.Tracing.EventCounter? EventSource { get { throw null; } }
+    }
+    public partial class EventCounterDisposedEventArgs : System.EventArgs
+    {
+        public EventCounterDisposedEventArgs() { }
+        public System.Diagnostics.Tracing.EventCounter? EventSource { get { throw null; } }
+    }
     public partial class EventSourceException : System.Exception
     {
         public EventSourceException() { }
@@ -282,6 +305,13 @@ namespace System.Diagnostics.Tracing
         public System.DateTime TimeStamp { get { throw null; } }
         public byte Version { get { throw null; } }
     }
+    public partial class EventCounterUpdatedEventArgs : System.EventArgs
+    {
+        internal EventCounterUpdatedEventArgs() { }
+        public System.Diagnostics.Tracing.EventWrittenEventArgs EventWrittenEventArgs { get { throw null; } }
+        public System.Diagnostics.Tracing.EventCounter EventCounter { get { throw null; } }
+        public double Metric { get { throw null; } }
+    }
     [System.AttributeUsageAttribute(System.AttributeTargets.Method)]
     public sealed partial class NonEventAttribute : System.Attribute
     {

Which adds 3 new events (and associated handlers) to inform the inheritor or consumer of the disposal of an EventSource, and the creation, or disposal, of an EventCounter. Further, it adds a new EventCounterUpdated event which is called whenever and EventWritten event is called, that is logically associated with an EventCounter, allowing for much easier consumption.

NOTE My proposal for EventCounterUpdatedEventArgs is a starting point for discussion, it could easily be expanded to provide more useful functionality, or direct access through to the underlying EventWrittenEventArgs; it is also an addition to correspond more readily to the life-time events for counters. As discussed below, it is not technically a requirement to add the EventCounterUpdated event and this class, but it seems like the opportune time to add such an event to reduce the boilerplate implementations currently required for handling counters and to reflect more accurately the EventSource API.

Usage Examples

Instead of the example found in https://docs.microsoft.com/en-us/dotnet/core/diagnostics/event-counters, the following would be possible, which exposes a simple dictionary of the currently active counters and their latest metric (NOTE This is for demonstration purposes, it hasn't been compiled obviously, and is not concerned regarding thread-safety):

public class SimpleEventListener : EventListener
{
    private readonly int _intervalSec;
    private readonly Dictionary<EventCounter, double> _counters = new Dictionary<EventCounter, double>();

    public IReadonlyDictionary<EventCounter, double> Counters => _counters;

    public SimpleEventListener(int intervalSec = 1) =>
        _intervalSec = intervalSec <= 0
            ? throw new ArgumentException("Interval must be at least 1 second.", nameof(intervalSec))
            : intervalSec;

    protected override void OnEventSourceCreated(EventSource source)
    {
        if (!source.Name.Equals("System.Runtime"))
        {
            return;
        }

        EnableEvents(source, EventLevel.Verbose, EventKeywords.All, new Dictionary<string, string>()
        {
            ["EventCounterIntervalSec"] = _intervalSec.ToString()
        });
    }

    protected override void OnEventCounterCreated(EventCounter counter)
    {
        _counters[counter] = 0D;
    }

    protected override void OnEventCounterUpdated(EventCounterUpdatedEventArgs eventArgs)
    {
        _counters[eventArgs.EventCounter] = eventArgs.Metric;
    }

    protected override void OnEventCounterDisposed(EventCounter counter)
    {
        _counters.Remove(counter);
    }
}

Alternative Designs

• Listeners could record the last time an event was received from a specific counter and infer that it was disposed after a suitable timeout. It would be a significantly inferior solution and would not allow accurate tracking of the lifetime of sources and counters.

• The current API doesn't actually expose EventCounters, but, instead, provides a breakdown of information via EventWrittenEventArgs, the proposed API addresses this by having a new EventCounterUpdated event, which is a subset of the EventWritten event. However, an alternative might be to provide similar data as found in EventWrittenEventArgs to the EventCounter lifetime events and not add the new EventCounterUpdated event.

Risks

I am not currently aware of the information currently passed through the inter-process pipes, as such implementing these new events is likely to increase the bandwidth consumed. However, that is likely insignificant compared to the bandwidth used by the regular event updates themselves. Further, there will be an increase in the number of events raised, though this too would be likely insignificant in comparison to the frequency of the EventWritten event.

Care needs to be taken that monitored producers may not yet send notification of the 3 new lifetime events for the listener to consume. This matters most for anyone expecting the EventCounterCreated event and anticipating being able to perform setup actions prior to receiving events from the specific counter. To mitigate this the event should be fired immediately prior to an EventWritten (and EventCounterUpdated - see above) event.

Simiarly, the two disposal events may not be triggered. This can happen because the monitored app has not yet been updated to a version supporting the proposed API; but can also happen because the process is terminated prematurely, or the objects are not correctly disposed. This is likely to be less problematic to code, however, it too could be mitigated by optionally supporting a timeout mechanism that will signal the relevant xxxDisposed event(s) when a counter or source has not been updated in a particular period. The above suggestion about triggering the relevant xxxCreated event(s), automatically, for an unexpected EventWritten would mitigate premature disposal.

Tagging @sywhang and @josalem as requested.

Author:	thargy
Assignees:	-
Labels:	api-suggestion, area-System.Diagnostics.Tracing, untriaged
Milestone:	-

[thargy]

A more compelling example usage would probably use SourceCache from DynamicData (effectively an observable dictionary), or an ObservableCollection.

[thargy]

Not going to lie, sad to see this bumped 😞.

[josalem]

I saw your comment @thargy and took a look through the history on this.

As you pointed out, this didn't make it in to .net6.

That being said, I think there a few things worth pointing out. EventSource isn't intended to be used in an instanced manner. As you mentioned in the other issue, it does implement IDisposable and you certainly can create instanced sources. Doing so, however, is rather expensive as there are substantial resources created depending on the use case and a lot of duplicate work is performed.

We're, however, looking into some changes to lifetimes of EventSource in .net7 on since there are some considerations around AssemblyLoadContexts that we probably need to address. If we choose to investigate this specific API suggestion it would most likely happen as part of that work.

All that aside, I think there is already a way to do what you want without having to keep track of the lifetimes or even expect that EventSources have lifetimes shorter than the process.

The following should roughly map to what it sounds like you're trying to do (caveat emptor -- I didn't actually run this code):

public sealed class MyComponentCounterSource : EventSource
{
	// you could also return the counter directly and track it in the Component itself rather than storing it on the EventSource in this dict
	private Dictionary<component, EventCounter> _counterDict = new();

	public static MyCounterSource Log = new();

	[NonEvent]
	public void OnComponentConstruction(MyComponent component)
	{
		lock (_counterDict)
		{
			_counterDict.Add(component, new PollingCounter(...));
		}
	}

	[NonEvent]
	public void OnComponentDispose(MyComponent component)
	{
		lock (_counterDict)
		{
			_counterDict.Remove(component, out EventCounter counter);
			counter.Dispose();
		}
	}
}

public class MyComponent : IDisposable
{
	public MyComponent()
	{
		// rest of init
		MyComponentCounterSource.Log.OnComponentConstruction(this);
	}

	public void Dispose()
	{
		MyComponentCounterSource.Log.OnComponentDispose(this);
		// other disposal
	}
}

Alternatively, you could have regular events on an EventSource and you have the events contain the instance name, the counter name on that instance, and the counter values. Though, that is effectively re-implementing EventCounters.

Hopefully, that helps! Again, we'll revisit this type of change in the 7.0 timeframe 😃