tool: Add watch-for-rts-spike

[marksvc]

This change is 

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.30%. Comparing base (413b58d) to head (b983095).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3383   +/-   ##
=======================================
  Coverage   82.30%   82.30%           
=======================================
  Files         613      613           
  Lines       36869    36869           
  Branches     6020     6020           
=======================================
  Hits        30345    30345           
  Misses       5642     5642           
  Partials      882      882           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[marksvc]

I have been using this script on our servers and it seems to be working well. I wanted to add it to our code repo to keep track of it.

Reviewable status: 0 of 1 files reviewed, all discussions resolved

---

scripts/watch-for-rts-spike.mts line 45 at r1 (raw file):

      }

      const aboveThreshold: boolean = memoryUsageMB >= this.options.thresholdMib;

Hmm, I would be inclined to change this from >= to >, except that this is what is running on the server right now :)

[marksvc]

This does not require testing because it is a developer tool.

Although it can be used / tested by running ./watch-for-rts-spike.mts. --help shows options. Deno needs to be installed.

[RaymondLuong3]

Mark and I looked at this together. It seems we are not able to get it working on my windows machine. I looked through the code and it seems to be doing the right things.

@RaymondLuong3 reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @marksvc)

---

scripts/watch-for-rts-spike.mts line 6 at r1 (raw file):

// sends SIGUSR2 signals at a configured interval so the target process can perform diagnostic actions.
// Designed to help capture data around sudden RealtimeServer memory spikes.

Could you add a usage line with the necessary environment variables here.

[marksvc]

Raymond and I had a look at how we might run this on Windows. pgrep was not available. ps -W was able to show Windows PIDs, but it did not appear to see the RealtimeServer process. A Windows specific task list program might be successful. /bin/kill -f -W was able to send signals to RealtimeServer, though when specifying -SIGUSR2, it killed RealtimeServer and did not generate .csv files. More work could be done to make the script multi-platform; though right now it just needs to run on the server.

[marksvc]

When testing, make sure to run dotnet with environment variable setting SF_SIGUSR2_ACTION=resourceUsage.

[marksvc]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @RaymondLuong3)

---

scripts/watch-for-rts-spike.mts line 6 at r1 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

Could you add a usage line with the necessary environment variables here.

I added a note about SF_SIGUSR2_ACTION, as well as a pointer to diagnostics.ts, which can be followed to learn about what happens when a report is requested.

[pmachapman]

@marksvc Just one change - can you please give examples of how to run it with command argument examples in the script header?

This script cannot run on Windows, as Windows does not have SIGUSR2, ps, pgrep, or kill, but it works great on my Linux PC.

@pmachapman reviewed all commit messages.
Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @RaymondLuong3)

[marksvc]

I added a bunch to the header about how to run the program on a Linux workstation or on the server. How's this?

I'd like for our tools to run on all of our development platforms. Not having SIGUSR2 is quite a barrier for getting this script to work. But that would explain why Raymond and I couldn't get it to work on his Windows machine.

But that makes me wonder how you trigger creating heap snapshots and profiling of SF on Windows if not by sending SIGUSR2. ? (i.e. in diagnostics.ts)

Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @RaymondLuong3)

[pmachapman]

I added a bunch to the header about how to run the program on a Linux workstation or on the server. How's this?

Looks good. Thank you!

But that makes me wonder how you trigger creating heap snapshots and profiling of SF on Windows if not by sending SIGUSR2. ? (i.e. in diagnostics.ts)

I always run with a debugger attached. If our prod environment was Windows (for some reason), we would have to implement another port for debugging and have our own protocol for doing this stuff.

@pmachapman reviewed 1 of 2 files at r3, all commit messages.
Reviewable status: 1 of 2 files reviewed, all discussions resolved (waiting on @RaymondLuong3)

[Nateowami]

@marksvc Is this going to be merged?