Skip to content

[GOBBLIN-2228] Emit job metrics at RM level #4143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[GOBBLIN-2228] Emit job metrics at RM level #4143

wants to merge 1 commit into from

Conversation

thisisArjit
Copy link
Contributor

@thisisArjit thisisArjit commented Sep 24, 2025 •
edited
Loading

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Emit Resource Manager level otel metrics like number of successful / failed jobs along with other dimensions like flow name, group, id, etc

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

@thisisArjit thisisArjit changed the title Emit job metrics at RM level [GOBBLIN-2228] Emit job metrics at RM level Sep 25, 2025
@thisisArjit thisisArjit marked this pull request as ready for review September 25, 2025 05:31
abhishekmjain
abhishekmjain reviewed Sep 25, 2025
View reviewed changes
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java
public static final String FLOW_UNSCHEDULE_KEY = "flow.unschedule";
public static final String FLOW_OWNING_GROUP_KEY = "flow.owningGroup";
public static final String FLOW_SPEC_EXECUTOR = "flow.edge.specExecutors";
public static final String RM_HOST = "hadoop.resource.manager.rpc";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: RM_HOST_KEY

abhishekmjain
abhishekmjain reviewed Sep 25, 2025
View reviewed changes
.../src/main/java/org/apache/gobblin/temporal/ddm/workflow/impl/ExecuteGobblinWorkflowImpl.java

Meter meter = otelMetrics.getMeter(GAAS_OBSERVABILITY_METRICS_GROUPNAME);
Attributes tags = getEventAttributes(jobState);
log.info("Emitting metrics for job: {}", jobState.getJobName());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we have it as debug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abhishekmjain abhishekmjain abhishekmjain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@thisisArjit @abhishekmjain