[4.10] Merit Certificate Not Issued for Some Users

[komalm]

Hello @pallakartheekreddy,

Environment Details:

Platform: SunbirdEd Portal

Version: 4.10

Background:
We are currently experiencing an issue where some users are not receiving Merit certificates after completing a course that includes an assessment. Interestingly, other users completing the same course and batch are receiving certificates successfully.

Issue Details:

We are using the following course, batch, and assessment:

Course ID: do_1136550052517314561308

Batch ID: 0136550586553057283

Assessment ID: do_1136456311780556801243

Observations:

1. In the Cassandra table sunbird_courses.user_enrolments, the user records appear as expected, for example:

{
  "userid": "{{userId}}",
  "courseid": "do_1136550052517314561308",
  "batchid": "0136550586553057283",
  "certificates": null,
  "certstatus": null,
  "completedon": "2024-11-25 17:08:10.833000+0000",
  "contentstatus": {
    "do_1136456311780556801243": 2,
    ...
  },
  "lastreadcontentid": "do_1136456311780556801243",
  "lastreadcontentstatus": 2,
  "progress": 25,
  "status": 2
}

The value 2 for the assessment in contentstatus confirms completion, but still no certificate is issued.

2. In the Cassandra table sunbird_courses.assessment_aggregator, no records exist for these users — even though they have completed the assessment (as seen in user_enrolments).

3. We attempted a manual certificate reissue, and checked logs in the certsign service, but there are no logs generated for these users.

Please note that this entire certificate generation flow is working correctly for other courses.

Request for Support:
Could you please guide us on the necessary steps to solve this issue.

We would appreciate your support in resolving this issue.

cc @jona-r @vinod303

[aimansharief]

Hey @komalm, while the user is consuming the course, could you check the logs for the following Flink jobs: relation-cache-updater-taskmanager and activity-aggregate-updater-taskmanager? Also, please verify if there's any lag in Kafka.

[jona-r]

Hello @aimansharief,
Thank you for the quick response.

We checked the following task managers for the logs:
1> Relation-Cache-Updater-Taskmanager
2> Activity-Aggregate-Updater-Taskmanager

Also there are no lags in kafka for the topics used by the above flink jobs

We also tried reissuing the certificate and re-attempting the assessment, but did not find any logs related to the issue.

Could you please guide us on how to proceed further?
If convenient, we’re happy to join a quick call to troubleshoot this together.
Looking forward to your support.

cc: @komalm @vinod303

[aimansharief]

Sure @jona-r , we can connect at 4 .

[jona-r]

Hi @aimansharief 4pm is good.

[Sunbird-Lern/Community] [4.10] Merit Certificate Not Issued for Some Users (Discussion #212)
Tuesday, May 20 · 4:00 – 4:30pm
Time zone: Asia/Kolkata
Google Meet joining info
Video call link: https://meet.google.com/huk-tbyj-iwf
Or dial: ‪(US) +1 470-729-3154‬ PIN: ‪901 994 789‬#
More phone numbers: https://tel.meet/huk-tbyj-iwf?pin=3050769775796

[ttpl-sagar]

Hello @aimansharief,
Thank you for the quick response.

We checked the following commands:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic dev.telemetry.assess.failed --partition 0 --offset 275 --max-messages 1

OUTPUT : NOT COME

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic dev.issue.certificate.failed --partition 0 --offset 275 --max-messages 1

OUTPUT :

[2025-05-21 13:07:10,821] WARN [Consumer clientId=consumer-console-consumer-96299-1, groupId=console-consumer-96299] Error while fetching metadata with correlation id 1 : {dev.issue.certificate.failed=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2025-05-21 13:07:10,925] WARN [Consumer clientId=consumer-console-consumer-96299-1, groupId=console-consumer-96299] Error while fetching metadata with correlation id 3 : {dev.issue.certificate.failed=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2025-05-21 13:07:11,027] WARN [Consumer clientId=consumer-console-consumer-96299-1, groupId=console-consumer-96299] Error while fetching metadata with correlation id 4 : {dev.issue.certificate.failed=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

Could you please guide us on how to proceed further?
Looking forward to your support.

cc: @jona-r @komalm

[komalm]

Hello @chethann007,

As per our discussion, the meeting has been rescheduled.
Kindly check your email and confirm the updated invite.

[komalm]

Hello @chethann007

Just checking in for a follow-up on this issue.

This is critical on our end, as many users are currently facing this issue and it's impacting their experience. We would greatly appreciate it if you could share any updates or possible workarounds as soon as possible.

Looking forward to your response.

[chethann007]

Hello @komalm, regarding the same can we have a call today at 12pm?

[komalm]

Hello @chethann007,
Yes, thank you!
I’ve shared the invitation for 5 PM — kindly check and confirm.

[chethann007]

Hello @komalm , yeah i have received the mail.

[NIUANULP]

@madhucr @pramodkvarma

We’re reaching out to request your help with the National Urban Learning Platform (NULP) ticket referenced above. We’re encountering the above challenges, and our users are not receiving certificates due to this. We kindly request Ekstep’s intervention to resolve this issue promptly.

[aimansharief]

@komalm - I agree that a contentstatus value of 2 indicates assessment completion. However, since this is related to the merit certificate, the assessment aggregator job is responsible for computing user progress and scores for course assessment content based on their consumption data. Could you please check if there are any issues with the job or with the data being stored in Cassandra while users are consuming the course?

Along with this, we have to do below analysis to understand the root cause and fix it properly.
As part of the Google Meet calls, we understood that, this issue is there for multiple courses where for a few users the merit certificate is issued but not for others who completed the course. So, let us know a few more details and analysis from your side.

Considering the challenge, it is environment specific issue.

1. Is the infrastructure stable? Are there any jobs/API services restarting?
2. Is this issue there only for the courses with Merit Certificate (With assessment to achieve more than configured score) or Completion Certificate Courses also have it?
3. Is there any lag for Assessment Aggregator and Activity Aggregator Job?
4. Did you check the logs of both the jobs. What are the observations?
5. For a selected user who didn't get the certificate, did we have the assessment computed row in Cassandra?

Please schedule a call along with these details to analyse and fix the issue.

cc: @maheshkumargangula

[komalm]

Hello @maheshkumargangula @aimansharief

Completion certificates

As per our discussion during yesterday’s call, I tried searching for courses that are configured to issue completion certificates. However, I was unable to locate any existing course that currently requires a completion certificate.

To investigate and validate the issue related to certificate issuance, we need access to a existing course that has this functionality enabled. Without such a course, it will not be possible to reproduce or debug the issue.

Note:
Completion certificates are currently being issued successfully for newly created/test courses.

Merit certificates

1. In the Cassandra table sunbird_courses.user_enrolments, the user records appear as expected. and The value 2 for the assessment in contentstatus confirms completion, but still no certificate is issued.
2. In the Cassandra table sunbird_courses.assessment_aggregator, no records exist for these users — even though they have completed the assessment (as seen in user_enrolments).
3. No lag for Assessment Aggregator and Activity Aggregator Job
4. And we didn't get any logs for those jobs
5. We checked the collection-cert-pre-processor and found the following log

2025-06-06 10:42:00,211 INFO  org.sunbird.job.collectioncert.functions.CollectionCertPreProcessorFn [] - issueCertificate i/p event =>JobRequest {map={actor=Map(id -> Course Certificate Generator, type -> System), eid=BE_JOB_REQUEST, edata=Map(courseId -> do_1141935778465628161923, reIssue -> false, iteration -> 1, userIds -> List(92d594c5-bca5-42ac-a96a-ae1dc74562f0), batchId -> 014197918018322432159, action -> issue-certificate, trigger -> auto-issue), ets=1749206520011, context=Map(pdata -> Map(ver -> 1.0, id -> org.sunbird.platform)), mid=LP.1749206520011.83545b95-b46e-44c3-a462-e8ab7bfe535b, object=Map(id -> 014197918018322432159_do_1141935778465628161923, type -> CourseCertificateGeneration)}}
2025-06-06 10:42:00,215 INFO  org.sunbird.job.collectioncert.functions.CollectionCertPreProcessorFn [] - User :: 92d594c5-bca5-42ac-a96a-ae1dc74562f0 did not match the criteria for batch :: 014197918018322432159 and course :: do_1141935778465628161923

[komalm]

Hello @maheshkumargangula, @aimansharief

Can we connect today?

[maheshkumargangula]

@komalm - I requested you to document the summary of our meeting (MoM). Here are the details what we discussed in our last meeting.

• The challenge of certificate issuing is there for both the types of certificate configurations i.e, for Completion Certificate and Merit Certificate configuration of the Course.
• There are no specific user enrolment identified for us to debug and understand the behaviour or challenge of why the certificate was not issued.

Considering the above context, we requested to identify a Course with just Completion certificate configuration (because this is easy to debug) and identify a user enrolment to the same course and batch where the user completed the course but certificate was not issued.

Considering the above comment, looks like there are no Courses and enrolments having issue with Completion Certificate.

Also, based on the data shared above, the enrolment record confirms that, the Course is completed but, the assessment details are not available for the same. Looks like, there is some challenge with the assessment aggregation job. We need to analyse this job and understand the challenge to solve this problem.

Based on the above inputs, please schedule a meeting today evening at 6pm. Please prepare the required information before the meeting to analyse the challenge.
Cc: @aimansharief @chethann007

[komalm]

@maheshkumargangula,
Thank you for summarizing the key points from our last meeting.

Yes, I’ll schedule the meeting for today at 6 PM as requested. I’ll also ensure the required information is prepared beforehand, especially around the assessment aggregation job, so we can analyze the root cause during the discussion.

Let me know if there’s anything specific you’d like me to include in the prep.

[komalm]

@maheshkumargangula @aimansharief @chethann007
I’ve shared the invitation for Today, 6 PM — kindly check and confirm.

[komalm]

Hello,
The certificate issue has been resolved, users are able to download certificates successfully.

Root Cause

The content state API is unstable and returning inconsistent responses such as:

• 401 Unauthorized
• 400 Bad Request
• 200 OK

This inconsistency is likely the reason why some users receive certificates while others do not, even after completing the same course.