[SPARK-50568][SS][TESTS] Fix KafkaMicroBatchSourceSuite to cover KafkaOffsetReaderConsumer

[ostronaut]

What changes were proposed in this pull request?

This was missed when the default value of configuration is changed via the following.

• [SPARK-40844][SS] Flip the default value of Kafka offset fetching config #38306

KafkaMicroBatchSourceSuite consists of set of different suites, where KafkaMicroBatchSourceSuiteBase based suite is defined. There are 4 implementations of this abstract class for now:

1. KafkaMicroBatchV1SourceSuite - V1 source that supposes to use KafkaOffsetReaderConsumer as KafkaOffsetReader.
2. KafkaMicroBatchV2SourceSuite - V2 source that supposes to use KafkaOffsetReaderConsumer as KafkaOffsetReader.
3. KafkaMicroBatchV1SourceWithAdminSuite - V1 source that uses KafkaOffsetReaderAdmin as KafkaOffsetReader.
4. KafkaMicroBatchV2SourceWithAdminSuite - V2 source that uses KafkaOffsetReaderAdmin as KafkaOffsetReader.
   But KafkaMicroBatchV1SourceSuite and KafkaMicroBatchV2SourceSuite are still running based on KafkaOffsetReaderAdmin, as USE_DEPRECATED_KAFKA_OFFSET_FETCHING is false be default. By switching it to true in beforeAll, we make sure that corresponding KafkaOffsetReader is in use.

Why are the changes needed?

To improve unit tests coverage for KafkaOffsetReaderConsumer

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit Tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Thank you for making a PR, @ostronaut . Could you file a JIRA issue and use it in the PR title with a prefix pattern, [SPARK-XXX]?

[HeartSaVioR]

Nice finding! Shall we do this a bit differently? Let's leave the base test class as it is (using default), and rename the derived test class to refer to Consumer and set the flag explicitly. I'm even OK to have a base test class as abstract, and have two derived test classes having config set for each. Maybe this is much clearer?

[dongjoon-hyun]

cc @gaborgsomogyi and @viirya too from the following

• [SPARK-40844][SS] Flip the default value of Kafka offset fetching config #38306

[viirya]

Good catch!

[gaborgsomogyi]

Thanks for catching it! After adapting the suggestions from @HeartSaVioR it will be good to go.

[ostronaut]

Thank you @HeartSaVioR for the review! I've refactored KafkaMicroBatchV1SourceSuite and KafkaMicroBatchV2SourceSuite to be abstract classes. All further implementations are inherit unit tests from those and only change config.

[ostronaut]

Hi @dongjoon-hyun. Good point, SPARK-50568 here it is!

[dongjoon-hyun]

+1, LGTM.

[HeartSaVioR]

+1

[HeartSaVioR]

Thanks! Merging to master.