feat: Add HDFS support

[Xuanwo]

Signed-off-by: Xuanwo [email protected]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

This PR will allow databend-query to use hdfs as storage backend.

Part of #5215

Changelog

• New Feature

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		May 9, 2022 at 4:25AM (UTC)

[mergify]

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

[Xuanwo]

@dantengsky Maybe I need to borrow some work from your PR to get JAVA setup correctly?

[BohuTANG]

This PR only makes the HDFS as a normal storage backend like AWS S3, it's not related to @dantengsky work on hive?

[Xuanwo]

From I understand, this PR makes the HDFS as a normal storage backend like AWS S3, it's not related to @dantengsky work on hive?

Yes. Hive's integration should happen in another PR.

[BohuTANG]

If we don't add the HdfsConfig item to *.toml, the config deserialize looks will crash?

[BohuTANG]

This is another problem, we don't want unused config items explicitly:

[storage]
# fs|s3
type = "s3"

[storage.fs] -- this config

[storage.s3]
bucket = "databend"
endpoint_url = "https://s3.amazonaws.com"
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"

[storage.azblob] -- this config

Is it possible to configure the item only we used, like [storage.s3] here?

[Xuanwo]

If we don't add the HdfsConfig item to *.toml, the config deserialize looks will crash?

If HdfsConfig is not added in *.toml, we will use the default value instead.

I tested this behavior locally: query is able to start without adding hdfs-related staff.

[BohuTANG]

That's great, I am out, thank you.
I will remove all the unused config items from the documents.

[dantengsky]

@dantengsky Maybe I need to borrow some work from your PR to get JAVA setup correctly?

Yeah, currently, hive PR's ut/it is not integrated with the github workflows yet. Just a local hadoop + hive cluster for the testings. there are no build-time dependencies on the JDK/jar files. A docker image seems to be able to cover it.

But for this PR, a docker image may not be enough. hope I get it right:

• to enable feature storage-hdfs at compiling
  we need a JDK(for libjvm.so and some header files of jvm)
• to enable feature storage-hdfs at runtime (or ut/it)
  an HDFS cluster (for the specified version) is needed

[Xuanwo]

@dantengsky Maybe I need to borrow some work from your PR to get JAVA setup correctly?

Yeah, currently, hive PR's ut/it is not integrated with the github workflows yet. Just a local hadoop + hive cluster for the testings. there are no build-time dependencies on the JDK/jar files. A docker image seems to be able to cover it.

But for this PR, a docker image may not be enough. hope I get it right:

• to enable feature storage-hdfs at compiling
  we need a JDK(for libjvm.so and some header files of jvm)
• to enable feature storage-hdfs at runtime (or ut/it)
  an HDFS cluster (for the specified version) is needed

Thanks for the advice! Maybe we can make databend-query compilable in this PR and test it in the next PR.

[BohuTANG]

👍