[SPARK-10655][SQL] Adding additional data type mappings to jdbc DB2dialect. #9162

sureshthalamati · 2015-10-19T00:12:10Z

This patch adds DB2 specific data type mappings for decfloat, real, xml , and timestamp with time zone (DB2Z specific type) types on read and for byte, short data types on write to the to jdbc data source DB2 dialect. Default mapping does not work for these types when reading/writing from DB2 database.

Added docker test, and a JDBC unit test case.

rxin · 2015-10-20T05:45:07Z

Jenkins, test this please.

SparkQA · 2015-10-20T07:44:00Z

Test build #1930 has finished for PR 9162 at commit ec8e546.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-10-20T07:48:07Z

sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala

isn't this mapping it to normal decimal with precision 31?

i.e. precision lost here?

Using this mapping DB2 will throw error if the precision of the value being written is > 31 during insert execution. But if the value scale is higher than 2 , value will rounded to scale of 2.

Other alternative was to let it fail with error when create table is executed (current behavior). I was not sure how common the decimal type system default (38, 18) is used to create the data frame. I thought it is better to map to DB2 max precision, instead of failing with error ,considering there is no way to write decimals of precision > 31 to DB2.

If you think it is better to fail on create , instead of surprises during execution. I will update the patch.

I am also working on creating pull request (SPARK-10849) to allow users to specify the target database column type. This will allow user to specify the decimal precision, and scale of their choice in these kind of scenarios.

sureshthalamati · 2015-10-28T17:08:05Z

Thank you for reviewing patch , Reynold. Please let me know if it needs any more changes.

rick-ibm · 2015-11-04T18:13:26Z

I think these are fine defaults. The work on SPARK-10849 gives fine-grained control to power users. LGTM. Thanks.

sureshthalamati · 2015-11-10T00:38:14Z

@rxin Wondering if I need any additional work on this patch ? Thanks.

JoshRosen · 2015-11-11T00:05:32Z

Hey @sureshthalamati, take a look at #9503, which we just merged to re-enable the Docker JDBC integration tests; you should be able to build on that patch to write integration tests that run against a real DB2 instance that's running in Docker.

sureshthalamati · 2015-11-16T08:25:06Z

Thank you for the information , Josh. Luciano seems to working on creating PR for DB2 docker test setup. I will check with him , and incorporate the type mapping tests into the docker test case.

@lresende

lresende · 2015-11-30T21:52:35Z

I have a wip pr #9893, and an open question that I think @JoshRosen could help about the JDBC drivers for the other docker tests. Anyway, these two PRs are independent of each other, no ?

sureshthalamati · 2016-05-21T07:30:42Z

@JoshRosen @rxin

Updated the PR with the DB2 docker test case for the DB2 specific type mappings added in this PR. TIME STAMP WITH TIME ZONE is DB2Z specific , I could not add the docker test case for this data type.

Can you please review the updated PR.

sureshthalamati · 2016-05-24T17:48:36Z

ping @JoshRosen @rxin

This issue was blocked due to docker relates issues for a while. If you can review the updated PR , that will be great.

lresende · 2016-05-24T18:17:37Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala

Is there a duplication of BooleanType line here ? probably copy + paste issue.

lresende · 2016-05-24T18:20:00Z

Other then the minor comment around BooleanType, it LGTM.

sureshthalamati · 2016-05-25T01:06:17Z

Thanks, Luciano. Addressed your comment.

sureshthalamati · 2016-06-01T23:04:06Z

@JoshRosen @rxin Any suggestions to improve this fix to get it merged ?

lresende · 2016-06-03T22:28:24Z

Jenkins retest this please

gatorsmile · 2016-06-03T22:33:32Z

.../docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala

How about DECFLOAT(16) and DECFLOAT(34)?

Thanks for reviewing @gatorsmile . Added test cases for those two variations of the DECFLOAT types.

SparkQA · 2016-09-26T23:08:12Z

Test build #65932 has finished for PR 9162 at commit f85c3d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- assert(types(8).equals(\"class java.math.BigDecimal\"))
- assert(types(9).equals(\"class java.math.BigDecimal\"))

gatorsmile · 2017-06-16T23:32:43Z

@sureshthalamati Could you please retest it using DB2?

gatorsmile · 2017-06-16T23:32:49Z

retest this please

SparkQA · 2017-06-17T02:23:39Z

Test build #78198 has finished for PR 9162 at commit f85c3d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
assert(types(8).equals(\"class java.math.BigDecimal\"))
assert(types(9).equals(\"class java.math.BigDecimal\"))

sureshthalamati · 2017-06-20T18:52:45Z

sure @gatorsmile . Thanks.

sureshthalamati · 2017-06-21T00:35:23Z

@gatorsmile I rebased and ran the DB2 docker test on my machine, it ran fine.

SparkQA · 2017-06-21T03:04:35Z

Test build #78337 has finished for PR 9162 at commit 8b8bc9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
assert(types(8).equals(\"class java.math.BigDecimal\"))
assert(types(9).equals(\"class java.math.BigDecimal\"))

gatorsmile · 2017-06-21T05:29:57Z

The scope of this PR is just to the DB2 dialect. The risk is pretty small even if the fix might not cover all the scenarios. Thanks! @sureshthalamati

LGTM

gatorsmile · 2017-06-21T05:35:58Z

Thanks! Merging to master.

sureshthalamati · 2017-06-21T23:07:41Z

Thank you @gatorsmile

…alect. This patch adds DB2 specific data type mappings for decfloat, real, xml , and timestamp with time zone (DB2Z specific type) types on read and for byte, short data types on write to the to jdbc data source DB2 dialect. Default mapping does not work for these types when reading/writing from DB2 database. Added docker test, and a JDBC unit test case. Author: sureshthalamati <[email protected]> Closes apache#9162 from sureshthalamati/db2dialect_enhancements-spark-10655.

rxin reviewed Oct 20, 2015
View reviewed changes

sureshthalamati force-pushed the db2dialect_enhancements-spark-10655 branch from ec8e546 to 729fb85 Compare November 25, 2015 23:03

sureshthalamati force-pushed the db2dialect_enhancements-spark-10655 branch from 27eeba0 to 5d022be Compare May 21, 2016 00:26

lresende reviewed May 24, 2016
View reviewed changes

gatorsmile reviewed Jun 3, 2016
View reviewed changes

sureshthalamati added 3 commits June 20, 2017 11:36

[SPARK-10655] Adding additional data type mappings to jdbc DB2Dialect.

4abba5c

Addressed review comments.

7eaaa30

Adding additional test cases for DECFLOAT(16) and DECFLOAT(34)

8b8bc9a

sureshthalamati force-pushed the db2dialect_enhancements-spark-10655 branch from f85c3d9 to 8b8bc9a Compare June 21, 2017 00:31

asfgit closed this in 9ce714d Jun 21, 2017

[SPARK-10655][SQL] Adding additional data type mappings to jdbc DB2dialect. #9162

[SPARK-10655][SQL] Adding additional data type mappings to jdbc DB2dialect. #9162

Conversation

sureshthalamati commented Oct 19, 2015 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rxin commented Oct 20, 2015

Uh oh!

SparkQA commented Oct 20, 2015

Uh oh!

rxin Oct 20, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Oct 20, 2015

Choose a reason for hiding this comment

Uh oh!

sureshthalamati Oct 20, 2015

Choose a reason for hiding this comment

Uh oh!

sureshthalamati commented Oct 28, 2015

Uh oh!

rick-ibm commented Nov 4, 2015

Uh oh!

sureshthalamati commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 11, 2015

Uh oh!

sureshthalamati commented Nov 16, 2015

Uh oh!

lresende commented Nov 30, 2015

Uh oh!

sureshthalamati commented May 21, 2016

Uh oh!

sureshthalamati commented May 24, 2016

Uh oh!

lresende May 24, 2016

Choose a reason for hiding this comment

Uh oh!

lresende commented May 24, 2016

Uh oh!

sureshthalamati commented May 25, 2016

Uh oh!

sureshthalamati commented Jun 1, 2016

Uh oh!

lresende commented Jun 3, 2016

Uh oh!

gatorsmile Jun 3, 2016

Choose a reason for hiding this comment

Uh oh!

sureshthalamati Jun 6, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

gatorsmile commented Jun 16, 2017

Uh oh!

gatorsmile commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

sureshthalamati commented Jun 20, 2017

Uh oh!

sureshthalamati commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

gatorsmile commented Jun 21, 2017

Uh oh!

gatorsmile commented Jun 21, 2017

Uh oh!

sureshthalamati commented Jun 21, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sureshthalamati commented Oct 19, 2015 •

edited

Loading