forked from apache/spark
-
Notifications
You must be signed in to change notification settings - Fork 0
For review #15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
For review #15
Changes from all commits
Commits
Show all changes
306 commits
Select commit
Hold shift + click to select a range
401a64c
Fixed CreateTable testcase problem and updated RowKeyParser
sboeschhuawei 7106713
add type conversion function
93af29f
Removed LogicalPlan and SchemaRDD from PhysicalPlans
sboeschhuawei 5e792ad
Working through HBase Snappy issues and HBaseSQLParser resolution issue
sboeschhuawei a9c22ff
Add content to test
4321d7e
Additional work on partitioning
sboeschhuawei bb8dc68
Removed LogicalPlan and SchemaRDD from PhysicalPlans
sboeschhuawei b46140a
Modify the workflow of InsertIntoTable
093e164
Incremental query testing
sboeschhuawei de26141
Working through issues with Catalog integration
sboeschhuawei 33fe712
Fixed Catalog bugs: namespace mixup (partial fix), RowKey in wrong or…
sboeschhuawei b768186
Disabled pred pushdown and able to reach ReaderRDD
sboeschhuawei d54fa22
Change the syntax of CreateTable
6871100
fix the namespace issues
ff9714e
add delete table function
96d0290
Add Drop
6d58edc
Fixed conn issues in HBaseSQLReaderRDD
sboeschhuawei d8444f7
Fixed conn issues in HBaseSQLReaderRDD
sboeschhuawei e959502
use catalyst data type instead of hbase data type
6b60109
remove hbase data type
a5fd662
Change the input to catalyst datatype
aee3401
RowKey and HBaseSQLReaderRDD fixes
sboeschhuawei bbec48e
fix the code style issue
c89bf27
Add verification to Hbase CreateTable
ecdfcb4
Basic query working
sboeschhuawei a6cbd95
Fixed conn issues in HBaseSQLReaderRDD
sboeschhuawei 4913a8e
add logical table exist check
37b4387
fix the data type issue
af30223
Fixed RowKeyParser write path - used by InsertIntoTable
sboeschhuawei 57bf401
Modify the verification and add HBaseAnalyzer for future development
eadc2b5
Ignore integration tests requiring external hbase access
sboeschhuawei 407e97d
create hbase table required for testing
d8acba2
add more data type tests
4d82fe1
code formatting
59a4414
Fixed select * path but order is incorrect
sboeschhuawei 45f799c
Fixed conn issues in HBaseSQLReaderRDD
sboeschhuawei 823b91c
Small test tweaks for preds
sboeschhuawei f3afe35
Refactored according to Yan's designs
sboeschhuawei 5d4df1a
Refactored according to Yan's designs
sboeschhuawei 5dae756
Removed unused/unnecessary classes and code
sboeschhuawei c120105
mv hbase files to the new old dir
7f2f032
code format and minor fix
scwf 04385a7
update with apache master and fix confilict
scwf ad798d8
Simplified version of HBaseSQLContext
3cd96cf
scala style fix
scwf a10f270
revert some diffs with apache master
scwf f2916ee
Merge pull request #2 from scwf/hbase
scwf ecf84d7
Delete HBaseAnalyzer
0fe8f02
Change the syntax of InsertIntoTable
e0d1621
Modify the workflow of CreateTable and DropTable
dbb16bb
Make it compatible with updated Apache Spark Code
cfdd604
add catalog file
b59f28e
add to cache
643da9c
Fix some compilation errors
238f2b3
Move Parser files to original dir
89d918e
add delete table
8a7d47f
adding HBaseOperators and HBaseStrageties
03f01f3
fix a compilation error
efeaa9c
addition of package.scala
2a8eefc
add skeleton of HBaseRelation
6f53486
fix the test cases and delete table
2550b49
Clean up the HBaseRelation
0f75a20
Merge HBaseCatalogTable into HBaseRelation
6f5c2d0
Fix the complition errors
91eccd5
change create table method
d056c16
Recover the CreateTable workflow
89ab616
fix the issue based on tests
c9f1547
persist metadata using object serialization
61838fd
Change the input order
2e1b9c9
refact the Query processing
af4b958
merged from the remote branch
9dfb009
remove the old dir
2cdb1bf
remove unnecessary tolowercase
a4a810c
new package execution and logical
scwf 1d10ddd
minore fix
scwf c49e64f
use string formatter
1302fdb
Fix the issue of getting HBaseRelation attributes
aa02d90
Merge pull request #5 from scwf/hbase
scwf c36082f
add row key support
108123d
file rename according sparksql convention
ccf6056
Rename HBaseCommands.scala to hbaseCommands.scala
ce5cb59
Rename HBaseOperators.scala to hbaseOperators.scala
949ef99
optimize import
0c346f3
use arraybuffer to improve performance
3ebd032
use allColumns as single parameter in the create method
9ac6578
Change the input parameter of CreateTable to catalog
944d88d
add close() to the function
57e3036
simplify HBaseRelation constructor
8295e54
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into hbase
b25addf
fix the test case issues after removing the parameters in the create …
d2e6e19
comment on HBaseSQLContext
202ffa5
Add the testcases to test RowKeyParser part and Add the test to inser…
4affc61
Removal of TestData.scala
3364342
add "val" to the class definition
2409ba7
Fix the bugs in doing Select
6bcfe0e
Simplify HBaseRelation ctor
59552df
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into hw…
46e2df1
refactory
34d548c
Add testcases for Select and temporarily change some HBaseCatalog int…
98ec286
fix the compilation errors
d083e27
fix a NPE when a table does not exist
3698247
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into hbase
ae2d31a
add INT keyword
567f494
Merge pull request #10 from jackylk/hbase-int
0f9ccb6
add bytes implementation
e09b52c
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into t1
34533b8
Merge pull request #7 from jackylk/rename
ba39b6c
initial support bulk loading
scwf 360f483
add interfaces for altering table
95e6520
reordering keywords according to community convention
3c10e53
rename SparkImmutableBytesWritable
scwf e99841b
[WIP] support for creating hbase user table in CREATE TABLE
efc9348
fix line exceed issue
386b600
add alter table support
d512f8c
add alter table support
2993d80
add hbase profile to Pom, now we can use sbt/sbt -Phbase assembly
scwf ef05b51
rename the file
0963d20
add test case for altering table
119f187
Add InsertIntoTable Support
153c123
Add one more Bytes conversion
0d41c67
Add Alter ADD and Alter Drop
5d95863
Merge pull request #11 from jackylk/reorder
5dd02ec
fix line length issue
e5ad775
adding FIELDS TERMINATED BY clause in bulkload
39ab8e3
Merge pull request #13 from jackylk/create-htable
53db574
new implemention of bytes utility
18c96f0
update the test cases
0306443
reformating
adecb45
Merge pull request #16 from jackylk/reformat
1651d41
change short implementation
1a3a39d
update test cases
7a337da
provide double/long implementation
65d1d91
Add buildFilter
a0c9299
Fix the issue in Byte conversion
6f5f4e3
make string2KV more general to use
scwf f4ec6a4
use new BytesUtil instead
scwf d712d9d
adding HBaseShuffledRDD
scwf ff5e5a4
remove unused variable
5f5ba9b
Merge pull request #20 from jackylk/remove-val
88955ed
refactory for helper
efaffa8
draft for optimized bulk loading
scwf 8f99fc6
change according to comment
048597f
Use of primary key in DDL
d186701
partial evaluation for partition pruning
3d2d345
partial evaluation for partition pruning
204ca57
Remove unnecessary change
f5b0bb8
Merge pull request #21 from jackylk/refactory-helper
3189c53
hot fix for compile
scwf cf63dd2
fix style
scwf 61cebfc
use buffer to speed up encode/decode
7bfc1a0
resolving conflict
79a3add
Update the pom
e9f9d9d
Format the pom file
d3df256
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into te…
56ec13e
modify plan execution
0b4fbd4
bulk load clean
scwf 0a88e09
adding FIELDS TERMINATED BY clause in bulkload
scwf cbe8d48
add FIELDS TERMINATED BY to phsical plan
scwf b13efbf
Add HBase CLI
sboeschhuawei fe96685
Add first version of HBaseIntegrationTestBase
sboeschhuawei a2d2ad9
Add first version of HBaseIntegrationTestBase (2)
sboeschhuawei 0e0540f
formatting
8c5187c
use listbuffer instead of arraybuffer
c474832
fix test for bulk loading
scwf 31ca595
fix list buffer compile error
scwf 783b19d
clean and add a play test for minicluster
scwf d283e51
add license title
scwf 94b1b7a
fix test: bulk loading test ok locally
scwf 3309aaa
clean debug code
scwf fec1a69
fix hbasepartitioner bug
scwf 895d6df
range comparison update
7190353
range comparison update
83bcea0
refactory bulk load logical plan class name
831b0be
add comments for range comparision
8f9435b
replace the default partitioning with range partitioning
e9186bf
Modify the PartialPredEval and Add testcases
0163ee4
basic CLI support
92bbb99
partial implement
47b5d6f
add todo
0b6aa6b
Fix the bugs in PartialEval
3fd88d3
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into sh…
88d988a
Add AND, OR and Not in PartialEval and Modify tryCompare
df19391
basic CLI support
19750db
Add Equal case to tryCompare and Change to use the correct equiv oper…
295f144
Small change on test case
e6dbd8a
fix the deprecated issue
18844c9
add basic CLI support
d12c8ab
fix bug in CREATE TABLE using int data type
cebc0f7
add support for SHOW TABLES
1235d6e
solve conflict
1fdbf7c
add support for INSERT VALUES
b09ee41
solve conflict
0289fe0
fix the style check errors
71b342e
initial code for points finder
a3f2fd5
code cleanup of HBaseRelation
7cc1b53
fix build failure
scwf 76cf17d
optimize the imports
39323e5
create HBaseAdmin instance just once
07bbec1
Interim testcase work
sboeschhuawei af9194a
Working version of HBaseIntegrationTestBase
sboeschhuawei e66c809
Working version of HBaseIntegrationTestBase
sboeschhuawei 5214bfe
add support for DESCEIBE
5dad6ee
let Analyzer resolve the relation in INSERT VALUES
37c9476
add completion support and help support in CLI
a017251
update with apache master
scwf 35bd58e
fix hbase pom
scwf 2f969dc
https -> http in pom so that locally agent work
scwf 678e4ff
fix compile error
scwf 84656bd
revert some no need change
scwf a532044
update with hbase
scwf d423718
fix scala style
scwf e4cdcc3
draft
scwf 8eb8cdc
draft for support user defined schema
scwf b3d35c7
fix comment
scwf c203ce2
adding test case
scwf 5378c5a
fix the style error
f21d6ce
Merge branch 'ddl' into hbase
scwf 824f953
adding HBaseRelation2
scwf 647c2bc
use ListBuffer in string2KV
987095a
refactory to use ListBuffer in string2KV
a78d573
recover the files from 'git push --force'
daa49aa
Revert "recover the files from 'git push --force'"
565ffb9
predicate pushdown: partial reduction, filter predicate classificatio…
069314a
predicate pushdown: partial reduction, filter predicate classificatio…
3c86a48
should have used partition-specific filter predicate
d46a517
temporarily revert a predicate pushdown related change
f96ebbc
fix the potential errors in the codes
a1f4acb
construct filter list (initial implementation)
65372d7
Critical Point Impl
dc9b574
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into hbase
3107e2f
handle non-key column properly using SingleColumnValueFilter
f79665d
Added CreateTableAndLoadData trait
sboeschhuawei c3b26d5
hot fix for test, not best solution
scwf b224a3a
Added workaround for Rowkey order bug and automated row data types/va…
sboeschhuawei 26cad54
CriticalPointFinder enhancement 1
baa34b3
Merge branch 'hbase' of https://github.com/Huawei-Spark/spark into hbase
fc482ab
Added filter, order-by, and aggregate queries: created QueriesSuiteBa…
sboeschhuawei 1749e9e
adding HBaseScanBuilder
scwf d7b6ae0
fix createRelation
scwf d54d466
adding source test
scwf 33a1bb8
update
scwf ec8e196
delete no use test files
scwf f958b22
adding test case
scwf dfb7309
add license head
scwf f0a2135
fix DecimalType bug
scwf dcf8fb5
some comment
scwf 675de86
merge and fix conflict
scwf 3835e5a
revert change by debug
scwf aa0522d
refactoring code structure
scwf 3d4548e
revert pom
scwf File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,6 +81,7 @@ set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%tools\target\scala-%SCALA_VERSION%\clas | |
| set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\classes | ||
| set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\classes | ||
| set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\classes | ||
| set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\hbase\target\scala-%SCALA_VERSION%\classes | ||
|
|
||
| set SPARK_TEST_CLASSES=%FWDIR%core\target\scala-%SCALA_VERSION%\test-classes | ||
| set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%repl\target\scala-%SCALA_VERSION%\test-classes | ||
|
|
@@ -91,6 +92,7 @@ set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%streaming\target\scala-%SCALA | |
| set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\test-classes | ||
| set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\test-classes | ||
| set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\test-classes | ||
| set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\hbase\target\scala-%SCALA_VERSION%\test-classes | ||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. revert this change |
||
| if "x%SPARK_TESTING%"=="x1" ( | ||
| rem Add test clases to path - note, add SPARK_CLASSES and SPARK_TEST_CLASSES before CLASSPATH | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -59,6 +59,7 @@ if [ -n "$SPARK_PREPEND_CLASSES" ]; then | |
| CLASSPATH="$CLASSPATH:$FWDIR/sql/hive/target/scala-$SPARK_SCALA_VERSION/classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/sql/hive-thriftserver/target/scala-$SPARK_SCALA_VERSION/classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/yarn/stable/target/scala-$SPARK_SCALA_VERSION/classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/sql/hbase/target/scala-$SCALA_VERSION/classes" | ||
| fi | ||
|
|
||
| # Use spark-assembly jar from either RELEASE or assembly directory | ||
|
|
@@ -130,6 +131,7 @@ if [[ $SPARK_TESTING == 1 ]]; then | |
| CLASSPATH="$CLASSPATH:$FWDIR/sql/catalyst/target/scala-$SPARK_SCALA_VERSION/test-classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/sql/core/target/scala-$SPARK_SCALA_VERSION/test-classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/sql/hive/target/scala-$SPARK_SCALA_VERSION/test-classes" | ||
| CLASSPATH="$CLASSPATH:$FWDIR/sql/hbase/target/scala-$SCALA_VERSION/classes" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. revert this. |
||
| fi | ||
|
|
||
| # Add hadoop conf dir if given -- otherwise FileSystem.*, etc fail ! | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| # | ||
| # Shell script for starting the Spark SQL for HBase CLI | ||
|
|
||
| # Enter posix mode for bash | ||
| set -o posix | ||
|
|
||
| CLASS="org.apache.spark.sql.hbase.HBaseSQLCLIDriver" | ||
|
|
||
| # Figure out where Spark is installed | ||
| FWDIR="$(cd "`dirname "$0"`"/..; pwd)" | ||
|
|
||
| function usage { | ||
| echo "Usage: ./bin/hbase-sql [options] [cli option]" | ||
| pattern="usage" | ||
| pattern+="\|Spark assembly has been built with hbase" | ||
| pattern+="\|NOTE: SPARK_PREPEND_CLASSES is set" | ||
| pattern+="\|Spark Command: " | ||
| pattern+="\|--help" | ||
| pattern+="\|=======" | ||
|
|
||
| "$FWDIR"/bin/spark-submit --help 2>&1 | grep -v Usage 1>&2 | ||
| echo | ||
| echo "CLI options:" | ||
| "$FWDIR"/bin/spark-class $CLASS --help 2>&1 | grep -v "$pattern" 1>&2 | ||
| } | ||
|
|
||
| if [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then | ||
| usage | ||
| exit 0 | ||
| fi | ||
|
|
||
| source "$FWDIR"/bin/utils.sh | ||
| SUBMIT_USAGE_FUNCTION=usage | ||
| gatherSparkSubmitOpts "$@" | ||
|
|
||
| exec "$FWDIR"/bin/spark-submit --class $CLASS "${SUBMISSION_OPTS[@]}" spark-internal "${APPLICATION_OPTS[@]}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To remove this.