Skip to content

Conversation

@panbingkun
Copy link
Contributor

@panbingkun panbingkun commented Sep 24, 2022

What changes were proposed in this pull request?

This PR aims to upgrade RocksDB JNI library from 7.5.3 to 7.6.0.

Why are the changes needed?

This version bring performance improvements(related to read) and some bug fix,
The Release Note

image

image

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

@github-actions github-actions bot added the BUILD label Sep 24, 2022
@panbingkun panbingkun changed the title [BUILD] Upgrade rocksdbjni from 7.5.3 to 7.6.0 [SPARK-40548][BUILD] Upgrade rocksdbjni from 7.5.3 to 7.6.0 Sep 24, 2022
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@HyukjinKwon
Copy link
Member

Merged to master.

@LuciferYang
Copy link
Contributor

@panbingkun we should use org.apache.spark.util.kvstore.RocksDBBenchmark to compare the performance of 7.5.3 and 7.6.0

@panbingkun
Copy link
Contributor Author

panbingkun commented Sep 26, 2022

@panbingkun we should use org.apache.spark.util.kvstore.RocksDBBenchmark to compare the performance of 7.5.3 and 7.6.0

Done!

@panbingkun
Copy link
Contributor Author

panbingkun commented Sep 29, 2022

cc @LuciferYang

** 7.5.3 **

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                        	count	mean	min	max	95th
dbClose                                 	4	1.896	1.138	3.488	3.488
dbCreation                              	4	508.155	178.442	1491.927	1491.927
naturalIndexCreateIterator              	1024	0.018	0.002	5.939	0.046
naturalIndexDescendingCreateIterator    	1024	0.018	0.008	1.070	0.046
naturalIndexDescendingIteration         	1024	0.044	0.005	23.978	0.050
naturalIndexIteration                   	1024	0.022	0.006	1.351	0.037
randomDeleteIndexed                     	1024	0.061	0.026	4.449	0.164
randomDeletesNoIndex                    	1024	0.020	0.015	0.526	0.022
randomUpdatesIndexed                    	1024	0.131	0.036	28.985	0.425
randomUpdatesNoIndex                    	1024	0.036	0.026	3.778	0.033
randomWritesIndexed                     	1024	0.197	0.040	79.184	0.424
randomWritesNoIndex                     	1024	0.072	0.030	19.508	0.098
refIndexCreateIterator                  	1024	0.006	0.004	1.002	0.005
refIndexDescendingCreateIterator        	1024	0.008	0.003	0.674	0.003
refIndexDescendingIteration             	1024	0.018	0.006	1.023	0.059
refIndexIteration                       	1024	0.028	0.006	9.666	0.011
sequentialDeleteIndexed                 	1024	0.046	0.024	1.150	0.098
sequentialDeleteNoIndex                 	1024	0.023	0.017	0.428	0.024
sequentialUpdatesIndexed                	1024	0.065	0.031	3.281	0.086
sequentialUpdatesNoIndex                	1024	0.048	0.028	3.574	0.041
sequentialWritesIndexed                 	1024	0.089	0.034	30.183	0.081
sequentialWritesNoIndex                 	1024	0.084	0.030	22.702	0.076

** 7.6.0 **

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                        	count	mean	min	max	95th
dbClose                                 	4	1.607	0.813	3.222	3.222
dbCreation                              	4	529.812	173.540	1541.769	1541.769
naturalIndexCreateIterator              	1024	0.018	0.002	5.584	0.035
naturalIndexDescendingCreateIterator    	1024	0.015	0.010	1.438	0.037
naturalIndexDescendingIteration         	1024	0.035	0.004	5.964	0.119
naturalIndexIteration                   	1024	0.020	0.005	1.415	0.023
randomDeleteIndexed                     	1024	0.065	0.026	4.339	0.177
randomDeletesNoIndex                    	1024	0.074	0.018	5.307	0.173
randomUpdatesIndexed                    	1024	0.132	0.035	27.710	0.382
randomUpdatesNoIndex                    	1024	0.103	0.028	11.059	0.270
randomWritesIndexed                     	1024	0.192	0.042	79.420	0.398
randomWritesNoIndex                     	1024	0.091	0.030	11.488	0.210
refIndexCreateIterator                  	1024	0.007	0.006	0.783	0.008
refIndexDescendingCreateIterator        	1024	0.005	0.003	0.809	0.004
refIndexDescendingIteration             	1024	0.018	0.006	0.495	0.065
refIndexIteration                       	1024	0.033	0.006	9.748	0.082
sequentialDeleteIndexed                 	1024	0.104	0.026	15.134	0.253
sequentialDeleteNoIndex                 	1024	0.032	0.017	1.521	0.079
sequentialUpdatesIndexed                	1024	0.066	0.033	3.390	0.142
sequentialUpdatesNoIndex                	1024	0.126	0.030	38.740	0.269
sequentialWritesIndexed                 	1024	0.088	0.034	30.329	0.119
sequentialWritesNoIndex                 	1024	0.072	0.031	17.704	0.072

@LuciferYang
Copy link
Contributor

sequentialDeleteIndexed                 	1024	0.046	0.024	1.150	0.098
sequentialDeleteNoIndex                 	1024	0.023	0.017	0.428	0.024
sequentialUpdatesNoIndex                	1024	0.048	0.028	3.574	0.041
sequentialDeleteIndexed                 	1024	0.104	0.026	15.134	0.253
sequentialDeleteNoIndex                 	1024	0.032	0.017	1.521	0.079
sequentialUpdatesNoIndex                	1024	0.126	0.030	38.740	0.269

hmm... seems the long tail fluctuates greatly, let me check it too

@LuciferYang
Copy link
Contributor

My test results as follow:
7.5.3

                                        	count	mean	min	max	95th
dbClose                                 	4	0.351	0.275	0.523	0.523
dbCreation                              	4	77.053	3.475	297.513	297.513
naturalIndexCreateIterator              	1024	0.005	0.002	1.522	0.007
naturalIndexDescendingCreateIterator    	1024	0.005	0.004	0.069	0.007
naturalIndexDescendingIteration         	1024	0.006	0.004	0.285	0.009
naturalIndexIteration                   	1024	0.006	0.004	0.055	0.009
randomDeleteIndexed                     	1024	0.026	0.020	0.302	0.035
randomDeletesNoIndex                    	1024	0.015	0.013	0.037	0.017
randomUpdatesIndexed                    	1024	0.081	0.033	31.324	0.083
randomUpdatesNoIndex                    	1024	0.034	0.031	0.585	0.038
randomWritesIndexed                     	1024	0.120	0.035	51.894	0.122
randomWritesNoIndex                     	1024	0.039	0.035	1.561	0.044
refIndexCreateIterator                  	1024	0.004	0.003	0.018	0.005
refIndexDescendingCreateIterator        	1024	0.003	0.002	0.026	0.004
refIndexDescendingIteration             	1024	0.006	0.005	0.049	0.008
refIndexIteration                       	1024	0.007	0.005	0.096	0.011
sequentialDeleteIndexed                 	1024	0.021	0.017	0.102	0.025
sequentialDeleteNoIndex                 	1024	0.015	0.012	0.042	0.018
sequentialUpdatesIndexed                	1024	0.043	0.037	0.782	0.049
sequentialUpdatesNoIndex                	1024	0.039	0.031	0.735	0.047
sequentialWritesIndexed                 	1024	0.048	0.042	1.875	0.054
sequentialWritesNoIndex                 	1024	0.039	0.032	2.284	0.041

7.6.0

                                        	count	mean	min	max	95th
dbClose                                 	4	0.366	0.282	0.559	0.559
dbCreation                              	4	79.553	3.415	308.188	308.188
naturalIndexCreateIterator              	1024	0.005	0.002	1.515	0.007
naturalIndexDescendingCreateIterator    	1024	0.006	0.005	0.067	0.007
naturalIndexDescendingIteration         	1024	0.006	0.004	0.254	0.009
naturalIndexIteration                   	1024	0.006	0.004	0.059	0.009
randomDeleteIndexed                     	1024	0.027	0.020	0.250	0.037
randomDeletesNoIndex                    	1024	0.015	0.012	0.043	0.018
randomUpdatesIndexed                    	1024	0.083	0.033	31.733	0.086
randomUpdatesNoIndex                    	1024	0.035	0.032	0.626	0.039
randomWritesIndexed                     	1024	0.123	0.035	54.316	0.124
randomWritesNoIndex                     	1024	0.040	0.034	1.556	0.045
refIndexCreateIterator                  	1024	0.004	0.004	0.027	0.006
refIndexDescendingCreateIterator        	1024	0.003	0.002	0.031	0.004
refIndexDescendingIteration             	1024	0.007	0.005	0.044	0.009
refIndexIteration                       	1024	0.008	0.005	0.059	0.010
sequentialDeleteIndexed                 	1024	0.031	0.017	3.271	0.039
sequentialDeleteNoIndex                 	1024	0.016	0.013	0.048	0.019
sequentialUpdatesIndexed                	1024	0.044	0.038	0.879	0.052
sequentialUpdatesNoIndex                	1024	0.040	0.031	0.737	0.048
sequentialWritesIndexed                 	1024	0.048	0.039	2.076	0.055
sequentialWritesNoIndex                 	1024	0.039	0.033	2.929	0.042

@panbingkun panbingkun deleted the upgrade_rocksdbjni branch November 7, 2022 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants