[SPARK-18097][SQL] Add exception catch to handle corrupted metadata scenario while dropping the table #16083
Conversation
Update from original
Handling the drop table command for meta data corrupted scenario.
|
This patch would be catching the exception happening while the metadatacheck(due to getting corrupted) and continue with a warning log and continue with the drop operation... |
| case _ => | ||
| } | ||
| } catch { | ||
| case e: QueryExecutionException => log.warn(e.toString, e) |
There was a problem hiding this comment.
Hi @gatorsmile, I tested this change against all existing test cases. Also @davies mentioned in the jira itself, we cannot purposefully corrupt the meta data. So I tested and verified, this change will not break any existing test cases. And we are expecting QueryExecutionException when the meta data is get corrupted in some way.
There was a problem hiding this comment.
The existing test cases do not cover this case. I am afraid this might not work as expected. We need to test it.
There was a problem hiding this comment.
@gatorsmile and @davies ; In that case, any suggestions to mock the metadata corrupt scenario?
There was a problem hiding this comment.
I do not know why we can hit QueryExecutionException in getTableMetadata.
There was a problem hiding this comment.
Maybe you can manually modify the metadata stored in the database and check whether it fails as expected.
|
Hi @jayadevanmurali, are you still working on this? It seems inactive for few months. Maybe, it might better be closed for now if you are currently not able to proceed further. |
|
Hi @HyukjinKwon , |
|
Actually, I don't know (or heard of) how the schema is corrupt. I guess any committer should verify this before merging it. How about asking this to dev/user mailing list if you are not sure? If it is expected to be open without the reproducible steps for a long time (like few weeks or months), I guess we maybe should better close this for now and then open it latter. |
|
Can one of the admins verify this patch? |
Closes apache#16819 Closes apache#13467 Closes apache#16083 Closes apache#17135 Closes apache#8785 Closes apache#16278 Closes apache#16997 Closes apache#17073 Closes apache#17220
5 participants
What changes were proposed in this pull request?
Add exception catch to handle corrupted meta data scenario while dropping the table
How was this patch tested?
Unit tests
Please review http://spark.apache.org/contributing.html before opening a pull request.