Skip to content

Enable PCRE UTF-8 validity string checks #26731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 18, 2018
Merged

Enable PCRE UTF-8 validity string checks #26731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions base/regex.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

include("pcre.jl")

const DEFAULT_COMPILER_OPTS = PCRE.UTF | PCRE.NO_UTF_CHECK | PCRE.ALT_BSUX
const DEFAULT_MATCH_OPTS = PCRE.NO_UTF_CHECK
const DEFAULT_COMPILER_OPTS = PCRE.UTF | PCRE.ALT_BSUX
const DEFAULT_MATCH_OPTS = zero(UInt32)

mutable struct Regex
pattern::String
Expand Down
16 changes: 16 additions & 0 deletions test/regex.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,19 @@ end

# Proper unicode handling
@test match(r"∀∀", "∀x∀∀∀").match == "∀∀"

@test_throws ErrorException match(r"a", "\xe2\x88") # 1 byte missing at end
@test_throws ErrorException match(r"a", "\xe2\x08\x80") # byte 2 top bits not 0x80
@test_throws ErrorException match(r"a", "\xf8\x89\x89\x80\x80") # 5-byte character is not allowed (RFC 3629)
@test_throws ErrorException match(r"a", "\xf4\x9f\xbf\xbf") # code points greater than 0x10ffff are not defined
@test_throws ErrorException match(r"a", "\Udfff") # code points 0xd800-0xdfff are not defined
@test_throws ErrorException match(r"a", "\xc0\x80") # overlong 2-byte sequence
@test_throws ErrorException match(r"a", "\xff") # illegal byte (0xfe or 0xff)

@test_throws ErrorException Regex("\xe2\x88") # 1 byte missing at end
@test_throws ErrorException Regex("\xe2\x08\x80") # byte 2 top bits not 0x80
@test_throws ErrorException Regex("\xf8\x89\x89\x80\x80") # 5-byte character is not allowed (RFC 3629)
@test_throws ErrorException Regex("\xf4\x9f\xbf\xbf") # code points greater than 0x10ffff are not defined
@test_throws ErrorException Regex("\Udfff") # code points 0xd800-0xdfff are not defined
@test_throws ErrorException Regex("\xc0\x80") # overlong 2-byte sequence
@test_throws ErrorException Regex("\xff") # illegal byte (0xfe or 0xff)