Skip to content

PHP 8.0 | Support union types in PHP Tokenizer and File class utility methods #3032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 2, 2020
Merged

PHP 8.0 | Support union types in PHP Tokenizer and File class utility methods #3032

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a0145ae
PHP 8.0 | Tokens: add new `T_TYPE_UNION` token
jrfnl Jul 15, 2020
63c2b22
PHP 8.0 | Tokenizer/PHP: tokenize the "|" for union types as T_TYPE_U…
jrfnl Jul 15, 2020
8112f56
PHP 8.0 | Tokenizer/PHP: array return type keyword to T_STRING vs PHP…
jrfnl Jul 7, 2020
dc6fe75
PHP 8.0 | Tokenizer/PHP: arrow function backfill vs PHP8 union types
jrfnl Jul 15, 2020
564b780
PHP 8.0 | File::getMethodParameters(): add support for "union" parame…
jrfnl Jul 5, 2020
38c01b3
PHP 8.0 | File::getMethodProperties(): add support for "union" return…
jrfnl Jul 5, 2020
b552768
PHP 8.0 | File::getMemberProperties(): add support for "union" types
jrfnl Jul 5, 2020
9a502fd
Docs: update "nullable_type" comments to clarify meaning
jrfnl Jul 6, 2020
53aa409
Tokenizer/PHP: add new token to the $knownLengths property
jrfnl Jul 26, 2020
25a2e44
Tests: update the Tokenizer/UndoNamespacedNameSingleToken test
jrfnl Sep 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions package.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,8 @@ http://pear.php.net/dtd/package-2.0.xsd">
<file baseinstalldir="" name="BackfillFnTokenTest.php" role="test" />
<file baseinstalldir="" name="BackfillNumericSeparatorTest.inc" role="test" />
<file baseinstalldir="" name="BackfillNumericSeparatorTest.php" role="test" />
<file baseinstalldir="" name="BitwiseOrTest.inc" role="test" />
<file baseinstalldir="" name="BitwiseOrTest.php" role="test" />
<file baseinstalldir="" name="NullsafeObjectOperatorTest.inc" role="test" />
<file baseinstalldir="" name="NullsafeObjectOperatorTest.php" role="test" />
<file baseinstalldir="" name="ScopeSettingWithNamespaceOperatorTest.inc" role="test" />
Expand Down Expand Up @@ -2006,6 +2008,8 @@ http://pear.php.net/dtd/package-2.0.xsd">
<install as="CodeSniffer/Core/Tokenizer/BackfillFnTokenTest.inc" name="tests/Core/Tokenizer/BackfillFnTokenTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/BackfillNumericSeparatorTest.php" name="tests/Core/Tokenizer/BackfillNumericSeparatorTest.php" />
<install as="CodeSniffer/Core/Tokenizer/BackfillNumericSeparatorTest.inc" name="tests/Core/Tokenizer/BackfillNumericSeparatorTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/BitwiseOrTest.php" name="tests/Core/Tokenizer/BitwiseOrTest.php" />
<install as="CodeSniffer/Core/Tokenizer/BitwiseOrTest.inc" name="tests/Core/Tokenizer/BitwiseOrTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/NullsafeObjectOperatorTest.php" name="tests/Core/Tokenizer/NullsafeObjectOperatorTest.php" />
<install as="CodeSniffer/Core/Tokenizer/NullsafeObjectOperatorTest.inc" name="tests/Core/Tokenizer/NullsafeObjectOperatorTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/ScopeSettingWithNamespaceOperatorTest.php" name="tests/Core/Tokenizer/ScopeSettingWithNamespaceOperatorTest.php" />
Expand Down Expand Up @@ -2071,6 +2075,8 @@ http://pear.php.net/dtd/package-2.0.xsd">
<install as="CodeSniffer/Core/Tokenizer/BackfillFnTokenTest.inc" name="tests/Core/Tokenizer/BackfillFnTokenTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/BackfillNumericSeparatorTest.php" name="tests/Core/Tokenizer/BackfillNumericSeparatorTest.php" />
<install as="CodeSniffer/Core/Tokenizer/BackfillNumericSeparatorTest.inc" name="tests/Core/Tokenizer/BackfillNumericSeparatorTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/BitwiseOrTest.php" name="tests/Core/Tokenizer/BitwiseOrTest.php" />
<install as="CodeSniffer/Core/Tokenizer/BitwiseOrTest.inc" name="tests/Core/Tokenizer/BitwiseOrTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/NullsafeObjectOperatorTest.php" name="tests/Core/Tokenizer/NullsafeObjectOperatorTest.php" />
<install as="CodeSniffer/Core/Tokenizer/NullsafeObjectOperatorTest.inc" name="tests/Core/Tokenizer/NullsafeObjectOperatorTest.inc" />
<install as="CodeSniffer/Core/Tokenizer/ScopeSettingWithNamespaceOperatorTest.php" name="tests/Core/Tokenizer/ScopeSettingWithNamespaceOperatorTest.php" />
Expand Down
18 changes: 15 additions & 3 deletions src/Files/File.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1288,7 +1288,8 @@ public function getDeclarationName($stackPtr)
* // or FALSE if there is no type hint.
* 'type_hint_end_token' => integer, // The stack pointer to the end of the type hint
* // or FALSE if there is no type hint.
* 'nullable_type' => boolean, // TRUE if the var type is nullable.
* 'nullable_type' => boolean, // TRUE if the type is preceded by the nullability
* // operator.
* 'comma_token' => integer, // The stack pointer to the comma after the param
* // or FALSE if this is the last param.
* )
Expand Down Expand Up @@ -1443,6 +1444,9 @@ public function getMethodParameters($stackPtr)
break;
case T_NAMESPACE:
case T_NS_SEPARATOR:
case T_TYPE_UNION:
case T_FALSE:
case T_NULL:
// Part of a type hint or default value.
if ($defaultStart === null) {
if ($typeHintToken === false) {
Expand Down Expand Up @@ -1533,7 +1537,8 @@ public function getMethodParameters($stackPtr)
* 'return_type' => '', // The return type of the method.
* 'return_type_token' => integer, // The stack pointer to the start of the return type
* // or FALSE if there is no return type.
* 'nullable_return_type' => false, // TRUE if the return type is nullable.
* 'nullable_return_type' => false, // TRUE if the return type is preceded by the
* // nullability operator.
* 'is_abstract' => false, // TRUE if the abstract keyword was found.
* 'is_final' => false, // TRUE if the final keyword was found.
* 'is_static' => false, // TRUE if the static keyword was found.
Expand Down Expand Up @@ -1631,8 +1636,11 @@ public function getMethodProperties($stackPtr)
T_SELF => T_SELF,
T_PARENT => T_PARENT,
T_STATIC => T_STATIC,
T_FALSE => T_FALSE,
T_NULL => T_NULL,
T_NAMESPACE => T_NAMESPACE,
T_NS_SEPARATOR => T_NS_SEPARATOR,
T_TYPE_UNION => T_TYPE_UNION,
];

for ($i = $this->tokens[$stackPtr]['parenthesis_closer']; $i < $this->numTokens; $i++) {
Expand Down Expand Up @@ -1700,7 +1708,8 @@ public function getMethodProperties($stackPtr)
* // or FALSE if there is no type.
* 'type_end_token' => integer, // The stack pointer to the end of the type
* // or FALSE if there is no type.
* 'nullable_type' => boolean, // TRUE if the type is nullable.
* 'nullable_type' => boolean, // TRUE if the type is preceded by the nullability
* // operator.
* );
* </code>
*
Expand Down Expand Up @@ -1815,8 +1824,11 @@ public function getMemberProperties($stackPtr)
T_CALLABLE => T_CALLABLE,
T_SELF => T_SELF,
T_PARENT => T_PARENT,
T_FALSE => T_FALSE,
T_NULL => T_NULL,
T_NAMESPACE => T_NAMESPACE,
T_NS_SEPARATOR => T_NS_SEPARATOR,
T_TYPE_UNION => T_TYPE_UNION,
];

for ($i; $i < $stackPtr; $i++) {
Expand Down
182 changes: 179 additions & 3 deletions src/Tokenizers/PHP.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,7 @@ class PHP extends Tokenizer
T_BACKTICK => 1,
T_OPEN_SHORT_ARRAY => 1,
T_CLOSE_SHORT_ARRAY => 1,
T_TYPE_UNION => 1,
];

/**
Expand Down Expand Up @@ -1475,6 +1476,7 @@ function return types. We want to keep the parenthesis map clean,
T_SELF => T_SELF,
T_PARENT => T_PARENT,
T_NAMESPACE => T_NAMESPACE,
T_STATIC => T_STATIC,
T_NS_SEPARATOR => T_NS_SEPARATOR,
];

Expand Down Expand Up @@ -1509,12 +1511,14 @@ function return types. We want to keep the parenthesis map clean,
}//end for

// Any T_ARRAY tokens we find between here and the next
// token that can't be part of the return type need to be
// token that can't be part of the return type, need to be
// converted to T_STRING tokens.
for ($x; $x < $numTokens; $x++) {
if (is_array($tokens[$x]) === false || isset($allowed[$tokens[$x][0]]) === false) {
if ((is_array($tokens[$x]) === false && $tokens[$x] !== '|')
|| (is_array($tokens[$x]) === true && isset($allowed[$tokens[$x][0]]) === false)
) {
break;
} else if ($tokens[$x][0] === T_ARRAY) {
} else if (is_array($tokens[$x]) === true && $tokens[$x][0] === T_ARRAY) {
$tokens[$x][0] = T_STRING;

if (PHP_CODESNIFFER_VERBOSITY > 1) {
Expand Down Expand Up @@ -1996,6 +2000,7 @@ protected function processAdditional()
T_PARENT => T_PARENT,
T_SELF => T_SELF,
T_STATIC => T_STATIC,
T_TYPE_UNION => T_TYPE_UNION,
];

$closer = $this->tokens[$x]['parenthesis_closer'];
Expand Down Expand Up @@ -2176,6 +2181,177 @@ protected function processAdditional()
}
}

continue;
} else if ($this->tokens[$i]['code'] === T_BITWISE_OR) {
/*
Convert "|" to T_TYPE_UNION or leave as T_BITWISE_OR.
*/

$allowed = [
T_STRING => T_STRING,
T_CALLABLE => T_CALLABLE,
T_SELF => T_SELF,
T_PARENT => T_PARENT,
T_STATIC => T_STATIC,
T_FALSE => T_FALSE,
T_NULL => T_NULL,
T_NS_SEPARATOR => T_NS_SEPARATOR,
];

$suspectedType = null;
$typeTokenCount = 0;

for ($x = ($i + 1); $x < $numTokens; $x++) {
if (isset(Util\Tokens::$emptyTokens[$this->tokens[$x]['code']]) === true) {
continue;
}

if (isset($allowed[$this->tokens[$x]['code']]) === true) {
++$typeTokenCount;
continue;
}

if ($typeTokenCount > 0
&& ($this->tokens[$x]['code'] === T_BITWISE_AND
|| $this->tokens[$x]['code'] === T_ELLIPSIS)
) {
// Skip past reference and variadic indicators for parameter types.
++$x;
continue;
}

if ($this->tokens[$x]['code'] === T_VARIABLE) {
// Parameter/Property defaults can not contain variables, so this could be a type.
$suspectedType = 'property or parameter';
break;
}

if ($this->tokens[$x]['code'] === T_DOUBLE_ARROW) {
// Possible arrow function.
$suspectedType = 'return';
break;
}

if ($this->tokens[$x]['code'] === T_SEMICOLON) {
// Possible abstract method or interface method.
$suspectedType = 'return';
break;
}

if ($this->tokens[$x]['code'] === T_OPEN_CURLY_BRACKET
&& isset($this->tokens[$x]['scope_condition']) === true
&& $this->tokens[$this->tokens[$x]['scope_condition']]['code'] === T_FUNCTION
) {
$suspectedType = 'return';
}

break;
}//end for

if ($typeTokenCount === 0 || isset($suspectedType) === false) {
// Definitely not a union type, move on.
continue;
}

$typeTokenCount = 0;
$unionOperators = [$i];
$confirmed = false;

for ($x = ($i - 1); $x >= 0; $x--) {
if (isset(Util\Tokens::$emptyTokens[$this->tokens[$x]['code']]) === true) {
continue;
}

if (isset($allowed[$this->tokens[$x]['code']]) === true) {
++$typeTokenCount;
continue;
}

// Union types can't use the nullable operator, but be tolerant to parse errors.
if ($typeTokenCount > 0 && $this->tokens[$x]['code'] === T_NULLABLE) {
continue;
}

if ($this->tokens[$x]['code'] === T_BITWISE_OR) {
$unionOperators[] = $x;
continue;
}

if ($suspectedType === 'return' && $this->tokens[$x]['code'] === T_COLON) {
$confirmed = true;
break;
}

if ($suspectedType === 'property or parameter'
&& (isset(Util\Tokens::$scopeModifiers[$this->tokens[$x]['code']]) === true
|| $this->tokens[$x]['code'] === T_VAR)
) {
// This will also confirm constructor property promotion parameters, but that's fine.
$confirmed = true;
}

break;
}//end for

if ($confirmed === false
&& $suspectedType === 'property or parameter'
&& isset($this->tokens[$i]['nested_parenthesis']) === true
) {
$parens = $this->tokens[$i]['nested_parenthesis'];
$last = end($parens);

if (isset($this->tokens[$last]['parenthesis_owner']) === true
&& $this->tokens[$this->tokens[$last]['parenthesis_owner']]['code'] === T_FUNCTION
) {
$confirmed = true;
} else {
// No parenthesis owner set, this may be an arrow function which has not yet
// had additional processing done.
if (isset($this->tokens[$last]['parenthesis_opener']) === true) {
for ($x = ($this->tokens[$last]['parenthesis_opener'] - 1); $x >= 0; $x--) {
if (isset(Util\Tokens::$emptyTokens[$this->tokens[$x]['code']]) === true) {
continue;
}

break;
}

if ($this->tokens[$x]['code'] === T_FN) {
for (--$x; $x >= 0; $x--) {
if (isset(Util\Tokens::$emptyTokens[$this->tokens[$x]['code']]) === true
|| $this->tokens[$x]['code'] === T_BITWISE_AND
) {
continue;
}

break;
}

if ($this->tokens[$x]['code'] !== T_FUNCTION) {
$confirmed = true;
}
}
}//end if
}//end if

unset($parens, $last);
}//end if

if ($confirmed === false) {
// Not a union type after all, move on.
continue;
}

foreach ($unionOperators as $x) {
$this->tokens[$x]['code'] = T_TYPE_UNION;
$this->tokens[$x]['type'] = 'T_TYPE_UNION';

if (PHP_CODESNIFFER_VERBOSITY > 1) {
$line = $this->tokens[$x]['line'];
echo "\t* token $x on line $line changed from T_BITWISE_OR to T_TYPE_UNION".PHP_EOL;
}
}

continue;
} else if ($this->tokens[$i]['code'] === T_STATIC) {
for ($x = ($i - 1); $x > 0; $x--) {
Expand Down
1 change: 1 addition & 0 deletions src/Util/Tokens.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@
define('T_ZSR', 'PHPCS_T_ZSR');
define('T_ZSR_EQUAL', 'PHPCS_T_ZSR_EQUAL');
define('T_FN_ARROW', 'T_FN_ARROW');
define('T_TYPE_UNION', 'T_TYPE_UNION');

// Some PHP 5.5 tokens, replicated for lower versions.
if (defined('T_FINALLY') === false) {
Expand Down
47 changes: 47 additions & 0 deletions tests/Core/File/GetMemberPropertiesTest.inc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,50 @@ class NSOperatorInType {
/* testNamespaceOperatorTypeHint */
public ?namespace\Name $prop;
}

$anon = class() {
/* testPHP8UnionTypesSimple */
public int|float $unionTypeSimple;

/* testPHP8UnionTypesTwoClasses */
private MyClassA|\Package\MyClassB $unionTypesTwoClasses;

/* testPHP8UnionTypesAllBaseTypes */
protected array|bool|int|float|NULL|object|string $unionTypesAllBaseTypes;

/* testPHP8UnionTypesAllPseudoTypes */
// Intentional fatal error - mixing types which cannot be combined, but that's not the concern of the method.
var false|mixed|self|parent|iterable|Resource $unionTypesAllPseudoTypes;

/* testPHP8UnionTypesIllegalTypes */
// Intentional fatal error - types which are not allowed for properties, but that's not the concern of the method.
public callable|static|void $unionTypesIllegalTypes;

/* testPHP8UnionTypesNullable */
// Intentional fatal error - nullability is not allowed with union types, but that's not the concern of the method.
public ?int|float $unionTypesNullable;

/* testPHP8PseudoTypeNull */
// Intentional fatal error - null pseudotype is only allowed in union types, but that's not the concern of the method.
public null $pseudoTypeNull;

/* testPHP8PseudoTypeFalse */
// Intentional fatal error - false pseudotype is only allowed in union types, but that's not the concern of the method.
public false $pseudoTypeFalse;

/* testPHP8PseudoTypeFalseAndBool */
// Intentional fatal error - false pseudotype is not allowed in combination with bool, but that's not the concern of the method.
public bool|FALSE $pseudoTypeFalseAndBool;

/* testPHP8ObjectAndClass */
// Intentional fatal error - object is not allowed in combination with class name, but that's not the concern of the method.
public object|ClassName $objectAndClass;

/* testPHP8PseudoTypeIterableAndArray */
// Intentional fatal error - iterable pseudotype is not allowed in combination with array or Traversable, but that's not the concern of the method.
public iterable|array|Traversable $pseudoTypeIterableAndArray;

/* testPHP8DuplicateTypeInUnionWhitespaceAndComment */
// Intentional fatal error - duplicate types are not allowed in union types, but that's not the concern of the method.
public int |string| /*comment*/ INT $duplicateTypeInUnion;
};
Loading