|
| 1 | +--- |
| 2 | +title: "Release Notes - Flink 1.13" |
| 3 | +--- |
| 4 | +<!-- |
| 5 | +Licensed to the Apache Software Foundation (ASF) under one |
| 6 | +or more contributor license agreements. See the NOTICE file |
| 7 | +distributed with this work for additional information |
| 8 | +regarding copyright ownership. The ASF licenses this file |
| 9 | +to you under the Apache License, Version 2.0 (the |
| 10 | +"License"); you may not use this file except in compliance |
| 11 | +with the License. You may obtain a copy of the License at |
| 12 | +
|
| 13 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 14 | +
|
| 15 | +Unless required by applicable law or agreed to in writing, |
| 16 | +software distributed under the License is distributed on an |
| 17 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 18 | +KIND, either express or implied. See the License for the |
| 19 | +specific language governing permissions and limitations |
| 20 | +under the License. |
| 21 | +--> |
| 22 | + |
| 23 | +# Release Notes - Flink 1.13 |
| 24 | + |
| 25 | +These release notes discuss important aspects, such as configuration, behavior, or dependencies, |
| 26 | +that changed between Flink 1.12 and Flink 1.13. Please read these notes carefully if you are |
| 27 | +planning to upgrade your Flink version to 1.13. |
| 28 | + |
| 29 | +### Failover |
| 30 | + |
| 31 | +#### Remove state.backend.async option. |
| 32 | + |
| 33 | +##### [FLINK-21935](https://issues.apache.org/jira/browse/FLINK-21935) |
| 34 | + |
| 35 | +The `state.backend.async` option is deprecated. Snapshots are always asynchronous now (as they were |
| 36 | +by default before) and there is no option to configure a synchronous snapshot any more. |
| 37 | + |
| 38 | +The constructors of `FsStateBackend` and `MemoryStateBackend` that take a flag for sync/async |
| 39 | +snapshots are kept for API compatibility, but the flags are ignored now. |
| 40 | + |
| 41 | +#### Unify binary format for Keyed State savepoints |
| 42 | + |
| 43 | +##### [FLINK-20976](https://issues.apache.org/jira/browse/FLINK-20976) |
| 44 | + |
| 45 | +Flink’s savepoint binary format is unified across all state backends. That means you can take a |
| 46 | +savepoint with one state backend and then restore it using another. |
| 47 | + |
| 48 | +If you want to switch the state backend you should first upgrade your Flink version to 1.13, then |
| 49 | +take a savepoint with the new version, and only after that, you can restore it with a different |
| 50 | +state backend. |
| 51 | + |
| 52 | +#### FailureRateRestartBackoffTimeStrategy allows one less restart than configured |
| 53 | + |
| 54 | +##### [FLINK-20752](https://issues.apache.org/jira/browse/FLINK-20752) |
| 55 | + |
| 56 | +The Failure Rate Restart Strategy was allowing 1 less restart per interval than configured. Users |
| 57 | +wishing to keep the current behavior should reduce the maximum number of allowed failures per |
| 58 | +interval by 1. |
| 59 | + |
| 60 | +#### Support rescaling for Unaligned Checkpoints |
| 61 | + |
| 62 | +##### [FLINK-17979](https://issues.apache.org/jira/browse/FLINK-17979) |
| 63 | + |
| 64 | +While recovering from unaligned checkpoints, users can now change the parallelism of the job. This |
| 65 | +change allows users to quickly upscale the job under backpressure. |
| 66 | + |
| 67 | +### SQL |
| 68 | + |
| 69 | +#### Officially deprecate the legacy planner |
| 70 | + |
| 71 | +##### [FLINK-21709](https://issues.apache.org/jira/browse/FLINK-21709) |
| 72 | + |
| 73 | +The old planner of the Table & SQL API is deprecated and will be dropped in Flink 1.14. This means |
| 74 | +that both the BatchTableEnvironment and DataSet API interop are reaching end of life. Use the |
| 75 | +unified TableEnvironment for batch and stream processing with the new planner, or the DataStream API |
| 76 | +in batch execution mode. |
| 77 | + |
| 78 | +#### Use TIMESTAMP_LTZ as return type for function PROCTIME() |
| 79 | + |
| 80 | +##### [FLINK-21714](https://issues.apache.org/jira/browse/FLINK-21714) |
| 81 | + |
| 82 | +Before Flink 1.13, the function return type of `PROCTIME()` is `TIMESTAMP`, and the return value is |
| 83 | +the `TIMESTAMP` in UTC time zone, e.g. the wall-clock shows `2021-03-01 12:00:00` at Shanghai, |
| 84 | +however the `PROCTIME()` displays `2021-03-01 04:00:00` which is wrong. Flink 1.13 fixes this issue |
| 85 | +and uses `TIMESTAMP_LTZ` type as return type of `PROCTIME()`, users don't need to deal time zone |
| 86 | +problems anymore. |
| 87 | + |
| 88 | +#### Support defining event time attribute on TIMESTAMP_LTZ column |
| 89 | + |
| 90 | +##### [FLINK-20387](https://issues.apache.org/jira/browse/FLINK-20387) |
| 91 | + |
| 92 | +Support defining event time attribute on TIMESTAMP_LTZ column, base on this, Flink SQL gracefully |
| 93 | +support the Daylight Saving Time. |
| 94 | + |
| 95 | +#### Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/LOCALTIME/LOCALTIMESTAMP/NOW() |
| 96 | + |
| 97 | +##### [FLINK-21713](https://issues.apache.org/jira/browse/FLINK-21713) |
| 98 | + |
| 99 | +The value of time function CURRENT_TIMESTAMP and NOW() are corrected from UTC time with `TIMESTAMP` |
| 100 | +type to epoch time with `TIMESTAMP_LTZ` type. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, |
| 101 | +CURRENT_TIME, CURRENT_TIMESTAMP and NOW() are corrected from evaluates for per record in batch mode |
| 102 | +to evaluate once at query-start for batch job. |
| 103 | + |
| 104 | +#### Disable problematic cast conversion between NUMERIC type and TIMESTAMP type |
| 105 | + |
| 106 | +##### [FLINK-21698](https://issues.apache.org/jira/browse/FLINK-21698) |
| 107 | + |
| 108 | +The CAST operation between `NUMERIC` type and `TIMESTAMP` type is problematic and is disabled now, |
| 109 | +e.g. |
| 110 | +`CAST(numeric AS TIMESTAMP(3))` is disabled and should use `TO_TIMESTAMP(FROM_UNIXTIME(numeric))` |
| 111 | +instead. |
| 112 | + |
| 113 | +#### Support USE MODULES syntax |
| 114 | + |
| 115 | +##### [FLINK-21298](https://issues.apache.org/jira/browse/FLINK-21298) |
| 116 | + |
| 117 | +The term MODULES is a reserved keyword now. Use backticks to escape column names and other |
| 118 | +identifiers with this name. |
| 119 | + |
| 120 | +#### Update TableResult.collect()/TableResult.print() to the new type system |
| 121 | + |
| 122 | +##### [FLINK-20613](https://issues.apache.org/jira/browse/FLINK-20613) |
| 123 | + |
| 124 | +`Table.execute().collect()` might return slightly different results for column types and row kind. |
| 125 | +The most important differences include: |
| 126 | + |
| 127 | +* Structured types are represented as POJOs of the original class and not Row anymore. |
| 128 | +* Raw types are serialized according to the configuration in TableConfig. |
| 129 | + |
| 130 | + |
| 131 | +#### Update the Row.toString method |
| 132 | + |
| 133 | +##### [FLINK-18090](https://issues.apache.org/jira/browse/FLINK-18090) |
| 134 | + |
| 135 | +The `Row.toSting()` method has been reworked. This is an incompatible change. If the legacy |
| 136 | +representation is still required for tests, the old behavior can be restored via the flag |
| 137 | +`RowUtils.USE_LEGACY_TO_STRING` for the local JVM. However, relying on the row's string |
| 138 | +representation for tests is not a good idea in general as field data types are not verified. |
| 139 | + |
| 140 | +#### Support start SQL Client with an initialization SQL file |
| 141 | + |
| 142 | +##### [FLINK-20320](https://issues.apache.org/jira/browse/FLINK-20320) |
| 143 | + |
| 144 | +The `sql-client-defaults.yaml` YAML file is deprecated and not provided in the release package. To |
| 145 | +be compatible, it's still supported to initialize the SQL Client with the YAML file if manually |
| 146 | +provided. But it's recommend to use the new introduced `-i` startup option to execute an |
| 147 | +initialization SQL file to setup the SQL Client session. The so-called initialization SQL file can |
| 148 | +use Flink DDLs to define available catalogs, table sources and sinks, user-defined functions, and |
| 149 | +other properties required for execution and deployment. The support of legacy SQL Client YAML file |
| 150 | +will be totally dropped in Flink 1.14. |
| 151 | + |
| 152 | +#### Hive dialect no longer supports Flink syntax for DML and DQL |
| 153 | + |
| 154 | +##### [FLINK-21808](https://issues.apache.org/jira/browse/FLINK-21808) |
| 155 | + |
| 156 | +Hive dialect supports HiveQL for DML and DQL. Please switch to default dialect in order to write in |
| 157 | +Flink syntax. |
| 158 | + |
| 159 | +### Runtime |
| 160 | + |
| 161 | +#### BoundedOneInput.endInput is called when taking synchronous savepoint |
| 162 | + |
| 163 | +##### [FLINK-21132](https://issues.apache.org/jira/browse/FLINK-21132) |
| 164 | + |
| 165 | +`endInput()` is not called anymore (on BoundedOneInput and BoundedMultiInput) when the job is |
| 166 | +stopping with savepoint. |
| 167 | + |
| 168 | +#### Remove JobManagerOptions.SCHEDULING_STRATEGY |
| 169 | + |
| 170 | +##### [FLINK-20591](https://issues.apache.org/jira/browse/FLINK-20591) |
| 171 | + |
| 172 | +The configuration parameter `jobmanager.scheduler.scheduling-strategy` has been removed, because |
| 173 | +the `legacy` scheduler has been removed from Flink 1.13.0. |
| 174 | + |
| 175 | +#### Warn user if System.exit() is called in user code |
| 176 | + |
| 177 | +##### [FLINK-15156](https://issues.apache.org/jira/browse/FLINK-15156) |
| 178 | + |
| 179 | +A new configuration value `cluster.intercept-user-system-exit` allows to log a warning, or throw an |
| 180 | +exception if user code calls `System.exit()`. |
| 181 | + |
| 182 | +This feature is not covering all locations in Flink where user code is executed. It just adds the |
| 183 | +infrastructure for such an interception. We are tracking this improvement in |
| 184 | +[FLINK-21307](https://issues.apache.org/jira/browse/FLINK-21307). |
| 185 | + |
| 186 | +### Docker |
| 187 | + |
| 188 | +#### Consider removing automatic configuration fo number of slots from docker |
| 189 | + |
| 190 | +##### [FLINK-21036](https://issues.apache.org/jira/browse/FLINK-21036) |
| 191 | + |
| 192 | +The docker images no longer set the default number of taskmanager slots to the number of CPU cores. |
| 193 | +This behavior was inconsistent with all other deployment methods and ignored any limits on the CPU |
| 194 | +usage set via docker. |
| 195 | + |
| 196 | +#### Rework jemalloc switch to use an environment variable |
| 197 | + |
| 198 | +##### [FLINK-21034](https://issues.apache.org/jira/browse/FLINK-21034) |
| 199 | + |
| 200 | +The docker switch for disabling the jemalloc memory allocator has been reworked from a script |
| 201 | +argument to an environment variable called DISABLE_JEMALLOC. If set to "true" jemalloc will not be |
| 202 | +enabled. |
| 203 | + |
| 204 | +### Connectors |
| 205 | + |
| 206 | +#### Remove swift FS filesystem |
| 207 | + |
| 208 | +##### [FLINK-21819](https://issues.apache.org/jira/browse/FLINK-21819) |
| 209 | + |
| 210 | +The Swift filesystem is no longer being actively developed and has been removed from the project and |
| 211 | +distribution. |
| 212 | + |
| 213 | +##### [FLINK-22133](https://issues.apache.org/jira/browse/FLINK-22133) |
| 214 | +The unified source API for connectors has a minor breaking change. The `SplitEnumerator.snapshotState()` |
| 215 | +method was adjusted to accept the *Checkpoint ID* of the checkpoint for which the snapshot is created. |
| 216 | + |
| 217 | +### Monitoring & debugging |
| 218 | + |
| 219 | +#### Introduce latency tracking state |
| 220 | + |
| 221 | +##### [FLINK-21736](https://issues.apache.org/jira/browse/FLINK-21736) |
| 222 | + |
| 223 | +State access latency metrics are introduced to track all kinds of keyed state access to help debug |
| 224 | +state performance. This feature is not enabled by default and can be turned on by |
| 225 | +setting `state.backend.latency-track.keyed-state-enabled` to true. |
| 226 | + |
| 227 | +#### Support for CPU flame graphs in web UI |
| 228 | + |
| 229 | +##### [FLINK-13550](https://issues.apache.org/jira/browse/FLINK-13550) |
| 230 | + |
| 231 | +Flink now offers flame graphs for each node in the job graph. Please enable this experimental feature |
| 232 | +by setting the respective configuration flag `rest.flamegraph.enabled`. |
| 233 | + |
| 234 | +#### Display last n exceptions/causes for job restarts in Web UI |
| 235 | + |
| 236 | +##### [FLINK-6042](https://issues.apache.org/jira/browse/FLINK-6042) |
| 237 | + |
| 238 | +Flink exposes the exception history now through the REST API and the UI. The amount of most-recently |
| 239 | +handled exceptions that shall be tracked can be defined through `web.exception-history-size`. |
| 240 | +Some values of the exception history's REST API Json response are deprecated as part of this effort. |
| 241 | + |
| 242 | +#### Create backPressuredTimeMsPerSecond metric |
| 243 | + |
| 244 | +##### [FLINK-20717](https://issues.apache.org/jira/browse/FLINK-20717) |
| 245 | + |
| 246 | +Previously `idleTimeMsPerSecond` was defined as the time task spent waiting for either the input or |
| 247 | +the back pressure. Now `idleTimeMsPerSecond` excludes back pressured time, so if the task is back |
| 248 | +pressured it is not idle. The back pressured time is now measured separately |
| 249 | +as `backPressuredTimeMsPerSecond`. |
| 250 | + |
| 251 | +#### Enable log4j2 monitor interval by default |
| 252 | + |
| 253 | +##### [FLINK-20510](https://issues.apache.org/jira/browse/FLINK-20510) |
| 254 | + |
| 255 | +The Log4j support for updating the Log4j configuration at runtime has been enabled by default. The |
| 256 | +configuration files are checked for changes every 30 seconds. |
| 257 | + |
| 258 | +#### ZooKeeper quorum fails to start due to missing log4j library |
| 259 | + |
| 260 | +##### [FLINK-20404](https://issues.apache.org/jira/browse/FLINK-20404) |
| 261 | + |
| 262 | +The Zookeeper scripts in the Flink distribution have been modified to disable the Log4j JMX |
| 263 | +integration due to an incompatibility between Zookeeper 3.4 and Log4j 2. To re-enable this feature, |
| 264 | +remove the line in the `zookeeper.sh` file that sets `zookeeper.jmx.log4j.disable`. |
| 265 | + |
| 266 | +#### Expose stage of task initialization |
| 267 | + |
| 268 | +##### [FLINK-17012](https://issues.apache.org/jira/browse/FLINK-17012) |
| 269 | + |
| 270 | +Task's RUNNING state was split into two states: INITIALIZING and RUNNING. Task is INITIALIZING while |
| 271 | +state is initialising and in case of unaligned checkpoints, until all the in-flight data has been |
| 272 | +recovered. |
| 273 | + |
| 274 | +### Deployment |
| 275 | + |
| 276 | +#### Officially deprecate Mesos support |
| 277 | + |
| 278 | +##### [FLINK-22352](https://issues.apache.org/jira/browse/FLINK-22352) |
| 279 | + |
| 280 | +The community decided to deprecate the Apache Mesos support for Apache Flink. It is subject to |
| 281 | +removal in the future. Users are encouraged to switch to a different resource manager. |
| 282 | + |
0 commit comments