Consider persist=False as default when creating table from DataFrame

[randerzander]

By default, when creating a table from a Dask Dataframe, the create_table function persists the Dask DataFrame in memory.

This was surprising (partly because I didn't see it in the doc examples) when working on a large ETL job. I suspect for most users (especially those running larger jobs on systems where data is larger than memory) this will be a surprising realization. In my case, I spent time trying to debug which compute tasks might be using more memory than expected only to eventually realize it was just the persist call which failed. Once I re-created the tables w/ persist=False, my workflow completed successfully.

Is there a rationale for persisting by default?

[nils-braun]

I am sorry to hear that this caused you problems! I do definitely see your reasons, let me give you my rationale, when I created this default:

• I assumed users will create a table once and then maybe do multiple queries on it. Persisting the content will speed up the process.
• data will be loaded at the moment they call create_table. If it changes later, they won't notice (which I assumed seemed reasonable).

I do not have any problem with changing the default if my assumptions are wrong here. But maybe it would also be ok to mention it better in the documentation? @rajagurunath, what do you think?

[rajagurunath]

Sorry for being late here, persist =True|False both have equal weights in my opinion, and agreed with both the points mentioned above.

• ETL - needs to create a table eagerly (on demand) and pull the latest data every time instead of persisting the table in cluster RAM
• Dashboard| BI tools - may benefit from using already created /cached tables to retrieve the data quickly.

One more thing I can think of, but not sure if it is really useful or not, can we allow the following set statements for setting the config globally

set create.table.presist = true;

As of now, making persist= False similar to frameworks like a spark and load the data whenever needed eagerly by specifying configs like persist = True in ETL pipeline, seems like a better solution and as mentioned by @nils-braun we will document this change.

Any other ideas/suggestions @nils-braun @randerzander?

[randerzander]

A last thought in support of making persist=False by default is Dask DataFrame also does not persist unless explicitly asked by the user. Since Dask SQL users are likely Dask users, persisting only when specified would be most consistent.

[nils-braun]

That @randerzander convinces me most :-)
Ok, I am now also in favour of changing the default. (if @rajagurunath is not against it). Would you like to do it @randerzander or shall we proceed?