Skip to content

Do not persist data to memory by default when creating tables #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 12, 2021
Merged

Do not persist data to memory by default when creating tables #245

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
efa492f
Add GPU tests, pytest `gpu` marker (#1)
charlesbluca Sep 24, 2021
bdd849d
Do not persist data to memory by default when creating tables
jdye64 Sep 24, 2021
64ff1a8
Revert "Add GPU tests, pytest `gpu` marker (#1)"
charlesbluca Oct 12, 2021
475ee7d
Merge remote-tracking branch 'upstream/main' into default_persist_false
charlesbluca Oct 12, 2021
8196ca3
update persist doc
jdye64 Oct 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions dask_sql/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def create_table(
table_name: str,
input_table: InputType,
format: str = None,
persist: bool = True,
persist: bool = False,
schema_name: str = None,
gpu: bool = False,
**kwargs,
Expand All @@ -144,8 +144,9 @@ def create_table(
Typical file formats are csv or parquet.
Any additional parameters will get passed on to the read method.
Please note that some file formats require additional libraries.
By default, the data will be loaded directly into the memory
of the nodes. If you do not want that, set persist to False.
By default, the data will be lazily loaded. If you would like to
load the data directly into memory you can do so by setting
persist=True.

See :ref:`data_input` for more information.

Expand Down Expand Up @@ -185,7 +186,7 @@ def create_table(
Specify the file format directly here if it can not be deduced from the extension.
If set to "memory", load the data from a published dataset in the dask cluster.
persist (:obj:`bool`): Only used when passing a string into the ``input`` parameter.
Set to false to turn off loading the file data directly into memory.
Set to true to turn on loading the file data directly into memory.
**kwargs: Additional arguments for specific formats. See :ref:`data_input` for more information.

"""
Expand Down