Skip to content

fix: escape column names and handle mismatched data types in D1 SQL dump #9866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 18, 2025

Conversation

invisal
Copy link
Contributor

@invisal invisal commented Jul 7, 2025

Fixes #8377, fixes #6357, fixes #5920
Might also related to #8226

This PR improves the D1 SQL dump logic

  • Properly escape column and table names to prevent syntax errors.
  • Handle SQLite's loose typing by escaping values based on value type instead of enforcing strict column types.
  • Add tests to verify SQL dump output and edge cases.

  • Tests
    • Tests included
    • Tests not necessary because:
  • Public documentation
    • Cloudflare docs PR(s):
    • Documentation not necessary because: It is a bugfix
  • Wrangler V3 Backport
    • Wrangler PR:
    • Not necessary because: not a Wrangler change

@invisal invisal requested a review from a team as a code owner July 7, 2025 05:48
Copy link

changeset-bot bot commented Jul 7, 2025

🦋 Changeset detected

Latest commit: e5c988a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
miniflare Patch
@cloudflare/pages-shared Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-pool-workers Patch
wrangler Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@invisal invisal marked this pull request as draft July 7, 2025 05:48
* @param id - The identifier to escape.
* @returns
*/
function escapeId(id: string) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQLite escapes double quotes in identifiers by doubling them using sqlite3_mprintf("\"%w\"", azCol[i])
https://github.com/sqlite/sqlite/blob/master/src/shell.c.in#L2914

The %w format doubles " as "":
https://github.com/sqlite/sqlite/blob/master/src/printf.c#L34

@emily-shen emily-shen marked this pull request as ready for review July 7, 2025 16:48
@github-project-automation github-project-automation bot moved this to Untriaged in workers-sdk Jul 8, 2025
@emily-shen emily-shen marked this pull request as draft July 8, 2025 10:36
Copy link

pkg-pr-new bot commented Jul 8, 2025

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@9866

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@9866

miniflare

npm i https://pkg.pr.new/miniflare@9866

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@9866

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@9866

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@9866

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@9866

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@9866

wrangler

npm i https://pkg.pr.new/wrangler@9866

commit: e5c988a

@invisal invisal force-pushed the invisal/d1-sql-dump-escape-columns branch from a1797b1 to 423b04a Compare July 15, 2025 13:51
@invisal invisal marked this pull request as ready for review July 15, 2025 13:51
@invisal invisal requested a review from a team as a code owner July 15, 2025 13:51
Copy link
Contributor

@alsuren alsuren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks for fixing this.

Part of me is tempted to assert that we can round-trip our export through the sqlite cli, reimport it and still get the same results. I doubt that our CI boxes have the sqlite cli installed though, so the way you've done it is probably the right balance.

@github-project-automation github-project-automation bot moved this from Untriaged to Approved in workers-sdk Jul 18, 2025
@alsuren alsuren merged commit 7e5585d into cloudflare:main Jul 18, 2025
51 of 56 checks passed
@github-project-automation github-project-automation bot moved this from Approved to Done in workers-sdk Jul 18, 2025
@workers-devprod workers-devprod added the contribution [Holopin] Recognizes an open-source contribution, big or small label Jul 18, 2025
Copy link

holopin-bot bot commented Jul 18, 2025

Congratulations @invisal, the maintainer of this repository has issued you a holobyte! Here it is: https://holopin.io/holobyte/cmd8uaetj1763207jvmdf7wiq6

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

petebacondarwin pushed a commit that referenced this pull request Jul 22, 2025
…ump (#9866)

* fix: escape column names and handle mismatched data types in D1 SQL dump

* fix(d1): allow exporting blob in tests and importing without errors

* fix(d1): correct test case by quoting table names with double quotes in SQL INSERT

* fix(d1): Add patch note for escape identifiers and handle dynamic typing

---------

Co-authored-by: visal <[email protected]>
emily-shen pushed a commit that referenced this pull request Jul 22, 2025
…ump (#9866) (#10035)

* fix: escape column names and handle mismatched data types in D1 SQL dump

* fix(d1): allow exporting blob in tests and importing without errors

* fix(d1): correct test case by quoting table names with double quotes in SQL INSERT

* fix(d1): Add patch note for escape identifiers and handle dynamic typing

---------

Co-authored-by: Visal .In <[email protected]>
Co-authored-by: visal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution [Holopin] Recognizes an open-source contribution, big or small
Projects
Archived in project
4 participants