Fix: multipart upload

[rraulinio]

Relates to: minio/mint#399.

Remove checksum headers from CreateMultipartUpload.

Problem

Multipart uploads fail with InvalidPart error:

One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.

Root Cause

Incorrectly existing checksum headers (recently added) to both CreateMultipartUpload and UploadPart requests. This causes ETag mismatch because:

Why it fails: When checksum headers are present on CreateMultipartUpload, the S3 server establishes expectations for how the final object's ETag should be calculated. When the same checksum headers are then sent with each UploadPart, the server calculates part ETags differently than expected from the initial multipart setup. During CompleteMultipartUpload, the server validates that uploaded part ETags match what was declared, but they don't match due to the conflicting checksum information, resulting in InvalidPart error.

Per S3 spec: Checksum headers should only be on individual UploadPart requests (where actual data is uploaded), not on CreateMultipartUpload (which only initializes the upload with metadata like storage class and encryption).

Solution

Remove headers.extend(checksum_headers) before CreateMultipartUpload call. Checksum headers should only be sent with UploadPart requests.

LE: Solution extended to filtering the headers in CreateMultipartUpload. On top of that, adding the missing checksum fields in CompleteMultipartUpload.

[rraulinio]

Mint PR pulling from here - minio/mint@2b140ce - resolves the problem for the minio-py and s3select related tests.

[rraulinio]

@balamurugana Even if you remove this line from the test - checksum=Algorithm.CRC32C - in the put_object request, the bug still happens, just realised now.
That's just because Algorithm.CRC32C gets picked as default if nothing is requested explicitly.

You can try also with checksum=Algorithm.SHA256 instead, just to run another explicit example. It will work just fine, because as mentioned, x-amz-checksum-algorithm is optional in the initial create request. S3 will infer the algorithm from the UploadPart requests. We can of course just leave this one in the request, instead of removing it as well (as per my current code change), but it makes no difference in reality.

If you have a strong preference, I can change to this:

headers.extend({k: v for k, v in checksum_headers.items()
                if k == "x-amz-sdk-checksum-algorithm"})

Fine either way.

[balamurugana]

I also tested with local minio from community version works!

$ PYTHONPATH=$PWD python c.py 
---------START-HTTP---------
GET /mybucket?location= HTTP/1.1
Host: localhost:9000
User-Agent: MinIO (Linux; x86_64) minio-py/7.2.19
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Content-Md5: 1B2M2Y8AsgTpgAmY7PhCfg==
X-Amz-Date: 20251114T121417Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20251114/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*

HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 128
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 1877DE88A8D57DB8
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3331
X-Ratelimit-Remaining: 3331
X-Xss-Protection: 1; mode=block
Date: Fri, 14 Nov 2025 12:14:17 GMT

<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></LocationConstraint>
----------END-HTTP----------
---------START-HTTP---------
POST /mybucket/myobject?uploads= HTTP/1.1
Content-Type: application/octet-stream
X-Amz-Sdk-Checksum-Algorithm: crc32c
X-Amz-Checksum-Crc32C: GUlcdQ==
Host: localhost:9000
User-Agent: MinIO (Linux; x86_64) minio-py/7.2.19
Content-Length: 0
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Content-Md5: 1B2M2Y8AsgTpgAmY7PhCfg==
X-Amz-Date: 20251114T121417Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20251114/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;content-type;host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=*REDACTED*

HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 339
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 1877DE88A8E87968
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3331
X-Ratelimit-Remaining: 3331
X-Xss-Protection: 1; mode=block
Date: Fri, 14 Nov 2025 12:14:17 GMT

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>mybucket</Bucket><Key>myobject</Key><UploadId>MjNkNDczOTQtMzZmZi00MmQ1LWIyZTktMzBhNWI3ZDRmNTA0LjI3ZmFiNjQwLWM0OWQtNDBmMS04Y2MxLWQxODQ3M2E2ZTExZHgxNzYzMTIyNDU3NjY5NjU4NTE2</UploadId></InitiateMultipartUploadResult>
----------END-HTTP----------
---------START-HTTP---------
PUT /mybucket/myobject?partNumber=1&uploadId=MjNkNDczOTQtMzZmZi00MmQ1LWIyZTktMzBhNWI3ZDRmNTA0LjI3ZmFiNjQwLWM0OWQtNDBmMS04Y2MxLWQxODQ3M2E2ZTExZHgxNzYzMTIyNDU3NjY5NjU4NTE2 HTTP/1.1
X-Amz-Sdk-Checksum-Algorithm: crc32c
X-Amz-Checksum-Crc32C: GUlcdQ==
Host: localhost:9000
User-Agent: MinIO (Linux; x86_64) minio-py/7.2.19
Content-Length: 5242880
Content-Type: application/octet-stream
X-Amz-Content-Sha256: f2e2ee4d13fd938e5d3e6fb985ff45843f1d74abad4ee12631249618263a5769
X-Amz-Date: 20251114T121417Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20251114/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=*REDACTED*

---------START-HTTP---------
PUT /mybucket/myobject?partNumber=2&uploadId=MjNkNDczOTQtMzZmZi00MmQ1LWIyZTktMzBhNWI3ZDRmNTA0LjI3ZmFiNjQwLWM0OWQtNDBmMS04Y2MxLWQxODQ3M2E2ZTExZHgxNzYzMTIyNDU3NjY5NjU4NTE2 HTTP/1.1
X-Amz-Sdk-Checksum-Algorithm: crc32c
X-Amz-Checksum-Crc32C: dIGSKw==
Host: localhost:9000
User-Agent: MinIO (Linux; x86_64) minio-py/7.2.19
Content-Length: 1048576
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e10a11f0f19b7b3c40d39738ecd97c59c1f2a890984f9d8c63456ebadfef17f4
X-Amz-Date: 20251114T121417Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20251114/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=*REDACTED*

HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 0
ETag: "6cbee1d380620e11b4f8002ec1990529"
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Checksum-Crc32c: GUlcdQ==
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 1877DE88AC0F8A08
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3331
X-Ratelimit-Remaining: 3331
X-Xss-Protection: 1; mode=block
Date: Fri, 14 Nov 2025 12:14:17 GMT


----------END-HTTP----------
HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 0
ETag: "261f9336675060e97255ef9f0b386aba"
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Checksum-Crc32c: dIGSKw==
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 1877DE88AE1862FC
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3331
X-Ratelimit-Remaining: 3331
X-Xss-Protection: 1; mode=block
Date: Fri, 14 Nov 2025 12:14:17 GMT


----------END-HTTP----------
---------START-HTTP---------
POST /mybucket/myobject?uploadId=MjNkNDczOTQtMzZmZi00MmQ1LWIyZTktMzBhNWI3ZDRmNTA0LjI3ZmFiNjQwLWM0OWQtNDBmMS04Y2MxLWQxODQ3M2E2ZTExZHgxNzYzMTIyNDU3NjY5NjU4NTE2 HTTP/1.1
Content-Type: application/xml
Content-Md5: y5BdTUJ4lOknM1m402FqPw==
Host: localhost:9000
User-Agent: MinIO (Linux; x86_64) minio-py/7.2.19
Content-Length: 271
X-Amz-Content-Sha256: 252397351168dab7a249dda6f3635a1f8ac3c01f56f2c83767812517fda7f69e
X-Amz-Date: 20251114T121417Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20251114/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*

<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><PartNumber>1</PartNumber><ETag>"6cbee1d380620e11b4f8002ec1990529"</ETag></Part><Part><PartNumber>2</PartNumber><ETag>"261f9336675060e97255ef9f0b386aba"</ETag></Part></CompleteMultipartUpload>

HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 311
Content-Type: application/xml
ETag: "34d8be078634c0cc2076feccfb0b6342-2"
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 1877DE88AF11CFA4
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3331
X-Ratelimit-Remaining: 3331
X-Xss-Protection: 1; mode=block
Date: Fri, 14 Nov 2025 12:14:17 GMT

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://localhost:9000/mybucket/myobject</Location><Bucket>mybucket</Bucket><Key>myobject</Key><ETag>&#34;34d8be078634c0cc2076feccfb0b6342-2&#34;</ETag></CompleteMultipartUploadResult>
----------END-HTTP----------
created myobject object; etag: 34d8be078634c0cc2076feccfb0b6342-2, version-id: None

[balamurugana]

The bug is CompleteMultipartUpload API is not called with checksums. minio-py should send like

<?xml version="1.0"?>
<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Part>
    <ChecksumCRC32C>GUlcdQ==</ChecksumCRC32C>
    <PartNumber>1</PartNumber>
    <ETag>6cbee1d380620e11b4f8002ec1990529</ETag>
  </Part>
  <Part>
    <ChecksumCRC32C>dIGSKw==</ChecksumCRC32C>
    <PartNumber>2</PartNumber>
    <ETag>261f9336675060e97255ef9f0b386aba</ETag>
  </Part>
</CompleteMultipartUpload>

[rraulinio]

The bug is CompleteMultipartUpload API is not called with checksums. minio-py should send like

<?xml version="1.0"?>
<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Part>
    <ChecksumCRC32C>GUlcdQ==</ChecksumCRC32C>
    <PartNumber>1</PartNumber>
    <ETag>6cbee1d380620e11b4f8002ec1990529</ETag>
  </Part>
  <Part>
    <ChecksumCRC32C>dIGSKw==</ChecksumCRC32C>
    <PartNumber>2</PartNumber>
    <ETag>261f9336675060e97255ef9f0b386aba</ETag>
  </Part>
</CompleteMultipartUpload>

That's just a different approach/fix to the same problem.

[rraulinio]

Indeed, for community edition, it works, but not for EOS.

[rraulinio]

Regardless, the bug in minio-py is real, so we need one of the two approaches - I will update now as such the code, then we can select.

[balamurugana]

@rraulinio We need to fix the bug not to remove checksum support.

[rraulinio]

@balamurugana Correct. So we either have this approach that I've just commited, in which we only allow the algo header, but we don't send wrongfully the hash of the first part anymore. Or the approach you suggested, they both solve the problem (I did not test your approach, but I take it will work).

[rraulinio]

Both fixes are now implemented. Everything passes, so problem solved. 💪

[rraulinio]

@balamurugana is working on a bigger PR that might supersede this one, leaving it open for now just in case.