HDDS-15179. Support Chunked Transfer without Content Length

[peterxcli]

What changes were proposed in this pull request?

Currently chunked transfer would fail if we don't specify content length

def _ev_add_te_header(request, **kwargs):
    request.headers.add_header('Transfer-Encoding', 'chunked')

def test_object_write_with_chunked_transfer_encoding():
    bucket_name = get_new_bucket()
    client = get_client()

    client.meta.events.register_first('before-sign.*.*', _ev_add_te_header)
    response = client.put_object(Bucket=bucket_name, Key='foo', Body='bar')

    assert response['ResponseMetadata']['HTTPStatusCode'] == 200

with the following error:

botocore.exceptions.ClientError: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: The provided 'x-amz-content-sha256' header does not match the computed hash.

https://github.com/ceph/s3-tests/blob/fb8b73092bb1dd8db829f1205a9e52e73bf9a232/s3tests/functional/test_s3.py#L1589-L1599

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15179

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)

[peterxcli]

I notice the jira ticket should be as a subtask under https://issues.apache.org/jira/browse/HDDS-8423, is there any way to fix it? I couldn't found the move option.

[adoroszlai]

Thanks @peterxcli for the patch.

[chungen0126]

Thanks @peterxcli for the patch.

Based on my understanding of S3, requests fall into one of two cases: they either include a Content-Length header or an x-amz-decoded-content-length header. That being said, I totally understand that this proposed change adds flexibility to the upload process, and for me it is ok.

[chungen0126]

Regarding S3, we should also consider the scenario where the Content-Length header is absent, but the x-amz-decoded-content-length header is provided. In this case, since we already know the exact size of the payload, we wouldn't need to manually calculate the upload length. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

For all requests, you must include the x-amz-decoded-content-length header, specifying the size of the object in bytes.

[chungen0126]

+1 LGTM. Let's wait for the CI to pass.

[ivandika3]

Is this going to copy the whole object into a temporary file on S3G machine? In other words, is S3G required to allocate some disk space for this length calculation?

[ivandika3]

If so, this might be problematic since firstly currently a lot of cluster (including mine) assume that S3G does not have variable disk requirement (this then require some kind of persistent volume implementation for S3G deployed in K8s). Additionally, putting the whole object into a file and then re-reading again might incur additional disk IO.

[ivandika3]

From what I see, the length is only need to check whether the length is 0 or not. In that case is it possible to simply check the first byte existence?

[ivandika3]

If so, this might be problematic since firstly currently a lot of cluster (including mine) assume that S3G does not have variable disk requirement (this then require some kind of persistent volume implementation for S3G deployed in K8s). Additionally, putting the whole object into a file and then re-reading again might incur additional disk IO.

[peterxcli]

thanks for the review, let me think about this.