Error when uploading a folder with files in a premium blob

[miguelgfierro]

I got this error:

HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a75c11550>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/connection.py", line 56, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/opt/az/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a75c11550>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a75c11550>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 269, in _perform_request
    response = self._httpclient.perform_request(request)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/_http/httpclient.py", line 92, in perform_request
    proxies=self.proxies)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a75c11550>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 369, in execute
    six.reraise(*sys.exc_info())
  File "/opt/az/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 343, in execute
    result = cmd(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
    result = op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/blob/baseblobservice.py", line 630, in create_container
    self._perform_request(request, expected_errors=[_CONTAINER_ALREADY_EXISTS_ERROR_CODE])
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 381, in _perform_request
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 311, in _perform_request
    raise AzureException(ex.args[0])
azure.common.AzureException: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a75c11550>: Failed to establish a new connection: [Errno -2] Name or service not known',))
uploading /data1/imagenet/ILSVRC2012_devkit_t12.tar.gz
HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets/imagenet/ILSVRC2012_devkit_t12.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f777f7b15f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/connection.py", line 56, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/opt/az/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f777f7b15f8>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets/imagenet/ILSVRC2012_devkit_t12.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f777f7b15f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 269, in _perform_request
    response = self._httpclient.perform_request(request)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/_http/httpclient.py", line 92, in perform_request
    proxies=self.proxies)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets/imagenet/ILSVRC2012_devkit_t12.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f777f7b15f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 369, in execute
    six.reraise(*sys.exc_info())
  File "/opt/az/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 343, in execute
    result = cmd(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
    result = op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 199, in storage_blob_upload_batch
    if_none_match=if_none_match, timeout=timeout)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/util.py", line 218, in wrapper
    return True, func(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 185, in _upload_blob
    return upload_blob(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 286, in upload_blob
    return type_func[blob_type]()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 279, in upload_block_blob
    return client.create_blob_from_path(**create_blob_args)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 463, in create_blob_from_path
    timeout=timeout)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 582, in create_blob_from_stream
    timeout=timeout)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 971, in _put_blob
    return self._perform_request(request, _parse_base_properties)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 381, in _perform_request
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 311, in _perform_request
    raise AzureException(ex.args[0])
azure.common.AzureException: HTTPSConnectionPool(host='datasets.blob.core.windows.net', port=443): Max retries exceeded with url: /datasets/imagenet/ILSVRC2012_devkit_t12.tar.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f777f7b15f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[miguelgfierro]

The script works with blob storage general purpose 1 and 2

[miguelgfierro]

@msalvaris by any chance do you have a script to upload to premium blobs? This script azure_blob_upload_folder_with_files.sh is working with storage general purpose 1 and 2, but not with premium

[msalvaris]

I use azcopy. Here is an example
azcopy --source /data/train
--destination https://${ACCOUNT_NAME}.blob.core.windows.net/${CONTAINER_NAME}/train
--dest-key ${azure_premium_storage_key} --quiet --recursive --exclude-older