Index Out of Rang errir

[nathaniela]

Though the backup procedure works, there is an error in the cloudwatch logs of the lambda function:

[ERROR] 2017-07-12T07:22:50.632Z e8984d42-66d2-11e7-a7af-510b0421530a Error: list index out of range

Here is the full log:

[DEBUG]	2017-07-12T07:22:49.907Z	Loading JSON file: /var/runtime/botocore/data/endpoints.json
[DEBUG]	2017-07-12T07:22:50.31Z	Loading JSON file: /var/runtime/botocore/data/ssm/2014-11-06/service-2.json
[DEBUG]	2017-07-12T07:22:50.64Z	Loading JSON file: /var/runtime/botocore/data/_retry.json
[DEBUG]	2017-07-12T07:22:50.65Z	Registering retry handlers for service: ssm
[DEBUG]	2017-07-12T07:22:50.68Z	Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x7faf2e12e848>
[DEBUG]	2017-07-12T07:22:50.69Z	The s3 config key is not a dictionary type, ignoring its value of: None
[DEBUG]	2017-07-12T07:22:50.70Z	Setting ssm timeout as (60, 60)
START RequestId: e8984d42-66d2-11e7-a7af-510b0421530a Version: $LATEST
[INFO]	2017-07-12T07:22:50.71Z	e8984d42-66d2-11e7-a7af-510b0421530a
{
    "account": "350785128185",
    "region": "us-east-1",
    "detail": {
        "LifecycleHookName": "ASGBackup",
        "AutoScalingGroupName": "awseb-e-mam8pnazxz-stack-AWSEBAutoScalingGroup-1RRZHOXULUX78",
        "LifecycleActionToken": "a860bb0a-3dfb-4cfc-b64f-86d228aea689",
        "LifecycleTransition": "autoscaling:EC2_INSTANCE_TERMINATING",
        "EC2InstanceId": "i-057beff453ba8887e"
    },
    "detail-type": "EC2 Instance-terminate Lifecycle Action",
    "source": "aws.autoscaling",
    "version": "0",
    "time": "2017-07-12T07:22:48Z",
    "id": "3e0b0358-9730-44c5-89a2-096e441c0c23",
    "resources": [
        "arn:aws:autoscaling:us-east-1:350785128185:autoScalingGroup:a8d17511-eec5-4b17-866b-ea0d5f1b0c36:autoScalingGroupName/awseb-e-mam8pnazxz-stack-AWSEBAutoScalingGroup-1RRZHOXULUX78"
    ]
}

[DEBUG]	2017-07-12T07:22:50.71Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG]	2017-07-12T07:22:50.72Z	e8984d42-66d2-11e7-a7af-510b0421530a	Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListDocuments'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG]	2017-07-12T07:22:50.72Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG]	2017-07-12T07:22:50.72Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG]	2017-07-12T07:22:50.73Z	e8984d42-66d2-11e7-a7af-510b0421530a	Calculating signature using v4 auth.
[DEBUG]	2017-07-12T07:22:50.76Z	e8984d42-66d2-11e7-a7af-510b0421530a	CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListDocuments

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
769c5616cb57fca6e74861906824d48410e9a1c6052187a8c2afaeda27fb55b6
[DEBUG]	2017-07-12T07:22:50.76Z	e8984d42-66d2-11e7-a7af-510b0421530a	StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
78f369959524afc6a18bbaba60b947d6c02a25f147319e2cbbf673cb7ab4b987
[DEBUG]	2017-07-12T07:22:50.77Z	e8984d42-66d2-11e7-a7af-510b0421530a	Signature:
dabe509b02c447b3d3612fc0837ba6ccc14c0fafa6e4be6518328e97a3b83840
[DEBUG]	2017-07-12T07:22:50.100Z	e8984d42-66d2-11e7-a7af-510b0421530a	Sending http request: <PreparedRequest [POST]>
[INFO]	2017-07-12T07:22:50.100Z	e8984d42-66d2-11e7-a7af-510b0421530a	Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com
[DEBUG]	2017-07-12T07:22:50.303Z	e8984d42-66d2-11e7-a7af-510b0421530a	"POST / HTTP/1.1" 200 167
[DEBUG]	2017-07-12T07:22:50.304Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response headers: {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG]	2017-07-12T07:22:50.316Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response body:
{
    "DocumentIdentifiers": [
        {
            "DocumentType": "Command",
            "DocumentVersion": "1",
            "Name": "ASGLogBackup",
            "Owner": "350785128185",
            "PlatformTypes": [
                "Linux"
            ],
            "SchemaVersion": "1.2"
        }
    ]
}

[DEBUG]	2017-07-12T07:22:50.317Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG]	2017-07-12T07:22:50.317Z	e8984d42-66d2-11e7-a7af-510b0421530a	No retry needed.
[INFO]	2017-07-12T07:22:50.317Z	e8984d42-66d2-11e7-a7af-510b0421530a	Documents list: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'350785128185', u'SchemaVersion': u'1.2'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}}}
[INFO]	2017-07-12T07:22:50.317Z	e8984d42-66d2-11e7-a7af-510b0421530a	Documents exists: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'350785128185', u'SchemaVersion': u'1.2'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}}}
[DEBUG]	2017-07-12T07:22:50.318Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG]	2017-07-12T07:22:50.318Z	e8984d42-66d2-11e7-a7af-510b0421530a	Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListDocuments'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG]	2017-07-12T07:22:50.336Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG]	2017-07-12T07:22:50.336Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG]	2017-07-12T07:22:50.337Z	e8984d42-66d2-11e7-a7af-510b0421530a	Calculating signature using v4 auth.
[DEBUG]	2017-07-12T07:22:50.337Z	e8984d42-66d2-11e7-a7af-510b0421530a	CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListDocuments

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
769c5616cb57fca6e74861906824d48410e9a1c6052187a8c2afaeda27fb55b6
[DEBUG]	2017-07-12T07:22:50.337Z	e8984d42-66d2-11e7-a7af-510b0421530a	StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
78f369959524afc6a18bbaba60b947d6c02a25f147319e2cbbf673cb7ab4b987
[DEBUG]	2017-07-12T07:22:50.337Z	e8984d42-66d2-11e7-a7af-510b0421530a	Signature:
dabe509b02c447b3d3612fc0837ba6ccc14c0fafa6e4be6518328e97a3b83840
[DEBUG]	2017-07-12T07:22:50.337Z	e8984d42-66d2-11e7-a7af-510b0421530a	Sending http request: <PreparedRequest [POST]>
[DEBUG]	2017-07-12T07:22:50.357Z	e8984d42-66d2-11e7-a7af-510b0421530a	"POST / HTTP/1.1" 200 167
[DEBUG]	2017-07-12T07:22:50.357Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response headers: {'x-amzn-requestid': 'e96e3ab5-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG]	2017-07-12T07:22:50.357Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response body:
{
    "DocumentIdentifiers": [
        {
            "DocumentType": "Command",
            "DocumentVersion": "1",
            "Name": "ASGLogBackup",
            "Owner": "350785128185",
            "PlatformTypes": [
                "Linux"
            ],
            "SchemaVersion": "1.2"
        }
    ]
}

[DEBUG]	2017-07-12T07:22:50.358Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG]	2017-07-12T07:22:50.358Z	e8984d42-66d2-11e7-a7af-510b0421530a	No retry needed.
[DEBUG]	2017-07-12T07:22:50.358Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event before-parameter-build.ssm.SendCommand: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG]	2017-07-12T07:22:50.359Z	e8984d42-66d2-11e7-a7af-510b0421530a	Making request for OperationModel(name=SendCommand) (verify_ssl=True) with params: {'body': '{"DocumentName": "ASGLogBackup", "InstanceIds": ["i-057beff453ba8887e"], "TimeoutSeconds": 120}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.SendCommand'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG]	2017-07-12T07:22:50.376Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event request-created.ssm.SendCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG]	2017-07-12T07:22:50.376Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event choose-signer.ssm.SendCommand: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG]	2017-07-12T07:22:50.377Z	e8984d42-66d2-11e7-a7af-510b0421530a	Calculating signature using v4 auth.
[DEBUG]	2017-07-12T07:22:50.377Z	e8984d42-66d2-11e7-a7af-510b0421530a	CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.SendCommand

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
f2237f6810448b76d44e17fd66b83b1b4371df545e6117e7f66eebb311631f28
[DEBUG]	2017-07-12T07:22:50.377Z	e8984d42-66d2-11e7-a7af-510b0421530a	StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
17b7fd589cca1a6a1548ff529c6ec0884756f72920c6cf5d79e9cf8b0d55c76b
[DEBUG]	2017-07-12T07:22:50.377Z	e8984d42-66d2-11e7-a7af-510b0421530a	Signature:
1cf64492c0ebe9ddcd021fa19fbcf5caeffe46284e55f8b853242c8b1ae8c9a1
[DEBUG]	2017-07-12T07:22:50.377Z	e8984d42-66d2-11e7-a7af-510b0421530a	Sending http request: <PreparedRequest [POST]>
[DEBUG]	2017-07-12T07:22:50.543Z	e8984d42-66d2-11e7-a7af-510b0421530a	"POST / HTTP/1.1" 200 528
[DEBUG]	2017-07-12T07:22:50.557Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response headers: {'x-amzn-requestid': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '528', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG]	2017-07-12T07:22:50.557Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response body:
{
    "Command": {
        "CommandId": "0ece375e-a643-4474-9642-81185bd7ff4a",
        "Comment": "",
        "CompletedCount": 0,
        "DocumentName": "ASGLogBackup",
        "ErrorCount": 0,
        "ExpiresAfter": 1499847890.54,
        "InstanceIds": [
            "i-057beff453ba8887e"
        ],
        "MaxConcurrency": "50",
        "MaxErrors": "0",
        "NotificationConfig": {
            "NotificationArn": "",
            "NotificationEvents": [],
            "NotificationType": ""
        },
        "OutputS3BucketName": "",
        "OutputS3KeyPrefix": "",
        "Parameters": {},
        "RequestedDateTime": 1499844170.54,
        "ServiceRole": "",
        "Status": "Pending",
        "StatusDetails": "Pending",
        "TargetCount": 1,
        "Targets": []
    }
}

[DEBUG]	2017-07-12T07:22:50.558Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event needs-retry.ssm.SendCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG]	2017-07-12T07:22:50.576Z	e8984d42-66d2-11e7-a7af-510b0421530a	No retry needed.
[INFO]	2017-07-12T07:22:50.576Z	e8984d42-66d2-11e7-a7af-510b0421530a	Command sent: {u'Command': {u'Comment': u'', u'Status': u'Pending', u'MaxErrors': u'0', u'Parameters': {}, u'ExpiresAfter': datetime.datetime(2017, 7, 12, 8, 24, 50, 540000, tzinfo=tzlocal()), u'ServiceRole': u'', u'DocumentName': u'ASGLogBackup', u'TargetCount': 1, u'OutputS3BucketName': u'', u'NotificationConfig': {u'NotificationArn': u'', u'NotificationEvents': [], u'NotificationType': u''}, u'CompletedCount': 0, u'Targets': [], u'StatusDetails': u'Pending', u'ErrorCount': 0, u'OutputS3KeyPrefix': u'', u'RequestedDateTime': datetime.datetime(2017, 7, 12, 7, 22, 50, 540000, tzinfo=tzlocal()), u'CommandId': u'0ece375e-a643-4474-9642-81185bd7ff4a', u'InstanceIds': [u'i-057beff453ba8887e'], u'MaxConcurrency': u'50'}, 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '528', 'content-type': 'application/x-amz-json-1.1'}}}
[DEBUG]	2017-07-12T07:22:50.577Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event before-parameter-build.ssm.ListCommandInvocations: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG]	2017-07-12T07:22:50.577Z	e8984d42-66d2-11e7-a7af-510b0421530a	Making request for OperationModel(name=ListCommandInvocations) (verify_ssl=True) with params: {'body': '{"InstanceId": "i-057beff453ba8887e", "CommandId": "0ece375e-a643-4474-9642-81185bd7ff4a", "Details": false}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListCommandInvocations'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG]	2017-07-12T07:22:50.596Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event request-created.ssm.ListCommandInvocations: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG]	2017-07-12T07:22:50.597Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event choose-signer.ssm.ListCommandInvocations: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG]	2017-07-12T07:22:50.597Z	e8984d42-66d2-11e7-a7af-510b0421530a	Calculating signature using v4 auth.
[DEBUG]	2017-07-12T07:22:50.597Z	e8984d42-66d2-11e7-a7af-510b0421530a	CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListCommandInvocations

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
06321604afedcd5513b2df3a174aad8faf73c23978da2c8fa03ff26689ddbaa0
[DEBUG]	2017-07-12T07:22:50.597Z	e8984d42-66d2-11e7-a7af-510b0421530a	StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
aeddc98a9eb7a4eaec8aa55d7eefe079d6f56c45766155b52abf22bdc4818c38
[DEBUG]	2017-07-12T07:22:50.597Z	e8984d42-66d2-11e7-a7af-510b0421530a	Signature:
e5fc7fac6dae183c6697431e16d769d2be5555680f04742f4f2783c8ec5abe7c
[DEBUG]	2017-07-12T07:22:50.598Z	e8984d42-66d2-11e7-a7af-510b0421530a	Sending http request: <PreparedRequest [POST]>
[DEBUG]	2017-07-12T07:22:50.631Z	e8984d42-66d2-11e7-a7af-510b0421530a	"POST / HTTP/1.1" 200 25
[DEBUG]	2017-07-12T07:22:50.631Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response headers: {'x-amzn-requestid': 'e995e711-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '25', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG]	2017-07-12T07:22:50.631Z	e8984d42-66d2-11e7-a7af-510b0421530a	Response body:
{
    "CommandInvocations": []
}

[DEBUG]	2017-07-12T07:22:50.632Z	e8984d42-66d2-11e7-a7af-510b0421530a	Event needs-retry.ssm.ListCommandInvocations: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG]	2017-07-12T07:22:50.632Z	e8984d42-66d2-11e7-a7af-510b0421530a	No retry needed.
[ERROR]	2017-07-12T07:22:50.632Z	e8984d42-66d2-11e7-a7af-510b0421530a	Error: list index out of range
END RequestId: e8984d42-66d2-11e7-a7af-510b0421530a
REPORT RequestId: e8984d42-66d2-11e7-a7af-510b0421530a	Duration: 561.05 ms	Billed Duration: 600 ms Memory Size: 128 MB	Max Memory Used: 40 MB

I couldn't find any more details to troubleshoot...

@diegonat @ovalba I am facing the same issue, is there an update to this script ?

[binoykrishnamedhi]

Change the function check_command as below :

def check_command(command_id, instance_id):
    timewait = 1
    time.sleep(timewait)
    timewait += timewait
    while True:
        response_iterator = ssm_client.list_command_invocations(
            CommandId = command_id, 
            InstanceId = instance_id, 
            Details=False
            )
        if check_response(response_iterator):
            response_iterator_status = response_iterator['CommandInvocations'][0]['Status']
            if response_iterator_status != 'Pending':
                if response_iterator_status == 'InProgress' or response_iterator_status == 'Success':
                    logging.info( "Status: %s", response_iterator_status)
                    return True
                else:
                    logging.error("ERROR: status: %s", response_iterator)
                    return False

[diegonat]

sorry for the late response.

Can you try the following and let me know?

def check_command(command_id, instance_id):
	timewait = 1
	time.sleep(timewait)
	while True:
		response_iterator = ssm_client.list_command_invocations(
			CommandId = command_id,
			InstanceId = instance_id,
			Details=False
			)
		if check_response(response_iterator):
			try:
				response_iterator_status = response_iterator['CommandInvocations'][0]['Status']
				logging.info( "CommandInvocations Status: %s", response_iterator['CommandInvocations'])
				if response_iterator_status != 'Pending':
					if response_iterator_status == 'InProgress' or response_iterator_status == 'Success':
						logging.info( "Status: %s", response_iterator_status)
						return True
					else:
						logging.error("ERROR: status: %s", response_iterator)
						return False
			except Exception as e:
				logging.error("ERROR: %s", str(e))
				logging.error("ERROR: status: %s", response_iterator)
				return False
		time.sleep(timewait)
		timewait += timewait

[monishamsingh]

Also, you can try the below:

{
"schemaVersion": "1.2",
"description": "Backup logs to S3",
"parameters": {},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": [
"",
"ASGNAME='xxxxx'",
"LIFECYCLEHOOKNAME='xxx'",
"BACKUPDIRECTORY='/var/log/'",
"S3BUCKET='xxxxx'",
"SNSTARGET='arn:aws:sns:ap-south-1:1234567:xxxxx'",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)",
"REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)",
"REGION='ap-south-1'",
"HOOKRESULT='CONTINUE'",
"MESSAGE='test_asg'",
"",
"tar -vcf /tmp/${INSTANCEID}.tar -P $BACKUPDIRECTORY &> /tmp/backup",
"if [ $? -ne 0 ]",
"then",
" MESSAGE=$(cat /tmp/backup)",
"else",
" aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup",
" MESSAGE=$(cat /tmp/backup)",
"fi",
"",
"aws sns publish --subject 'ASG Backup' --message "$MESSAGE" --target-arn ${SNSTARGET} --region ${REGION}",
"aws autoscaling complete-lifecycle-action --lifecycle-hook-name ${LIFECYCLEHOOKNAME} --auto-scaling-group-name ${ASGNAME} --lifecycle-action-result ${HOOKRESULT} --instance-id ${INSTANCEID} --region ${REGION}"
]
}
]
}
}
}

[monishamsingh]

If u have multiple Auto-scaling groups,

{
"schemaVersion": "1.2",
"description": "Backup logs to S3",
"parameters": {},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": [
"",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)",
"ASGNAME=aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCEID | grep AutoScalingGroupName | cut -d: -f2 | tr -d ',' | sed \"s\/^[ \\t]*\/\/\" | tr -d '\"'",
"LIFECYCLEHOOKNAME='xxxxx'",
"BACKUPDIRECTORY='/var/log/'",
"S3BUCKET='xxxxx'",
"SNSTARGET='arn:aws:sns:ap-south-1:1234567:xxxxxx'",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)",
"REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)",
"REGION='ap-south-1'",
"HOOKRESULT='CONTINUE'",
"MESSAGE='xxxxx'",
"",
"tar -vcf /tmp/${INSTANCEID}.tar -P $BACKUPDIRECTORY &> /tmp/backup",
"if [ $? -ne 0 ]",
"then",
" MESSAGE=$(cat /tmp/backup)",
"else",
" aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup",
" MESSAGE=$(cat /tmp/backup)",
"fi",
"",
"aws sns publish --subject 'ASG Backup' --message "$MESSAGE" --target-arn ${SNSTARGET} --region ${REGION}",
"aws autoscaling complete-lifecycle-action --lifecycle-hook-name ${LIFECYCLEHOOKNAME} --auto-scaling-group-name ${ASGNAME} --lifecycle-action-result ${HOOKRESULT} --instance-id ${INSTANCEID} --region ${REGION}"
]
}
]
}
}
}

You need to add the region to boto3 client commands:

1. Declare a global variable for aws region

import os
AWS_REGION = os.environ['AWS_REGION']

2. Update boto3 client to use region

line 8: ssm_client = boto3.client("ssm", region_name=AWS_REGION)
line 96: asg_client = boto3.client('autoscaling', region_name=AWS_REGION)

[matrix200]

I am also having this error, so what is the ultimate fix?

[diegonat]

I'd kindly ask you to use the code above and see if it works

[matrix200]

I am just a little confused from various code snippets.
Are you referring to the one you posted on April the 11th?

[matrix200]

OK so after using your code from April 11th, this error is gone, but I still get nothing in the bucket.
Here are the logs:
15:17:46
[DEBUG] 2018-08-30T15:17:46.992Z Loading variable profile from defaults.
[DEBUG] 2018-08-30T15:17:46.992Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.992Z Loading variable config_file from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.992Z Loading variable credentials_file from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.992Z Loading variable data_path from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable region from environment with value 'us-west-2'.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults.
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable ca_bundle from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable api_versions from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable credentials_file from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable config_file from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.994Z Loading variable metadata_service_timeout from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.994Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.994Z Loading variable metadata_service_num_attempts from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.998Z Loading variable profile from defaults.
15:17:46
[DEBUG] 2018-08-30T15:17:46.998Z Looking for credentials via: env
15:17:46
[INFO] 2018-08-30T15:17:46.998Z Found credentials in environment variables.
15:17:47
[DEBUG] 2018-08-30T15:17:47.0Z Loading JSON file: /var/runtime/botocore/data/endpoints.json
15:17:47
[DEBUG] 2018-08-30T15:17:47.29Z Loading variable profile from defaults.
15:17:47
[DEBUG] 2018-08-30T15:17:47.29Z Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f78a5e53cf8>
15:17:47
[DEBUG] 2018-08-30T15:17:47.211Z Loading JSON file: /var/runtime/botocore/data/ssm/2014-11-06/service-2.json
15:17:47
[DEBUG] 2018-08-30T15:17:47.290Z Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x7f78a5e8e320>
15:17:47
[DEBUG] 2018-08-30T15:17:47.291Z The s3 config key is not a dictionary type, ignoring its value of: None
15:17:47
[DEBUG] 2018-08-30T15:17:47.293Z Setting ssm timeout as (60, 60)
15:17:47
[DEBUG] 2018-08-30T15:17:47.294Z Loading JSON file: /var/runtime/botocore/data/_retry.json
15:17:47
[DEBUG] 2018-08-30T15:17:47.297Z Registering retry handlers for service: ssm
15:17:47
START RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432 Version: $LATEST
15:17:47
[INFO] 2018-08-30T15:17:47.298Z d95427a2-ac67-11e8-bfed-bdb957ac3432 {"account": "215691954377", "region": "us-west-2", "detail": {"LifecycleHookName": "ASGBackup", "AutoScalingGroupName": "multicastingNewServer", "LifecycleActionToken": "4b215a7d-f41b-4050-9ec0-b80230e88588", "LifecycleTransition": "autoscaling:EC2_INSTANCE_TERMINATING", "EC2InstanceId": "i-071f6ae185996bc6f"}, "detail-type": "EC
15:17:47
[DEBUG] 2018-08-30T15:17:47.298Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50>
15:17:47
[DEBUG] 2018-08-30T15:17:47.299Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exec-env/AWS_Lambda_python2.7
15:17:47
[DEBUG] 2018-08-30T15:17:47.307Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>>
15:17:47
[DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60>
15:17:47
[DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth.
15:17:47
[DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A
15:17:47
[DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 7f18f52c48499da58730fcb7881f155261bd1435451672a6f3804ec57a6bc433
15:17:47
[DEBUG] 2018-08-30T15:17:47.309Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 7f32c91638e59cc92792c6498357803ded2e5ede7502cedc8ceb93260282569e
15:17:47
[DEBUG] 2018-08-30T15:17:47.337Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]>
15:17:47
[INFO] 2018-08-30T15:17:47.338Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Starting new HTTPS connection (1): ssm.us-west-2.amazonaws.com
15:17:47
[DEBUG] 2018-08-30T15:17:47.563Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 201
15:17:47
[DEBUG] 2018-08-30T15:17:47.564Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': '1f875bf1-b251-448b-a3e0-e521f139449b', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '201', 'content-type': 'application/x-amz-json-1.1'}
15:17:47
[DEBUG] 2018-08-30T15:17:47.564Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"DocumentIdentifiers":[{"DocumentFormat":"JSON","DocumentType":"Command","DocumentVersion":"1","Name":"ASGLogBackup","Owner":"215691954377","PlatformTypes":["Linux"],"SchemaVersion":"1.2","Tags":[]}]}
15:17:47
[DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10>
15:17:47
[DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed.
15:17:47
[INFO] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Documents list: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'Tags': [], u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'215691954377', u'SchemaVersion': u'1.2', u'DocumentFormat': u'JSON'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1f8
15:17:47
[INFO] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Documents exists: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'Tags': [], u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'215691954377', u'SchemaVersion': u'1.2', u'DocumentFormat': u'JSON'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1
15:17:47
[DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50>
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exec-env/AWS_Lambda_python2.7
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>>
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60>
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth.
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A
15:17:47
[DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 7f18f52c48499da58730fcb7881f155261bd1435451672a6f3804ec57a6bc433
15:17:47
[DEBUG] 2018-08-30T15:17:47.567Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 7f32c91638e59cc92792c6498357803ded2e5ede7502cedc8ceb93260282569e
15:17:47
[DEBUG] 2018-08-30T15:17:47.567Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]>
15:17:47
[DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 201
15:17:47
[DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': 'c7ba175a-7f5f-467a-a195-971a1ada437a', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '201', 'content-type': 'application/x-amz-json-1.1'}
15:17:47
[DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"DocumentIdentifiers":[{"DocumentFormat":"JSON","DocumentType":"Command","DocumentVersion":"1","Name":"ASGLogBackup","Owner":"215691954377","PlatformTypes":["Linux"],"SchemaVersion":"1.2","Tags":[]}]}
15:17:47
[DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10>
15:17:47
[DEBUG] 2018-08-30T15:17:47.629Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed.
15:17:47
[DEBUG] 2018-08-30T15:17:47.629Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.SendCommand: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50>
15:17:47
[DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=SendCommand) (verify_ssl=True) with params: {'body': '{"DocumentName": "ASGLogBackup", "InstanceIds": ["i-071f6ae185996bc6f"], "TimeoutSeconds": 120}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exe
15:17:47
[DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.SendCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>>
15:17:47
[DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.SendCommand: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60>
15:17:47
[DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth.
15:17:47
[DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A
15:17:47
[DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 839c71b66e35d2072a1b08f639b7e3eb3888bd325114c8301d3e40db2773a071
15:17:47
[DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 156c189eb80f4ee393375987eaa9209d196a2143332141a5bfeec85886842f43
15:17:47
[DEBUG] 2018-08-30T15:17:47.690Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]>
15:17:47
[DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 684
15:17:47
[DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': 'c4a72f3b-6b89-4c34-939b-f59637f9b408', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '684', 'content-type': 'application/x-amz-json-1.1'}
15:17:47
[DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"Command":{"CloudWatchOutputConfig":{"CloudWatchLogGroupName":"","CloudWatchOutputEnabled":false},"CommandId":"653f9d43-0a30-4204-9d4c-865d0a481983","Comment":"","CompletedCount":0,"DeliveryTimedOutCount":0,"DocumentName":"ASGLogBackup","DocumentVersion":"","ErrorCount":0,"ExpiresAfter":1.535645987832E9,"Instance
15:17:47
[DEBUG] 2018-08-30T15:17:47.867Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.SendCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10>
15:17:47
[DEBUG] 2018-08-30T15:17:47.868Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed.
15:17:47
[INFO] 2018-08-30T15:17:47.868Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Command sent: {u'Command': {u'MaxErrors': u'0', u'Parameters': {}, u'DocumentName': u'ASGLogBackup', u'OutputS3BucketName': u'', u'OutputS3KeyPrefix': u'', u'StatusDetails': u'Pending', u'RequestedDateTime': datetime.datetime(2018, 8, 30, 15, 17, 47, 832000, tzinfo=tzlocal()), u'Status': u'Pending', u'TargetCount': 1, u'Notificat
15:17:48
[DEBUG] 2018-08-30T15:17:48.869Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListCommandInvocations: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50>
15:17:48
[DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListCommandInvocations) (verify_ssl=True) with params: {'body': '{"InstanceId": "i-071f6ae185996bc6f", "CommandId": "653f9d43-0a30-4204-9d4c-865d0a481983", "Details": false}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119
15:17:48
[DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListCommandInvocations: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>>
15:17:48
[DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListCommandInvocations: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60>

15:17:48
[DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth.
15:17:48
[DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151748Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A
15:17:48
[DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151748Z 20180830/us-west-2/ssm/aws4_request 6089f8a64d5097bd60ea9c671d2fba8c8cd1aa2063b6c180f2b7c0a4941bd07a
15:17:48
[DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 5860ada82ebd6599a535605ad545111abfb53455e75f2b69eb37d7f31fecf031
15:17:48
[DEBUG] 2018-08-30T15:17:48.888Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]>
15:17:48
[DEBUG] 2018-08-30T15:17:48.906Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 540
15:17:48
[DEBUG] 2018-08-30T15:17:48.907Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': '6fa68de8-f293-4754-8217-c15230edda82', 'date': 'Thu, 30 Aug 2018 15:17:48 GMT', 'content-length': '540', 'content-type': 'application/x-amz-json-1.1'}
15:17:48
[DEBUG] 2018-08-30T15:17:48.907Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"CommandInvocations":[{"CloudWatchOutputConfig":{"CloudWatchLogGroupName":"","CloudWatchOutputEnabled":false},"CommandId":"653f9d43-0a30-4204-9d4c-865d0a481983","CommandPlugins":[],"Comment":"","DocumentName":"ASGLogBackup","DocumentVersion":"","InstanceId":"i-071f6ae185996bc6f","InstanceName":"","NotificationCon
15:17:48
[DEBUG] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListCommandInvocations: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10>
15:17:48
[DEBUG] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed.
15:17:48
[INFO] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CommandInvocations Status: [{u'Comment': u'', u'Status': u'InProgress', u'CommandPlugins': [], u'ServiceRole': u'', u'CloudWatchOutputConfig': {u'CloudWatchLogGroupName': u'', u'CloudWatchOutputEnabled': False}, u'InstanceId': u'i-071f6ae185996bc6f', u'DocumentName': u'ASGLogBackup', u'NotificationConfig': {u'NotificationArn': u'
15:17:48
[INFO] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Status: InProgress
15:17:48
[INFO] 2018-08-30T15:17:48.930Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Lambda executed correctly
15:17:48
END RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432
15:17:48
REPORT RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432 Duration: 1631.98 ms Billed Duration: 1700 ms Memory Size: 128 MB Max Memory Used: 48 MB

The only thing that kind of looks like an error is
The s3 config key is not a dictionary type, ignoring its value of: None
Also no tar files are created on the server while the script runs.
What am I missing?

[diegonat]

it seems that the commend was running. Did you check the logs of the SSM? Do you see any error? If you look at [{u'Comment': u'', u'Status': u'InProgress', u'Command.... the status is in progress. Does the instance has the permission to write to S3?

[matrix200]

Not sure about the S3, I thought no permission was necessary as long as the account itself has write permission?
But the problem is obviously before S3 as the files aren't even getting created on the instance (the tar archives).
You are right, I have got to go over SSM logs.
Will do in a few.

[matrix200]

Here is output of the command in SSM logs:
% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 26874 0 --:--:-- --:--:-- --:--:-- 19000

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 10 100 10 0 0 21097 0 --:--:-- --:--:-- --:--:-- 10000

/var/lib/amazon/ssm/i-071f6ae185996bc6f/document/orchestration/653f9d43-0a30-4204-9d4c-865d0a481983/awsrunShellScript/0.awsrunShellScript/_script.sh: 8: /var/lib/amazon/ssm/i-071f6ae185996bc6f/document/orchestration/653f9d43-0a30-4204-9d4c-865d0a481983/awsrunShellScript/0.awsrunShellScript/_script.sh: Bad substitution

failed to run commands: exit status 2

[diegonat]

this error is not related to the lambda_backup.py script. It is an error with the document that is ran within the instance. Try to run it in an instance that is not part of ASG to troubleshoot the issue. Please open another issue.

[akhileshjoshi2]

getting the below error

[DEBUG] 2018-10-23T09:41:35.581Z d37829b0-d6a7-11e8-b0eb-f5d458a80990 The s3 config key is not a dictionary type, ignoring its value of: None