diff --git a/src/docs/_sidebar.md b/src/docs/_sidebar.md index 763c32edde..eac1988e84 100644 --- a/src/docs/_sidebar.md +++ b/src/docs/_sidebar.md @@ -20,7 +20,6 @@ - [**Elastigroup**](elastigroup/) - [**Elastigroup Stateful Node**](managed-instance/) - [Ocean](ocean/) -- [**Ocean for Apache Spark**](ocean-spark/) - [**Eco**](eco/) - [**Security**](spot-security/) - [**Spot Connect**](spot-connect/) diff --git a/src/docs/connect-your-cloud-provider/README.md b/src/docs/connect-your-cloud-provider/README.md index c72cfc64c2..e29abca440 100644 --- a/src/docs/connect-your-cloud-provider/README.md +++ b/src/docs/connect-your-cloud-provider/README.md @@ -2,14 +2,13 @@ ## Products -* [Ocean](ocean/) -* [Elastigroup](/elastigroup/) -* [Elastigroup Stateful Node](managed-instance/) -* [Eco](eco/) -* [Ocean for Apache Spark](ocean-spark/) -* [Spot Security](spot-security/) -* [Spot Connect](spot-connect/) -* [Spot Storage for AWS](spot-storage/) +- [Ocean](ocean/) +- [Elastigroup](/elastigroup/) +- [Elastigroup Stateful Node](managed-instance/) +- [Eco](eco/) +- [Spot Security](spot-security/) +- [Spot Connect](spot-connect/) +- [Spot Storage for AWS](spot-storage/) ## Getting started @@ -21,6 +20,6 @@ ## More links -* [Integration tools](tools-and-provisioning/) -* [Spot OpenAPI specification](https://docs.spot.io/api/) -* [FAQs](faqs/) +- [Integration tools](tools-and-provisioning/) +- [Spot OpenAPI specification](https://docs.spot.io/api/) +- [FAQs](faqs/) diff --git a/src/docs/faqs/faqs-ocean.md b/src/docs/faqs/faqs-ocean.md index 6cb322f36f..2aa9b43dc8 100644 --- a/src/docs/faqs/faqs-ocean.md +++ b/src/docs/faqs/faqs-ocean.md @@ -34,16 +34,16 @@ us-east1, us-east1, us-east1, us-east4, us-east4, us-east4, us-central1, us-cent When shutdown hours end and Ocean needs to launch a node, it searches for a virtual node group with these characteristics: -* It is not in shutdown hours. -* Has no taints (Ocean will not launch a virtual node group that has a taint). -* `maxInstanceCount > 0` (or not set). +- It is not in shutdown hours. +- Has no taints (Ocean will not launch a virtual node group that has a taint). +- `maxInstanceCount > 0` (or not set). Ocean sorts the groups as follows: -* Highest max instance count. -* Highest spot percentage. -* Highest number of AZs. -* Highest number of possible instance types defined. +- Highest max instance count. +- Highest spot percentage. +- Highest number of AZs. +- Highest number of possible instance types defined. **Virtual Node Group shutdown hours:** @@ -59,8 +59,6 @@ See also [Set Shutdown Hours](https://docs.spot.io/ocean/tutorials/set-running-h - -
AWS, Azure, GCP: What's the difference between allocation and utilization for Ocean right sizing? @@ -108,31 +106,35 @@ This can happen if your virtual node group was deleted in Terraform. When you de
- If you get a `snapshotId cannot be modified on the root device` error: +If you get a `snapshotId cannot be modified on the root device` error: - 1. In the Spot console, go to **Ocean** > **Cloud Clusters**, and select the cluster. - 2. On the Virtual Nodes Groups tab, select the virtual node group. - 3. Click **JSON**. - 4. In the blockDeviceMappings, update the snapshotID or remove it: +1. In the Spot console, go to **Ocean** > **Cloud Clusters**, and select the cluster. +2. On the Virtual Nodes Groups tab, select the virtual node group. +3. Click **JSON**. +4. In the blockDeviceMappings, update the snapshotID or remove it: - ````json - "blockDeviceMappings": [ - { - "deviceName": "/dev/xvda", - "ebs": { - "deleteOnTerminaspoton": true, - "encrypted": false, - "iops": 3000, - "throughput": 125, - "snapshotId": "snap-1234", - "volumeSize": 100, - "volumeType": "GP3" - } + ```json + "blockDeviceMappings": [ + { + "deviceName": "/dev/xvda", + "ebs": { + "deleteOnTerminaspoton": true, + "encrypted": false, + "iops": 3000, + "throughput": 125, + "snapshotId": "snap-1234", + "volumeSize": 100, + "volumeType": "GP3" } + } ], - ```` + ``` + + ``` + + ``` - 5. Click **Save**. +5. Click **Save**.
@@ -145,11 +147,11 @@ This can happen if your virtual node group was deleted in Terraform. When you de No, you will get this error: -```` +``` Virtual Node Group configuration failed to update. Reason: Error while trying to create LaunchSpec. spotPercentage cannot be set on both ocean cluster and launch spec -```` +``` -The parameter spotPercentage cannot be used for both a cluster and one of its virtual node groups at the same time. This is intentional. Either remove it from the cluster or from the virtual node group. +The parameter spotPercentage cannot be used for both a cluster and one of its virtual node groups at the same time. This is intentional. Either remove it from the cluster or from the virtual node group. @@ -162,14 +164,14 @@ The parameter spotPercentage cannot be used for both a cluster and one of You can get this message when the group or cluster is scaling up instances: -```` +``` Can't Spin Instances: Code: ValidationError, Message: can't spin spot due to duplicate tags error -```` +``` This happens if you have duplicate tags configured: -* The cluster has more than one of the same custom tags. -* You created a custom tag key with spotinst—Spot automatically creates scaling tags that start with spotinst, resulting in multiple identical tags. +- The cluster has more than one of the same custom tags. +- You created a custom tag key with spotinst—Spot automatically creates scaling tags that start with spotinst, resulting in multiple identical tags. @@ -199,9 +201,9 @@ Draining timeout is the time in seconds to allow the instance or node to be drai The default draining for: -* Elastigroup is 120 seconds -* Ocean is 300 seconds -* ECS (Elastigroup/Ocean) is 900 seconds +- Elastigroup is 120 seconds +- Ocean is 300 seconds +- ECS (Elastigroup/Ocean) is 900 seconds @@ -222,20 +224,24 @@ You can see the list of permissions required for Spot in [Sample AWS policies](h
- You can stream Elastigroup logs to an AWS S3 bucket. Then, you can configure Elasticsearch and Kibana to collect logs from the S3 bucket: - - * [Ocean](/ocean/features/log-integration-with-s3) - * [Elastigroup](https://docs.spot.io/api/#tag/Elastigroup-AWS/operation/elastigroupAwsCreate) add this code to the JSON: - - ````json - "logging": { - "export": { - "s3": { - "id": "di-123" - } - } - } - ```` +You can stream Elastigroup logs to an AWS S3 bucket. Then, you can configure Elasticsearch and Kibana to collect logs from the S3 bucket: + +- [Ocean](/ocean/features/log-integration-with-s3) +- [Elastigroup](https://docs.spot.io/api/#tag/Elastigroup-AWS/operation/elastigroupAwsCreate) add this code to the JSON: + + ```json + "logging": { + "export": { + "s3": { + "id": "di-123" + } + } + } + ``` + + ``` + + ```
@@ -248,54 +254,54 @@ You can see the list of permissions required for Spot in [Sample AWS policies](h You can change your volume type to gp3 by: -* Adding a block device mapping for a single virtual node group in the Spot console: +- Adding a block device mapping for a single virtual node group in the Spot console: - 1. In the Spot console, go to **Ocean** > **Cloud Cluster**s and select the cluster. - 2. On the Virtual Nodes Groups tab, select the virtual node group. - 3. Go to **Advanced** > **Block Device Mapping**. - 4. Add the block device mapping and click **Save**. - 5. [Roll the virtual node group](ocean/features/roll-gen?id=roll-per-node-or-vng) if you want the changes to apply immediately on new nodes. + 1. In the Spot console, go to **Ocean** > **Cloud Cluster**s and select the cluster. + 2. On the Virtual Nodes Groups tab, select the virtual node group. + 3. Go to **Advanced** > **Block Device Mapping**. + 4. Add the block device mapping and click **Save**. + 5. [Roll the virtual node group](ocean/features/roll-gen?id=roll-per-node-or-vng) if you want the changes to apply immediately on new nodes. -* Changing the AMI to an AMI with gp3 volume type: +- Changing the AMI to an AMI with gp3 volume type: - 1. In the Spot console, go to **Ocean** > **Cloud Cluster**s and select the cluster. - 2. On the Virtual Nodes Groups tab, select the virtual node group. - 3. Go to **Advanced** > **Image**. - 4. Select an AMI with gp3. + 1. In the Spot console, go to **Ocean** > **Cloud Cluster**s and select the cluster. + 2. On the Virtual Nodes Groups tab, select the virtual node group. + 3. Go to **Advanced** > **Image**. + 4. Select an AMI with gp3. -* Making the [default virtual node group](ocean/features/launch-specifications?id=default-virtual-node-group) gp3 by adding a block device mapping at the cluster level. +- Making the [default virtual node group](ocean/features/launch-specifications?id=default-virtual-node-group) gp3 by adding a block device mapping at the cluster level. - 1. Add the block device mapping: + 1. Add the block device mapping: - * In the JSON: select the cluster > **Actions** > **Edit Cluster** > **Review** > **JSON** > **Edit Mode**. - * Using the [Ocean AWS cluster update API](https://docs.spot.io/api/#tag/Ocean-AWS/operation/OceanAWSClusterUpdate). + - In the JSON: select the cluster > **Actions** > **Edit Cluster** > **Review** > **JSON** > **Edit Mode**. + - Using the [Ocean AWS cluster update API](https://docs.spot.io/api/#tag/Ocean-AWS/operation/OceanAWSClusterUpdate). Keep in mind, you cannot use both [block device mapping](ocean/tutorials/manage-virtual-node-groups?id=advanced-parameters) and [root volume size](ocean/tutorials/manage-virtual-node-groups?id=configuration-parameters) at the same time. Sample block device mapping: - ````json - { - "group": { - "compute": { - "launchSpecification": { - "blockDeviceMappings": [ - { - "deviceName": "/dev/sda1", - "ebs": { - "deleteOnTermination": true, - "volumeSize": 24, - "volumeType": "gp2" - } + ```json + { + "group": { + "compute": { + "launchSpecification": { + "blockDeviceMappings": [ + { + "deviceName": "/dev/sda1", + "ebs": { + "deleteOnTermination": true, + "volumeSize": 24, + "volumeType": "gp2" } - ] } - } - } - } - ```` + ] + } + } + } + } + ``` - 2. Make sure to [roll the cluster](ocean/features/roll-gen) to replace the current instance gracefully with the changes. + 2. Make sure to [roll the cluster](ocean/features/roll-gen) to replace the current instance gracefully with the changes. @@ -309,8 +315,9 @@ You can change your volume type to gp3 by: You can have on-demand instances running in your group/cluster using reserved instance/savings plan even if you have set utilizeCommitments: false. This happens because of: -* **AWS commitments coverage**: When an on-demand instance launches in AWS, if there are any existing reservation or savings plan AWS may use them. AWS has its own way of deciding if an instance can be covered by a commitment plan. If the instance meets certain criteria, it will be covered if there's available space. This is how AWS handles reservations and savings plans. This happens even if you select utilizeCommitments: false. -* **Elastigroup/Ocean’s explicit commitment utilization**: If you’ve selected utilizeCommitments: true, Spot imitates AWS’s method to help you utilize all the commitment plans for your AWS account. If there is free space in the commitment plan and markets, your on-demand instances run reserved instances/savings plans. + +- **AWS commitments coverage**: When an on-demand instance launches in AWS, if there are any existing reservation or savings plan AWS may use them. AWS has its own way of deciding if an instance can be covered by a commitment plan. If the instance meets certain criteria, it will be covered if there's available space. This is how AWS handles reservations and savings plans. This happens even if you select utilizeCommitments: false. +- **Elastigroup/Ocean’s explicit commitment utilization**: If you’ve selected utilizeCommitments: true, Spot imitates AWS’s method to help you utilize all the commitment plans for your AWS account. If there is free space in the commitment plan and markets, your on-demand instances run reserved instances/savings plans. An on-demand instance marked as a reserved instance/savings plan doesn't always mean it will launch as a commitment plan. There can be other reasons for launching on-demand instances, such as when there is no spot capacity available or when certain requirements in Ocean need an on-demand instance. Then, if the on-demand instance is eligible, it will automatically use a commitment plan if there's space. @@ -325,9 +332,9 @@ Spot cannot control how AWS automatically handles commitment plan utilization. I
- When is an on-demand (OD) instance a reserved instance (RI), savings plan (SP), or full-priced on demand? - - When launching an on-demand instance, you cannot specifically request it to run as a reserved instance or savings plan. +When is an on-demand (OD) instance a reserved instance (RI), savings plan (SP), or full-priced on demand? + +When launching an on-demand instance, you cannot specifically request it to run as a reserved instance or savings plan. AWS decides according to: @@ -338,7 +345,7 @@ AWS decides according to: 5. Otherwise, the instance will run as a full-price on-demand instance. Throughout the lifetime of an instance, it can change its “price” whenever there’s any change in the commitments utilization rate. For example, if an instance is running as a full price on-demand instance, and another instance that was utilizing a compute savings plan commitment was terminated, the first instance will start utilizing this commitment if its hourly price rate has enough free space under this commitment. It might take a couple of minutes for this change to show, but since the billing is being calculated retroactively, in practice it’s starting to utilize the commitment right away. - +
@@ -348,7 +355,7 @@ Throughout the lifetime of an instance, it can change its “price” whenever t
- Yes, a cluster roll will override the spotinst.io/restrict-scale-down label. Nodes containing pods with the spotinst.io/restrict-scale-down label will be replaced during a cluster roll. +Yes, a cluster roll will override the spotinst.io/restrict-scale-down label. Nodes containing pods with the spotinst.io/restrict-scale-down label will be replaced during a cluster roll. Nodes can be replaced during a cluster roll even if the [instance is locked](elastigroup/features/core-features/instance-actions?id=lock-an-instance). Instance lock only protects the instance from autoscaling actions. Cluster roll is a manually triggered action that requires replacing all the cluster’s instances. @@ -361,7 +368,7 @@ Nodes can be replaced during a cluster roll even if the [instance is locked](ela
-When you’re in a cluster or group, you only see roles associated with the instance profile. +When you’re in a cluster or group, you only see roles associated with the instance profile.
@@ -380,7 +387,8 @@ Instance metadata service version 2 (IMDSv2) addresses security concerns and vul You can define metadata for autoscaling groups in AWS that gets imported when you import the groups from AWS to Spot. You can manually configure them in Spot to use IMDSv2. 1. Follow the [Ocean AWS Cluster Create](https://docs.spot.io/api/#tag/Ocean-AWS/operation/OceanAWSClusterCreate) or [Elastigroup AWS Create](https://docs.spot.io/api/#tag/Elastigroup-AWS/operation/elastigroupAwsCreate) API instructions and add this configuration for the cluster: - ````json + + ```json "compute": { "launchSpecification": { "instanceMetadataOptions": { @@ -390,11 +398,11 @@ You can define metadata for autoscaling groups in AWS that gets imported when yo } } } - ```` + ``` 2. Apply these changes to the currently running instances so the clusters are restarted and have the new definitions: - * [Deploy an Elastigroup](https://docs.spot.io/elastigroup/tutorials/elastigroup-actions-menu/deploy-or-roll-elastigroup?id=deploy-an-elastigroup) - * [Roll an Ocean cluster](https://docs.spot.io/ocean/features/roll-gen) + - [Deploy an Elastigroup](https://docs.spot.io/elastigroup/tutorials/elastigroup-actions-menu/deploy-or-roll-elastigroup?id=deploy-an-elastigroup) + - [Roll an Ocean cluster](https://docs.spot.io/ocean/features/roll-gen) **Scenario 2: Stateful Node** @@ -403,21 +411,23 @@ When a stateful managed node is imported from AWS, Spot creates an image from th You can use your own AMI and configure IMDSv2 on it. All instances launched after recycling will have IMDSv2 by default. 1. Configure IMDSv2 on your AMI: - * If you're creating a new AMI, you can add IMDSv2 support using AWS CLI: - ```` - aws ec2 register-image Let me know if there is anything else I can help you with. - --name my-image \ - --root-device-name /dev/xvda \ - --block-device-mappings DeviceName=/dev/xvda,Ebs={SnapshotId=snap-0123456789example} \ - --imds-support v2.0 - ```` - - * If you use an existing AMI, you can add IMDSv2 using AWS CLI: - ```` - aws ec2 modify-image-attribute \ - --image-id ami-0123456789example \ - --imds-support v2.0 - ```` + + - If you're creating a new AMI, you can add IMDSv2 support using AWS CLI: + + ``` + aws ec2 register-image Let me know if there is anything else I can help you with. + --name my-image \ + --root-device-name /dev/xvda \ + --block-device-mappings DeviceName=/dev/xvda,Ebs={SnapshotId=snap-0123456789example} \ + --imds-support v2.0 + ``` + + - If you use an existing AMI, you can add IMDSv2 using AWS CLI: + ``` + aws ec2 modify-image-attribute \ + --image-id ami-0123456789example \ + --imds-support v2.0 + ``` 2. In the Spot console, [create a stateful node](https://docs.spot.io/managed-instance/getting-started/create-a-new-managed-instance) with the custom AMI. @@ -432,20 +442,19 @@ You can use your own AMI and configure IMDSv2 on it. All instances launched afte When you use autoTag in CloudFormation, Spot adds these tracking tags to instances provisioned as part of the custom resource: -* `spotinst:aws:cloudformation:logical-id` -* `spotinst:aws:cloudformation:stack-name` -* `spotinst:aws:cloudformation:stack-id` +- `spotinst:aws:cloudformation:logical-id` +- `spotinst:aws:cloudformation:stack-name` +- `spotinst:aws:cloudformation:stack-id` You can see examples of autotagging in: -* [Ocean](tools-and-provisioning/cloudformation/template-structure/parameters?id=request-json-example-adding-auto-tags-to-a-kubernetes-ocean-cluster) -* [Elastigroup](tools-and-provisioning/cloudformation/template-structure/parameters?id=request-json-example-adding-auto-tags-to-elastigroup) +- [Ocean](tools-and-provisioning/cloudformation/template-structure/parameters?id=request-json-example-adding-auto-tags-to-a-kubernetes-ocean-cluster) +- [Elastigroup](tools-and-provisioning/cloudformation/template-structure/parameters?id=request-json-example-adding-auto-tags-to-elastigroup)
-
@@ -453,14 +462,13 @@ You can see examples of autotagging in:
- See the following topic: +See the following topic: - * [Startup Taints](https://docs.spot.io/ocean/features/labels-and-taints?id=startup-taints) +- [Startup Taints](https://docs.spot.io/ocean/features/labels-and-taints?id=startup-taints)
-
AWS: Why doesn’t Spot gracefully terminate instances if AWS gives a 2-minute termination notice? @@ -473,8 +481,8 @@ When AWS terminates an instance, the machine status is updated regardless of the You can get higher availability by including: -* More instance types and availability zones for the group/cluster -* Fallback to on-demand +- More instance types and availability zones for the group/cluster +- Fallback to on-demand @@ -488,8 +496,8 @@ If you change the Spot % to 0, your already running spot instances do not automa You need to: -* [Deploy an Elastigroup](elastigroup/tutorials/elastigroup-actions-menu/deploy-or-roll-elastigroup?id=deploy-an-elastigroup) -* [Roll an Ocean cluster](ocean/features/roll-gen) +- [Deploy an Elastigroup](elastigroup/tutorials/elastigroup-actions-menu/deploy-or-roll-elastigroup?id=deploy-an-elastigroup) +- [Roll an Ocean cluster](ocean/features/roll-gen) The automatic process only happens when changing the Spot % from on-demand instances to spot (fix strategy in [Elastigroup](elastigroup/features/core-features/market-scoring-managing-interruptions?id=fix-strategy), [Ocean](ocean/features/dynamic-commitments-aws)). @@ -523,16 +531,16 @@ If an instance type isn’t [EBS-optimized by default](https://docs.aws.amazon.c If you’re getting this message: -```` +``` Can't Spin Spot Instances: Message: The tag policy does not allow the specified value for the following tag key: 'XXX'. -```` +``` It means a tag defined in your Elastigroup or cluster doesn’t comply with AWS’s tag policy. 1. In the Spot console, go to: - * **Elastigroup** > **Groups** > click on the Elastigroup > **Log**. - * **Ocean** > **Cloud Clusters** > click on the cluster > **Log**. + - **Elastigroup** > **Groups** > click on the Elastigroup > **Log**. + - **Ocean** > **Cloud Clusters** > click on the cluster > **Log**. 2. Identify the problematic tag keys/values. @@ -540,8 +548,8 @@ It means a tag defined in your Elastigroup or cluster doesn’t comply with AWS 4. In the Spot console, update the tag keys/values: - * **Elastigroup** > **Groups** > click on the Elastigroup > **Actions** > **Edit Configuration** > **Compute** > **Advanced Settings**. - * **Ocean** > **Cloud Clusters** > click on the cluster > **Actions** > **Edit Cluster** > **Compute**. + - **Elastigroup** > **Groups** > click on the Elastigroup > **Actions** > **Edit Configuration** > **Compute** > **Advanced Settings**. + - **Ocean** > **Cloud Clusters** > click on the cluster > **Actions** > **Edit Cluster** > **Compute**. The instance will be launched when the tags in Spot clusters/groups comply with the tag policy defined in AWS. @@ -556,18 +564,18 @@ The instance will be launched when the tags in Spot clusters/groups comply with You can get these messages when the group or cluster is scaling up instances: -* `Can’t Spin Instances: Message: You are not authorized to perform this operation. Encoded authorization failure message` -* `Can’t Spin On-Demand Instances: Message: You are not authorized to perform this operation. Encoded authorization failure message` +- `Can’t Spin Instances: Message: You are not authorized to perform this operation. Encoded authorization failure message` +- `Can’t Spin On-Demand Instances: Message: You are not authorized to perform this operation. Encoded authorization failure message` These messages could be related to [service control policies](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) (SCP). Keep in mind, Spot doesn’t get SCP information from AWS, so doesn’t know which instance types AWS blocks because of the SCP restrictions. As a result, Spot cannot launch a new instance of a different type. 1. You need to [identify the reason for the error](https://docs.aws.amazon.com/STS/latest/APIReference/API_DecodeAuthorizationMessage.html) in AWS. 2. In the Spot console, update the instance types: - * [Ocean](ocean/tips-and-best-practices/manage-machine-types?id=opt-out-of-machine-types) - * [Elastigroup](elastigroup/features/compute/preferred-instance-types) - * [Ocean ECS cluster update API](https://docs.spot.io/api/#tag/Ocean-ECS/operation/OceanECSClusterUpdate) - * [Elastigroup AWS update API](https://docs.spot.io/api/#tag/Elastigroup-AWS/operation/elastigroupAwsUpdate) + - [Ocean](ocean/tips-and-best-practices/manage-machine-types?id=opt-out-of-machine-types) + - [Elastigroup](elastigroup/features/compute/preferred-instance-types) + - [Ocean ECS cluster update API](https://docs.spot.io/api/#tag/Ocean-ECS/operation/OceanECSClusterUpdate) + - [Elastigroup AWS update API](https://docs.spot.io/api/#tag/Elastigroup-AWS/operation/elastigroupAwsUpdate) @@ -580,23 +588,23 @@ These messages could be related to [service control policies](https://docs.aws.a You can get this message when the group or cluster is scaling up instances: -```` +``` Can't spin spot instance: Code: UnsupportedOperation, Message: The instance configuration for this AWS Marketplace product is not supported. Please see the AWS Marketplace site for more information about supported instance types, regions, and operating systems. -```` +``` This typically happens if the group/cluster AMI product doesn’t support specific instance types in the group/cluster instance list. 1. Identify the AMI: - * [Search AWS Marketplace for the AMI ID](https://aws.amazon.com/marketplace/search/results?ref_=nav_search_box&searchTerms=ami). - * **Elastigroup**: in the Spot console, go to **Elastigroup** > **Groups** > select the group > **Group Information** and click **Details** > **productCodeId**. - * **Ocean**: in the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Compute** > **Instance specifications** > **View AMI details** > **productCodeId**. + - [Search AWS Marketplace for the AMI ID](https://aws.amazon.com/marketplace/search/results?ref_=nav_search_box&searchTerms=ami). + - **Elastigroup**: in the Spot console, go to **Elastigroup** > **Groups** > select the group > **Group Information** and click **Details** > **productCodeId**. + - **Ocean**: in the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Compute** > **Instance specifications** > **View AMI details** > **productCodeId**. 2. [Troubleshoot AWS Marketplace AMIs](https://repost.aws/knowledge-center/ami-marketplace-troubleshoot). For example, check the instance types, regions, and availability zones. You can compare the instance types in AWS with the Spot console: - * **Elastigroup**: in the Spot console, go to **Elastigroup** > **Groups** > select the group > **Compute** > **Instance types**. - * **Ocean**: in the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Compute** > **Instance types**. - + - **Elastigroup**: in the Spot console, go to **Elastigroup** > **Groups** > select the group > **Compute** > **Instance types**. + - **Ocean**: in the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Compute** > **Instance types**. +
@@ -608,9 +616,9 @@ This typically happens if the group/cluster AMI product doesn’t support specif You may get this message when creating or importing an Elastigroup or cluster if you reach your AWS service quota limit for security groups per network interface: -```` +``` POST https://api.spotinst.io/aws/ec2/group?accountId=act-xxxxx: 400 (request: "xxxxx") SecurityGroupLimitExceeded: You have exceeded the number of VPC security groups allowed per instance. -```` +``` You can [request a quota increase from AWS](https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html). @@ -625,14 +633,14 @@ You can [request a quota increase from AWS](https://docs.aws.amazon.com/vpc/late You can get this log message if: -* The instance is scaled down because of AWS’s capacity. -* An instance replacement was initiated because of AWS’s capacity. A new instance is launched to replace an instance that was taken back because of AWS’s capacity. -* An instance is manually terminated in AWS. +- The instance is scaled down because of AWS’s capacity. +- An instance replacement was initiated because of AWS’s capacity. A new instance is launched to replace an instance that was taken back because of AWS’s capacity. +- An instance is manually terminated in AWS. This means that there are no [spot markets](elastigroup/features/core-features/market-scoring-managing-interruptions?id=fix-strategy) available to launch spot instances. You can add more spot markets to improve availability: -* For Elastigroup, [instance types](elastigroup/features/compute/preferred-instance-types?id=preferred-instance-types) and [availability zones](elastigroup/features/compute/preferred-availability-zones). -* For Ocean, [instance types](ocean/features/vngs/attributes-and-actions-per-vng?id=preferred-instance-types-per-virtual-node-group-aws) and [availability zones](ocean/features/avail-zones-scores?id=configure-your-availability-zones-recommendations). +- For Elastigroup, [instance types](elastigroup/features/compute/preferred-instance-types?id=preferred-instance-types) and [availability zones](elastigroup/features/compute/preferred-availability-zones). +- For Ocean, [instance types](ocean/features/vngs/attributes-and-actions-per-vng?id=preferred-instance-types-per-virtual-node-group-aws) and [availability zones](ocean/features/avail-zones-scores?id=configure-your-availability-zones-recommendations). @@ -645,9 +653,9 @@ This means that there are no [spot markets](elastigroup/features/core-features/m You can get this message if the key pair is missing or not valid: -```` +``` Can't Spin On-Demand Instances: Code: InvalidKeyPair.NotFound, Message: The key pair 'xxxxx' does not exist -```` +``` Update the key pair: @@ -690,8 +698,8 @@ One of the reasons this can happen is if you’re using enhanced networking and Yes, you can connect using SSH to a VM running: -* [Linux](https://learn.microsoft.com/en-us/azure/virtual-machines/linux-vm-connect?tabs=Linux) -* [Windows](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/connect-ssh?tabs=azurecli) +- [Linux](https://learn.microsoft.com/en-us/azure/virtual-machines/linux-vm-connect?tabs=Linux) +- [Windows](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/connect-ssh?tabs=azurecli) @@ -704,9 +712,9 @@ Yes, you can connect using SSH to a VM running: You got this error in the logs, and it’s not possible for the cluster to perform any scaling actions: -```` +``` Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app -```` +``` In Azure Kubernetes Service (AKS), there are two kinds of secrets: client secret ID and client secret value. @@ -729,12 +737,12 @@ Cooldown is set at the cluster level and is applied across all virtual node grou You can set the cooldown period: -* In the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Review** > **JSON** > **Edit Mode**. -* Using the APIs: +- In the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Edit Cluster** > **Review** > **JSON** > **Edit Mode**. +- Using the APIs: - * [Ocean AWS cluster update](https://docs.spot.io/api/#tag/Ocean-AWS/operation/OceanAWSClusterUpdate) - * [Ocean ECS cluster update](https://docs.spot.io/api/#tag/Ocean-ECS/operation/OceanECSClusterUpdate) - * [Ocean GKE cluster update](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterUpdate) + - [Ocean AWS cluster update](https://docs.spot.io/api/#tag/Ocean-AWS/operation/OceanAWSClusterUpdate) + - [Ocean ECS cluster update](https://docs.spot.io/api/#tag/Ocean-ECS/operation/OceanECSClusterUpdate) + - [Ocean GKE cluster update](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterUpdate) @@ -751,7 +759,6 @@ Cluster roll randomly chooses the nodes and divides the instances between batche -
ECS, EKS: How do I create spot interruption notifications? @@ -782,16 +789,17 @@ You can use AWS EventBridge to send spot interruption warnings to the Spot platf This message is shown in the console logs if Ocean attempts to scale up a certain spot instance type in a particular availability zone. This happens because of a lack of capacity on the AWS side. -```` +``` Can't Spin Spot Instances: Code: InsufficientInstanceCapacity, Message: We currently do not have sufficient m5.2xlarge capacity in the Availability Zone you requested (us-east-1a). Our system will be working on provisioning additional capacity. You can currently get m5.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-1b, us-east-1c, us-east-1d, us-east-1f. -```` +``` Ocean is aware of a pending pod and is spinning up an instance. Based on your current instance market, Ocean chooses the instance type in a particular availability zone and attempts to scale up. If it fails due to a lack of capacity, the error message is shown in the console logs. You can solve this by: -* Having many instance types so Ocean can choose the best available markets. -* Having multiple availability zones to provide more availability. -* For workloads that are not resilient to disruptions, configure the [on demand label](https://docs.spot.io/ocean/features/labels-and-taints?id=spotinstionode-lifecycle) `spotinst.io/node-lifecycle`. + +- Having many instance types so Ocean can choose the best available markets. +- Having multiple availability zones to provide more availability. +- For workloads that are not resilient to disruptions, configure the [on demand label](https://docs.spot.io/ocean/features/labels-and-taints?id=spotinstionode-lifecycle) `spotinst.io/node-lifecycle`. @@ -804,27 +812,24 @@ You can solve this by: You have scaling up instances for your Elastigroup or Ocean clusters and you get this message: -```` +``` ERROR, Can't Spin Instances: Code: InvalidSnapshot.NotFound, Message: The snapshot 'snap-xyz' does not exist. -```` +``` If you have a block device that is mapped to a snapshot ID of an Elastigroup or Ocean cluster and the snapshot isn't available, you will get this error. This can happen if the snapshot is deleted. cant-spin-instances-invalidsnapshot1 - If you have another snapshot, then you can use that snapshot ID for the block device mapping. If not, you can remove the snapshot ID, and then the instance is launched using the AMI information. -* **Elastigroup**: on the Elastigroup you want to change, [open the creation wizard](https://docs.spot.io/elastigroup/features/compute/block-device-mapping) and update the snapshot ID. +- **Elastigroup**: on the Elastigroup you want to change, [open the creation wizard](https://docs.spot.io/elastigroup/features/compute/block-device-mapping) and update the snapshot ID. cant-spin-instances-invalidsnapshot2 - -* **Ocean**: on the virtual node group you want to change, update the snapshot ID. +- **Ocean**: on the virtual node group you want to change, update the snapshot ID. cant-spin-instances-invalidsnapshot3 -
@@ -834,17 +839,16 @@ If you have another snapshot, then you can use that snapshot ID for the block de
- You can get this message if AWS's spot service limit is reached: - - ```` - Can't Spin Spot Instances:Code: MaxSpotInstanceCountExceeded, Message: Max spot instance count exceeded -```` +You can get this message if AWS's spot service limit is reached: + +``` +Can't Spin Spot Instances:Code: MaxSpotInstanceCountExceeded, Message: Max spot instance count exceeded +``` You may also get an email from Spot: Spot Proactive Monitoring | Max Spot Instance Count Exceeded. This email includes instructions for opening a support request with AWS, such as the instance type and region that triggered the error. You can read the AWS documentation on [spot instance quotas](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-limits.html). -
@@ -856,12 +860,12 @@ You can read the AWS documentation on [spot instance quotas](https://docs.aws.am You can get this error when the group's device name (for Block Device Mapping) and the AMI's device name do not match: -```` +``` Can't Spin Spot Instance: Code: InvalidBlockDeviceMapping, Message: The device 'xvda' is used in more than one block-device mapping -```` +``` -* AMI - "deviceName": "xvda" -* Group's configuration - "deviceName": "/dev/xvda" +- AMI - "deviceName": "xvda" +- Group's configuration - "deviceName": "/dev/xvda" Change the device name from `xvda` to `/dev/xvda` on the group's side. In the stateful node, go to **Actions** > **Edit Configuration** > **Review** > **JSON** > **Edit Mode**. Change the device name from `xvda` to `/dev/xvda` and click **Update**. @@ -874,17 +878,17 @@ Change the device name from `xvda` to `/dev/xvda` on the group's side. In the st
-When you import Fargate services with more than 5 security groups, you get an error: +When you import Fargate services with more than 5 security groups, you get an error: -```` +``` Failed to import Fargate services into Ocean. Please verify Spot IAM policy has the right permissions and try again. -```` +``` In Spot, you see this warning: -```` +``` Fargate import failed for xxx-xxxxxx, due to Failed to import services, too many security groups. Import less services to this group (Group ID: xxxx-xxxxxx). -```` +``` You can have up to 5 security groups in each service according to this [article](https://spot.io/blog/import-ecs-fargate-into-spot-ocean/#:~:text=more%20than%20five-,security,-groups%20as%20only). This means that if more than 5 security groups are defined in one of the services, the import doesn’t succeed. @@ -915,7 +919,6 @@ Yes, you can launch an instance with a specific launch specification or virtual When a new instance is launched, it will be from the dedicated virtual node group. -
@@ -925,10 +928,10 @@ When a new instance is launched, it will be from the dedicated virtual node grou
- An ECS cluster launches an instance just for a single task, even when there is capacity on the nodes currently running in the cluster. This can happen if a task has placement constraints called distinctInstance, which causes each task in the group to run on its own instance. - - You can [define which container instances Amazon ECS uses for tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html). The placementConstraints may be defined in one of these actions [CreateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html), [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html), and/or [RunTask](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html). - +An ECS cluster launches an instance just for a single task, even when there is capacity on the nodes currently running in the cluster. This can happen if a task has placement constraints called distinctInstance, which causes each task in the group to run on its own instance. + +You can [define which container instances Amazon ECS uses for tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html). The placementConstraints may be defined in one of these actions [CreateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html), [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html), and/or [RunTask](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html). +
@@ -938,20 +941,20 @@ When a new instance is launched, it will be from the dedicated virtual node grou
- If your virtual node group has more on-demand instances than defined, your extra instances are reverted to spot instances when they become available. This is called the fix strategy. +If your virtual node group has more on-demand instances than defined, your extra instances are reverted to spot instances when they become available. This is called the fix strategy. If you see this message in the log: -```` +``` DEBUG, Replacement of type Out of strategy for instance i-xxx has been canceled. Reason for cancelation: Instance contains stand-alone tasks, and the group's configuration doesn't allow termination of stand-alone tasks. -```` +``` It means that your strategy cannot be fixed and your spot instances cannot be reverted to spot instances. This is because you have standalone tasks in the instances, and the group's configuration can't stop standalone tasks. The autoscaler cannot scale down these instances. Update the cluster [in the API](https://docs.spot.io/api/#tag/Ocean-ECS/operation/OceanECSClusterUpdate) or in the cluster's JSON file to include `"shouldScaleDownNonServiceTasks": true`. The standalone task and instance are terminated and are not redeployed because they weren't created as part of a service. - +
@@ -963,11 +966,11 @@ The standalone task and instance are terminated and are not redeployed because t Your container instances may be unregistered if the newly launched Ocean ECS container instance: -* Has unregistered contain instance events -* Doesn’t have a Container Instance ID -* Is eventually scaled down -* CPU and memory resource allocations are 0% -* Status: Can’t determine +- Has unregistered contain instance events +- Doesn’t have a Container Instance ID +- Is eventually scaled down +- CPU and memory resource allocations are 0% +- Status: Can’t determine @@ -980,42 +983,41 @@ Your container instance must be registered with an ECS cluster. If the container If your container is unregistered, you should make sure: -* **User Data** - +- **User Data** + 1. Go to the cluster in the Spot console and click **Actions** > **Edit Configuration** > **Compute**. 2. Add this script to **User Data**, using your cluster name. - ```` - #!/bin/bash - echo ECS_CLUSTER="xxxxx" >> /etc/ecs/ecs.config - ```` - -* **AMI** + ``` + #!/bin/bash + echo ECS_CLUSTER="xxxxx" >> /etc/ecs/ecs.config + ``` + +- **AMI** ECS is optimized and Agent (similar to the controller in Kubernetes) is configured in the AMI. - -* **Security group and specific ports** - * **Port 22 (SSH)** is required if you want to connect to your container instances using Secure Shell (SSH) for troubleshooting or maintenance. + +- **Security group and specific ports** + + - **Port 22 (SSH)** is required if you want to connect to your container instances using Secure Shell (SSH) for troubleshooting or maintenance. It is not directly related to ECS cluster registration, but it's commonly included for administrative access to the instances. - * **Port 2375 (TCP)** is used for the ECS container agent to communicate with the ECS control plane. It allows the agent to register the container instance with the cluster, send heartbeats, and receive instructions for task placement and management. - * **Port 2376 (TCP)** is used for secure communication between the ECS container agent and the ECS control plane. It enables encrypted communication and is recommended for improved security when managing your ECS cluster. + - **Port 2375 (TCP)** is used for the ECS container agent to communicate with the ECS control plane. It allows the agent to register the container instance with the cluster, send heartbeats, and receive instructions for task placement and management. + - **Port 2376 (TCP)** is used for secure communication between the ECS container agent and the ECS control plane. It enables encrypted communication and is recommended for improved security when managing your ECS cluster. -* **IAM role** +- **IAM role** Configure an instance profile with relevant permissions. unregistered-container-instance3 -* **IP** +- **IP** Make sure you configured Public IP according to subnet, and have NAT gateway. If you change the configuration in the virtual node group, such as tags/user data, it immediately overrides the cluster's configuration. unregistered-container-instance4 - -* [AWS troubleshooting](https://aws.amazon.com/premiumsupport/knowledge-center/ecs-instance-unable-join-cluster/) - +- [AWS troubleshooting](https://aws.amazon.com/premiumsupport/knowledge-center/ecs-instance-unable-join-cluster/) @@ -1026,8 +1028,8 @@ If your container is unregistered, you should make sure:
- A non-service task is a standalone task that isn't part of a service. It's typically used for batch processing or one-time jobs rather than continuous, long-running services. When an independent task runs in a cluster, and there aren't enough resources available, the task may fail to launch due to CPU or memory errors. This means that no service is continuously attempting to launch tasks to meet the required number of tasks. Instead, the task will be launched later when resources become available. - +A non-service task is a standalone task that isn't part of a service. It's typically used for batch processing or one-time jobs rather than continuous, long-running services. When an independent task runs in a cluster, and there aren't enough resources available, the task may fail to launch due to CPU or memory errors. This means that no service is continuously attempting to launch tasks to meet the required number of tasks. Instead, the task will be launched later when resources become available. +
@@ -1065,18 +1067,18 @@ As a result, the new instances have auto-assign public IP disabled. Headroom can only be scheduled if there are enough instance types. If you’re using [manual headroom](ocean/features/headroom?id=manual-headroom) and there aren’t enough instance types, you may get this message: -```` +``` WARN, AutoScaler - Attempt Scale Up, Task service:spotinst-headroom-task-ols-e72002a2-4 is pending but could not find any applicable instance type to scale up in order to schedule the pending Task. -```` +``` You can: -* Add more [instance types](ocean/features/vngs/attributes-and-actions-per-vng?id=preferred-spot-instance-types) (bigger instance types) to the virtual node group, which gives Ocean more options to choose from. This can reduce your costs. -* Decrease the **Reserve**, **CPU**, **Memory**: +- Add more [instance types](ocean/features/vngs/attributes-and-actions-per-vng?id=preferred-spot-instance-types) (bigger instance types) to the virtual node group, which gives Ocean more options to choose from. This can reduce your costs. +- Decrease the **Reserve**, **CPU**, **Memory**: - 1. In the Spot console, go to **Ocean** > **Cloud Clusters** and select the cluster. - 2. On the Virtual Node Groups tab, click on the virtual node group. - 3. Go to **Advanced** > **Headroom** and update the **Reserve**, **CPU**, and/or **Memory**. + 1. In the Spot console, go to **Ocean** > **Cloud Clusters** and select the cluster. + 2. On the Virtual Node Groups tab, click on the virtual node group. + 3. Go to **Advanced** > **Headroom** and update the **Reserve**, **CPU**, and/or **Memory**. @@ -1089,9 +1091,9 @@ You can: You may get this message if you create a custom virtual node group and then change the AMI: -```` +``` error: The Virtual Node Group’s architecture doesn’t match the Virtual Node Group Template filter. -```` +``` This can happen if the new AMI architecture does not support the instance types set in the default virtual node group. @@ -1121,34 +1123,35 @@ If a node only has one task running, then it causes the node to be underutilized Example service: -````json +```json "placementConstraints": [], "placementStrategy": [], -```` +``` The task definition doesn't have constraints to spread tasks across nodes. -````json +```json "placementConstraints": [ { "type": "memberOf", "expression": "attribute:nd.type == default" } ], -```` +``` Check the **portMappings: hostPort** value in the task/service defintion. Port mappings allow containers to access ports on the host container instances to send or receive traffic. This configuration can be found in the task definition. The hostPort value in port mapping is normally left blank or set to 0. Example: -````json + +```json "portMappings": [ { "protocol": "tcp", "hostPort": 0, "containerPort": 443 -```` +``` However, if the hostPort value equals the containerPort value, then each task needs its own container. Any pending tasks trigger a scale-up, and the number of launched instances is equal to the number of pending tasks. This leads to underutilized instances and higher costs. @@ -1195,7 +1198,7 @@ You can also [push the ECS agent logs to CloudWatch](https://docs.aws.amazon.com
- You can safely disconnect Ocean from an existing EKS Cluster: +You can safely disconnect Ocean from an existing EKS Cluster: 1. Increase the number of instances in the ASG attached to the EKS cluster. This way, the pods that run on the nodes managed by Spot will be able to reschedule on the new instances and avoid downtime. 2. In the Spot console, go to **Ocean** > **Cloud Clusters**, and select the cluster. @@ -1206,9 +1209,10 @@ You can also [push the ECS agent logs to CloudWatch](https://docs.aws.amazon.com oceandisconnectcluster The instances managed by Ocean will be detached and the pods will be rescheduled on the new instances launched by AWS ASG. + 6. In the Spot console, go to **Ocean** > **Cloud Clusters**, and select the cluster. 7. Click **Actions** > **Delete**. - +
@@ -1226,7 +1230,6 @@ Add a [node group to your EKS cluster](https://docs.aws.amazon.com/eks/latest/us -
EKS: How can I get the AMI ID for EKS-optimized Amazon Linux? @@ -1251,15 +1254,14 @@ Update your tolerances in the DaemonSet YAML so you can schedule DaemonSet pods For example, you can update your [DaemonSet pod YAML](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to include: -````json +```json spec: tolerations: - key: dedicated operator: Equal value: statefulset effect: NoSchedule -```` - +``` @@ -1274,7 +1276,7 @@ Yes, you can use `autoScaler: resourceLimits: maxInstanceCount: 10` to set capac For example: -````json +```json apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: @@ -1287,7 +1289,7 @@ name: ng1 resourceLimits: maxInstanceCount: 10 # ... -```` +``` @@ -1301,18 +1303,19 @@ name: ng1 Kubernetes nodes in the cluster have Unhealthy status—the node has a Node Name but the Kubernetes status is Unhealthy. You can debug unhealthy Kubernetes nodes: -* Check the nodes' status by running this command in CLI: `kubectl get nodes` - Look for nodes in a NotReady or Unknown state. This indicates that the nodes are unhealthy or experiencing issues. -* Get detailed information about the problematic nodes by running the `kubectl describe` command: `kubectl describe node `. - Look for any error messages or warnings that can help identify the problem. Pay attention to resource allocation issues, network connectivity problems, or other relevant information. -* Verify the health of cluster components such as the kubelet, kube-proxy, and container runtime (for example, Docker, containerd). Check your local logs and the status of these components to identify any errors or issues. -* Examine the resource utilization of your nodes, including CPU, memory, and disk usage. High resource utilization can lead to node instability or unresponsiveness. Use tools like Prometheus or Grafana to monitor resource metrics. -* Ensure that network connectivity is properly configured and functioning between the Kubernetes control plane and the nodes. Verify that nodes can reach each other and communicate with external services. -* Use the `kubectl get events` command to check for cluster-level events that might provide insights into the node health issues. Events often contain helpful information about the state of your cluster and its components. -* Examine the logs of individual pods running on the problematic nodes. Logs can provide clues about any application-specific issues or errors that might be impacting node health. Use the `kubectl logs` command to retrieve pod logs. -* Verify that the node configurations (for example, kubelet configuration, network settings) are correct and aligned with the cluster requirements. -* Ensure that the container runtime (such as Docker, containerd) is properly installed and functioning on the nodes. Check the runtime logs for any errors or warnings. -* If you think that a specific component is causing the node health issues, consider updating or reinstalling that component to resolve any known bugs or conflicts. + +- Check the nodes' status by running this command in CLI: `kubectl get nodes` + Look for nodes in a NotReady or Unknown state. This indicates that the nodes are unhealthy or experiencing issues. +- Get detailed information about the problematic nodes by running the `kubectl describe` command: `kubectl describe node `. + Look for any error messages or warnings that can help identify the problem. Pay attention to resource allocation issues, network connectivity problems, or other relevant information. +- Verify the health of cluster components such as the kubelet, kube-proxy, and container runtime (for example, Docker, containerd). Check your local logs and the status of these components to identify any errors or issues. +- Examine the resource utilization of your nodes, including CPU, memory, and disk usage. High resource utilization can lead to node instability or unresponsiveness. Use tools like Prometheus or Grafana to monitor resource metrics. +- Ensure that network connectivity is properly configured and functioning between the Kubernetes control plane and the nodes. Verify that nodes can reach each other and communicate with external services. +- Use the `kubectl get events` command to check for cluster-level events that might provide insights into the node health issues. Events often contain helpful information about the state of your cluster and its components. +- Examine the logs of individual pods running on the problematic nodes. Logs can provide clues about any application-specific issues or errors that might be impacting node health. Use the `kubectl logs` command to retrieve pod logs. +- Verify that the node configurations (for example, kubelet configuration, network settings) are correct and aligned with the cluster requirements. +- Ensure that the container runtime (such as Docker, containerd) is properly installed and functioning on the nodes. Check the runtime logs for any errors or warnings. +- If you think that a specific component is causing the node health issues, consider updating or reinstalling that component to resolve any known bugs or conflicts. @@ -1335,12 +1338,12 @@ If there is no active migration, after the configured unhealthy duration ends (t
- Ocean doesn't actually have a horizontal pod autoscaling (HPA) policy. The HPA is essentially operating on the Kubernetes side so Ocean itself doesn't have an HPA. +Ocean doesn't actually have a horizontal pod autoscaling (HPA) policy. The HPA is essentially operating on the Kubernetes side so Ocean itself doesn't have an HPA. The cluster autoscaler only takes care of provisioning the required number of nodes. Essentially, if the load increases on your cluster, then Kubernetes will create more replicas, and Ocean will launch nodes for the new pods. Kubernetes HPA will create pods and Ocean will launch new nodes for pods to be scheduled. - +
@@ -1353,15 +1356,17 @@ Essentially, if the load increases on your cluster, then Kubernetes will create AWS node termination handler is a DaemonSet pod that is deployed on each spot instance. It detects the instance termination notification signal so that there will be a graceful termination of any pod running on that node, drain from load balancers, and redeploy applications elsewhere in the cluster. AWS node termination handler makes sure that the Kubernetes control plane responds as it should to events that can cause EC2 instances to become unavailable. Some reasons EC2 instances may become unavailable include: -* EC2 maintenance events -* EC2 spot interruptions -* ASG scale-in -* ASG AZ rebalance -* EC2 instance termination using the API or Console + +- EC2 maintenance events +- EC2 spot interruptions +- ASG scale-in +- ASG AZ rebalance +- EC2 instance termination using the API or Console If not handled, the application code may not stop gracefully, take longer to recover full availability, or accidentally schedule work to nodes going down. The workflow of the node termination handler DaemonSet is: + 1. Identify that a spot instance is being reclaimed. 2. Use the 2-minute notification window to prepare the node for graceful termination. 3. Taint the node and cordon it off to prevent new pods from being placed. @@ -1370,7 +1375,7 @@ The workflow of the node termination handler DaemonSet is: Ocean does not conflict with aws-node-termination-handler. It is possible to install it, but using aws-node-termination-handler is not required. Ocean continuously analyzes how your containers use infrastructure, automatically scaling compute resources to maximize utilization and availability. Ocean ensures that the cluster resources are utilized and scales down underutilized nodes to optimize maximal cost. - + @@ -1381,7 +1386,7 @@ Ocean ensures that the cluster resources are utilized and scales down underutili
The JSON for a virtual node group has all the parameters from the Ocean template/default virtual node group. Any items you haven’t defined yet have a value of null. This way, you can edit the existing parameters. - +
@@ -1400,7 +1405,6 @@ You can update this line in the SDK to debug: -
EKS: Why can’t I see EKS clusters in Ocean in the Spot console when I’m importing to Ocean? @@ -1408,11 +1412,11 @@ You can update this line in the SDK to debug: When [importing EKS clusters to Ocean](ocean/getting-started/eks/join-an-existing-cluster) in the Spot console, some of your clusters may not show in the list you can import from. Make sure: -* The EKS cluster is in the region you’re trying to import from. -* You have the [correct permissions](ocean/getting-started/eks/join-an-existing-cluster?id=add-required-permissions) and the most [current Spot policy](administration/api/spot-policy-in-aws). -* The Kubernetes cluster has an EKS version that is [supported by Amazon](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Spot supports an EKS version two months after the Amazon [EKS release date](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). A version is considered deprecated for Spot when Amazon [ends standard support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). A version is considered retired for Spot when Amazon [ends extended support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). -* You have at least one node group in your EKS cluster. There don’t need to be any nodes running in the node group, just configured in the AWS console. -* If you’re using [ASG](ocean/tutorials/manage-virtual-node-groups?id=create-a-vng-from-an-asg) in your EKS cluster, you need to import the EKS cluster using the legacy design: +- The EKS cluster is in the region you’re trying to import from. +- You have the [correct permissions](ocean/getting-started/eks/join-an-existing-cluster?id=add-required-permissions) and the most [current Spot policy](administration/api/spot-policy-in-aws). +- The Kubernetes cluster has an EKS version that is [supported by Amazon](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Spot supports an EKS version two months after the Amazon [EKS release date](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). A version is considered deprecated for Spot when Amazon [ends standard support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). A version is considered retired for Spot when Amazon [ends extended support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar). +- You have at least one node group in your EKS cluster. There don’t need to be any nodes running in the node group, just configured in the AWS console. +- If you’re using [ASG](ocean/tutorials/manage-virtual-node-groups?id=create-a-vng-from-an-asg) in your EKS cluster, you need to import the EKS cluster using the legacy design:
  1. In the Spot console, go to Ocean > Cloud Clusters > Create Cluster.
    2. @@ -1475,13 +1479,13 @@ Ocean Insights is intended for unmanaged clusters. If you get a `Maximum Pods configuration reached` message for a node in the console: -* It usually means that you reached the EKS [maximum pod limit](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/shared/runtime/eni-max-pods.txt). For example, the EKS maximum pod limit for r4.large is 29. +- It usually means that you reached the EKS [maximum pod limit](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/shared/runtime/eni-max-pods.txt). For example, the EKS maximum pod limit for r4.large is 29. - You can [increase the EKS maximum pods](https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/) in AWS. You can see more information about the number of pods per EKS instance on [Stack Overflow](https://stackoverflow.com/questions/57970896/pod-limit-on-node-aws-eks#:~:text=For%20t3.,22%20pods%20in%20your%20cluster). + You can [increase the EKS maximum pods](https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/) in AWS. You can see more information about the number of pods per EKS instance on [Stack Overflow](https://stackoverflow.com/questions/57970896/pod-limit-on-node-aws-eks#:~:text=For%20t3.,22%20pods%20in%20your%20cluster). -* If the node has fewer pods than the EKS maximum pod limit, then check if the max-pods limit is set at the user data level in the Ocean configuration. +- If the node has fewer pods than the EKS maximum pod limit, then check if the max-pods limit is set at the user data level in the Ocean configuration. - Increase this limit for the user data in Ocean: + Increase this limit for the user data in Ocean:
    1. Go to the cluster in the Spot console and click Actions > Edit Configuration > Compute.
      2. @@ -1493,7 +1497,7 @@ If you get a `Maximum Pods configuration reached` message for a node in the cons
    2. Roll the cluster.
    - If you continue to get this error, [roll the cluster](ocean/features/roll-gen) again and disable [Respect Pod Disruption Budget (PDB)](ocean/features/roll-gen?id=respect-pod-disruption-budget). You can also manually terminate the node. + If you continue to get this error, [roll the cluster](ocean/features/roll-gen) again and disable [Respect Pod Disruption Budget (PDB)](ocean/features/roll-gen?id=respect-pod-disruption-budget). You can also manually terminate the node. @@ -1509,7 +1513,7 @@ You may get an Invalid IAMInstanceProfile error when you're [creating an If you want to use IAMInstanceProfileName in Terraform, set use_as_template_only to true. Once the cluster is configured to use the default virtual node group as a template, IAMInstanceProfileName can be used instead of Invalid IAMInstanceProfile. - +
@@ -1521,15 +1525,16 @@ Once the cluster is configured to use the default virtual node group as a templa If you have unregistered nodes and are getting log messages such as: -```` +``` /var/lib/cloud/instance/scripts/part-001: line 5: unexpected EOF while looking for matching `"' - + /var/lib/cloud/instance/scripts/part-001: line 9: syntax error: unexpected end of file Feb 01 14:03:05 cloud-init[2517]: util.py[WARNING]: Running module scripts-user () failed -```` +``` Make sure: + 1. The parameters are configured correctly (such as labels, AMI, IP, user data). 2. The user data script is executable and working properly. @@ -1544,21 +1549,21 @@ Make sure: You may get this message in Kubernetes: -```` -Failed to create pod sandbox: rpc error: code = Unknown desc = +``` +Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "xxxxx" -network for pod "coreservice-xxxxx": -networkPlugin cni failed to set up pod "coreservice-xxxxx" +network for pod "coreservice-xxxxx": +networkPlugin cni failed to set up pod "coreservice-xxxxx" network: add cmd: failed to assign an IP address to container -```` +``` -Each node on Kubernetes has a [different number of elastic network interfaces (ENI) available](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt). For example, M5.Large can only have 29+2*31 ENIs. +Each node on Kubernetes has a [different number of elastic network interfaces (ENI) available](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt). For example, M5.Large can only have 29+2\*31 ENIs. You can create a script to dynamically calculate the `--max-pods` value based on the instance type and CNI version. For example: -```` +``` CNI_VERSION= MAX_PODS=$(/etc/eks/max-pods-calculator.sh --instance-type-from-imds --cni-version $CNI_VERSION) -```` +``` `--instance-type-from-imds` gets the instance type from the instance metadata service (IMDS). @@ -1579,14 +1584,13 @@ Defining a static value for `--max-pods` in the user data startup script for a v You can: -* [Download the Spot provider plugin](tools-and-provisioning/terraform/getting-started/install-terraform) and update it. -* [Update the plugin from Terraform](tools-and-provisioning/terraform/getting-started/install-terraform#update-terraform-provider). +- [Download the Spot provider plugin](tools-and-provisioning/terraform/getting-started/install-terraform) and update it. +- [Update the plugin from Terraform](tools-and-provisioning/terraform/getting-started/install-terraform#update-terraform-provider). -
GKE: How do zones and regions work with clusters? @@ -1598,8 +1602,8 @@ In Spot, when you import a regional cluster, the cluster is not integrate Keep in mind: -* The control planes are managed in GKE and are replicated when a regional cluster is selected. This gives you high reliability in the control planes. -* Ocean autoscaler chooses the best markets available for the pending pods. Ocean quickly launches instances in a different zone if there's a zonal outage. +- The control planes are managed in GKE and are replicated when a regional cluster is selected. This gives you high reliability in the control planes. +- Ocean autoscaler chooses the best markets available for the pending pods. Ocean quickly launches instances in a different zone if there's a zonal outage. @@ -1614,8 +1618,8 @@ You can set up committed use discounts (CUDs) for clusters in Ocean and groups i Set up committed use discounts for: -* [Ocean](ocean/features/committed-use-discount) -* [Elastigroup](elastigroup/features/gcp/commit-use-discount) +- [Ocean](ocean/features/committed-use-discount) +- [Elastigroup](elastigroup/features/gcp/commit-use-discount) @@ -1638,16 +1642,18 @@ Set up committed use discounts for:
Some of the common reasons your GKE nodes can be unregistered are if: -* You have shielded nodes. [Shutdown hours](ocean/features/running-hours?id=scaling-behavior-ocean-for-kubernetes) are not supported for GKE clusters with shielded nodes. If you use shutdown hours with shielded nodes, make sure that the Ocean controller is available at the end of the off time by checking that it runs on a node that Ocean does not manage. This is because the controller is part of the node registration process and requires an available node to run on. -* The cluster is in a private network. You need to configure NAT gateway on the cluster in GKE so it’ll have access to the internet. + +- You have shielded nodes. [Shutdown hours](ocean/features/running-hours?id=scaling-behavior-ocean-for-kubernetes) are not supported for GKE clusters with shielded nodes. If you use shutdown hours with shielded nodes, make sure that the Ocean controller is available at the end of the off time by checking that it runs on a node that Ocean does not manage. This is because the controller is part of the node registration process and requires an available node to run on. +- The cluster is in a private network. You need to configure NAT gateway on the cluster in GKE so it’ll have access to the internet. Make sure the cluster has external-nat and ONE_TO_ONE_NAT set: - * In the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Action** > **Edit Cluster** > **Review** > **JSON** - * In the [API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterGet) + - In the Spot console, go to **Ocean** > **Cloud Clusters** > select the cluster > **Action** > **Edit Cluster** > **Review** > **JSON** + - In the [API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterGet) + + For example: - For example: -````json +```json "compute": { "networkInterfaces": [ { @@ -1667,7 +1673,7 @@ Some of the common reasons your GKE nodes can be unregistered are if: "projectId": "projectId" } ], -```` +```
@@ -1697,9 +1703,9 @@ Ocean then detects the pending pods and launches virtual node groups for the nod If Ocean isn’t launching a VM, you might get this log message: -```` +``` Can’t Spin Instance: Name: sin-abcd. Code: Error, Message: Invalid resource usage: 'Requested boot disk architecture (X86_64) is not compatible with machine type architecture (ARM64).' -```` +``` This can happen because Ocean doesn’t validate VM architecture for GCP. You can [troubleshoot this error](https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-arm-vms#errors_when_updating_vms) in GCP. @@ -1714,10 +1720,10 @@ This can happen because Ocean doesn’t validate VM architecture for GCP. You ca You may get this log message when a VM is trying to scale up or launch VMs: -```` +``` Can't Spin Instance: Name: abcde. Code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, Message: The zone 123 does not have enough resources available to fulfill the request, '(resource type:compute)'. -```` +``` This can happen if the specific VM family and size aren’t available for a certain zone at the moment. Elastigroup or Ocean will try to automatically spin up a different VM in a different zone to compensate. @@ -1732,15 +1738,16 @@ This can happen if the specific VM family and size aren’t available for a cert If you update the Kubernetes version and pods launch with the old version, you may get these errors: -* `ERROR, Failed to update the launchSpec ols-f775236b with the latest changes in GKE cluster tagging-stg-eu1-1. Reason: Node pool tagging-stg-eu1-1-pool does not exist.` +- `ERROR, Failed to update the launchSpec ols-f775236b with the latest changes in GKE cluster tagging-stg-eu1-1. Reason: Node pool tagging-stg-eu1-1-pool does not exist.` -* `ERROR, Failed to update the group with the latest changes in GKE cluster tagging-stg-eu1-1. Reason: Node pool tagging-stg-eu1-1-pool does not exist.` +- `ERROR, Failed to update the group with the latest changes in GKE cluster tagging-stg-eu1-1. Reason: Node pool tagging-stg-eu1-1-pool does not exist.` This can happen if the original node pool is deleted, which prevents Ocean from fetching/updating the new GKE configuration. In the future, [preserve the original node pool](ocean/getting-started/gke?id=preserve-original-node-pool) instead of deleting it. To resolve the errors, you can either: -* [Create a new node pool](https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools) with the original pool name. It doesn’t need to run any nodes. -* Delete the cluster in the Spot console (Actions > Delete Cluster) or using [the Spot API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterDelete), then import the cluster in the [Spot console](ocean/getting-started/gke) or using the [Spot API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/reImportGke). + +- [Create a new node pool](https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools) with the original pool name. It doesn’t need to run any nodes. +- Delete the cluster in the Spot console (Actions > Delete Cluster) or using [the Spot API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/OceanGKEClusterDelete), then import the cluster in the [Spot console](ocean/getting-started/gke) or using the [Spot API](https://docs.spot.io/api/#tag/Ocean-GKE/operation/reImportGke). Every 30 minutes, [an automatic process](ocean/features/auto-update-process-gke) runs to update the GKE configuration in the control plane manager. You can [trigger the process manually](https://docs.spot.io/api/#tag/Ocean-GKE/operation/reImportGke). @@ -1755,9 +1762,9 @@ Every 30 minutes, [an automatic process](ocean/features/auto-update-process-gke) You can get this message if the instance type is not compatible with the boot disk type: -```` +``` ERROR, Can't Spin Instance: Name: sin-xxxx. Code: Error, Message: [pd-standard] features and [instance_type: VIRTUAL_MACHINE family: COMPUTE_OPTIMIZED generation: GEN_3 cpu_vendor: INTEL architecture: X86_64 ] InstanceTaxonomies are not compatible for creating instance. -```` +``` [Compare the machine family](https://cloud.google.com/compute/docs/machine-resource#machine_type_comparison) and PD-standard disk type to decide which is appropriate for your workload. @@ -1774,9 +1781,9 @@ Contact support to decide on the selected instance type for launching and to rem You may get this message when scaling up instances: -```` +``` ERROR, Can't Spin Instance: Name: sin-xxxxx. Code: QUOTA_EXCEEDED, Message: Quota 'M1_CPUS' exceeded. Limit: 0.0 in region us-east4 -```` +``` GCP has [allocation quotas](https://cloud.google.com/compute/resource-usage), which limit the number of resources that your project has access to. The limit is per region. @@ -1793,8 +1800,8 @@ The prefix in some of the [machine names changed from n1 to m1](https://cloud.go
- 1. [Change the cgroup_mode in the GKE node pool](https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options). - 2. [Reimport the cluster configuration to Ocean](https://docs.spot.io/api/#tag/Ocean-GKE/operation/reImportGke) (or [roll the cluster/virtual node group](ocean/features/roll-gen?id=roll-per-node-or-vng) for all nodes so they have the latest changes). +1. [Change the cgroup_mode in the GKE node pool](https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options). +2. [Reimport the cluster configuration to Ocean](https://docs.spot.io/api/#tag/Ocean-GKE/operation/reImportGke) (or [roll the cluster/virtual node group](ocean/features/roll-gen?id=roll-per-node-or-vng) for all nodes so they have the latest changes).
@@ -1824,11 +1831,12 @@ For example, if you have a 600 second terminationGracePeriodSeconds, make sure y
- When the `useAsTemplateOnly` parameter is true, you cannot edit the target capacity in the Ocean cluster configuration. - +When the `useAsTemplateOnly` parameter is true, you cannot edit the target capacity in the Ocean cluster configuration. + Keep in mind that it may not be necessary to increase the target capacity because Ocean automatically scales instances up and down as needed. If you want to edit the target capacity: + 1. In the Spot console, go to **Ocean** > **Cloud Clusters**, and select the cluster. 2. Click **Actions** > **Edit**. 3. On the Review tab, click **JSON** > **Edit Mode**. @@ -1892,10 +1900,10 @@ You can set a static endpoint to use with Ocean Controller Version 2: If you get this message when you’re [upgrading the Ocean Controller Version 2](ocean/tutorials/spot-kubernetes-controller/ocean-controller-two-install?id=install-via-helm) using Helm: -```` +``` Release "ocean-controller" does not exist. Installing it now. Error: parse error at (ocean-kubernetes-controller/templates/_helpers.tpl:320): unclosed action -```` +``` You need to: @@ -1916,7 +1924,7 @@ If these don’t work, add the `--set metrics-server.deloyChart=false` flag to t After you upgrade to Ocean Controller Version 2, you may get many SIEM alerts due to SelfSubjectAccessReview requests to your API server. This is expected behavior. -With the Version 2 Ocean Controller, Spot gets reports for any custom resource you gave it access to through the controller cluster role. For example, an Argo Rollouts custom resource or a VerticalPodAutoscaler for rightsizing. These require Spot to list the custom resources in the cluster and make sure there's read access. This happens when the controller starts up and on a regular basis when it's running. +With the Version 2 Ocean Controller, Spot gets reports for any custom resource you gave it access to through the controller cluster role. For example, an Argo Rollouts custom resource or a VerticalPodAutoscaler for rightsizing. These require Spot to list the custom resources in the cluster and make sure there's read access. This happens when the controller starts up and on a regular basis when it's running.
@@ -1946,8 +1954,8 @@ The Ocean Controller saves up to 8 days of logs. The logs for each day are about 1. Make sure you’re using the [latest version of the controller](ocean/tutorials/spot-kubernetes-controller/ocean-controller-two-update). It takes around 4 days for the metrics to show after upgrading. 2. If you’re using an EKS cluster, make sure you have 2 [security groups](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html): - * Worker node group with an inbound rule that allows communication with the control plane’s security group through port 443. - * Cluster’s control plane. + - Worker node group with an inbound rule that allows communication with the control plane’s security group through port 443. + - Cluster’s control plane. 3. Check the [common issues with the metrics server](https://repost.aws/knowledge-center/eks-metrics-server). @@ -1962,9 +1970,9 @@ The Ocean Controller saves up to 8 days of logs. The logs for each day are about If a node replacement is canceled, you may see this log message in the cluster in the Spot console: -```` +``` DEBUG, Replacement of type Out of strategy for instance has been canceled. Reason for cancellation: A pod with the restrict-scale-down label is currently running on the node. -```` +``` You can also get this message if you’re using the `cluster-autoscaler.kubernetes.io/safe-to-evict` label. It works the same as the `restrict-scale-down` label. When you have one of those labels, the node is not scaled down or replaced. @@ -1979,9 +1987,9 @@ Make sure that labels and annotations don’t prevent scaling down [on the virtu
- You can use a programmatic token for creating Ocean cluster controllers. The benefit of programmatic tokens is they aren't linked to a specific user. If the user is deleted, it doesn't affect the Ocean controller. This helps prevent interruptions and heartbeat issues. +You can use a programmatic token for creating Ocean cluster controllers. The benefit of programmatic tokens is they aren't linked to a specific user. If the user is deleted, it doesn't affect the Ocean controller. This helps prevent interruptions and heartbeat issues. - At minimum, the token must have **account viewer** [permissions](/administration/policies/). Viewer permission is the only permission required for a cluster controller to operate. Cluster controllers don't manage resources in Ocean, the autoscaler does. If you want this same programmatic user to manage other resources in your cluster, additional permission policies are required. +At minimum, the token must have **account viewer** [permissions](/administration/policies/). Viewer permission is the only permission required for a cluster controller to operate. Cluster controllers don't manage resources in Ocean, the autoscaler does. If you want this same programmatic user to manage other resources in your cluster, additional permission policies are required. For a network client, only the **account viewer** permission is required for the client to operate. @@ -2028,10 +2036,10 @@ You may get this event in your Kubernetes cluster: This can happen because: -* Kubernetes needs [storage classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) to create the [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for [persistent volume claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVCs) dynamically. Make sure you have storage classes configured unless you’re using static persistent volume claims. -* The [persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) and [persistent volume claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes-1) access modes don’t match. -* The persistent volume [capacity](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#capacity) is less than the persistent volume claim. -* The total number of persistent volume claims is higher than the persistent volume. +- Kubernetes needs [storage classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) to create the [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for [persistent volume claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVCs) dynamically. Make sure you have storage classes configured unless you’re using static persistent volume claims. +- The [persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) and [persistent volume claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes-1) access modes don’t match. +- The persistent volume [capacity](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#capacity) is less than the persistent volume claim. +- The total number of persistent volume claims is higher than the persistent volume.
@@ -2044,21 +2052,21 @@ This can happen because: If the Ocean autoscaler scales up an instance for your pod at least 5 times, but the Kubernetes scheduler can’t schedule the pod, you may get this message: -```` +``` WARN, Pod Metrics-Server-xxxxx Has Failed To Schedule For 76 Minutes. Autoscaling Disabled For Pod Metrics-Server-xxxxx WARN, Pod Redis-0 Has Failed To Schedule For 76 Minutes. Autoscaling Disabled For Pod Redis-0 WARN, Pod Kube-Dns-Autoscaler-xxxxx Has Failed To Schedule For 76 Minutes. Autoscaling Disabled For Pod Kube-Dns-Autoscaler-xxxxx WARN, Pod Worker-Deployment-xxxxx Has Failed To Schedule For 76 Minutes. Autoscaling Disabled For Pod Worker-Deployment-xxxxx WARN, Pod Kube-Dns-xxxxx Has Failed To Schedule For 76 Minutes. Autoscaling Disabled For Pod Kube-Dns-xxxxx - ```` +``` Ocean stops trying to scale up this pod to prevent infinite scaling. This can happen if: -* Ocean launches instances for the pending pod but they don’t fully register to the Kubernetes cluster because the pod has no node to schedule. -* You’re using AWS ebs-csi-driver PV/PVC. It’s possible that the [ebs-csi-node](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) DaemonSet pods are not running on the nodes. This can happen if the DaemonSet object is having issues, the DaemonSet pods are not running, or if taints on a custom virtual node group are stopping the DaemonSet pods from being scheduled on the node. If you’re using DaemonSet, then the DaemonSet pods must run on every node if a pending pod has a PVC. -* You’re using GPU nodes. The [Nvidia GPU DaemonSet](https://github.com/NVIDIA/k8s-device-plugin) is required to run on every GPU node for the nodes to expose their GPU resources. If a pending node is requesting GPU, then Ocean launches a GPU instance. You need to make sure the nodes are exposing the GPU resources. Typically, you do this with the Nvidia GPU DaemonSet. If the DaemonSet has issues, then the pod may not be scheduled on the node because the node won’t be exposing the GPU. +- Ocean launches instances for the pending pod but they don’t fully register to the Kubernetes cluster because the pod has no node to schedule. +- You’re using AWS ebs-csi-driver PV/PVC. It’s possible that the [ebs-csi-node](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) DaemonSet pods are not running on the nodes. This can happen if the DaemonSet object is having issues, the DaemonSet pods are not running, or if taints on a custom virtual node group are stopping the DaemonSet pods from being scheduled on the node. If you’re using DaemonSet, then the DaemonSet pods must run on every node if a pending pod has a PVC. +- You’re using GPU nodes. The [Nvidia GPU DaemonSet](https://github.com/NVIDIA/k8s-device-plugin) is required to run on every GPU node for the nodes to expose their GPU resources. If a pending node is requesting GPU, then Ocean launches a GPU instance. You need to make sure the nodes are exposing the GPU resources. Typically, you do this with the Nvidia GPU DaemonSet. If the DaemonSet has issues, then the pod may not be scheduled on the node because the node won’t be exposing the GPU. @@ -2071,10 +2079,10 @@ This can happen if: You may see this message in the logs if you use Prometheus to scrape Ocean metrics: -```` +``` ERROR 1 --- java.lang.OutOfMemoryError: Java heap space with root cause java.lang.OutOfMemoryError: Java heap space -```` +``` This means the application ran out of Java heap space, and the pod will crash temporarily. You may also see that the target on the [Prometheus](ocean/tools-and-integrations/prometheus/scrape) dashboard is down. @@ -2084,7 +2092,6 @@ Set the amounts according to the needs of your pods. -
@@ -2096,14 +2103,14 @@ Set the amounts according to the needs of your pods. You get this error in the log: -```` -Kubernetes Autoscaler, Deadlock for Pod: '{pod-name}' -Can't scale up an Instance since PersistentVolumeClaim: -'{PVC-name}' -VolumeId: '{vol-name}' is already attached to an existing Instance: -'{instance-ID}' Please consider using a new PersistentVolumeClaim or open a +``` +Kubernetes Autoscaler, Deadlock for Pod: '{pod-name}' +Can't scale up an Instance since PersistentVolumeClaim: +'{PVC-name}' +VolumeId: '{vol-name}' is already attached to an existing Instance: +'{instance-ID}' Please consider using a new PersistentVolumeClaim or open a support ticket. -```` +``` This can happen when the pod has a claim for a specific volume attached to a different instance, and that instance does not have free space for the pod. @@ -2139,7 +2146,6 @@ A [limit range](https://kubernetes.io/docs/concepts/policy/limit-range/) is a po -
AKS, EKS, GKE: Can I configure headroom for a node? @@ -2147,8 +2153,8 @@ A [limit range](https://kubernetes.io/docs/concepts/policy/limit-range/) is a po You cannot add headroom at a node level. Headroom is intended for: -* Fast scaling: the infrastructure is ready, no need to wait for scaling. -* Interruption: there is available capacity for the pod. If the headroom is all on one node and the node is interrupted, then there is no headroom that is readily available. +- Fast scaling: the infrastructure is ready, no need to wait for scaling. +- Interruption: there is available capacity for the pod. If the headroom is all on one node and the node is interrupted, then there is no headroom that is readily available. @@ -2161,15 +2167,15 @@ You cannot add headroom at a node level. Headroom is intended for: You can configure [automatic headroom](ocean/features/headroom) using kOps at the cluster level, not at a virtual node group level. Add these [metadata labels](/ocean/tools-and-integrations/kops/metadata-labels): -```` +``` spotinst.io/autoscaler-auto-config: "true" spotinst.io/autoscaler-auto-headroom-percentage : {Value} spotinst.io/ocean-default-launchspec: "true" -```` +``` Here's an example of a config file: -````json +```json apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: @@ -2190,7 +2196,7 @@ spec: role: Node maxSize: 1 minSize: 1 -```` +``` @@ -2202,15 +2208,17 @@ minSize: 1
You can restrict specific pods from scaling down by configuring Ocean and Kubernetes. The instance will be replaced only if: -* It goes into an unhealthy state. -* Forced by a cloud provider interruption. + +- It goes into an unhealthy state. +- Forced by a cloud provider interruption. There are two options for restricting pods from scaling down: -* Kubernetes deployments/pods: spotinst.io/restrict-scale-down: true + +- Kubernetes deployments/pods: spotinst.io/restrict-scale-down: true Use the `spotinst.io/restrict-scale-down` label set to true to block proactive scaling down for more efficient bin packing. This will leave the instance running as long as possible. It gets defined as a label in the pod's configuration. See [restrict scale down](ocean/features/labels-and-taints?id=spotinstiorestrict-scale-down). -* Virtual node group (VNG): restrict scale down (only available for AWS, ECS, and GKE) +- Virtual node group (VNG): restrict scale down (only available for AWS, ECS, and GKE) You can configure [Restrict Scale Down](ocean/features/vngs/attributes-and-actions-per-vng) at the virtual node group level so the nodes and pods within the virtual node group are not replaced or scaled down due to the auto scaler resource optimization. Create a virtual node group, go to the Advanced tab, then select **Restrict Scale Down**. @@ -2228,11 +2236,11 @@ You can stop the autoscaler and recoveries: 1. Disable [autoscaling](ocean/features/scaling-kubernetes?id=customize-scaling-configuration). 2. Stop recoveries: -
    -
  1. In the Spot console, go to Ocean > Cloud Clusters and select a group.
    2. -
  2. On the Nodes tab, select the node and click Actions > Detach.
    3. -
  3. In the AWS console, detach any new nodes.
    4. -
+
    +
  1. In the Spot console, go to Ocean > Cloud Clusters and select a group.
    2. +
  2. On the Nodes tab, select the node and click Actions > Detach.
    3. +
  3. In the AWS console, detach any new nodes.
    4. +
If you need to restart autoscaling and recoveries, enable [autoscaling](ocean/features/scaling-kubernetes?id=customize-scaling-configuration). @@ -2249,9 +2257,9 @@ You cannot update the instance types in the default virtual node group. For exam If you do, you’ll get this error: -```` +``` Launch spec ols-xxxxxxxx instance types are not a subset of ocean cluster -```` +``` Remove the instance types at the cluster level, add m5d.xlarge and m6i.xlarge instance types, and then update the cluster. @@ -2267,9 +2275,10 @@ Instance types of the virtual node group are always a subset of the Ocean cluste
You can include or exclude certain instance types in your Ocean cluster. Typically, you do it from the cluster configuration. -* **Blacklist**: instance types to block launching in the Ocean cluster. It cannot be used with a permit list. -* **Whitelist**: instance types allowed in the Ocean cluster. It cannot be used with a deny list. -* **Filtering**: list of filters. The instance types that match with all filters make up the Ocean's whitelist parameter. Filtering cannot be used with allow or block lists. + +- **Blacklist**: instance types to block launching in the Ocean cluster. It cannot be used with a permit list. +- **Whitelist**: instance types allowed in the Ocean cluster. It cannot be used with a deny list. +- **Filtering**: list of filters. The instance types that match with all filters make up the Ocean's whitelist parameter. Filtering cannot be used with allow or block lists. You can allow, [block](https://docs.spot.io/ocean/tips-and-best-practices/manage-machine-types?id=opt-out-of-machine-types), or [filter](https://docs.spot.io/ocean/tips-and-best-practices/manage-machine-types?id=select-instance-types-with-advanced-filters) instance types in the cluster configuration in compute: instanceTypes in the cluster’s JSON or using an API. @@ -2307,14 +2316,14 @@ Initially, the costs are compared with the on demand value of the instance types If you have shutdown hours set up and autoscaler is disabled, you may see one of these messages in the Spot console: -* `Info Instances: [i-xxxxx] have been launched. Reason: Shutdown hours period finished` -* `Info Instances: [i-xxxxx] have been detached. Reason: Scale-down as part of instance recovery` -* `Info Instances: [i-xxxxx] have been launched. Reason: Scale-up as part of instance recovery` +- `Info Instances: [i-xxxxx] have been launched. Reason: Shutdown hours period finished` +- `Info Instances: [i-xxxxx] have been detached. Reason: Scale-down as part of instance recovery` +- `Info Instances: [i-xxxxx] have been launched. Reason: Scale-up as part of instance recovery` If shutdown hours are set up and autoscaler is disabled, new nodes are not scaled up based on pending pods. [An existing node is still launched](ocean/features/running-hours?id=scaling-behavior-kubernetes): -* At the end of the shutdown hours. -* If the spot node launched at the end of shutdown hours has an interruption or recovery. +- At the end of the shutdown hours. +- If the spot node launched at the end of shutdown hours has an interruption or recovery. You can disable shutdown hours in the Spot console: go to **Ocean** > **Cloud Clusters** > select the cluster > **Actions** > **Customize Scaling** > **Cluster Shutdown Hours**. @@ -2329,14 +2338,14 @@ You can disable shutdown hours in the Spot console: go to **Ocean** > **Cloud Cl If your Ocean cluster won’t scale up, you may see a message like this in the Spot console logs: -```` +``` Failed to perform scale up for virtual node group xxxxx (vng-xxxxx). Got status code different from SC_OK : 400 Body { "code": "BadRequest", "details": null, "message": "Client Error: error parsing version(1.26). If you would like to use alias minor version, please use api version starting from 2022-03-02-preview", "subcode": "" } -```` +``` This happens when the Ocean cluster tries to create a node pool using a specific Kubernetes version. In this message, it’s version 1.26. -* If you want to use a specific version, you also need to give the exact patch version (the alias minor version). -* You also need to make sure your [AKS API version](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#alias-minor-version) is at least the version mentioned in the message. +- If you want to use a specific version, you also need to give the exact patch version (the alias minor version). +- You also need to make sure your [AKS API version](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#alias-minor-version) is at least the version mentioned in the message.
@@ -2349,14 +2358,15 @@ This happens when the Ocean cluster tries to create a node pool using a specific If your pods are scheduled on [B-series nodes](https://learn.microsoft.com/en-us/azure/virtual-machines/b-series-cpu-credit-model/b-series-cpu-credit-model), the nodes and VMs are burstable. This means that they outperform their actual limits for a short period. After the burst credits are used, the pods will fail, and you may see this message: -```` +``` The node was low on resource: memory. Threshold quantity: 750Mi, available: 757424Ki. Container terrakube-registry was using 339556Ki, request is 0, has larger consumption of memory. -```` +``` You can: -* Make sure your resource allocation is set up correctly. You can use the [resource quotas and limit ranges](https://kubernetes.io/docs/concepts/policy/resource-quotas/) as a reference. -* [Exclude b-series (Bs) nodes](ocean/tutorials/manage-virtual-nd-groups-aks?id=vm-selection) from your virtual node groups. -* Set up [rightsizing recommendations](ocean/features/ocean-cluster-right-sizing-recom-tab). + +- Make sure your resource allocation is set up correctly. You can use the [resource quotas and limit ranges](https://kubernetes.io/docs/concepts/policy/resource-quotas/) as a reference. +- [Exclude b-series (Bs) nodes](ocean/tutorials/manage-virtual-nd-groups-aks?id=vm-selection) from your virtual node groups. +- Set up [rightsizing recommendations](ocean/features/ocean-cluster-right-sizing-recom-tab).
@@ -2369,22 +2379,22 @@ You can: If your pods are not registering in your AKS cluster, you may get this message: -```` +``` Could not scale up for pending pod xxxxx due to technical failure to launch required instances. Scale down has been disabled in the cluster until pod is scheduled. ERROR Failed to perform scale up for virtual node group xxxxx (vng-xxxxx). Got status code different from SC_OK : 400 Body { "code": "UDRWithNodePublicIPNotAllowed", "details": null, "message": "OutboundType UserDefinedRouting can not be combined with Node Public IP.", "subcode": "" } ERROR Failed to scale up 1 new nodes as part of scaling the virtual node groups vng-xxxxx (xxxxx). -```` +``` You cannot use **enableNodePublicIP** set to True with **userDefinedRouting** set to outboundType. If you’re using [outbound types of userDefinedRouting](https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype#outbound-type-of-userdefinedrouting), change `"enableNodePublicIP": true`, to false. For example: -````json +```json "nodePoolProperties": { "maxPodsPerNode": 250, "enableNodePublicIP": false, } -```` +``` @@ -2397,12 +2407,12 @@ If you’re using [outbound types of userDefinedRouting](https://learn.microsoft The Start Migration button can be grayed out for an AKS Ocean cluster if: -* The cluster has system node pools, which must run as regular nodes and don’t require scaling. -* The Kubernetes cluster isn’t running on AKS infrastructure. -* Kubernetes cluster isn’t connected to an Ocean cluster. You can [import an AKS cluster to Ocean](ocean/getting-started/aks/?id=import-an-aks-cluster-to-ocean). -* The Ocean Controller wasn’t installed, updated, and running in the cluster. -* Cluster or virtual node group doesn’t have a [supported Kubernetes version](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar). -* You don’t have dedicated [virtual node groups](ocean/features/vngs/?id=virtual-node-groups) for your workload to let Ocean autoscaler scale up nodes. +- The cluster has system node pools, which must run as regular nodes and don’t require scaling. +- The Kubernetes cluster isn’t running on AKS infrastructure. +- Kubernetes cluster isn’t connected to an Ocean cluster. You can [import an AKS cluster to Ocean](ocean/getting-started/aks/?id=import-an-aks-cluster-to-ocean). +- The Ocean Controller wasn’t installed, updated, and running in the cluster. +- Cluster or virtual node group doesn’t have a [supported Kubernetes version](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar). +- You don’t have dedicated [virtual node groups](ocean/features/vngs/?id=virtual-node-groups) for your workload to let Ocean autoscaler scale up nodes. @@ -2419,8 +2429,6 @@ If you’re seeing an unable to migrate status in workload migration, check if t
- -
AKS: Can I create VMs with specific architecture in Ocean AKS? @@ -2435,24 +2443,23 @@ However, it’s not possible to do with Ocean AKS clusters because you cannot ch 1. Create a new virtual node group in the Ocean AKS cluster and configure it manually or import the configuration of a node pool. 2. Add vmSizes to the virtual node group JSON file. - ````json - "vmSizes": { - "filters": { - "architectures": [ - "x86_64" - ], - "series": [] - } - } - ```` - - * Architectures is a list of strings, and the values can be a combination of x86_64 (includes both intel64 and amd64), intel64, amd64, and arm64. + ```json + "vmSizes": { + "filters": { + "architectures": [ + "x86_64" + ], + "series": [] + } + } + ``` + + - Architectures is a list of strings, and the values can be a combination of x86_64 (includes both intel64 and amd64), intel64, amd64, and arm64. - * Add series with the VM series for the particular architecture. + - Add series with the VM series for the particular architecture. For example, run VMs with arm64 and launch the VMs with Dps_V5 as the series. - - + @@ -2471,16 +2478,16 @@ Do not set this up on production clusters because if the admission controller po 1. Edit the webhook configuration: `kubectl edit MutatingWebhookConfiguration spot-admission-controller.kube-system.svc`. -3. Make sure this object is in the configuration file: +2. Make sure this object is in the configuration file: - ````yaml + ```yaml objectSelector: matchExpressions: - - key: app.kubernetes.io/name - operator: NotIn - values: - - spot-admission-controller - ```` + - key: app.kubernetes.io/name + operator: NotIn + values: + - spot-admission-controller + ``` 3. If the object is not there, [reinstall the Spot admission controller](ocean/getting-started/aks/?id=step-4-automatic-spot-tolerance-injection-optional). 4. Change `failurePolicy` to Fail (`failurePolicy: Fail`). @@ -2503,126 +2510,3 @@ AKS only launches spot nodes if the admission controller is enabled and Spot tol
- - - -## Ocean for Apache Spark - -
- Can I set the number of retries for a stage in Ocean Spark? - -
- -If there is a stage failure when a job runs in Ocean Spark, there’s a [retry mechanism](https://spark.apache.org/docs/3.5.2/configuration.html#:~:text=2.0.3-,spark.stage.maxConsecutiveAttempts,-4). You can change the number of retries for a stage: - -1. In the Spot console, go to **Ocean for Spark** > **Configuration Templates**. -2. Select the configuration template of the application you need to change. -3. Add `spark.stage.maxConsecutiveAttempts` with the number of retries. - - -
- -
- -
- Can I run Spark jobs on the driver, not on executors? - -
- -Yes, you can define your configuration template to run your Spark application on the driver and not on the executors. - -Define a [Jupyter kernel](ocean-spark/tools-integrations/connect-jupyter-notebooks?id=define-jupyter-kernels-with-configuration-templates) with a low idle timeout so it’s scaled down quickly if it’s not in use: - -```` -"spark.dynamicAllocation.enabled": "true", -"spark.dynamicAllocation.maxExecutors": "1", -"spark.dynamicAllocation.minExecutors": "0", -"spark.dynamicAllocation.initialExecutors": "0", -"spark.dynamicAllocation.executorIdleTimeout": "10s" -```` - -
- -
- - -
- Why are my pods going to the wrong virtual node group? - -
- -If your Ocean Spark pods are going to the wrong virtual node group, it’s typically because the virtual node group was updated or deleted. - -You can either recreate the Ocean Spark cluster or update the labels and taints. These are the definitions for virtual node group labels and taints: - -**ocean-spark-system** - -````json - "labels": [ - { - "key": "nodegroup-name", - "value": "ofas-system" - } - ], - "taints": [], -```` - -**ocean-spark-on-demand** - -````json - "labels": [ - { - "key": "bigdata.spot.io/vng", - "value": "ocean-spark" - }, - { - "key": "nodegroup-name", - "value": "ocean-spark-on-demand" - } - ], - "taints": [ - { - "key": "bigdata.spot.io/unschedulable", - "value": "ocean-spark", - "effect": "NoSchedule" - } - ], -```` - -**ocean-spark-spot** - -````json - "labels": [ - { - "key": "bigdata.spot.io/vng", - "value": "ocean-spark" - }, - { - "key": "nodegroup-name", - "value": "ocean-spark-spot" - } - ], - "taints": [ - { - "key": "bigdata.spot.io/unschedulable", - "value": "ocean-spark", - "effect": "NoSchedule" - } - ], -```` - - -
- -
- -
- What are the minimum permissions for creating a workspace? - -
- -You can give some of your users [access to a workspace](ocean-spark/configure-permissions/?id=set-permissions-for-workspace-users) but not allow them to make changes to a cluster. - -
- -
diff --git a/src/docs/ocean-spark/README.md b/src/docs/ocean-spark/README.md deleted file mode 100644 index d91f77d012..0000000000 --- a/src/docs/ocean-spark/README.md +++ /dev/null @@ -1,74 +0,0 @@ - - -# Ocean for Apache Spark - -Ocean for Apache Spark (also referred to in the user documentation as Ocean Spark) is a managed cloud-native Spark platform that can be deployed in your cloud account. As of December 2021, Ocean Spark is available on AWS, Azure, and GCP, and is an alternative to managed services like Databricks, EMR, Google Dataproc, Azure HDInsight, or DIY Spark infrastructures. - -Running on top of Ocean, Spot’s serverless infrastructure engine for containers, Ocean Spark makes it easy for your data teams to be successful with Apache Spark on Kubernetes, without dealing with the complexity of managing servers. - -Ocean Spark features intuitive UIs from which you can view your applications configurations, logs, key metrics, Spark UI, and costs. It provides reliable, automated, and continuously optimized cloud infrastructure and Spark configurations, resulting in significant time and cost savings. - -## Key Features - -Ocean Spark makes it easy to provision, configure, and monitor Kubernetes clusters in your cloud account, and then run containerized Spark applications on top of it. Here are some of the key features. - -### Spark-centric observability layer - -Like with any Ocean cluster, the Spot console gives you visibility over your Kubernetes cluster(s): nodes, pods, scaling activity, costs at the cluster-level. - -With Ocean for Spark, a unique Spark-centric observability layer is added, giving you visibility over your Spark applications’ configurations, logs, Spark UI, and key metrics (CPU, Memory, I/O, Spark efficiency ratio, Shuffle). This information is available both while the app is running and after it is completed. - -### Spark jobs configuration optimization - -A job is a set of Spark applications - typically the same application code that you run regularly. Since the ID of a job is a required field when you submit a Spark application through our API, you explicitly define the grouping of applications in jobs. - -Ocean Spark automatically tunes certain infrastructure parameters (e.g. container sizes, # of executors) and Spark configurations (e.g. I/O optimization, memory management, shuffle, Spark feature flags) based on past execution of your Spark jobs. - -You can track the evolution of each of your job’s performance, stability, costs, and other key metrics in a dedicated dashboard. - -### Automatically scaled and optimized infrastructure - -Leveraging advanced AI algorithms, Ocean Spark automatically scales your cluster(s) based on the real-time load, choosing the highest-performance, lowest-cost instances (including spot nodes) matching your workload requirements. - -The scheduling of pods (containers) onto nodes (instances) is optimized with a bin-packing algorithm to maximize efficiency and reduce your costs. An automatic headroom can be configured to guarantee that your Spark applications can start instantaneously without waiting for new capacity to be provisioned. - -### External Integrations - -Ocean for Spark includes pre-built integrations with Jupyter notebooks (Jupyter Enterprise Gateway) and popular scheduling tools like Airflow. Our REST API lets you securely submit Spark applications from anywhere. In addition, Spark can be configured to read/write data to all popular data storages, as well as to use external Hive metastore. - -The open and flexible architecture of Ocean lets you leverage popular cloud-native technologies in networking, security, observability and CI/CD. - -### Fleet of Spark Docker images to build upon - -The Ocean for Spark team maintains a fleet of optimized Docker images for Apache. These images contain the Spark distribution itself (Spark 2.4 and later) as well as popular connectors for data sources (S3, GCS, ADLS, Snowflake, Delta Lake, Kafka, and more) as well as Scala, Java, Python and Hadoop. - -You can use these images directly, or choose to build your own custom Docker images on top of them. You can run these images locally, or at scale on the Kubernetes cluster, and benefit from a reliable and fast developer workflow. - -## Architecture Overview - -The diagram below shows an architectural overview of Ocean Spark. - - - -### End Users - -You can start interactive Spark sessions by connecting Jupyter notebooks (Jupyter, JupyterLab, JupyterHub), or submit batch or streaming applications through the REST API. We provide integrations with popular schedulers such as Airflow, but it’s also easy to implement to your own custom workflow orchestrator. They can then track the execution of their Spark applications by logging in to the Spot console. - -### Spot.io Control Plane (Backend) - -The Spot console, the REST API, and the Kubernetes and Spark optimization logic are hosted in the Spot.io control plane. These services continuously monitor your Ocean for Spark cluster(s) to enable the key features such as the management, monitoring, and optimization of your cluster and Spark applications. - -### Ocean Spark Cluster - -The Ocean Spark cluster itself is a Kubernetes cluster in your cloud account. This is where the Spark applications are running inside pods, Docker containers. You can run hundreds of Spark applications in parallel, using a variety of Spark versions and dependencies (each Spark application is fully isolated and independent), on a heterogeneous infrastructure made of an optimized mix of different instances (instance family, instance size, spot or on-demand). - -### Data Sources - -Spark can read data from (and write data to) a wide range of data sources and formats including object stores, data warehouses, streaming sources, relational and non-relational databases. - -You can leverage cloud security best-practices (such as IAM role and Kubernetes secrets) to give your Spark applications permission to access the data in the most secure manner. The sensitive data Spark works with stays protected in your cloud account all the time. - -## What’s Next? - -- If you are new to Spot, [connect your cloud provider](connect-your-cloud-provider/aws-account) to Spot. -- If you are already a Spot user, go ahead and [Get Started with Ocean Spark](ocean-spark/getting-started/). diff --git a/src/docs/ocean-spark/_media/configure-cluster-aws.png b/src/docs/ocean-spark/_media/configure-cluster-aws.png deleted file mode 100644 index 7649126e79..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-cluster-aws.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-cluster-gcp.png b/src/docs/ocean-spark/_media/configure-cluster-gcp.png deleted file mode 100644 index b9db118c3f..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-cluster-gcp.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-jobs-01.png b/src/docs/ocean-spark/_media/configure-jobs-01.png deleted file mode 100644 index bac380eaa1..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-jobs-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-spark-applications-01.png b/src/docs/ocean-spark/_media/configure-spark-applications-01.png deleted file mode 100644 index 6e75d916af..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-spark-applications-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-01.png b/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-01.png deleted file mode 100644 index 1f59153246..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-02.png b/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-02.png deleted file mode 100644 index 97731b506d..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-spark-apps-access-your-data-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-spark-apps-memory-&-cores-01.png b/src/docs/ocean-spark/_media/configure-spark-apps-memory-&-cores-01.png deleted file mode 100644 index 12c6d33253..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-spark-apps-memory-&-cores-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/configure-spark-apps-package-spark-code-01.png b/src/docs/ocean-spark/_media/configure-spark-apps-package-spark-code-01.png deleted file mode 100644 index 990f691f59..0000000000 Binary files a/src/docs/ocean-spark/_media/configure-spark-apps-package-spark-code-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/getting-started-01.png b/src/docs/ocean-spark/_media/getting-started-01.png deleted file mode 100644 index b4ea9b651d..0000000000 Binary files a/src/docs/ocean-spark/_media/getting-started-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/getting-started-02.png b/src/docs/ocean-spark/_media/getting-started-02.png deleted file mode 100644 index 165afec4e6..0000000000 Binary files a/src/docs/ocean-spark/_media/getting-started-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/jupyter-vscode-ide-1.png b/src/docs/ocean-spark/_media/jupyter-vscode-ide-1.png deleted file mode 100644 index a7d8df766a..0000000000 Binary files a/src/docs/ocean-spark/_media/jupyter-vscode-ide-1.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/jupyter-vscode-ide-2.png b/src/docs/ocean-spark/_media/jupyter-vscode-ide-2.png deleted file mode 100644 index bbdbe8a697..0000000000 Binary files a/src/docs/ocean-spark/_media/jupyter-vscode-ide-2.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/monitor-jobs-01.png b/src/docs/ocean-spark/_media/monitor-jobs-01.png deleted file mode 100644 index d158d805e8..0000000000 Binary files a/src/docs/ocean-spark/_media/monitor-jobs-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/ocean-spark-monitor-applications-01.png b/src/docs/ocean-spark/_media/ocean-spark-monitor-applications-01.png deleted file mode 100644 index 5b500110e1..0000000000 Binary files a/src/docs/ocean-spark/_media/ocean-spark-monitor-applications-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/ocean-spark-overview-01.png b/src/docs/ocean-spark/_media/ocean-spark-overview-01.png deleted file mode 100644 index bb76845fa6..0000000000 Binary files a/src/docs/ocean-spark/_media/ocean-spark-overview-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/overview-01.png b/src/docs/ocean-spark/_media/overview-01.png deleted file mode 100644 index 386c83aae1..0000000000 Binary files a/src/docs/ocean-spark/_media/overview-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/overview-02.png b/src/docs/ocean-spark/_media/overview-02.png deleted file mode 100644 index 20949fc629..0000000000 Binary files a/src/docs/ocean-spark/_media/overview-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/product-tour-01.png b/src/docs/ocean-spark/_media/product-tour-01.png deleted file mode 100644 index 780b692854..0000000000 Binary files a/src/docs/ocean-spark/_media/product-tour-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/run-your-first-app-01.png b/src/docs/ocean-spark/_media/run-your-first-app-01.png deleted file mode 100644 index 274abb4562..0000000000 Binary files a/src/docs/ocean-spark/_media/run-your-first-app-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/run-your-first-app-02.png b/src/docs/ocean-spark/_media/run-your-first-app-02.png deleted file mode 100644 index a5fd117726..0000000000 Binary files a/src/docs/ocean-spark/_media/run-your-first-app-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-01.png b/src/docs/ocean-spark/_media/tools-airflow-01.png deleted file mode 100644 index 95249054b7..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-02.png b/src/docs/ocean-spark/_media/tools-airflow-02.png deleted file mode 100644 index df644d1688..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-03.png b/src/docs/ocean-spark/_media/tools-airflow-03.png deleted file mode 100644 index 415a83d062..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-04.png b/src/docs/ocean-spark/_media/tools-airflow-04.png deleted file mode 100644 index f0318334b2..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-05.png b/src/docs/ocean-spark/_media/tools-airflow-05.png deleted file mode 100644 index 97231d854f..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-05.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-06.png b/src/docs/ocean-spark/_media/tools-airflow-06.png deleted file mode 100644 index fcf425a51c..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-06.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-07.png b/src/docs/ocean-spark/_media/tools-airflow-07.png deleted file mode 100644 index 7462629779..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-07.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-airflow-08.png b/src/docs/ocean-spark/_media/tools-airflow-08.png deleted file mode 100644 index b195f96377..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-airflow-08.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-01.png b/src/docs/ocean-spark/_media/tools-aws-glue-catalog-01.png deleted file mode 100644 index 64d9ec69c3..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-02.png b/src/docs/ocean-spark/_media/tools-aws-glue-catalog-02.png deleted file mode 100644 index 6c32907347..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-03.png b/src/docs/ocean-spark/_media/tools-aws-glue-catalog-03.png deleted file mode 100644 index de1f4d3562..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-04.png b/src/docs/ocean-spark/_media/tools-aws-glue-catalog-04.png deleted file mode 100644 index 61f5e2a0ac..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-05.png b/src/docs/ocean-spark/_media/tools-aws-glue-catalog-05.png deleted file mode 100644 index 65af84ba26..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-aws-glue-catalog-05.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-01.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-01.png deleted file mode 100644 index 22e3bac33a..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-02.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-02.png deleted file mode 100644 index 89369ffe8e..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-03.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-03.png deleted file mode 100644 index 6c085d7a46..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-04.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-04.png deleted file mode 100644 index 157a840771..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-05.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-05.png deleted file mode 100644 index 3e64703f49..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-05.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-06.png b/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-06.png deleted file mode 100644 index d9e6769dc3..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-connect-jupyter-notebooks-06.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-hive-metastore-01.png b/src/docs/ocean-spark/_media/tools-hive-metastore-01.png deleted file mode 100644 index 11e4e8d1e9..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-hive-metastore-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tools-hive-metastore-02.png b/src/docs/ocean-spark/_media/tools-hive-metastore-02.png deleted file mode 100644 index 9af192f76b..0000000000 Binary files a/src/docs/ocean-spark/_media/tools-hive-metastore-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-01.png b/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-01.png deleted file mode 100644 index 923b6d702a..0000000000 Binary files a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-02.png b/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-02.png deleted file mode 100644 index 851447e2b9..0000000000 Binary files a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-03.png b/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-03.png deleted file mode 100644 index 1d83c07467..0000000000 Binary files a/src/docs/ocean-spark/_media/troubleshoot-cluster-deployment-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01.png deleted file mode 100644 index 6917396d23..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01a.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01a.png deleted file mode 100644 index db23a57374..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01b.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01b.png deleted file mode 100644 index 37d206a7c6..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-01b.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02.png deleted file mode 100644 index 1705511196..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-1.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-1.png deleted file mode 100644 index 57f7e9df7e..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-1.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-2.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-2.png deleted file mode 100644 index e29ac08bc9..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-2.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-3.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-3.png deleted file mode 100644 index b33f59b804..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a-3.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a.png deleted file mode 100644 index 495eb13c19..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-4.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-4.png deleted file mode 100644 index 1e8bfed612..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-4.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-5.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-5.png deleted file mode 100644 index 9339db0fbb..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-5.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-6.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-6.png deleted file mode 100644 index 6df79f07fb..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b-6.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b.png deleted file mode 100644 index 08fe0ecc08..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-02b.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03.png deleted file mode 100644 index b4dfcab54d..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03a.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03a.png deleted file mode 100644 index 100020febd..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03b.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03b.png deleted file mode 100644 index 5dc02b3292..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-03b.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04.png deleted file mode 100644 index 2c1349c3fc..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04a.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04a.png deleted file mode 100644 index b2515c8e66..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04b.png b/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04b.png deleted file mode 100644 index 1afcc3d8d3..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorial-wave-cost-analysis-04b.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-cluster-02.png b/src/docs/ocean-spark/_media/tutorials-cluster-02.png deleted file mode 100644 index 8eea2e09d3..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-cluster-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-cluster-02a.png b/src/docs/ocean-spark/_media/tutorials-cluster-02a.png deleted file mode 100644 index c9ce561c2e..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-cluster-02a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-cluster-03.png b/src/docs/ocean-spark/_media/tutorials-cluster-03.png deleted file mode 100644 index 934d4d2971..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-cluster-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-cluster-04.png b/src/docs/ocean-spark/_media/tutorials-cluster-04.png deleted file mode 100644 index fc47936226..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-cluster-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-clusters-00.png b/src/docs/ocean-spark/_media/tutorials-clusters-00.png deleted file mode 100644 index bbcddffca2..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-clusters-00.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-clusters-01-1.png b/src/docs/ocean-spark/_media/tutorials-clusters-01-1.png deleted file mode 100644 index b8f89f614c..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-clusters-01-1.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-clusters-01.png b/src/docs/ocean-spark/_media/tutorials-clusters-01.png deleted file mode 100644 index 5b81120560..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-clusters-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-clusters-01a.png b/src/docs/ocean-spark/_media/tutorials-clusters-01a.png deleted file mode 100644 index 2fc210e6d4..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-clusters-01a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-spark-app-01.png b/src/docs/ocean-spark/_media/tutorials-spark-app-01.png deleted file mode 100644 index 5c686205d9..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-spark-app-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-spark-app-02.png b/src/docs/ocean-spark/_media/tutorials-spark-app-02.png deleted file mode 100644 index 896b953b86..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-spark-app-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-spark-app-03.png b/src/docs/ocean-spark/_media/tutorials-spark-app-03.png deleted file mode 100644 index ea1ce2ebad..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-spark-app-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-wave-clusters-01.png b/src/docs/ocean-spark/_media/tutorials-wave-clusters-01.png deleted file mode 100644 index 30a6f02b36..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-wave-clusters-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-wave-clusters-01a.png b/src/docs/ocean-spark/_media/tutorials-wave-clusters-01a.png deleted file mode 100644 index bce580f589..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-wave-clusters-01a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-wave-clusters-02.png b/src/docs/ocean-spark/_media/tutorials-wave-clusters-02.png deleted file mode 100644 index 776a8971e1..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-wave-clusters-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/tutorials-wave-clusters-02a.png b/src/docs/ocean-spark/_media/tutorials-wave-clusters-02a.png deleted file mode 100644 index 20298338fc..0000000000 Binary files a/src/docs/ocean-spark/_media/tutorials-wave-clusters-02a.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-01.png b/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-01.png deleted file mode 100644 index c1916dc6d3..0000000000 Binary files a/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-02.png b/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-02.png deleted file mode 100644 index ef08b843ba..0000000000 Binary files a/src/docs/ocean-spark/_media/use-vngs-with-ocean-spark-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-app-details-01.png b/src/docs/ocean-spark/_media/view-app-details-01.png deleted file mode 100644 index d7cabeb5a7..0000000000 Binary files a/src/docs/ocean-spark/_media/view-app-details-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-app-details-02.png b/src/docs/ocean-spark/_media/view-app-details-02.png deleted file mode 100644 index bd8f72cade..0000000000 Binary files a/src/docs/ocean-spark/_media/view-app-details-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-app-details-03.png b/src/docs/ocean-spark/_media/view-app-details-03.png deleted file mode 100644 index 3e329eac79..0000000000 Binary files a/src/docs/ocean-spark/_media/view-app-details-03.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-app-details-04.png b/src/docs/ocean-spark/_media/view-app-details-04.png deleted file mode 100644 index ad307a8cce..0000000000 Binary files a/src/docs/ocean-spark/_media/view-app-details-04.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-app-details-05.png b/src/docs/ocean-spark/_media/view-app-details-05.png deleted file mode 100644 index 8e9fe0a16c..0000000000 Binary files a/src/docs/ocean-spark/_media/view-app-details-05.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-job-details-01-1.png b/src/docs/ocean-spark/_media/view-job-details-01-1.png deleted file mode 100644 index b978f9d20e..0000000000 Binary files a/src/docs/ocean-spark/_media/view-job-details-01-1.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-job-details-01.png b/src/docs/ocean-spark/_media/view-job-details-01.png deleted file mode 100644 index 62f13d40f3..0000000000 Binary files a/src/docs/ocean-spark/_media/view-job-details-01.png and /dev/null differ diff --git a/src/docs/ocean-spark/_media/view-job-details-02.png b/src/docs/ocean-spark/_media/view-job-details-02.png deleted file mode 100644 index bbc05f0c54..0000000000 Binary files a/src/docs/ocean-spark/_media/view-job-details-02.png and /dev/null differ diff --git a/src/docs/ocean-spark/_sidebar.md b/src/docs/ocean-spark/_sidebar.md deleted file mode 100644 index a4a157d528..0000000000 --- a/src/docs/ocean-spark/_sidebar.md +++ /dev/null @@ -1,49 +0,0 @@ - - - - -- Back to Home -- [Ocean for Apache Spark](ocean-spark/) - - [Getting Started](ocean-spark/getting-started/) - - [Create an Ocean Spark Cluster](ocean-spark/getting-started/create-cluster) - - [Run Your First App](ocean-spark/getting-started/run-your-first-app) - - [Troubleshoot Cluster Deployment](ocean-spark/getting-started/troubleshoot-cluster-deployment) - - [Product Tour](ocean-spark/product-tour/) - - [Manage Clusters](ocean-spark/product-tour/manage-clusters) - - [View Cluster Details](ocean-spark/product-tour/view-cluster-details) - - [Analyze Costs](ocean-spark/product-tour/analyze-costs) - - [Monitor Applications](ocean-spark/product-tour/monitor-applications) - - [View Application Details](ocean-spark/product-tour/view-application-details) - - [Monitor Jobs](ocean-spark/product-tour/monitor-jobs) - - [View Job Details](ocean-spark/product-tour/view-job-details) - - [Use Virtual Node Groups](ocean-spark/product-tour/use-vngs) - - [Configure Spark Applications](ocean-spark/configure-spark-apps/) - - [Access Your Data](ocean-spark/configure-spark-apps/access-your-data) - - [Package Spark Code](ocean-spark/configure-spark-apps/package-spark-code) - - [Configure Pod Sizes](ocean-spark/configure-spark-apps/memory-&-cores) - - [Common Spark Configurations](ocean-spark/configure-spark-apps/common-spark-configs) - - [Secrets & Environment Variables](ocean-spark/configure-spark-apps/secrets-environment-variables) - - [Docker Images](ocean-spark/configure-spark-apps/docker-images) - - [Mount Volumes](ocean-spark/configure-spark-apps/mount-volumes) - - [Configure Clusters](ocean-spark/configure-clusters/) - - [Configure Permissions](ocean-spark/configure-permissions/) - - [Tools & Integrations](ocean-spark/tools-integrations/) - - [Connect Jupyter Notebooks](ocean-spark/tools-integrations/connect-jupyter-notebooks) - - [Run Apps from Airflow](ocean-spark/tools-integrations/run-apps-from-airflow) - - [Spark Connect](ocean-spark/tools-integrations/spark-connect) - - [JDBC](ocean-spark/tools-integrations/jdbc) - - [Shuffle Plugin](ocean-spark/tools-integrations/shuffle-plugin) - - [Hive Metastore](ocean-spark/tools-integrations/hive-metastore) - - [AWS Glue Data Catalog](ocean-spark/tools-integrations/aws-glue-catalog) - - [Docker Images Release Notes](ocean-spark/docker-images-release-notes/) - - [gen25](ocean-spark/docker-images-release-notes/gen25.md) - - [gen24](ocean-spark/docker-images-release-notes/gen24.md) - - [gen23](ocean-spark/docker-images-release-notes/gen23.md) - - [gen22](ocean-spark/docker-images-release-notes/gen22.md) - - [gen21](ocean-spark/docker-images-release-notes/gen21.md) - - [gen20](ocean-spark/docker-images-release-notes/gen20.md) - - [gen19](ocean-spark/docker-images-release-notes/gen19.md) - - [gen18](ocean-spark/docker-images-release-notes/gen18.md) - - [Legacy Images](ocean-spark/docker-images-release-notes/legacy-dm-images) - - [Cluster Release Notes](ocean-spark/data-plane-release-notes/) - - [Get Support](ocean-spark/support/) diff --git a/src/docs/ocean-spark/configure-clusters/README.md b/src/docs/ocean-spark/configure-clusters/README.md deleted file mode 100644 index ce8a306523..0000000000 --- a/src/docs/ocean-spark/configure-clusters/README.md +++ /dev/null @@ -1,558 +0,0 @@ - - -# Configure Cluster - -This section shows you how to configure your Ocean for Apache Spark (OfAS) cluster. You can configure the cluster ingress, log collection, Spark application namespaces, and more. You can also refer to the [API documentation](https://docs.spot.io/api/#tag/Ocean-Spark) and the [Terraform](https://registry.terraform.io/modules/spotinst/ocean-spark/spotinst/latest) module to see the different cluster configuration options. - -## Ingress - -Some of the Ocean for Apache Spark features require inbound connectivity to be set up on the cluster. These features are: - -- Notebooks. -- Live driver and Kubernetes log streams while the Spark application is running. -- Access to the Spark UI while the Spark application is running. - -The Ocean for Apache Spark installation provisions a public load balancer by default. The load balancer is configured to only accept traffic from the Ocean for Apache Spark control plane IP address and the communication is protected by mutual TLS. - -The following configuration options enable you to customize this setup: - -### Load Balancer Service Annotations - -Ocean for Apache Spark creates a `LoadBalancer` service called `ofas-ingress-nginx-controller` in the `spot-system` namespace by default. This service triggers the creation of a load balancer by your cloud provider. To customize the load balancer that gets provisioned, you can set additional annotations on the `LoadBalancer` service. Different annotations are supported on different cloud providers. - -**Example of a cluster configuration:** - -```json -{ - "cluster": { - "config": { - "ingress": { - "loadBalancer": { - "serviceAnnotations": { - "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags": "Environment=dev,Team=data-science", - "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "ip", - "service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing", - "service.beta.kubernetes.io/aws-load-balancer-target-group-attributes": "preserve_client_ip.enabled=true", - "service.beta.kubernetes.io/aws-load-balancer-type": "external" - } - } - } - } - } -} -``` - -The example above for AWS shows the choice to provision a Network Load Balancer instead of the default Classic Load Balancer. You can also set additional tags on the load balancer and enable client IP preservation. - -Note: For this to work, install the AWS Load Balancer Controller on your cluster. - -If you are adding configuration annotations to an existing load balancer service, you may need to re-create it in some cases for them to work properly. The easiest way to do this is to uninstall the `ofas-ingress-nginx` component. It is automatically re-installed with the new configuration. - -```sh -helm delete ofas-ingress-nginx -n spot-system -``` - -### AWS PrivateLink - -You can configure your cluster so that the inbound connections go over an [AWS PrivateLink](https://aws.amazon.com/privatelink/) if you are using AWS. You can use this [Terraform example](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/from-scratch-with-private-link) to set this up easily. Read on for more details. - -**Prerequisites** - -The infrastructure necessary for the AWS PrivateLink connection must be set up in your AWS account and must include: - -- A target group with IP address target type, handling IPv4 traffic over TCP on port 443. -- A network load balancer that forwards traffic to the target group. This load balancer can be internal. -- A VPC endpoint service. This VPC endpoint service should be connected to the network load balancer. You can set acceptance required to false and add the OfAS AWS account as an allowed principal (arn:aws:iam::066597193667:root). See the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html) for more details. -- The AWS Load Balancer Controller must be installed on your cluster. - -**Example of a cluster configuration:** - -```json -{ - "cluster": { - "config": { - "ingress": { - "loadBalancer": { - "managed": false, - "targetGroupArn": "arn:aws:elasticloadbalancing:region:XXXXXXXXXXXX:targetgroup/target-group-name/XXXXXXXXXXXXXXXX" - }, - "privateLink": { - "enabled": true, - "vpcEndpointService": "com.amazonaws.vpce.region.vpce-svc-XXXXXXXXXXXXXXXXX" - } - } - } - } -} -``` - -Explanation of the example: - -- `loadBalancer.managed = false` – OfAS does not provision a load balancer for the OfAS cluster. -- `loadBalancer.targetGroupArn` - The ARN of the target group linked with the load balancer you want to use. OfAS deploys a `TargetGroupBinding` called `ocean-spark-ingress` in the `spot-system` namespace linking the ingress controller service with the target group. -- `privateLink.enabled = true` - Enable AWS PrivateLink connection between the OfAS control plane and OfAS cluster. -- `privateLink.vpcEndpointService` - The VPC endpoint service name. - -An endpoint connection appears on your endpoint service, with the owner `066597193667`. The endpoint connection should be marked as available. - -In addition, ensure that you set up your node security group to allow traffic from the load balancer. - -### Use an Existing Load Balancer - -If you are using AWS, you can have OfAS use an existing load balancer instead of provisioning a new one. OfAS will still install an ingress controller, but it will be linked with an existing load balancer using a `TargetGroupBinding`. - -**Prerequisites** - -- A target group with IP address target type, handling IPv4 traffic over TCP on port 443. -- An internet facing (network) load balancer that forwards traffic to the target group. -- The AWS Load Balancer Controller should be installed in your cluster. - -**Example of a cluster configuration:** - -```json -{ - "cluster": { - "config": { - "ingress": { - "loadBalancer": { - "managed": false, - "targetGroupArn": "arn:aws:elasticloadbalancing:region:XXXXXXXXXXXX:targetgroup/XXX/XXX" - }, - "customEndpoint": { - "enabled": true, - "address": "custom-address.example.com" - } - } - } - } -} -``` - -Explanation of the example: - -- `loadBalancer.managed = false` - OfAS will not provision a load balancer for the OfAS cluster. -- `loadBalancer.targetGroupArn` - The ARN of the target group linked with the load balancer you want to use. OfAS will deploy a `TargetGroupBinding` called `ocean-spark-ingress` in the `spot-system` namespace linking the ingress controller service with the target group. -- `customEndpoint`- Set `enabled` to true and in the `address` field, enter the address that the OfAS control plane will use to communicate with the OfAS cluster. This is the DNS name of your public load balancer. - -Ensure that you set up your node security group to allow traffic from the load balancer. - -### Node Security Group - -Your node security group must allow traffic from the load balancer, over TCP on port 443. For the load balancer health checks, you can either allow traffic from your VPC CIDR, or the private IP addresses used by the load balancer nodes. See the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html#target-security-groups) for more information. - -If everything is configured correctly, you should see a healthy target in your target group. The target IP address should be the IP address of the `ofas-ingress-nginx-controller` pod running in the `spot-system` namespace. - -In addition, ensure that client IP preservation is enabled and open your node security group to traffic from the OfAS control plane. See the section on allowing traffic from the OfAS control plane for more details. - -### Use an Existing Ingress Controller - -Ocean for Apache Spark installs an `ingress-nginx` ingress controller by default. The ingress controller resides in the `spot-system` namespace and is called `ofas-ingress-nginx`. The `ofas-ingress-nginx` controller is configured to only manage Ocean for Apache Spark ingresses using an ingress class with the name `spot-bigdata-nginx`. You can therefore safely allow Ocean for Apache Spark to deploy its own ingress controller, it will live alongside your existing ingress controller. - -However, you can configure the Ocean for Apache Spark installation to use an existing ingress controller instead of installing a new one. For example, this is useful if you want to use Istio on your OfAS cluster. - -**Example of a cluster configuration:** - -```json -{ - "cluster": { - "config": { - "ingress": { - "controller": { - "managed": false - }, - "loadBalancer": { - "managed": false - }, - "customEndpoint": { - "enabled": true, - "address": "custom-address.example.com" - } - } - } - } -} -``` - -Explanation of the example: - -- `controller.managed = false` - OfAS will not install an ingress controller on the OfAS cluster. -- `loadBalancer.managed = false` - OfAS will not provision a load balancer for the OfAS cluster. -- `customEndpoint` - Set `enabled` to true and the `address` field should contain the address that the OfAS control plane will use to communicate with the OfAS cluster. For example, this can be the `loadBalancer.ingress.hostname` of the `LoadBalancer` service on your OfAS cluster. - -#### Setting up Ingresses - -If you use your own ingress controller, the OfAS installation will not provision ingresses for the services that need to be exposed. This must be done separately. - -Ingresses are protected by mutual TLS. The `bigdata-operator` component is responsible for creating and updating a secret called `spot-bigdata-tls`, which contains the TLS certificates. - -By default, this secret is created in the `spot-system` namespace. If this secret is required in a different namespace, you can label the namespace with `operator.bigdata.spot.io/injectTLSSecret=true`: - -```sh -kubectl label namespace example-namespace operator.bigdata.spot.io/injectTLSSecret=true -``` - -Connections from the OfAS control plane to the OfAS cluster use a hostname of the form `org-$SPOT_ORGANIZATION_ID-$OCEAN_SPARK_CLUSTER_ID.bigdata.svc.cluster.local`, where `$SPOT_ORGANIZATION_ID` is the Spot organization ID and `$OCEAN_SPARK_CLUSTER_ID` is the OfAS cluster ID. - -Example: `org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local` - -To find your Spot organization ID, in the menu in the top right corner of the Spot console click My Organzation. The OfAS cluster ID is in the cluster list in the Spot console in the form `osc-XXXXXXXX`. - -##### Ingress-nginx - -Example ingress definitions (change the host name and the ingress class): - -```yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - annotations: - nginx.ingress.kubernetes.io/auth-tls-secret: spot-system/spot-bigdata-tls - nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" - nginx.ingress.kubernetes.io/proxy-read-timeout: "600" - nginx.ingress.kubernetes.io/rewrite-target: /$1 - name: bigdata-notebook-service - namespace: spot-system -spec: - ingressClassName: nginx - rules: - - host: org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local - http: - paths: - - backend: - service: - name: bigdata-notebook-service - port: - number: 80 - path: /bigdata-notebook-service/?(.*) - pathType: Prefix - tls: - - hosts: - - org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local - secretName: spot-bigdata-tls ---- -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - annotations: - nginx.ingress.kubernetes.io/auth-tls-secret: spot-system/spot-bigdata-tls - nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" - nginx.ingress.kubernetes.io/rewrite-target: /$1 - name: bigdata-proxy - namespace: spot-system -spec: - ingressClassName: nginx - rules: - - host: org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local - http: - paths: - - backend: - service: - name: bigdata-proxy - port: - number: 80 - path: /bigdata-proxy/?(.*) - pathType: Prefix - tls: - - hosts: - - org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local - secretName: spot-bigdata-tls -``` - -##### Istio Ingress Gateway - -The TLS secret must be in the namespace that your gateway is installed in. Ensure to enable TLS secret injection for that namespace, for example: - -```sh -kubectl label namespace istio-ingress operator.bigdata.spot.io/injectTLSSecret=true -``` - -If this is not done, you may see warning logs like this in the log output of your `istiod` service: - -```sh -$ kubectl logs istiod-584b74f7f9-XXXX -n istio-system - -istiod-584b74f7f9-g7wkz discovery 2023-03-22T12:46:56.728332Z warn ads failed to fetch ca certificate for kubernetes://spot-bigdata-tls-cacert: secret istio-ingress/spot-bigdata-tls not found -istiod-584b74f7f9-g7wkz discovery 2023-03-22T12:46:56.728347Z warn ads failed to fetch key and certificate for kubernetes://spot-bigdata-tls: secret istio-ingress/spot-bigdata-tls not found -``` - -Now you can create a `Gateway` and `VirtualService` to expose the `bigdata-proxy` and `bigdata-notebook-service` services. For example (remember to change the host name): - -```yaml -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: ocean-spark-gateway - namespace: spot-system -spec: - selector: - istio: ingress - servers: - - port: - number: 443 - name: https - protocol: HTTPS - tls: - mode: MUTUAL - credentialName: spot-bigdata-tls - hosts: - - org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local ---- -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: ocean-spark-virtual-service - namespace: spot-system -spec: - hosts: - - org-XXXXXXXXXXXX-osc-XXXXXXXX.bigdata.svc.cluster.local - gateways: - - ocean-spark-gateway - http: - - match: - - uri: - prefix: /bigdata-proxy/ - rewrite: - uri: / - route: - - destination: - host: bigdata-proxy.spot-system.svc.cluster.local - port: - number: 80 - - match: - - uri: - prefix: /bigdata-notebook-service/ - rewrite: - uri: / - route: - - destination: - host: bigdata-notebook-service.spot-system.svc.cluster.local - port: - number: 80 -``` - -Ensure that the Gateway selector matches your installation (`istio: ingress` in the example above). It must match the ingress gateway pod labels. See the [Istio documentation](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/) for more information. - -Also ensure to turn off Istio sidecar injection in your Spark application namespaces. - -### Install the AWS Load Balancer Controller - -To use advanced load balancer configurations on AWS, install the AWS Load Balancer Controller on your cluster. See [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) for instructions. - -Ensure that your node security group allows traffic from the Kubernetes control plane security group over TCP on port 9443 (default). If not, you may see errors in the AWS Load Balancer Controller logs indicating that the AWS Load Balancer Controller webhook cannot be reached. - -### Configure Timeouts - -When a notebook starts, the OfAS control plane issues a long running request that waits for the notebook to start running. Therefore, Spot recommends to increase the request timeout. One way to do this is with an annotation on your LoadBalancer service, for example: - -```yaml -# AWS -service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "1800" - -# Azure -service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "30" -``` - -### Allow Traffic from the OfAS Control Plane - -You can ensure that only the OfAS control plane can reach your nodes by setting the `loadBalancerSourceRanges` of your `LoadBalancer` service to the IP address of the OfAS control plane (54.198.192.164/32): - -```yaml -spec: - loadBalancerSourceRanges: - - 54.198.192.164/32 -``` - -You can also do this manually, by adding an entry to your node security group to allow traffic over TCP on port 443 from source `54.198.192.164/32`. - -In addition, ensure that client IP addresses are preserved. - -On AWS, you can find this under Target Group > Attributes > Preserve client IP addresses. You can also set an annotation on your `LoadBalancer` service: - -```sh -service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true -``` - -If this is not done and you have allowed traffic from the load balancer to your nodes over TCP on port 443 (for health checks), all inbound traffic will be allowed through the load balancer. - -See the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) for more information. - -## Log Collection - -The OfAS system collects logs from Spark applications. These logs are: - -- The Spark event log. -- The Spark driver log. -- The Spark executor logs. - -The Spark event log contains information from the Spark runtime. It is used to calculate metrics for the Spark application, analyze its efficiency and identify issues. It is also used to power the Spark UI, after the Spark application has finished. - -The Spark driver and executor logs are the log streams from the Spark pods. They are collected and made available for download after the Spark application has finished. - -You can turn off the driver and executor log collection on your OfAS cluster. - -**Example of a cluster configuration:** - -```json -{ - "cluster": { - "config": { - "logCollection": { - "collectDriverLogs": false - } - } - } -} -``` - -## Spark Application Namespaces - -By default, Ocean for Apache Spark runs Spark applications in the spark-apps namespace. - -You can configure additional Spark application namespaces in your OfAS cluster. - -**Example of a cluster configuration**: - -```json -{ - "cluster": { - "config": { - "spark": { - "appNamespaces": ["extra-spark-app-ns-1", "extra-spark-app-ns-2"] - } - } - } -} -``` - -The `spark-apps` namespace is always the default if no namespace is specified in the Spark application submission. It is not necessary to create these namespaces beforehand, the `bigdata-operator` component creates them if they do not exist. - -### Istio Sidecar Injection - -If you are using Istio, disable the Istio sidecar injection in the Spark application namespaces since the sidecar can interfere with the Spark pod lifecycle. - -For example: - -```sh -kubectl label namespace spark-apps istio-injection=disabled --overwrite -``` - -See the [Istio documentation](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#controlling-the-injection-policy) for more information. - -## Configure Webhook - -The Spark Operator runs a `MutatingAdmissionWebhook` to mutate Spark pods as they are submitted to the cluster. - -If you are using AWS EKS and using a custom CNI like Calico, you may need to enable host networking for the webhook to work. - -An example of a cluster configuration: - -```json -{ - "cluster": { - "config": { - "webhook": { - "useHostNetwork": true, - "hostNetworkPorts": [25554] - } - } - } -} -``` - -Ensure that your node security group allows traffic from the Kubernetes control plane security group over TCP on the host network port. If it is not allowed, you may see errors like `Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-apps] failed` and `java.net.SocketTimeoutException: timeout` in the Spark Operator logs. - -## Configure VNGs - -This section shows you how to configure Virtual Node Groups (VNGs) for Ocean for Apache Spark. - -### Automatically Created VNGs - -Ocean for Apache Spark creates three Virtual Node Groups (VNGs) by default on your Ocean cluster when the OfAS cluster is created: - -- ocean-spark-on-demand -- ocean-spark-spot -- ocean-spark-system - -If you prefer to create these VNGs using other methods, like Terraform, you can disable this. - -The automatically created VNGs use taints by default to prevent non-Spark workloads from running on them. You can disable them. - -**Example of a cluster configuration**: - -```json -{ - "cluster": { - "config": { - "compute": { - "createVngs": true/false, - "useTaints": true/false - } - } - } -} -``` - -Note: These configuration options only affect the initial cluster creation call. - -## Create ARM VNGs - -To run ARM workloads on Ocean for Apache Spark, you first need to create VNGs to manage your ARM instances. Similarly to AMD, you need to create two VNGs, one for On-demand instances and one for Spot instances. The last step is to dedicate the VNGs to Ocean for Apache Spark. - -### ARM VNG setup - -This section shows you how to set up ARM VNGs on the different cloud providers. - -#### AWS - -Select an Amazon Machine Image suitable for ARM. View [this list](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) to select the right image according to the AMI ID in your region. - - - -#### GCP - -1. Create a GKE node pool. Select the T2A machine type from that node pool. -2. Create a new VNG by importing that node pool into your cluster. - - - -### Configure App - -When ARM based VNGs are created in your cluster, **it is important to note that all your apps, including existing ones, should now explicitly target a specific machine type, even AMD-based apps**. If this is not done, the AMD apps might run on ARM instances and vice versa. This must be done for both driver and executors in the app configuration. Use the `vngIds` field to target your ARM or AMD VNGs accordingly. - -```json -"driver": { - "coreRequest": "500m", - "memory": "1000m", - "spot": false, - "vngIds": ["ols-XXXXXXXX"] -}, -"executor": { - "cores": 2, - "coreRequest": "500m", - "memory": "1000m", - "vngIds": ["ols-XXXXXXXX"] -} -``` - -Alternatively, you can limit the choice of instance type by using the `instanceAllowList` field: - -```json -"driver": { - "coreRequest": "500m", - "memory": "1000m", - "spot": false, - "instanceAllowList": ["c6gn", "c6g", "m6g", "r6g"] -}, -"executor": { - "cores": 2, - "coreRequest": "500m", - "memory": "1000m", - "instanceAllowList": ["c6gn", "c6g", "m6g", "r6g"] -} -``` - -When setting a value for the `image` field, check that ARM64 compatible images are being used. Starting with `gen19`, the spark images that Ocean for Apache Spark provides are multi-architecture and compatible with ARM64. These images are used by default if you don't provide a value for the `image` field. - -## What's next? - -Learn how to [access your data](ocean-spark/configure-spark-apps/access-your-data). diff --git a/src/docs/ocean-spark/configure-permissions/README.md b/src/docs/ocean-spark/configure-permissions/README.md deleted file mode 100644 index 5beab9d058..0000000000 --- a/src/docs/ocean-spark/configure-permissions/README.md +++ /dev/null @@ -1,378 +0,0 @@ - - -# Configure Permissions - -This topic describes setting up permissions for **Ocean for Apache Spark** product users. - -If you're unfamiliar with how permissions work in Spot, see [Permission Policies](https://docs.spot.io/administration/policies/). - -## Overview - -Ocean for Apache Spark lets you configure permission policies for **Actions** for these resource types: - -- Clusters -- Apps -- Jobs -- Configuration templates -- Workspaces - -All these resource types are at least bound to a **Cluster** resource. -In Ocean for Apache Spark, clusters are identified by an id with the format `osc-xxxxxxxx` - -## Account-Level Managed Permission Policies - -To get started, you can use one of the following managed policies, which apply at the account level. -If you want to set permissions with more granularity, see [Granular Permissions Policies How-to Guides](#granular-permissions-policies-how-to-guides). - -| Policy | Level | Effect | Product Scope | Ocean Spark Resource Scope | -| ------------------------------------- | ------------ | --------------------------------------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| Account Viewer | Spot account | Give read access | All Spot products including Ocean for Apache Spark | All Resources | -| Account Editor | Spot account | Give edit access
**including app submission** | All Spot products including Ocean for Apache Spark | All Resources | - -## Granular Permissions Policies How-to Guides - -The patterns described below are for "edit" actions only. -Users should have read-access managed policies attached to their profile. - -### How to Define Permissions for One or Several Clusters - -Cluster is the default resource to scope a permission policy. -You can do it by using the `resources` field in the policy : - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": ["spark:*"], - "resources": [ - "sparkClusterId:osc-cluster1", - "sparkClusterId:osc-cluster2", - "sparkConfigTemplateId:*", - "sparkJobId:*" - ] - } - ] -} -``` - -### How to Define Permissions for a Specific Set of config-templates - -You can specify a clusterId and several configTemplateId like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": ["spark:*"], - "resources": [ - "sparkClusterId:osc-cluster1", - "sparkConfigTemplateId:config-template-1", - "sparkConfigTemplateId:config-template-2", - "sparkJobId:*" - ] - } - ] -} -``` - -You can also use a wildcard `*` in the config-template resource value like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": ["spark:*"], - "resources": [ - "sparkClusterId:osc-cluster1", - "sparkConfigTemplateId:config-template-*-abc-*", - "sparkJobId:*" - ] - } - ] -} -``` - -> **Important Note 1:** If you specify several clusters and several config-templates, -> it will allow users to do operations related to any config-template specified against any cluster specified, even if you declare several `statements` with different `resources` arrays - -> **Important Note 2:** If you want to force users to use a `configTemplateId` in-app submissions, use the `condition` field as described below. - -Use the 'condition' field to allow operations only for a specific set of config templates against a specific set of clusters. -For example, to allow a user to: - -- use config-templates with pattern `ct-team-a-*` against `cluster-1` and `cluster-2` -- use config-templates with pattern `ct-notebook-*` only against `cluster-1` - -The permission might look like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": ["spark:*"], - "resources": ["*"], - "condition": { - "Or": [ - { - "And": [ - { - "Or": [ - { - "StringEquals": { - "sparkClusterId": "osc-cluster1" - } - }, - { - "StringEquals": { - "sparkClusterId": "osc-cluster2" - } - } - ] - }, - { - "StringPatternMatch": { - "sparkConfigTemplateId": "ct-team-a-*" - } - } - ] - }, - { - "And": [ - { - "StringEquals": { - "sparkClusterId": "osc-cluster1" - } - }, - { - "StringPatternMatch": { - "sparkConfigTemplateId": "ct-notebook-*" - } - } - ] - } - ] - } - } - ] -} -``` - -## Use Cases - -### Set Permissions to Isolate App Submission by Team - -Let's say you want each of your teams to have their own config-templates and to be able only to submit Spark applications using their config-templates. - -For each team, you can: - -1. Create config-templates using a specific pattern for the id - For example : `team-A-config-template-1`, `team-A-config-template-2`, `team-B-config-template-1` - -2. Attach a managed policy like `Account Viewer` to your users. - -3. Create and attach one of the following policies by team: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", // optional - "spark:createConfigTemplate", - "spark:updateConfigTemplate", - "spark:deleteConfigTemplate" - ], - "resources": ["sparkClusterId:*", "sparkConfigTemplateId:team-a*"] - } - ] -} -``` - -If you also want to restrict which cluster users are allowed to submit applications to: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", // optional - "spark:createConfigTemplate", - "spark:updateConfigTemplate", - "spark:deleteConfigTemplate" - ], - "resources": [ - "sparkClusterId:osc-xxxxxx", - "sparkConfigTemplateId:team-a*" - ] - } - ] -} -``` - -If you want to **force users to use a config-template when submitting an app**, use the `condition` field like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", // optional - "spark:createConfigTemplate", - "spark:updateConfigTemplate", - "spark:deleteConfigTemplate" - ], - "resources": ["*"], - "condition": { - "And": [ - { - "StringEquals": { - "sparkClusterId": "osc-xxxxxx" - } - }, - { - "StringPatternMatch": { - "sparkConfigTemplateId": "ct-team-a-*" - } - } - ] - } - } - ] -} -``` - -### Set Permissions for Notebook Users - -If you want to give your users access to the notebook feature using local Jupyter notebooks or a JupyterLab instance, -you can use this policy: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", - "spark:createNotebook", - "spark:updateNotebook", - "spark:deleteNotebook" - ], - "resources": ["*"] - } - ] -} -``` - -If you want to allow notebook use only for a subset of config-templates, you can do it like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", - "spark:createNotebook", - "spark:updateNotebook", - "spark:deleteNotebook" - ], - "resources": [ - "sparkClusterId:osc-xxxxxx", - "sparkConfigTemplateId:team-a*" - ] - } - ] -} -``` - -### Set Permissions for Workspace Users - -If you want to give your users only access to the integrated notebook workspace feature, -you can use this policy. - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", - "spark:createNotebook", - "spark:updateNotebook", - "spark:deleteNotebook", - "spark:createWorkspace", - "spark:updateWorkspace", - "spark:deleteWorkspace", - "spark:createWorkspaceProxy", - "spark:updateWorkspaceProxy", - "spark:deleteWorkspaceProxy" - ], - "resources": ["*"] - } - ] -} -``` - -If you want to allow workspace use only for a subset of config-templates, you can do it like this: - -```json -{ - "statements": [ - { - "effect": "ALLOW", - "actions": [ - "spark:createApplication", - "spark:deleteApplication", - "spark:createNotebook", - "spark:updateNotebook", - "spark:deleteNotebook", - "spark:createWorkspace", - "spark:updateWorkspace", - "spark:deleteWorkspace", - "spark:createWorkspaceProxy", - "spark:updateWorkspaceProxy", - "spark:deleteWorkspaceProxy" - ], - "resources": [ - "sparkClusterId:osc-xxxxxx", - "sparkConfigTemplateId:team-a*" - ] - } - ] -} -``` - -## Advanced Policy Patterns - -If you want more complex rules and combinations between resources, you can use the `condition` field. -Learn more here: [Policy conditions](/administration/policies/create-new-policy?id=policy-conditions) - -## Reference - -All actions listed below are "edit" permissions - -| Actions | Resources bound | -| -------------------------------------------------------------------------------------------- | ----------------------------------------------- | -| `spark:createApplication` | - `sparkClusterId`
- `sparkConfigTemplateId` | -| `spark:deleteApplication` | - `sparkClusterId` | -| `spark:createCluster`
`spark:updateCluster`
`spark:deleteCluster` | - `sparkClusterId` | -| `spark:createConfigTemplate` | - `sparkClusterId` | -| `spark:updateConfigTemplate`
`spark:deleteConfigTemplate` | - `sparkClusterId`
- `sparkConfigTemplateId` | -| `spark:createNotebook`
`spark:updateNotebook`
`spark:deleteNotebook` | - `sparkClusterId` | -| `spark:createWorkspace`
`spark:updateWorkspace`
`spark:deleteWorkspace` | - `sparkClusterId` | -| `spark:createWorkspaceProxy`
`spark:updateWorkspaceProxy`
`spark:deleteWorkspaceProxy` | - `sparkClusterId` | -| `spark:updateJob`
`spark:updateConfig` | - `sparkClusterId`
- `sparkJobId` | -| `spark:createVirtualNodeGroup`
`spark:deleteVirtualNodeGroup` | - `sparkClusterId` | diff --git a/src/docs/ocean-spark/configure-spark-apps/README.md b/src/docs/ocean-spark/configure-spark-apps/README.md deleted file mode 100644 index 42ce07acc5..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/README.md +++ /dev/null @@ -1,164 +0,0 @@ - - -# Configure Spark Applications - -This section shows you how to configure critical aspects of your Spark applications, such as how to control permissions to [access data](ocean-spark/configure-spark-apps/access-your-data), how to [package your code](ocean-spark/configure-spark-apps/package-spark-code) (and install libraries), how to configure the size of your Spark containers, and more. - -Before diving into these topics, it is important to realize that the final configuration of a Spark application is the result of applying multiple sources of configuration inputs. By order of precedence: - -### Source #1: Config Overrides (Highest precedence) - -This is an application-specific configuration that you specify directly in your [API request](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). -This source of configuration takes precedence over the other sources. - -### Source #2: Job Configurations - -Configurations defined at the level of a job are automatically applied to the future executions of the job. -In other words, applications inherit the configurations defined at the job level. - -Job configurations are a handy way to define fields such as the `mainApplicationFile`, the file corresponding to your job. You can also insert specific configurations at this level to improve the performance of your jobs. -For example, if a job requires a lot of memory, you may modify your job configuration to set the `instanceAllowList` field to target specifically high-memory instances. - -### Source #3: Auto-tuning - -These configurations are applied by Ocean for Spark to improve the performance and stability of your workloads. Some of these optimizations are static. For example, some Spark configurations are adjusted based on the Spark version you selected, while others are dynamically determined by the history of the past executions of a job or the real-time characteristics of your infrastructure. Auto-tuning takes precedence over configuration templates, but is overridden by config overrides. - -### Source #4: Configuration Templates (Lowest precedence) - -These are fragments of configuration that you can define in the Ocean Spark UI or API and then reuse across many notebooks and jobs. - -## Config Overrides - -Config overrides are fragments of configuration that you define when submitting Spark applications through the API. They are specified directly in the body of the POST request (https://api.spotinst.io/ocean/spark/cluster/{clusterId}/app): - -```yaml -curl -k -X POST \ - 'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{ - "jobId": "spark-pi", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.0", - "mainApplicationFile": "local:///opt/spark/examples/jars/examples.jar", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "mainClass": "org.apache.spark.examples.SparkPi", - "arguments": ["1000"] - } -}' -``` - -Config overrides have higher precedence than all other sources of configuration. As a result, they are useful: - -- To specify arguments that change at every execution of your [Spark job](ocean-spark/product-tour/monitor-jobs). For example, if you have an ETL pipeline processing data for a specific date, you should pass this date using config overrides. -- To forcefully ensure a specific configuration is applied (so that it cannot be changed by auto-tuning or a configuration template). For example, you may have a technical reason to force Spark to use a certain codec for compression. - -All other configurations are better placed in the configuration templates. To know more about all the configurations you can set, check out the [API reference](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). - -## Configuration Templates - -Configuration templates are fragments of Spark application configuration stored in Ocean Spark. This is useful when you need to share a large default configuration between multiple applications, or to avoid storing configurations on your side. - -Following the example above, we'll use a configuration template to store stable configurations, i.e., those that don't change at every execution. - -There are two ways you can manage the configuration templates in your deployment: through the Spot Console or through the API. - -
- Spot Console - -To manage your configuration templates in the Spot console, go to Ocean for Spark in the menu tree and click Configuration Templates. You should see your current list of configuration templates. - - - -Click on "New Template" in the upper right corner and create a configuration template called `my-template` with the following content: - -```yaml -{ "sparkVersion": "3.2.0" } -``` - -To know more about the all configurations you can set in a template, check out the [API reference](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). - -
- -
- API - -The API routes under `https://api.spotinst.io/ocean/spark/cluster/{your-cluster-id}/configTemplate` let you manage configuration templates as a REST resource. - -To know more about the API routes and parameters, check out the [API reference](https://docs.spot.io/api/#tag/Ocean-Spark). - -The following command creates a configuration template with the ID my-template containing this block of Spark application configuration: - -```yaml -curl -X POST \ - https://api.spotinst.io/ocean/spark/cluster//configTemplate \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer \ - -d '{ - "id": "my-template", - "config": { - "sparkVersion": "3.2.0", - } -}' -``` - -
- -The configuration template can now be used as a kernel for a Jupyter notebook or referenced when submitting a Spark application using the field `configTemplateId`: - -```yaml -curl -X POST \ - 'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer \ - -d '{ - "jobId": "daily-upload", - "configTemplateId": "my-template", - "configOverrides": { - "type": "Scala", - "mainApplicationFile": "s3a://acme-assets/processing-1.0.0.jar", - "mainClass": "com.acme.processing.DailyUpload", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "arguments": ["2022-01-01"] - } -}' -``` - -Ocean Spark merges the configurations from the configuration template and the config overrides. The configurations in configOverrides have higher precedence than the configuration template. - -## Job Configurations - -Just like configuration templates, you can define job-specific configurations using the Spot console or the API (see API docs). - -To edit the configuration for a job from the Spot console, go to Ocean Spark in the menu tree, click on Jobs, find the Job that you're interested in, and then click on the Configuration Tab. - - - -Job configurations have a higher precedence than configuration templates, but a lower precedence than config overrides. - -### Auto-tuning - -Ocean Spark automatically tunes the infrastructure parameters and Spark configurations of your applications to improve their performance and stability. - -Some of these optimizations are static. For example certain sparkConf feature flags are adjusted based on the Spark version you use. Global defaults are also inserted, e.g., to make sure your applications run smoothly and efficiently on Kubernetes. - -Other optimizations are dynamically determined by the historical performance of previous executions of your job. After each application finishes, Ocean Spark analyzes the Spark events logs (the logs powering the Spark UI) automatically detect performance inefficiencies or stability issues. This is a dynamic learning process that requires multiple executions of the same workload (grouped within the same Job). Execution after execution, this auto-tuning process continues and adapts to the evolution of your Spark application, without any action on your part. - -You can track these automated tuning of configurations by going to the “Configuration” tab of a Spark application and clicking on “Show Sources”. For each configuration, you will see whether it came from a config override, a configuration template, or an auto-tuning setting. - -To give you control over the tuning process, the config overrides have precedence over auto-tuning, while configuration templates have not. So if you want to force a parameter (and disable auto-tuning), you can set it in the config overrides. On the other hand, parameters found in configuration templates will be tuned by Ocean Spark when it makes sense. The values you put in the configuration template will serve as a starting point for the algorithm. - -For example, if you set the following configuration fragment in the configuration template of your application: - -```yaml -{ "executor": { "instances": 10 } } -``` - -Ocean Spark might change the number of executors, in order to improve efficiency of your job. If you put this fragment in the config overrides section of your API call, then the number of executors will never change. - -We recommend that you put all performance parameters in a configuration template, so that auto-tuning can adjust them over time. - -## What’s Next? - -Learn more about how to [access your data](ocean-spark/configure-spark-apps/access-your-data). diff --git a/src/docs/ocean-spark/configure-spark-apps/access-your-data.md b/src/docs/ocean-spark/configure-spark-apps/access-your-data.md deleted file mode 100644 index cdb882bf67..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/access-your-data.md +++ /dev/null @@ -1,468 +0,0 @@ - - -# Access Your Data - -This page shows how to run your own code and access data hosted in your cloud account. It assumes that you know how to [run a Spark application](ocean-spark/getting-started/?id=run-your-first-app) on Ocean for Apache Spark. -Specify data in your arguments -Suppose you want to run a word count application against files hosted in an S3 bucket. - -The application file is hosted at s3a://my-example-bucket/word_count.py and takes two arguments, an input folder and an output folder. - -Here’s the payload you would submit: - -```yaml -curl -X POST \ - 'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{ - "jobId": "spark-pi", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.1", - "mainApplicationFile": "s3a://my-example-bucket/word_count.py", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "arguments": ["s3a://my-example-bucket/input/*", "s3a://my-example-bucket/output"] - } -}' -``` - -The application above will likely fail because the Spark pods do not have sufficient permissions to access the code and the data. - -Below are two ways to grant the Spark pods access to your data. - -## Grant permissions to node instances - -Spark pods running in Kubernetes inherit the permissions of the nodes they run on. So a simple solution is to grant the Kubernetes nodes access to your data. - -### AWS: Your data is in the same AWS account as the Ocean Spark cluster - -To allow the Kubernetes nodes and Spark pods to access your S3 buckets, complete the following steps: - -1. Create a data access policy for your S3 buckets. -2. Create an IAM role and attach the data access policy to it. -3. Configure the Ocean [Virtual Node Group](ocean/features/launch-specifications) of your Ocean Spark cluster to use the IAM role. - -To create a policy for your cluster nodes: -Sign in to the [IAM console](https://console.aws.amazon.com/iam/) with a user having administrator permissions. - -1. In the navigation pane, choose Policies. -2. In the content pane, choose Create policy. -3. Choose the JSON tab and define the policy. An example of policy for Kubernetes nodes could be: - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Effect": "Allow", - "Action": ["s3:GetBucketLocation", "s3:ListAllMyBuckets"], - "Resource": "arn:aws:s3:::*" - }, - { - "Effect": "Allow", - "Action": "s3:*", - "Resource": - ["arn:aws:s3:::", "arn:aws:s3:::/*"] - } - ] -} -``` - -Now create an IAM role and attach the policy to it: - -1. Sign in to the AWS Management Console and open the IAM console. -2. In the navigation pane, choose Roles and then click Create role. -3. Select AWS service as the type of trusted entities, and the EC2. Click Next: Permissions -4. Search for the policy you created above and mark it. -5. Click Next as many times as needed, choose a name for the IAM role, and create it. - -Finally, configure the Ocean Virtual Node Group (VNG) of your Ocean Spark cluster to use the IAM role: - -1. Sign in to the AWS Management Console and open the IAM console. -2. In the navigation pane, choose Roles -3. Search the roles you created above and open it. -4. Collect the name of the instance profile you associated with the role. - - - -5. In the [Spot console](https://console.spotinst.com/), navigate to the [configuration of the VNG](ocean/tutorials/manage-virtual-node-groups?id=view-vngs) used in your Ocean Spark cluster. -6. Configure the VNG to use the instance profile that you noted down above - - - -The Spark application will now be able to access the S3 bucket specified in the IAM policy you created. - -> **Note**: If after modifying the instance profile used by the Ocean Spark VNGs, the Kubernetes nodes failed to launch and register with the cluster, you may need to modify your aws-auth config map. The new role should be added to the groups `system:bootstrappers` and `system:nodes`. - -### AWS: Your data is in an AWS account other than the Ocean Spark cluster - -> **Tip**: This section assumes you are familiar with the previous one, "Your data is in the same AWS account as the Ocean Spark cluster”. The previous section provides more detailed explanations and acts as a tutorial. This section assumes a good knowledge of AWS Identity and Access Management (IAM). - -To allow the Kubernetes nodes and Spark pods to access your S3 buckets, complete the following steps: - -1. Create an IAM role in the AWS account where the data resides ("data account”). Note down the instance profile associated with this IAM role. -2. Add the following policy to the IAM role to grant it data access. - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Effect": "Allow", - "Action": ["s3:*"], - "Resource": - [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" - ] - } - ] -} -``` - -3. Create an IAM role in the AWS account where the Ocean Spark cluster resides ("cluster account”). -4. In the data account, authorize the cluster account IAM role to trust the data account IAM role. To do so, add the following to the data account IAM role’s trust policy: - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Effect": "Allow", - "Principal": - { - "AWS": "arn:aws:iam:::role/", - }, - "Action": "sts:AssumeRole" - } - ] -} -``` - -5. Conversely, in the cluster account, grant the cluster account IAM role the ability to assume the data account IAM role. - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Sid": "Stmt1487884001000", - "Effect": "Allow", - "Action": ["sts:AssumeRole"], - "Resource": - ["arn:aws:iam:::role/"] - } - ] -} -``` - -6. From Spot console, configure the Virtual Node Group. -7. In the Spot console, configure the Virtual Node Group (VNG) of your Ocean Spark cluster to use the instance profile associated to the cluster account IAM role. -8. Eventually, modify the submission payload of your Spark application to assume the data account IAM role. To do so, merge the following JSON fragment into the configuration of your Spark application: - -```yaml -{ - "hadoopConf": - { - "fs.s3a.stsAssumeRole.arn": "arn:aws:iam:::role/", - "fs.s3a.assumed.role.arn": "arn:aws:iam:::role/", - "fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider", - "fs.s3a.assumed.role.credentials.provider": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider", - }, -} -``` - -> **Note**: If after modifying the instance profile used by the Ocean Spark VNGs, the Kubernetes nodes failed to launch and register with the cluster, you may need to modify your aws-auth config map. The new role should be added to the groups `system:bootstrappers` and `system:nodes`. - -### GCP - -Find the service account used by GCE instances running as Kubernetes nodes. Depending on your setup, it can be the default Compute Engine service account, of the form - -``` -PROJECT_NUMBER-compute@developer.gserviceaccount.com -``` - -or another service account that you created yourself of the form - -``` -SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com -``` - -The default Compute Engine service account is used by default in a GKE cluster. -You can specify another service account for Spark applications in the settings of the [Virtual Node Group](ocean/features/launch-specifications) (VNGs) dedicated to Spark. - -A quick way to find out the service account name is to run a Spark application accessing your data. -When the application fails, the authorization error in the driver log usually shows the service account name. - -Once you have found the service account, grant it sufficient permissions using [IAM roles](https://cloud.google.com/iam/docs/overview). The list of IAM roles for GCS is available [here](https://cloud.google.com/storage/docs/access-control/iam-roles). - -## Grant permissions using Kubernetes secrets - -### AWS - -Spark pods can impersonate an IAM user (AWS) when provided with the user’s credentials. This technique completely overrides the IAM role assumed by the Kubernetes nodes (see previous section, Granting permissions to node instances). - -To protect those credentials, you will store them in Kubernetes secrets and configure Spark to mount those secrets into all the driver and executor pods as [environment variables](ocean-spark/configure-spark-apps/secrets-environment-variables). - -To let your Spark pods access your S3 buckets, complete the following steps: - -1. Create a data access policy for your S3 buckets. -2. Create a user that is granted the data access policy. -3. Create an access key for the user. -4. Create a Kubernetes secret that contains the access key. -5. Configure Spark to use the Kubernetes secret. - -Create a policy for your cluster nodes: - -1. Sign in to the [IAM console](https://console.aws.amazon.com/iam/) with a user having administrator permissions. -2. In the navigation pane, choose Policies. -3. In the content pane, choose Create policy. -4. Choose the JSON tab and define the policy. An example of policy for cluster nodes could be: - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Effect": "Allow", - "Action": ["s3:GetBucketLocation", "s3:ListAllMyBuckets"], - "Resource": "arn:aws:s3:::*" - }, - { - "Effect": "Allow", - "Action": "s3:*", - "Resource": - ["arn:aws:s3:::", "arn:aws:s3:::/*"] - } - ] -} -``` - -Create a user having the data access policy: - -1. Sign in to the AWS Management Console and open the IAM console . -2. In the navigation pane, choose Users, and click Add User. -3. Give it a name and check "Programmatic access". -4. Click "Attach existing policies directly" and attach the policy you just created. -5. Complete the user creation process. - -A user with the correct policy is now created. - -Create an access key for the user: - -1. Sign in to the AWS Management Console and open the IAM console. -2. In the navigation pane, choose Users, and click on the user you just created. -3. Go to the "Security credentials" tab and click "Create access key". -4. Note down the access key ID and the access key secret. - -#### Create a Kubernetes secret that contains the access key - -The [kubectl command](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) below generates a secret from the access key ID and the access key secret created in the previous steps: - -``` -kubectl create secret -n spark-apps generic data-access \ - --from-literal 'AWS_ACCESS_KEY_ID=' \ - --from-literal 'AWS_SECRET_ACCESS_KEY=' -``` - -#### Configure Spark to use the Kubernetes secret - -You can now modify the payload to launch the Spark application in order to reference the secret: - -```bash -curl -X POST \ -'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{ - "job-id": "spark-pi", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.1", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "mainApplicationFile": "s3a://my-example-bucket/word_count.py", - "arguments": ["s3a://my-example-bucket/input/*", "s3a://my-example-bucket/output"], - "driver": { - "envSecretKeyRefs": { - "AWS_ACCESS_KEY_ID": { - "name": "data-access", - "key": "AWS_ACCESS_KEY_ID" - }, - "AWS_SECRET_ACCESS_KEY": { - "name": "data-access", - "key": "AWS_SECRET_ACCESS_KEY" - } - } - }, - "executor": { - "envSecretKeyRefs": { - "AWS_ACCESS_KEY_ID": { - "name": "data-access", - "key": "AWS_ACCESS_KEY_ID" - }, - "AWS_SECRET_ACCESS_KEY": { - "name": "data-access", - "key": "AWS_SECRET_ACCESS_KEY" - } - } - } - } - } -}' -``` - -### GCP - -Create a service account in the GCP console, and grant it sufficient permissions using [IAM roles](https://cloud.google.com/iam/docs/overview). The list of IAM roles for GCS is [here](https://cloud.google.com/storage/docs/access-control/iam-roles). - -This bash script will create an access key for the service account, and store it in a secret called `data-access` in the Kubernetes namespace `spark-apps` where Spark applications are run: - -``` -TMP_FILE=$(mktemp) -gcloud iam service-accounts keys create $TMP_FILE --iam-account @.iam.gserviceaccount.com -kubectl create secret -n spark-apps generic data-access --from-file=key.json=$TMP_FILE) -``` - -Modify the payload to launch the Spark application in order to reference the secret: - -```bash -curl -X POST \ -'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{ - "job-id": "spark-pi", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.1", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "mainApplicationFile": "s3a://my-example-bucket/word_count.py", - "arguments": ["s3a://my-example-bucket/input/*", "s3a://my-example-bucket/output"], - "driver": { - "secrets": [ - { - "name": "data-access", - "path": "/mnt/secrets", - "secretType": "GCPServiceAccount" - } - ] - }, - "executor": { - "secrets": [ - { - "name": "data-access", - "path": "/mnt/secrets", - "secretType": "GCPServiceAccount" - } - ] - } - } - } -}' -``` - -## Grant permissions with a custom Kubernetes service account (AWS only) - -On EKS, [a Kubernetes service account can be associated with an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). -The Ocean Spark API allows you to leverage this feature to grant your Spark applications data access in a secure way using service accounts. - -This section explains the required steps: - -1. Create an IAM OIDC provider for your cluster (one-time operation) -2. Create an IAM role and grant it access to your data -3. Create a Kubernetes service account in the EKS cluster -4. Associate the Kubernetes service account with the IAM role -5. Configure your Spark application to use the custom service account - -#### Create an IAM OIDC provider for your cluster - -> **Info**: This operation must be performed only once for every EKS cluster. - -Please refer to [this page](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) of the AWS documentation. - -#### Create and IAM role and grant it access to your data - -- Create an IAM role in the AWS account. Note down the ARN of this role. In the rest of the document, the example ARN will be called `arn:aws:iam::111122223333:role/replace-me`. -- Add the following policy to the IAM role to grant it data access. - -```yaml -{ - "Version": "2012-10-17", - "Statement": - [ - { - "Effect": "Allow", - "Action": ["s3:*"], - "Resource": - [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" - ] - } - ] -} -``` - -#### Create a Kubernetes service account in the EKS cluster - -```bash -kubectl create serviceaccount -n spark-apps data-writer -kubectl create rolebinding -n spark-apps data-writer-pod-manager-rb --role pod-manager --serviceaccount spark-apps:data-writer -``` - -The above bash snippet does the following: - -- Create a service account `data-writer` (this is an example name) in namespace `spark-apps`, where the Spark applications live. -- Bind the Kubernetes role `pod-manager` to service account `data-writer`. This is required so that the Spark driver can interact with Kubernetes, requesting executor pods for example. - -#### Associate the Kubernetes service account with the IAM role - -To tell EKS to associate a Kubernetes service account with an IAM role, an annotation must be added to the service account. - -```bash -kubectl annotate -n spark-apps serviceaccounts data-writer \ - eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/replace-me -``` - -Continuing our example, the above snippet tells EKS to associate service account `data-writer` to IAM role `arn:aws:iam::111122223333:role/replace-me`. This is an example ARN and should be replaced with the ARN noted down in section "Create and IAM role and grant it access to your data". - -#### Configure your Spark application to use the custom service account - -Use the field `serviceAccountName` exposed by the Ocean Spark API to configure the Spark application to use service account `data-writer`. - -Additionally, Hadoop (on which Spark relies to interacts with S3) must be configured to use the OIDC provider created in the first step of this section. -This is achieved by setting the Hadoop configuration `fs.s3a.aws.credentials.provider` to `com.amazonaws.auth.WebIdentityTokenCredentialsProvider`. - -Here is a full example: - -```bash -curl -X POST \ -'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{ - "job-id": "spark-pi", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.1", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "mainApplicationFile": "s3a://my-example-bucket/word_count.py", - "arguments": ["s3a://my-example-bucket/input/*", "s3a://my-example-bucket/output"], - "serviceAccountName": "data-writer", - "hadoopConf": { - "fs.s3a.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider" - } - } - } -}' -``` - -## What's Next? - -Learn how to [package Spark code](ocean-spark/configure-spark-apps/package-spark-code). diff --git a/src/docs/ocean-spark/configure-spark-apps/common-spark-configs.md b/src/docs/ocean-spark/configure-spark-apps/common-spark-configs.md deleted file mode 100644 index d1a43a1053..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/common-spark-configs.md +++ /dev/null @@ -1,142 +0,0 @@ - - -# Common Spark Configurations - -This page describes some common Spark configurations relevant to Ocean Spark. - -## Control the number of executors - -A Spark application can either use a fixed number of executors, or it can dynamically adjust the number -of Spark executors at runtime based on the load (dynamic allocation). - -Dynamic allocation is enabled by default for interactive notebooks. For applications submitted through the API, -by default Ocean Spark will use 6 executors. - -You can control how many executors to use by modifying this configuration: -```json -{ - "sparkConf": { - "spark.dynamicAllocation.enabled": "false" - }, - "executor": { - "instances": 6 - } -} -``` - -## Dynamic allocation - -For Spark versions 3.0 and above, dynamic allocation is enabled by default on your notebooks. - -It will cause the Spark driver to dynamically adjust the number of Spark executors at runtime based on load: -- When there are pending tasks, the Spark driver will request more executors. -- When an executor is idle for a while (not running any task), it is removed. - -Here is an example configuration fragment to enable dynamic allocation. The fields should be self-explanatory. - -```json -{ - "sparkConf": { - "spark.dynamicAllocation.enabled": "true", - "spark.dynamicAllocation.minExecutors": "0", - "spark.dynamicAllocation.maxExecutors": "25", - "spark.dynamicAllocation.initialExecutors": "1" - } -} -``` - -Dynamic Allocation works both for batch and for streaming queries. You can read more about it in the [Apache Spark documentation](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation). - -Dynamic allocation is a great way to save on your cloud costs for interactive workloads (notebooks) where using a fixed number of executors often leads to wasted resources. - -## Application Timeout - -You can configure a duration after which a Spark application will be forcibly terminated (timeout). -By default, Ocean Spark sets a 24-hour (1440 minutes) timeout on Spark applications. - -You can change this timeout duration by using the following configuration: -```json -"timeout": { - "minutes": 120, - "policy": "KILL" -} -``` - -You can also disable this timeout entirely - for example if you're running streaming applications, as follows: -```json -"timeout": "DISABLED" -``` - -Additional notes: -- Timed out applications wll enter the terminal "Timed Out" state. -- The timeout clock starts ticking once you make the API call to submit a Spark application, or once you -open up a notebook. This can be a few seconds or a few minutes before your Spark code starts running. -- Ocean Spark checks applications every 5 minutes to enforce their timeout. As a result, setting a very short -timeout (or a very precise timeout) may not produce the desired effect. Applications should be -timed out a few minutes after they reach their configuration timeout duration. - -## Enable Adaptive Query Execution (AQE) - -[Adaptive Query Execution](https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution) is a Spark performance optimization feature available from Spark 3.0 and enabled by default from Spark 3.2. You can enable or disable it by switching the corresponding sparkConf flag: - -```json -{ - "sparkConf": { - "spark.sql.adaptive.enabled": "true" - } -} -``` - -## Graceful executor decommissioning - -This Spark feature is available, and automatically enabled, for Spark versions 3.1 and above. -When enabled, an executor will try to move its shuffle and RDD blocks to another executor before exiting. -In particular, in the event of a spot instance termination, Ocean Spark will leverage the termination notice period -given by the cloud provider, to proactively move the shuffle files and hence avoid any disruption. - -Here are the 4 main Spark configuration flags to enable this feature: - -```json -{ - "sparkConf": { - "spark.decommission.enabled": "true", - "spark.storage.decommission.enabled": "true", - "spark.storage.decommission.rddBlocks.enabled": "true", - "spark.storage.decommission.shuffleBlocks.enabled": "true" - } -} -``` - -You can add an additional flag to handle the situation where a single executor is decomissioned. -In this scenario, there's no other executor who can receive the shuffle files, so you can configure to use -an object storage as fallback storage. Here's an example configuration: - -```json -{ - "sparkConf": { - "spark.storage.decommission.fallbackStorage.path": "s3a:///decom/" - } -} -``` - -Spark executors will need to have read and write permissions to the target storage. - -## Using the S3A protocol instead of S3 - -The [S3 protocol has been deprecated in favor of S3A since Hadoop 3.x](https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html#Introducing_the_Hadoop_S3A_client.), because S3A provides better performance and security. - -You should therefore always use S3 paths starting with "s3a://", attempting to use an "s3://" path would give you an error "No FileSystem for scheme 's3'". - -If you can't change the path, there's a workaround to instruct Spark to actually use the S3AFileSystem when it encounters an "s3://" path, by adding the following configuration to your applications: - -```json -{ - "sparkConf": { - "spark.hadoop.fs.s3.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem" - } -} -``` - -## What’s Next? - -Learn more about [secrets and environment variables](ocean-spark/configure-spark-apps/secrets-environment-variables). diff --git a/src/docs/ocean-spark/configure-spark-apps/docker-images.md b/src/docs/ocean-spark/configure-spark-apps/docker-images.md deleted file mode 100644 index 43973ccdfc..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/docker-images.md +++ /dev/null @@ -1,123 +0,0 @@ - - -# Docker Images - -Ocean Spark maintains a popular fleet of Docker images for Apache Spark. - -## What's a Docker image for Spark? - -When Spark runs on Kubernetes, the driver and executors are Docker containers that execute a Docker image specifically built to run Spark. - -## What’s in these Docker Images? - -In addition to a version of Spark itself, the `spark:platform` images include connectors to popular [object stores](ocean-spark/configure-spark-apps/docker-images?id=data-source-connectors): (S3, GCS, ADLS), Snowflake, Delta Lake, Python support with pip and conda, Jupyter notebook support, Hadoop, AWS Glue Catalog, and more. - -### Images to start with - -| Full Image name | Spark Version | Scala Version | Python Version | Hadoop Version | -| :------------------------------------------: | :-----------: | :-----------: | :------------: | :------------: | -| gcr.io/ocean-spark/spark:platform-3.5-latest | 3.5.3 | 2.12 | 3.10 | 3.3.6 | -| gcr.io/ocean-spark/spark:platform-3.4-latest | 3.4.4 | 2.12 | 3.10 | 3.3.6 | - -### How to use those images for your apps and jobs? - -When submitting Spark apps on Ocean for Apache Spark, you can: - -- Omit the image field. In this case, `spark:platform` will be used by default according to the Spark version specified in the `sparkVersion` field. If both image and sparkVersion fields are specified, the Spark version of the image takes precedence. -- Specify the image in your configuration with the image field (using a `configOverrides` directly in your API call, or using a configuration template). - -### Need another image? - -To match different dependencies and version requirements you can find more images in Spot's [Docker registry](https://gcr.io/ocean-spark/spark). - -All these dependencies can have different versions. A combination of dependency versions is called a flavor of spark:platform in this page. The image tag indicates the flavor of the image and can be adjusted to fit your needs. Here are two examples of image tags: - -``` -gcr.io/ocean-spark/spark:platform-3.3.0-latest -gcr.io/ocean-spark/spark:platform-3.3.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-latest -``` - -### Need to build your own Image? - -You should use one of the spark:platform images as a base. Once your custom image is in your local docker repository you have to Tag and Push it, see [Set up a Docker registry](ocean-spark/configure-spark-apps/package-spark-code?id=set-up-a-docker-registry-and-push-your-image) and push your image. - -## Data source connectors - -The image tags `gcr.io/ocean-spark/spark:platform` supports for the following data sources: - -- AWS S3 (s3a:// or s3:// scheme) -- Google Cloud Storage (gs:// scheme) -- Azure Blob Storage (wasbs:// scheme) -- Azure Datalake generation 1 (adl:// scheme) -- Azure Datalake generation 2 (abfss:// scheme) -- [Snowflake](https://docs.snowflake.com/en/user-guide/spark-connector.html) -- [Delta Lake](https://docs.delta.io/latest/index.html) -- [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) - -To check the versions used by an image, see the [release notes](ocean-spark/docker-images-release-notes/). - -## Python support - -The image tag `gcr.io/ocean-spark/spark:platform` supports Pyspark applications. When [building a custom image](ocean-spark/configure-spark-apps/package-spark-code) or working from a notebook, additional Python packages can be installed with pip or conda. - -## Image tags and flavors - -### Flavors - -Spot maintains Spark Docker images for multiple combinations of the versions of Spark and some of its dependencies (Hadoop, Java, Scala and Python). These combinations are called flavors. - -Note that not all the combinations of versions exist. To list all the flavors for a given image, check out the [Docker registry](https://gcr.io/ocean-spark/spark:platform). - -### Generations - -When Spot updates images, Spot creates a new generation. Generations can be identified by the suffix `-genXX` in the image tags (`-latest` always points to the latest generation). Each generation contains the following flavors: - -- Latest (at the time of the generation's release) Spark 3.x: - - - Java 8 flavor - - Java 11 flavor - -- Latest Spark 2.x: - - - Scala 2.11 flavor - - Scala 2.12 flavor - -- New versions of Spark 3.x that were released since the previous generation. - -- EMR compatible flavors: - - - Flavor compatible with latest EMR 6.x - - Flavor compatible with latest EMR 5.x - -### Best practices - -There are both long-form tags (like `gcr.io/ocean-spark/spark:platform-3.1.1-java-8-scala-2.12-hadoop-3.2.0-python-3.8-latest`) where all versions are explicitly listed, as well as short-form tags (like `gcr.io/ocean-spark/spark:platform-3.1-latest`). - -In general Spot encourages starting with Spot's short-form tags: - -- `gcr.io/ocean-spark/spark:platform-3.1.3-latest` contains a Spark 3.1.3 distribution and all other dependencies are set to the latest compatible version. For example, `platform-3.1.3-latest` contains Hadoop 3.2.0, Python 3.8, Scala 2.12, and Java 11. Spot reserves the right to upgrade the version of a dependency if a new, compatible version is released. For example, Spot can upgrade `platform-3.1.3-latest` to Hadoop 3.3.0 once it is compatible with Spark 3.1.3. -- `gcr.io/ocean-spark/spark:platform-3.1-latest` contains the latest Spark version of the 3.1 minor. - -Use a long-form only if you need a specific combination. For instance, you can require a specific combination of versions when migrating an existing Scala or Java project to Spark on Kubernetes. On the other hand, new JVM projects and Pyspark projects should work fine with short-form tags. - -For production workloads: - -- Spot doesn't recommend using the `-latest` tags. To keep the image stable you should use images with an explicit version suffix like `-gen18` below. The following images are the same: - - `gcr.io/ocean-spark/spark:platform-3.2-gen18` - - `gcr.io/ocean-spark/spark:platform-3.2.1-gen18` - - `gcr.io/ocean-spark/spark:platform-3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen18` -- Long-form tag images without the suffix version can change to the exclusion of the Spark, Hadoop, Java, Scala and Python versions specified in the image tag. - -See the release notes below to learn about the changes introduced by each version. - -## Alternative repositories - -In addition to Spot's [GCR](https://gcr.io/ocean-spark/spark) repository, Spot's images are also hosted in a public AWS ECR repository: [public.ecr.aws/ocean-spark/spark](https://gallery.ecr.aws/ocean-spark/spark), and a public Azure repository: `oceanspark.azurecr.io/spark`. - -## Release notes - -Release notes for each generation can be found [here](ocean-spark/docker-images-release-notes/). - -## What’s Next? - -Learn how to [add volumes to your Spark applications](ocean-spark/configure-spark-apps/mount-volumes). diff --git a/src/docs/ocean-spark/configure-spark-apps/memory-&-cores.md b/src/docs/ocean-spark/configure-spark-apps/memory-&-cores.md deleted file mode 100644 index 97a12ebc95..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/memory-&-cores.md +++ /dev/null @@ -1,184 +0,0 @@ - - -# Configure Pod Sizes - -This page describes how to configure your Spark pod sizes and select the instances they run on. - -## Concepts - -Your Ocean Spark cluster uses a variety of instances, such as a combination of instance families (for example on AWS: m5, r5, i3, …), sizes (for example, on AWS: x.large, 2x.large, …) and availability (spot, on-demand). - -These instances (called nodes in Kubernetes terminology) are dynamically added to the cluster as they are requested by your Spark applications, and are automatically terminated when they are unused so they do not incur any costs. This is all managed by Ocean through the concept of [Virtual Node Groups](ocean/features/launch-specifications) (VNGs). - -A Spark application consists of exactly one Spark driver pod and a varying number (0 to thousands) of Spark executor pods. You can configure your Spark driver independently of your Spark executors. For example, you can request a small container size for the Spark driver, and large container sizes for the Spark executors, or vice-versa. All Spark executors have the same configuration. - -## Run on Spot or On-demand Nodes - -Your Ocean Spark cluster should have at least two VNGs dedicated to Spark applications: one configured to use only on-demand nodes, one configured to use only spot nodes. - -For each application, you can control whether to use spot or on-demand nodes. For example, the following configuration requests that the Spark driver be placed on the on-demand VNG, while the executors will be placed on the Spot VNGs. Note that this is also the default API behavior if you omit these fields. - -```json -{ "driver": { "spot": "false" }, "executor": { "spot": "true" } } -``` - -You can switch the flags to change the behavior. Note that running the Spark driver on spot nodes is risky. If the spot node is terminated, your Spark application will fail. - -## Configure the Number of Cores - -To control the size of your pods, the main API field is cores, which corresponds to the number of CPU cores allocated to the Spark driver or Spark executor. This field also corresponds to the number of Spark tasks which can be executed in parallel on a Spark executor. -For example, the following configuration requests two cores for the Spark driver and four cores for each Spark executor. - -```json -{ "driver": { "cores": 2 }, "executor": { "cores": 4 } } -``` - -Note that the cores field is optional. If omitted, the Spark driver will have 1 core by default. This is a reasonable default as usually the Spark driver does not do much work and so it is more cost-effective to keep the Spark driver small. If you plan to run heavy operations on the Spark driver, such as running pure Python or Scala code, or collecting large results on it, you should increase the number of cores allocated to it. - -For executors, the default number of cores is four. There is no need to change this, unless you have a specific requirement. Instead, you can set the `instances` field to control how many executors to use. Read [How to Control the number of executors](ocean-spark/configure-spark-apps/common-spark-configs?id=control-the-number-of-executors) to learn more about this. - -## Configure the Type of Instances - -The instanceAllowList field lets you control which type of instances the pods will be placed on. It accepts a list of instance family names or specific instance types. - -For example, the configuration below requests: - -- That the Spark driver be placed on an r5.large instance (which it would fill entirely, given this instance type has 2 available CPU cores) or an r5.xlarge instance (which it would fill at 50% capacity, leaving room for another pod running on the same node). -- That the 20 requested Spark executors be placed on any of nine families of instances (m5, m5a, ...). For example you could have 20 executors each using an m5.xlarge instance, or you could have 10 executors using m5.xlarge instances, and 10 executors running on 5 m5.2xlarge instances. - -```json -{ - "driver": { - "cores": 2, - "instanceAllowList": ["r5.large", "r5.xlarge"] - }, - "executor": { - "cores": 4, - "instances": 20, - "instanceAllowList": [ - "m5", - "m5a", - "m5ad", - "m5d", - "m5dn", - "m5n", - "m5zn", - "m6a", - "m6i" - ] - } -} -``` - -Ocean Spark will optimize the choice of nodes (instances) to lower your cloud costs (efficient bin-packing, reusing existing capacity when available, using spot instances as much as possible) and optimize the stability of spot nodes (picking spot instances with a low risk of spot-interruption). - -If you have a specific need, you can pick a specific instance type or family, but in general we recommend that you to let Ocean pick which nodes to use across a large list of families. This gives Ocean flexibility to pick an optimal instance type based on the instances available in your cluster and the current Spot market dynamics. - -If you omit the instanceAllowList field, your Spark executor pods will be able to run on any instance type, preferably filling up nodes which have available capacity, before adding new nodes to the cluster. This gives Ocean Spark a lot of flexibility to pick Spot nodes at the lowest cost. - -## Configure the Memory - -You do not need to explicitly request an amount of memory, as Ocean Spark will automatically determine the optimal amount of memory based on the cores and instanceAllowList fields to optimize bin packing. - -For example, if you request: - -```json -{ "executor": { "cores": 4, "instanceAllowList": ["m5"] } } -``` - -Ocean Spark will determine how much memory to request so that each Spark executor exactly utilizes the memory available on an m5.xlarge instance (which has 4 available cores), or half of an m5.2xlarge instance (which has eight available cores). - -If you allow instance families with different memory/core ratios, Ocean spark will determine the memory amount corresponding to the highest memory/core ratio. - -The example below shows an instanceAllowList allowing a wide range of high-memory families for the Spark driver, and an instanceAllowList allowing a wide range of regular-sized families for the executors. - -```json -{ - "driver": { - "instanceAllowList": [ - "r5", - "r5a", - "r5ad", - "r5b", - "r5d", - "r5dn", - "r5n", - "r6i", - "i3" - ] - }, - "executor": { - "instanceAllowList": [ - "m5", - "m5a", - "m5ad", - "m5d", - "m5dn", - "m5n", - "m5zn", - "m6a", - "m6i" - ] - } -} -``` - -The configured amount of memory will be smaller than the value advertised by the cloud provider due to the following reasons: - -- Some memory is reserved for the instance operating system and Kubernetes internal operations. -- The memory field in our UI and API shows you the maximum heap size of the Spark Java Virtual Machine. This is not the same thing as the pod memory request. Read the next section on [Container Memory Overhead](ocean-spark/configure-spark-apps/memory-&-cores?id=container-memory-overhead)) for details about this. - -Should you want to control precisely yourself the amount of memory (heap-size) to allocate to the Spark driver or executors, you can configure it as follows: - -```json -{ "driver": { "cores": 4, "memory": "8.5g" } } -``` - -Be careful when entering memory settings manually, as it is easy to make mistakes. You can use the Ocean UI to view your nodes and pods and verify your understanding. In general, we recommend that you only select your pod sizes by using the cores and instanceAllowList fields. - -If you would like to investigate some of these configurations further, the official Apache Spark documentation page on [Running Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html) contains useful information. - -### Memory Auto-tuning Strategies - -In addition to hard-coded memory string, two strategies are available to -dynamically adjust the executor memory. These strategies analyze the previous -apps' performance to adjust the memory automatically. - -The oomRecovery strategy works like the default autotuning mechanism but it -automatically bumps the memory request up (doubling it by default) if the -previous app with the same job name failed with OOM (Out Of Memory) errors. - -```json -{ "executor": { "strategy": "oomRecovery" } } -``` - -The autotuning strategy expands on the oomRecovery strategy by additionally -adjusting memory request down to match the observed memory usage of previous -apps. The autotuning strategy is only available for Spark 3 apps as it relies on -spark metrics to monitor the memory usage of your apps. - -```json -{ "executor": { "strategy": "autotuning" } } -``` - -You can find more details on how to configure these strategies in the [Ocean Spark API reference ](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). - -## Container Memory Overhead - -If your Spark Driver or Executors are abruptly terminated with a Docker exit code 137, the memory used by the processes running inside your containers have exceeded the memory limit controlled by Kubernetes. This is known as an OOM-Kill (OutOfMemory-kill). - - - -This can typically occur when you use PySpark because some of your code will be executed by Python processes (one per core) running inside your container (alongside the main Spark executor JVM process). - -This also occurs when using Scala or Java because the JVM uses some amount of memory outsides of its heap in [certain situations](https://plumbr.io/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx). - -To remediate this issue, you should increase the memoryOverheadFactor configuration, for example, by using the setting below. - -```json -{ "memoryOverheadFactor": "0.5" } -``` - -## What's Next? - -Learn more about [common Spark configurations](ocean-spark/configure-spark-apps/common-spark-configs). diff --git a/src/docs/ocean-spark/configure-spark-apps/mount-volumes.md b/src/docs/ocean-spark/configure-spark-apps/mount-volumes.md deleted file mode 100644 index ad1d2427e2..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/mount-volumes.md +++ /dev/null @@ -1,178 +0,0 @@ - - -# Mount Volumes - -This page shows how to add volumes to your Spark applications. - -## Standard configuration - -By default, Spark uses temporary storage to spill data to disk during shuffles and other operations when no volume is present. With Spark on Kubernetes, pods are initialized with an emptyDir volume for each directory listed in spark.local.dir. When the pod is deleted, the ephemeral storage will be cleared and the data removed. - -For certain Spark applications with larger shuffle workfloads or a need to persist data beyond the life of the application, it may be desirable to mount an external volume. Depending on your cloud provider, there are several volume options you can choose from to make your Spark workspaces more dynamic and scalable. - -By default, if you select a node type that includes local SSDs on AWS, such as `d` instances (e.g., m5d, r5d, c5d), Ocean Spark will help you leverage the performance of local SSDs. Read the next section to learn more. - -## Mounting local SSDs on AWS - -For certain Spark applications with larger shuffle workfloads, the speed of the disk is the main bottleneck to achieve good Spark performance. This can be remediated by using faster disk types like NVMe-based SSDs. - -Instances like `d` instances (e.g., m5d, r5d, c5d) or i3 instances all come with local SSDs. However, they must be mounted into the file system of the instance. - -Ocean Spark provides a [user data script](https://raw.githubusercontent.com/spotinst/ocean-spark-examples/master/local_ssd_user_data_script/startup-script.sh) that you can use to mount SSDs automatically. This script must be added in the User Data section of the Virtual Node Group (VNG) configuration tab in Ocean. - -Be sure to fill in the variable EKS_CLUSTER_NAME with the name of your EKS cluster. - -We recommend adding this [user data script](https://raw.githubusercontent.com/spotinst/ocean-spark-examples/master/local_ssd_user_data_script/startup-script.sh) by default to all VNGs to be used for Spark applications. - -Most likely, the Spark-dedicated VNGs in your EKS cluster already use this script (Ocean Spark adds it by default). You can verify this by browsing to the Virtual Node Group tab in the Ocean section of the Spot console. - -Once the user data script is installed in the VNGs, Ocean Spark will automatically and transparently configure Spark to use the local SSDs for shuffle data. Two examples are provided below. - -In this sample piece of configuration, m5d instances are used as executors: - -``` - "executor": { - "instanceSelector": "m5d" - } -``` - -Ocean Spark will use the local SSDs for shuffle data. This will greatly enhance the performance of the Spark application if it uses a lot of shuffle. - -In this sample piece of configuration, m5 instances are used as executors: - -``` - "executor": { - "instanceSelector": "m5" - } -``` - -Ocean Spark will understand that there are no local SSDs and will use a non-SSD location on the host file system for shuffle data. If the CPU profile of the Spark application in the Spot console indicates a high amount of shuffle or IO, consider switching to an instance type with SSDs, like m5d. - -> **Tip**: The use of local SSDs for shuffle files is only supported in Spark 3 and above. - -## Mount secrets as files in volumes - -Instead of setting environment variables from [Kubernetes secrets](ocean-spark/configure-spark-apps/secrets-environment-variables), you can also mount secrets directly as files into a volume. - -First, create a kubernetes secret in your cluster, and make sure to use the namespace spark-apps so your application can access it. Here is an example basic-auth secret below: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - namespace: spark-apps - name: basic-auth -data: - password: cGFzc3dvcmQK # password - user: dXNlcgo= # user -type: Opaque -``` - -In your application json config, add a volume object that references the secret name, data key you would like to include in the volume, and a name for the volume reference. In your executor or driver config, add a volumeMounts object that includes the volume name referenced in the previous step and path where you would like the file to be mounted. You can find an example application json config below that creates a volume named volume-secret that references the user key in the basic-auth secret and mounts that secret file to /opt/spark/work-dir/secrets on the executor pod(s). - -```json -{ - "executor": { - "cores": 1, - "instances": 3, - "instanceType": "r5.xlarge", - "volumeMounts": [ - { - "mountPath": "/opt/spark/work-dir/secrets", - "name": "volume-secret" - } - ] - }, - "volumes": [ - { - "secret": { - "items": [ - { - "key": "user", - "path": "secret.yaml" - } - ], - "secretName": "basic-auth" - }, - "name": "volume-secret" - } - ] -} -``` - -If you `kubectl exec` into one of the executor pods, you will find the `secrets.yaml` file located at `/opt/spark/work-dir/secrets`. - -## Mount persistent cloud volumes to your container - -For larger volume needs, sharing volumes across applications, or persisting data beyond an application's lifecycle, you can mount a cloud provider volume to your container. Instead of passing in a mount path, find the appropriate key for the cloud volume of your choice, and add the necessary data to the object. You can find the full list of supported volume options in the [API reference](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit) for submitting applications. - -To enable volume mounting, you must first create the volume instance in your respective cloud provider. Then, you must make sure your cluster role has read/write access to the volume. - -On AWS, you can create an AWS Elastic Block Storage in the AWS console, or by running the following command: - -``` -aws ec2 create-volume --availability-zone -``` - -You can see more configuration options in the [EBS documentation](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-volume.html). - -To mount an EBS volume to your spark application, add the following json to your application submission: - -```json -{ - "volumes": [ - { - "name": "spark-aws-dir", - "awsElasticBlockStore": { - "fsType": "type of file system", - "readOnly": false, - "volumeID": "id of ebs" - } - } - ] -} -``` - -## Dynamically provision volumes using Persistent Volume Claims - -As of Spark 3.1, you can dynamically provision and mount volumes to both your executor and driver pods using persistent volume claims. Persistent volume claims are kubernetes resources that allow you to allocate and mount resources of an elastic disk to your kubernetes pods. When your application is complete, kubernetes will remove the pod and delete the volume from your cloud provider. You can read more about the different configuration options in [Running Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes). - -To enable PVC provisioning, you must add a few configuration settings to your sparkConf: - -```json -{ - "sparkConf": { - "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName": "OnDemand", - "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass": "standard", - "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit": "500Gi", - "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path": "/var/data", - "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly": "false" - } -} -``` - -In the sample above, the claimName must be set to OnDemand. You can set the mount path to anywhere you like on the container, just make sure spark is aware of the drive and actually using it. You may need to alter the setting spark.local.dir to the location of your mount point. - -In addition, you must also attach a cluster role of edit to the spark-driver service account. You can run the following command to attach the cluster role: - -``` -kubectl create clusterrolebinding --clusterrole=edit --serviceaccount=spark-apps:spark-driver --namespace=default - -``` - -## Dynamic PVC Reuse - -With the release of Spark 3.2, you can now dynamically reuse persistent volume claims within the same application. This becomes particularly beneficial when using spot instances. In a normal Spark application leveraging spot instances for executors, if you lose a node to a spot kill, you will lose any shuffle or output data that was stored on that node, and Spark will be forced to recompute the results. With PVC reuse enabled, the Spark driver will maintain the PVC(s) of the lost spot instance, add new executors to the cluster, and attach the PVC(s) to the new executors, maintaining the work and progress of the previous node. This feature can significantly reduce application runtime and cost, offsetting one of the major drawbacks of using spot instances. To enable dynamic PVC reuse, add the following two lines, in addition to the configuration above, to your Spark conf: - -``` -{ - "sparkConf": { - "spark.kubernetes.driver.reusePersistentVolumeClaim":"true", - "spark.kubernetes.driver.ownPersistentVolumeClaim":"true" - } -} -``` - -## What’s Next? - -Take the [Product Tour](ocean-spark/product-tour/) and learn about everything you can do in the Ocean Spark console. diff --git a/src/docs/ocean-spark/configure-spark-apps/package-spark-code.md b/src/docs/ocean-spark/configure-spark-apps/package-spark-code.md deleted file mode 100644 index 9a3213d6d9..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/package-spark-code.md +++ /dev/null @@ -1,322 +0,0 @@ - - -# Package Spark Code - -In this page, we describe how to package your Spark code so that it can be run on an Ocean Spark cluster. - -There are two options available: - -- Add your code to a Docker image -- Host your code on an object storage - -> **Tip**: You need to call spark.stop() at the end of your application code, where spark can be your Spark session or Spark context. Otherwise your application may keep running indefinitely. - -## Add your code to a Docker image - -Using Docker images makes dependency management easy, particularly for Python workloads. Docker images let you have tight control over your environment. You can run the same Docker image locally during development and on an Ocean Spark cluster for production. - -In this section, you will learn how to build a Docker image from your code, set up a container registry, and push the Docker image to the container registry. - -### Build a Docker image and run it locally - -You must have [Docker](https://www.docker.com/get-started) installed on your machine. - -For compatibility reasons, you must use one of our published Docker images as a base, then add your dependencies on top. Building an entirely custom Docker image is not supported. - -[Docker images](https://gcr.io/ocean-spark/spark) are offered by Ocean Spark and documented in the [user documentation](ocean-spark/configure-spark-apps/docker-images). - -
- Python - -In this example, the Python project uses the main Docker image offered by Ocean Spark, `spark:platform`. It includes Python support and connectors to popular data sources. The latest image is `gcr.io/ocean-spark/spark:platform-3.2.0-latest`. - -We'll assume your project directory has the following structure: - -- A main Python file e.g., `main.py` -- A `requirements.txt` file specifying project dependencies -- A global Python package called `src`, containing all project sources. This package can contain modules and packages and does not require source files to be flattened. Because `src` is a p - Python package, it must contain an `__init__.py file`. - -``` -|____ main.py -|____ requirements.txt -|____ src/ - |____ __init__.py - |____ mod1.py - |____ mod2.py - |____ pkg1/ - |____ pkg1_mod1.py - |____ ... - |___ ... -``` - -1. Add a file called Dockerfile to the project directory with the following content: - -``` -FROM gcr.io/ocean-spark/spark:platform-3.2.0-latest` - -COPY requirements.txt . -RUN pip3 install -r requirements.txt - -COPY src/ src/ -COPY main.py . -``` - -2. Build the Docker image by running this command in the project directory: - -`docker build -t my-app:dev` - -3. Run it locally with: - -`docker run -e SPARK_LOCAL_IP=127.0.0.1 my-app:dev driver local:///opt/spark/work-dir/main.py ` - -where `` are the arguments to be passed to the main script `main.py`. - -> **Tip**: The environment variable `SPARK_LOCAL_IP=127.0.0.1` is only required when running the image locally with docker. - -

- -
- Java & Scala - -We'll assume you have assembled your application into a fat or [uber JAR](https://stackoverflow.com/questions/11947037/what-is-an-uber-jar) called `main.jar`. - -For this example project, we'll use the main Docker image offered by Ocean for Spark, `spark:platform`. It includes Python support and connectors to popular data sources. The latest image is `gcr.io/ocean-spark/spark:platform-3.2.0-latest`. - -1. Add a file called Dockerfile to the directory where `main.jar` resides: - -``` -FROM gcr.io/ocean-spark/spark:platform-3.2.0-latest - -COPY main.jar . -``` - -2. Build the Docker image by running this command in the project directory: - -`docker build -t my-app:dev` - -3. Run it locally with - -`docker run -e SPARK_LOCAL_IP=127.0.0.1 my-app:dev driver --class local:///opt/spark/work-dir/main.jar ` - -where `` are the arguments to be passed to the application main class ``. - -> **Tip**: The environment variable `SPARK_LOCAL_IP=127.0.0.1` is only required when running the image locally with Docker. - -

- -### Set up a Docker registry and push your image - -The simplest option on AWS is to use the Elastic Container Registry (ECR) of the account where the Ocean Spark platform is deployed. This way, the Spark pods can pull the Docker images without needing extra permissions. - -1. Navigate to the [ECR console](https://console.aws.amazon.com/ecr/repositories) and create a repository with name my-app in the account where the Ocean Spark cluster is deployed. Make sure to create it in the same region as the Ocean Spark cluster to avoid transfer costs. Please refer to the [AWS documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html) in case of issue. -2. Generate a temporary token so that Docker can access ECR for 12 hours with the following: - -`aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com` - -This complex command can be found in the AWS console by clicking the "View push commands" button. - - - -3. You can now re-tag the Docker image we built above and push it to the ECR repository: - -``` -docker tag my-app:dev .dkr.ecr..amazonaws.com/my-app:dev -docker push .dkr.ecr..amazonaws.com/my-app:dev -``` - -Please refer to the [AWS documentation about ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html) in case of issue. - -### Run your image on Ocean Spark - -The Spark application can now be run on Ocean Spark: - -
- Python - -``` -curl -X POST \ -'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer - --data-raw '{ - "jobId": "my-job", - "configOverrides": { - "type": "Python", - "sparkVersion": "3.2.0", - "image": ".dkr.ecr..amazonaws.com/my-app:dev", - "mainApplicationFile": "local:///opt/spark/work-dir/main.py", - "arguments": [] - } - }' -``` - -

- -
- Java & Scala - -``` -curl -X POST \ - 'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer - --data-raw '{ - "jobId": "my-job", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.0", - "image": ".dkr.ecr..amazonaws.com/my-app:dev", - "mainApplicationFile": "local:///opt/spark/work-dir/main.jar", - "mainClass": "", - "arguments": [] - } - }' -``` - -
- -## Host your code on an object storage - -In this section, you will learn how to package your code, upload it to an object storage, and make it accessible to an Ocean Spark cluster. - -> **Tip**: If possible, use [Building a Docker image](ocean-spark/configure-spark-apps/package-spark-code?id=build-a-docker-image-and-run-it-locally) containing your source code. It is more robust and more convenient, especially for Python. - -
- Python - -### Project structure - -In order to run on your cluster, your Spark application project directory must fit the following structure: - -- A main python file e.g., `main.py` -- A `requirements.txt` file specifying project dependencies -- A global python package named `src` containing all project sources. This package can contain modules and packages and does not require source files to be flattened. Because src is a python package it must contain a `__init__.py` file. - -### Package Python libraries - -Run the following command at the root of your project, where the requirements.txt file is located. - -``` -rm -rf tmp_libs -pip wheel -r requirements.txt -w tmp_libs -cd tmp_libs -for file in $(ls) ; do - unzip $file - rm $file -done -zip -r ../libs.zip . -cd .. -rm -rf tmp_libs -``` - -All your dependencies are now zipped into a libs.zip file. - -### Package project source files - -Zip your project source files from the global package src. This package will be consumed by your Spark application main file using python imports such as: - -- import src.your_module -- from src.your_package.your_module import your_object - -Zip the src global package: - -`zip -r ./src.zip ./src` - -All your sources modules/packages are now zipped into a src.zip file. - -### Upload project files - -Upload prepared files to your cloud storage: - -``` -aws s3 cp libs.zip s3:///libs.zip -aws s3 cp src.zip s3:///src.zip -aws s3 cp s3:/// -``` - -### Run the application - -All required files are uploaded in your cloud storage. The Spark application can now be started: - -``` -curl -X POST \ - 'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer - --data-raw '{ - "jobId": "my-job", - "configOverrides": { - "type": "Python", - "sparkVersion": "3.2.0", - "image": ".dkr.ecr..amazonaws.com/my-app:dev", - "mainApplicationFile": "s3a:///", - "deps": { - "pyFiles": [ - "s3a:///libs.zip", - "s3a:///src.zip", - ] - } - } - }' -``` - -Note that Ocean Spark automatically chooses a Spark image for your app based on the sparkVersion. - -For AWS, if you are referencing s3 for the main application file or Dockerfile, you must use the file format s3a, otherwise spark will throw an exception. - -You can access the Ocean Spark console in order to monitor your Spark application execution. - -

- -
- Java & Scala - -The procedure is simpler for JVM-based languages, as Spark has been designed with these in mind. -Once your application is compiled, upload it to your cloud storage: - -``` -aws s3 cp .jar s3:///.jar -``` - -Reference your JAR (and its dependencies if it has any) in the configuration of your Spark application: - -``` -curl -X POST \ - https://api.spotinst.io/ocean/spark/cluster/osc-e4089a00/app \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer - --data-raw '{ - "jobId": "my-job", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.2.0", - "mainApplicationFile": "s3a:///.jar", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "deps": { - "jars": [ - "s3a:///.jar", - "s3a:///.jar" - ] - } - } - }' -``` - -Note that Ocean Spark automatically chooses a Spark image for your app based on the sparkVersion. - -For AWS, if you are referencing s3 for the main application file or Dockerfile, you must use the file format s3a, otherwise spark will throw an exception. - -You can access the [Ocean Spark console]() in order to monitor your Spark application execution. - -If you need to import a dependency directly from a repository like Maven, the `deps->jars` list accepts URLs, like: - -``` -https://repo1.maven.org/maven2/org/influxdb/influxdb-java/2.14/influxdb-java-2.14.jar -``` - -

- -## What's Next? - -Learn more about [memory and cores](ocean-spark/configure-spark-apps/memory-&-cores). diff --git a/src/docs/ocean-spark/configure-spark-apps/secrets-environment-variables.md b/src/docs/ocean-spark/configure-spark-apps/secrets-environment-variables.md deleted file mode 100644 index b9198cf8a3..0000000000 --- a/src/docs/ocean-spark/configure-spark-apps/secrets-environment-variables.md +++ /dev/null @@ -1,84 +0,0 @@ - - -# Secrets and Environment Variables - -This page describes the configuration of secrets and environment variables. - -## Set environment variables - -Environment variables can easily be set by inserting the lines below in a [configuration template](ocean-spark/configure-spark-apps/?id=configuration-templates) or in [config overrides](ocean-spark/configure-spark-apps/?id=config-overrides): - -```json -{ - "driver": { - "envVars": { - "ENV_VAR_KEY": "ENV_VAR_VALUE" - } - }, - "executor": { - "envVars": { - "ENV_VAR_KEY": "ENV_VAR_VALUE" - } - } -} -``` - -## Set environment variables using Kubernetes secrets - -If you have [defined Kubernetes secrets](ocean-spark/configure-spark-apps/access-your-data?id=grant-permissions-using-kubernetes-secrets), you can pass them to your Spark applications as an environment variable. Merge the following configuration segment into a configuration template or into config overrides: - -```json -{ - "driver": { - "envSecretKeyRefs": { - "ENV_VAR_KEY": { - "name": "secret-name", - "key": "secret-field" - } - } - }, - "executor": { - "envSecretKeyRefs": { - "ENV_VAR_KEY": { - "name": "secret-name", - "key": "secret-field" - } - } - } -} -``` - -This will create an environment variable with the key ENV_VAR_KEY and the value being the content of the secret secret-name at the field secret-field. - -The secret value is not visible in the application configuration, and it will be redacted in the Spark UI. - -## Retrieve environment variables in your code - -This is how you can retrieve environment variables in your Spark application code: - -
- Python - -```python -import os -env_vars = os.environ # Dictionary of key-value pairs -value = os.environ['ENV_VAR_KEY'] # ENV_VAR_VALUE -``` - -

- -
- Java and Scala - -```java -val envVars = System.getenv() // Map[String, String] of key-value pairs -val value = System.getenv("ENV_VAR_KEY") // ENV_VAR_VALUE -``` - -
- -To list all configurations you can set in Ocean Spark, check out the [API reference](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). - -## What’s Next? - -Learn more about [Docker images for Spark](ocean-spark/configure-spark-apps/docker-images). diff --git a/src/docs/ocean-spark/data-plane-release-notes/README.md b/src/docs/ocean-spark/data-plane-release-notes/README.md deleted file mode 100644 index 4833a113e8..0000000000 --- a/src/docs/ocean-spark/data-plane-release-notes/README.md +++ /dev/null @@ -1,411 +0,0 @@ - - -# Cluster Release Notes - -## [0.5.6-86] - 2025-01-09 - -Changelog - -- [0.5.8] bigdata-proxy - - vulnerability fixes -- [0.5.6] bigdata-operator - - support workspaces upgrade - -## [0.5.4-85] - 2024-12-16 - -Changelog - -- [0.6.4] bigdata-spark-watcher - - add cleaning up stuck failed apps and orphaned driver pods -- [0.5.7] bigdata-proxy - - support configuring the storage class for the workspace PVC - -## [0.5.2-84] - 2024-11-26 - -Changelog - -- [0.6.3] bigdata-spark-watcher - - spark-apps kube event collector fix GCP and Azure regional storage. -- [0.4.6] bigdata-notebook-service - - scala notebook kernel disconnection fix - -## [0.5.0-83] - 2024-11-17 - -Changelog - -- [0.5.0] bigdata-operator - - workspace state reporting -- [0.4.5] bigdata-notebook-service - - kernel persistence fix - - always create telemetry ConfigMap -- [0.4.14] bigdata-proxy - - always create telemetry ConfigMap -- [0.6.2] bigdata-spark-watcher - - always create telemetry ConfigMap -- [0.1.34] spark-operator - - always create telemetry ConfigMap - -## [0.4.24-82] - 2024-11-04 - -Changelog - -- [0.6.1] bigdata-spark-watcher - - spark-apps kube event collector bug fix - -## [0.4.23-81] - 2024-09-30 - -Changelog - -- [0.6.0] bigdata-spark-watcher - - - spark-apps kube event collector performance improvement - -## [0.4.22-80] - 2024-09-16 - -Changelog - -- [0.4.22] bigdata-operator - - - bug fixes - - handle multiple bigdata-environments - -- [0.5.21] bigdata-spark-watcher - - - vulnerability fixes - -- [0.1.32] spark-operator - - - vulnerability fixes - - handle double spark-application submission - -- [0.4.13] bigdata-proxy - - - vulnerability fixes - -- [0.4.3] bigdata-notebook-service - - vulnerability fixes - - a failsafe has been added to make sure notebook pods are always killed when kernel is shutdown - -## [0.4.21-79] - 2024-07-24 - -Changelog - -- [0.4.21] bigdata-operator - - - bug fixes - - handle extra spark app namespaces - -- [0.5.20] bigdata-spark-watcher - - - check for existence of spark-application CR before running spark submit - - enable spark-apps kube event collector - - vulnerability fixes - -- [1.11.0] ofas-ingress-nginx - - vulnerability fixes - -## [0.4.20-78] - 2024-07-11 - -Changelog - -- [0.1.32] spark-operator - - - vulnerability fixes - - custom configs for the telemetry sidecars - -- [0.4.12] bigdata-proxy - - - custom configs for the telemetry sidecars - -- [0.4.1] bigdata-notebook-service - - custom configs for the telemetry sidecars - -## [0.4.20-77] - 2024-07-02 - -Changelog - -- [0.4.20] bigdata-operator - - - use spark watcher 0.5.17 - - restart no longer required when installing new Ocean controller - -- [0.5.17] bigdata-spark-watcher - - restart no longer required when installing new Ocean controller - -## [0.4.19-76] - 2024-06-11 - -Changelog - -- [0.4.19] bigdata-operator - - - use notebook service 0.4.0 - -- [0.4.0] bigdata-notebook-service - - uses updated notebook config-templates endpoints - -## [0.4.18-75] - 2024-05-30 - -Changelog - -- [0.4.18] bigdata-operator - - non-root telemetry image - -## [0.4.17-75] - 2024-05-20 - -Changelog - -- [0.4.17] bigdata-operator - - use MapKubeAPIs to handle API deprecations when upgrading components - -## [0.4.16-75] - 2024-05-14 - -Changelog - -- [0.4.16] bigdata-operator - - - upgrade to go 1.21 - - modify logic to allow removal of following charts: - - `bigdata-notebook-service-storage-server` - - `bigdata-notebook-service-storage` - -- [0.3.2] bigdata-notebook-service - - - Use new backend session storage - - max port moved down to 50100 - - use appVersion 0.82.3 - -- [0.1.3] bigdata-notebook-service-static - - - max port moved down to 50100 - -- [0.5.15] bigdata-spark-watcher - - - support for new Ocean controller (rbac) - -- [removed] bigdata-notebook-service-storage-server - -- [removed] bigdata-notebook-service-storage - -- [0.4.15] bigdata-operator - - - deployerNamespace fixes - - use helm force when upgrading the spark-operator-static - - go package upgrade, security fixes - -- [0.4.13] bigdata-operator - - support new ocean-controller - - update `bigdata-operator-cluster-manager` cluster role - - telemetry fixes - -## [0.4.11-74] - 2024-04-08 - -Changelog - -- [0.5.12] bigdata-spark-watcher - - - support for new Ocean controller - - k8s event logs collector - - custom configs for the telemetry sidecars - -- [0.4.11] bigdata-operator - - custom configs for the telemetry sidecars - -## [0.4.10-73] - 2024-04-03 - -Changelog - -- [0.4.7] bigdata-proxy - - Fix workspace save with large notebook file - -## [0.4.10-72] - 2024-03-27 - -Changelog - -- [0.4.6] bigdata-proxy - - - Fix workspace large file upload - -- [0.4.10] bigdata-operator - - - Support for embedded helm charts - -- [0.4.9] bigdata-operator - - Support for running multiple replicas - -## [0.4.8-71] - 2024-02-28 - -Changelog - -- [0.5.9] bigdata-spark-watcher - - - enable leader election for the high-availability - -- [0.1.26] spark-operator - - enable leader election for the high-availability - -## [0.4.8-70] - 2024-02-20 - -Changelog - -- [0.4.8] bigdata-operator - - - store cluster cloud provider and region in CM - -- [0.5.8] bigdata-spark-watcher - - - Annotate spark driver pod exit code and exit time - -- [0.1.25] spark-operator - - run as non-root - -## [0.4.7-69] - 2024-01-24 - -Changelog - -- [0.4.7] bigdata-operator - - - enable telemetry - - run as non-root - -- [0.2.4] bigdata-notebook-service - - - upgrade workflow notebook image to JupyterLab 4 - - show kernel launchers in the JupyterLab UI for each Spark Connect app running in the cluster - - enable telemetry - -- [0.4.5] bigdata-proxy - - - run as non-root - - enable telemetry - -- [0.1.24] spark-operator - - - enable telemetry - -- [0.5.4] bigdata-spark-watcher - - run as non-root - - enable telemetry - - performance improvements - -## [0.4.4-68] - 2023-11-27 - -Changelog - -- [0.2.1] bigdata-notebook-service - - increase port range for notebook service -- [0.1.2] bigdata-notebook-service-static - - increase port range for notebook service - -## [0.4.4-67] - 2023-11-14 - -Changelog - -- [0.4.4] bigdata-operator - - retrieve the deployer namespace - -## [0.4.2-67] - 2023-10-24 - -Changelog - -- [0.4.3] bigdata-proxy - - update the secret token when starting a Notebook Workspace -- [0.4.2] bigdata-operator - - stop refreshing ocean-controller namespace - -## [0.4.1-66] - 2023-10-12 - -Changelog - -- [0.1.22] spark-operator - - performance improvements -- [0.4.2] bigdata-proxy - - increase ingress read timeout to 600 seconds by default - -## [0.4.1-65] - 2023-09-28 - -Changelog - -- [0.4.1] bigdata-proxy - - Workspace reverse proxy - -## [0.4.1-64] - 2023-09-13 - -Changelog - -- [0.4.1] bigdata-operator - - - respect proxy-url from `spotinst-kubernetes-cluster-controller-config` - -- [0.5.0] bigdata-spark-watcher -- [0.2.0] bigdata-notebook-service - - Support for HTTP_PROXY and HTTPS_PROXY environment variables - -## [0.4.0-63] - 2023-09-12 - -Changelog - -- [0.4.6] bigdata-spark-watcher - - fix bug in the executor storm handling -- [0.4.0] bigdata-operator - - manage bigdata CRDs via Helm chart - -## [0.3.0-62] - 2023-09-08 - -Changelog - -- [0.3.0] bigdata-operator - - support bigdatacomponents.bigdata.spot.io CRD - -## [0.2.8-62] - 2023-09-06 - -Changelog - -- [0.4.0] bigdata-proxy - - support for notebook workspace management - -## [0.2.8-61] - 2023-09-04 - -Changelog - -- [0.4.3] bigdata-spark-watcher - - executor storm handling improvements - -## [0.2.8-60] - 2023-08-25 - -Changelog - -- [0.3.5] bigdata-proxy - - add ability to delete Spark application pods cluster wide - -## [0.2.8-59] - 2023-08-17 - -Changelog - -- [0.2.8] bigdata-operator - - add telemetry fluentbit-sidecar to collect and output logs -- [0.1.30] bigdata-notebook-service - - - logger improvements - - add telemetry fluentbit-sidecar - -- [0.1.10] bigdata-notebook-service-storage-server - - - add telemetry fluentbit-sidecar - -- [0.3.4] bigdata-proxy - - - add telemetry fluentbit-sidecar - -- [0.4.2] bigdata-spark-watcher - - - add telemetry fluentbit-sidecar - -- [0.1.21] spark-operator - - add telemetry fluentbit-sidecar - -## [0.2.7-58] - 2023-08-07 - -Changelog - -- [0.4.1] bigdata-spark-watcher - - send regular heartbeats diff --git a/src/docs/ocean-spark/docker-images-release-notes/README.md b/src/docs/ocean-spark/docker-images-release-notes/README.md deleted file mode 100644 index 1906a2557c..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/README.md +++ /dev/null @@ -1,7 +0,0 @@ - - -# Docker Images Release Notes - -Please choose a specific release in the sidebar on the left. - -For an introduction on how to use these images, please refer to the [Docker images](ocean-spark/configure-spark-apps/docker-images) page. diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen18.md b/src/docs/ocean-spark/docker-images-release-notes/gen18.md deleted file mode 100644 index c825cec75d..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen18.md +++ /dev/null @@ -1,898 +0,0 @@ - - -# gen18 release notes - -## Changelog - -Generation 18 is a port of the [legacy dm18](ocean-spark/docker-images-release-notes/legacy-dm-images) generation. There are not changes compared to this generation. - -## Available tags - -### platform-3.3.0-hadoop-3.3.2-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.3-gen18` -- `platform-3.3.0-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.0 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.2-gen18` -- `platform-3.2.1-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.1 | -| Hadoop | 3.3.1 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.2.0-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.0 | -| Hadoop | 3.3.1 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.1-gen18` -- `platform-3.1.3-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.3 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.1.2-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.2 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.1.1-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.1 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.0-gen18` -- `platform-3.0.3-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.3 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.0.2-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.2 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.0.1-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.1 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18 - -#### Additional tags - -- `platform-3.0.0-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.0 | -| Hadoop | 3.2.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18 - -#### Additional tags - -- `platform-2.4.7-gen18` -- `platform-2.4-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.7 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18 - -#### Additional tags - -- `platform-2.4.6-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.6 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18 - -#### Additional tags - -- `platform-2.4.5-gen18` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.5 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-3.3.0-hadoop-3.3.2-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.0 | -| Hadoop | 3.3.2 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.1 | -| Hadoop | 3.3.1 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.0 | -| Hadoop | 3.3.1 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.3 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.2 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.1.1 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.3 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.2 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.1 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.0.0 | -| Hadoop | 3.2.0 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.7 | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.375 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.0 | -| AWS Glue | Hive 2.3.7 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.10.0 | - -### platform-2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.7 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.6 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.5 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 3.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen19.md b/src/docs/ocean-spark/docker-images-release-notes/gen19.md deleted file mode 100644 index 6ffb2cb465..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen19.md +++ /dev/null @@ -1,294 +0,0 @@ - - -# gen19 release notes - -## Changelog - -- support multi-platform `linux/amd64` and `linux/arm64` -- available on public acr, ecr and gcr registries: - - `oceanspark.azurecr.io/spark` - - https://gallery.ecr.aws/ocean-spark/spark - - https://gcr.io/ocean-spark/spark - -## Available tags - -### platform-3.3.1-hadoop-3.3.2-java-8-scala-2.12-python-3.8-gen19 - -#### Additional tags - -- `platform-3.3-gen19` -- `platform-3.3.1-gen19` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.1 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.1.1 | -| Snowflake | Spark Snowflake 2.11.0 | - -### platform-3.2.2-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen19 - -#### Additional tags - -- `platform-3.2-gen19` -- `platform-3.2.2-gen19` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.2 | -| Hadoop | 3.3.1 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.11.0 | - -### platform-3.3.1-hadoop-3.3.2-java-11-scala-2.12-python-3.8-gen19 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.1 | -| Hadoop | 3.3.2 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.1.1 | -| Snowflake | Spark Snowflake 2.11.0 | - -### platform-3.2.2-hadoop-3.3.1-java-11-scala-2.12-python-3.8-gen19 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.2.2 | -| Hadoop | 3.3.1 | -| Java | 11 | -| Scala | 2.12 | -| Python | 3.8 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.901 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.11.0 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen19 - -#### Additional tags - -- `platform-2.4.8-gen19` -- `platform-2.4-gen19` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Hive Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen19 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Hive Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-3.3.0-hadoop-3.3.3-java-8-scala-2.12-python-3.7-gen19 - -#### Additional tags - -- `platform-aws-emr-6.9-gen19` -- `platform-aws-emr-6-gen19` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.0 | -| Hadoop | 3.3.3 | -| Java | 8 | -| Scala | 2.12 | -| Python | 3.7 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additonnal formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.1.1 | -| Snowflake | Spark Snowflake 2.11.0 | - -### platform-2.4.8-hadoop-2.10.1-java-8-scala-2.11-python-3.7-gen19 - -#### Additional tags - -- `platform-aws-emr-5.36-gen19` -- `platform-aws-emr-5-gen19` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 2.10.1 | -| Java | 8 | -| Scala | 2.11 | -| Python | 3.7 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | Not supported | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.3 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive Not supported | - -#### Additonnal formats - -| name | version | -| :-------: | :-------------------: | -| Delta | Not supported | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen20.md b/src/docs/ocean-spark/docker-images-release-notes/gen20.md deleted file mode 100644 index f845f3019a..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen20.md +++ /dev/null @@ -1,299 +0,0 @@ - - -# gen20 release notes - -## Changelog - -- Added support spark 3.4 -- Updated spark 3.3.X to 3.3.2 -- Deprecated spark 3.2.X images - -## Available tags - -### platform-3.3.2-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen20 - -#### Additional tags - -- `platform-3.3-gen20` -- `platform-3.3.2-gen20` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.2 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.11.3 | - -### platform-3.3.2-hadoop-3.3.2-java-11-scala-2.12-python-3.10-gen20 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.2 | -| Hadoop | 3.3.2 | -| Java | 11 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.11.3 | - -### platform-3.4.0-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen20 - -#### Additional tags - -- `platform-3.4-gen20` -- `platform-3.4.0-gen20` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.0 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | Not supported | -| Snowflake | Not supported | - -### platform-3.3.2-nvidia-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen20 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.2 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.11.3 | - -#### GPU Nvidia - -| name | version | -| :----------------: | :-----: | -| Cuda Driver | 11.8.89 | -| Rapids-4-Spark Lib | 23.04.1 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen20 - -#### Additional tags - -- `platform-2.4.8-gen20` -- `platform-2.4-gen20` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12.10 | -| Python | 3.7.13 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen20 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.13 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-3.3.0-hadoop-3.3.3-java-8-scala-2.12-python-3.7-gen20 - -#### Additional tags - -- `platform-aws-emr-6.9-gen20` -- `platform-aws-emr-6-gen20` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.0 | -| Hadoop | 3.3.3 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.7.13 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.11.3 | - -### platform-2.4.8-hadoop-2.10.1-java-8-scala-2.11-python-3.7-gen20 - -#### Additional tags - -- `platform-aws-emr-5.36-gen20` -- `platform-aws-emr-5-gen20` - -#### Dependency versions - -| name | version | -| :-----: | :----------: | -| Spark | 2.4.8 | -| Hadoop | 2.10.1 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.13 | -| Hive | 1.2.1.spark2 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | Not supported | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.3 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen21.md b/src/docs/ocean-spark/docker-images-release-notes/gen21.md deleted file mode 100644 index 513c1aa1e1..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen21.md +++ /dev/null @@ -1,334 +0,0 @@ - - -# gen21 release notes - -## Changelog - -Generation 21 contains images for Apache Spark version 3.3.3, 3.4.1 and 3.5.0. All the older images have also been rebased on `eclipse-temurin` following the deprecation of `openjdk` docker images. - -## Available tags - -### platform-3.3.3-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen21 - -#### Additional tags - -- `platform-3.3-gen21` -- `platform-3.3.3-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.3 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -### platform-3.3.3-hadoop-3.3.2-java-11-scala-2.12-python-3.10-gen21 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.3 | -| Hadoop | 3.3.2 | -| Java | 11 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -### platform-3.4.1-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen21 - -#### Additional tags - -- `platform-3.4-gen21` -- `platform-3.4.1-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.1 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -### platform-3.5.0-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen21 - -#### Additional tags - -- `platform-3.5-gen21` -- `platform-3.5.0-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.0 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 0.8.0 | -| Snowflake | Not supported | - -### platform-3.3.3-nvidia-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen21 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.3 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -#### GPU Nvidia - -| name | version | -| :----------------: | :-----: | -| Cuda Driver | 11.8.89 | -| Rapids-4-Spark Lib | 23.10.0 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen21 - -#### Additional tags - -- `platform-2.4.8-gen21` -- `platform-2.4-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12.10 | -| Python | 3.7.13 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen21 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.13 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-3.3.0-hadoop-3.3.3-java-8-scala-2.12-python-3.7-gen21 - -#### Additional tags - -- `platform-aws-emr-6.9-gen21` -- `platform-aws-emr-6-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.0 | -| Hadoop | 3.3.3 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.7.13 | -| Hive | 2.3.9 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -### platform-2.4.8-hadoop-2.10.1-java-8-scala-2.11-python-3.7-gen21 - -#### Additional tags - -- `platform-aws-emr-5.36-gen21` -- `platform-aws-emr-5-gen21` - -#### Dependency versions - -| name | version | -| :-----: | :----------: | -| Spark | 2.4.8 | -| Hadoop | 2.10.1 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.13 | -| Hive | 1.2.1.spark2 | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | Not supported | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.3 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen22.md b/src/docs/ocean-spark/docker-images-release-notes/gen22.md deleted file mode 100644 index 8044144ca8..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen22.md +++ /dev/null @@ -1,83 +0,0 @@ - - -# gen22 release notes - -## Changelog - -Gen22 is a bugfix release to correct a build issue that affected 3.4.1 and 3.5 images in gen21. - -## Available tags - -### platform-3.4.1-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen22 - -#### Additional tags - -- `platform-3.4-gen22` -- `platform-3.4.1-gen22` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.1 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -### platform-3.5.0-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen22 - -#### Additional tags - -- `platform-3.5-gen22` -- `platform-3.5.0-gen22` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.0 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.10 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 0.8.0 | -| Snowflake | Not supported | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen23.md b/src/docs/ocean-spark/docker-images-release-notes/gen23.md deleted file mode 100644 index c875196c57..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen23.md +++ /dev/null @@ -1,336 +0,0 @@ - - -# gen23 release notes (2024-08-02) - -## Changelog - -- Update java 8 base image to Temurin v8u422-b05 -- Update java 11 base image to Temurin v11.0.24_8 -- Add image for spark v3.3.4 -- Add image for spark v3.4.3 -- Add image for spark v3.5.1 -- Correct wrong version of delta included in spark v3.5.0 -- Make sure versions of AWS Java SDK jars are coherent -- Update Delta to v3.2 for spark v3.5.0 and v3.5.1 -- Release notes now include release date - -## Available tags - -### platform-3.3.4-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen23 - -#### Additional tags - -- `platform-3.3-gen23` -- `platform-3.3.4-gen23` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -### platform-3.3.4-hadoop-3.3.2-java-11-scala-2.12-python-3.10-gen23 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.2 | -| Java | 11 | -| Scala | 2.12.15 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -### platform-3.4.3-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen23 - -#### Additional tags - -- `platform-3.4-gen23` -- `platform-3.4.3-gen23` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.3 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -### platform-3.5.0-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen23 - -#### Additional tags - -- `platform-3.5.0-gen23` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.0 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 3.2.0 | -| Snowflake | Not supported | - -### platform-3.5.1-hadoop-3.3.4-java-11-scala-2.12-python-3.10-gen23 - -#### Additional tags - -- `platform-3.5-gen23` -- `platform-3.5.1-gen23` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.1 | -| Hadoop | 3.3.4 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 3.2.0 | -| Snowflake | Not supported | - -### platform-3.5.1-hadoop-3.3.4-java-8-scala-2.12-python-3.10-gen23 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.1 | -| Hadoop | 3.3.4 | -| Java | 8 | -| Scala | 2.12.18 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.262 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 3.2.0 | -| Snowflake | Not supported | - -### platform-3.3.4-nvidia-hadoop-3.3.2-java-8-scala-2.12-python-3.10-gen23 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.2 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.14 | -| Hive | 2.3.9 | -| Pyarrow | 11.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.1026 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.13.0 | - -#### GPU Nvidia - -| name | version | -| :----------------: | :-----: | -| Cuda Driver | 11.8.89 | -| Rapids-4-Spark Lib | 24.06.0 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen23 - -#### Additional tags - -- `platform-2.4.8-gen23` -- `platform-2.4-gen23` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12.10 | -| Python | 3.7.10 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen23 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.10 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.1.5 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen24.md b/src/docs/ocean-spark/docker-images-release-notes/gen24.md deleted file mode 100644 index 09449dd77b..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen24.md +++ /dev/null @@ -1,306 +0,0 @@ - - -# gen24 release notes (2024-10-16) - -## Changelog - -- Update hadoop to hadoop v3.3.6 -- Update spark 3.5 to v3.5.3 -- Update hadoop gcs connector v2.2.25 -- Update guava to v32.0 -- spark_snowflake to: - - v2.16 for spark 3.3 - - v3.0 for spark 3.4 and 3.5 -- update scala 2.12 to v2.12.20 -- update scala 2.13 to v2.13.15 -- update nvidia rapids to v24.08.1 -- update to python 3.10 for 2.4 images -- Python updates: - - updated future to v1.0.0 - - update jupyter_client to 8.6.3 - - update pandas to 2.2.3 - -## Available tags - -### platform-3.3.4-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen24 - -#### Additional tags - -- `platform-3.3-gen24` -- `platform-3.3.4-gen24` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.6 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.16.0 | - -### platform-3.3.4-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen24 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.6 | -| Java | 11 | -| Scala | 2.12.15 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.16.0 | - -### platform-3.4.3-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen24 - -#### Additional tags - -- `platform-3.4-gen24` -- `platform-3.4.3-gen24` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.3 | -| Hadoop | 3.3.6 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -### platform-3.5.3-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen24 - -#### Additional tags - -- `platform-3.5-gen24` -- `platform-3.5.3-gen24` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.3 | -| Hadoop | 3.3.6 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | Not supported | -| Snowflake | Not supported | - -### platform-3.5.3-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen24 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.3 | -| Hadoop | 3.3.6 | -| Java | 8 | -| Scala | 2.12.18 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | Not supported | -| Snowflake | Not supported | - -### platform-3.3.4-nvidia-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen24 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.3.4 | -| Hadoop | 3.3.6 | -| Java | 8 | -| Scala | 2.12.15 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 17.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :--------------------: | -| Delta | 2.3.0 | -| Snowflake | Spark Snowflake 2.16.0 | - -#### GPU Nvidia - -| name | version | -| :----------------: | :-----: | -| Cuda Driver | 11.8.89 | -| Rapids-4-Spark Lib | 24.08.1 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen24 - -#### Additional tags - -- `platform-2.4.8-gen24` -- `platform-2.4-gen24` - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.12.10 | -| Python | 3.7.10 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | - -### platform-2.4.8-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen24 - -#### Dependency versions - -| name | version | -| :-----: | :-----------: | -| Spark | 2.4.8 | -| Hadoop | 3.1.0 | -| Java | 8 | -| Scala | 2.11.12 | -| Python | 3.7.10 | -| Hive | Not supported | -| Pyarrow | 8.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.11.271 | -| GCS (`gs://`) | 2.2.25 | -| ADLS gen1 (`adl://`) | ADLS SDK 2.2.5 | -| ADLS gen2 (`abfss://`) | Azure Storage 5.4.0 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 5.4.0 | -| AWS Glue | Not supported | - -#### Additional formats - -| name | version | -| :-------: | :-------------------: | -| Delta | 0.6.1 | -| Snowflake | Spark Snowflake 2.9.3 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/gen25.md b/src/docs/ocean-spark/docker-images-release-notes/gen25.md deleted file mode 100644 index 840aafd6b0..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/gen25.md +++ /dev/null @@ -1,161 +0,0 @@ - - -# gen25 release notes (2024-11-26) - -## Changelog - -- upgrade spark 3.4 images to spark 3.4.4 -- upgrade Nvidia images from spark 3.3.4 to spark 3.4.3 -- images now have a named 'spark' user matching user id 185 -- aws sdk upgraded to v1.12.777 -- pyarrow upgraded to v18.0.0 -- ipython upgraded to v8.29.0 -- guava upgraded to v32.1.3 -- deltalake upgraded to v3.2.1 - -## Available tags - -### platform-3.4.4-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen25 - -#### Additional tags - -- `platform-3.4-gen25` -- `platform-3.4.3-gen25` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.4 | -| Hadoop | 3.3.6 | -| Java | 11 | -| Scala | 2.12.17 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 18.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.11-shaded | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -### platform-3.5.3-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen25 - -#### Additional tags - -- `platform-3.5-gen25` -- `platform-3.5.3-gen25` - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.3 | -| Hadoop | 3.3.6 | -| Java | 11 | -| Scala | 2.12.18 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 18.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.14-shaded | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | Not supported | -| Snowflake | Not supported | - -### platform-3.5.3-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen25 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.5.3 | -| Hadoop | 3.3.6 | -| Java | 8 | -| Scala | 2.12.18 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 18.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.14-shaded | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | Not supported | -| Snowflake | Not supported | - -### platform-3.4.3-nvidia-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen25 - -#### Dependency versions - -| name | version | -| :-----: | :-----: | -| Spark | 3.4.3 | -| Hadoop | 3.3.6 | -| Java | 8 | -| Scala | 2.12.17 | -| Python | 3.10.15 | -| Hive | 2.3.9 | -| Pyarrow | 18.0.0 | - -#### Supported connectors - -| name | version | -| :-----------------------------: | :-----------------: | -| S3 (`s3a://` or `s3://`) | AWS 1.12.367 | -| GCS (`gs://`) | 2.2.11-shaded | -| ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | -| ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | -| Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | -| AWS Glue | Hive 2.3.9 | - -#### Additional formats - -| name | version | -| :-------: | :-----------: | -| Delta | 2.4.0 | -| Snowflake | Not supported | - -#### GPU Nvidia - -| name | version | -| :----------------: | :-----: | -| Cuda Driver | 11.8.89 | -| Rapids-4-Spark Lib | 24.10.0 | diff --git a/src/docs/ocean-spark/docker-images-release-notes/legacy-dm-images.md b/src/docs/ocean-spark/docker-images-release-notes/legacy-dm-images.md deleted file mode 100644 index 68320789a1..0000000000 --- a/src/docs/ocean-spark/docker-images-release-notes/legacy-dm-images.md +++ /dev/null @@ -1,49 +0,0 @@ - - -# Legacy images release notes - -Before generation 18, images were available in the `gcr.io/datamechanics/spark` repository. - -## Release dm18 - -- Add AWS Glue support for Spark 3 images (Spark 2.4 image do not support Glue) - -## Release dm17 - -- Add new images with Spark 3.1.3 and Spark 3.2.1 -- Upgrade Delta version to 1.0.1 with Spark 3.1.x -- Upgrade Snowflake to 2.9.3 for Spark 2 and 2.10.0 for Spark 3 - -## Release dm16 - -- Add new images with Spark 3.0.3 and Spark 3.1.2 -- Upgrade Snowflake connector to 2.9.2 -- Use the newly released Delta version 1.1.0 with Spark 3.2.0 - -## Release dm15 - -- Add new images with Spark 3.2.0 and Hadoop 3.3.1 -- Upgrade OS packages to apply latest security patches -- Use a JDK base image instead of a JRE base image to include tools like `jstack` -- Upgrade Snowflake connector to 2.9.1 -- Pin pip version for Python 2 images. Latest pip versions are not compatible with Python 2 anymore. - -## Release dm14 - -- Use Delta 1.0 on Spark ≥ 3.1 - -## Release dm13 - -- `pyarrow` support is added to all images. The version of `pyarrow` is 3.0.0. -- Delta is upgraded to version 0.8.0 for all images with Spark version ≥ 3.0.0. The Delta version for Spark 2.4.x images is 0.6.1 (unchanged). Please note that Delta is still incompatible with Spark 3.1.1 at the time of this release. - -## Release dm12 - -This release includes the first generation of images made available to the public. - -- Snowflake version: 2.8.4 -- AWS connector: determined by the Hadoop version -- Azure connector: determined by the Hadoop version -- GCS connector: 2.1.5 -- Guava version: 29.0 -- Delta version: 0.7.0 for Spark 3.0.0 and above, 0.6.1 for Spark 2.4.x diff --git a/src/docs/ocean-spark/getting-started/README.md b/src/docs/ocean-spark/getting-started/README.md deleted file mode 100644 index bc64eaa3b3..0000000000 --- a/src/docs/ocean-spark/getting-started/README.md +++ /dev/null @@ -1,21 +0,0 @@ - - -# Get Started with Ocean for Apache Spark - -Ocean for Apache Spark (also referred to as Ocean Spark) is a managed Spark platform deployed on a Kubernetes cluster in your cloud account. - -This documentation section walks you through the steps to get started with Ocean Spark. - -## Create a Spot Account - -If you don’t have a Spot account yet, you should [create a Spot account](https://console.spotinst.com/spt/auth/signUp). You can do this for free and then add additional users to your Spot organization. - -## Communicate your Spot Organization ID to the Ocean Spark Team. - -If the Ocean Spark menu does not appear in the Spot console navigation bar, please provide your Spot organization ID to the Ocean Spark team, so that they can enable the menu for you. If you do not have a contact in the Ocean Spark team yet, [schedule a first call](https://calendly.com/oceanspark/demo) with us. - -## Connect your Cloud Account to Spot - -Connect the cloud account in which you intend to deploy Ocean Spark: ([AWS](connect-your-cloud-provider/aws-account)) - -> **Tip**: Other Spot products such as [Eco](https://docs.spot.io/eco/) can give you visibility into your entire cloud spend and help you optimize it. Therefore, these products may require connecting multiple accounts (or your master billing account). For Ocean Spark, connecting only the account in which you intend to deploy an Ocean Spark cluster is enough. diff --git a/src/docs/ocean-spark/getting-started/create-cluster.md b/src/docs/ocean-spark/getting-started/create-cluster.md deleted file mode 100644 index 37a87edabd..0000000000 --- a/src/docs/ocean-spark/getting-started/create-cluster.md +++ /dev/null @@ -1,121 +0,0 @@ - - -# Create an Ocean Spark Cluster - -There are several ways to deploy an Ocean Spark cluster: - -- Create a new Kubernetes cluster from scratch -- Import an existing Kubernetes cluster to Ocean Spark -- Import an existing Ocean cluster to Ocean Spark - -Each method is described below. Choose the method right for you. - -## Create a New Kubernetes Cluster from Scratch - -### Using spotctl (AWS only) - -1. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) (and configure it for your AWS account), the Kubernetes [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) utility, and the [spotctl command-line tool](https://github.com/spotinst/spotctl#installation). - -2. Create a cluster by running the command: - -``` -$ spotctl ocean spark create cluster --region $YOUR_REGION --cluster-name $MY_CLUSTER_NAME -``` - -This command will create a new EKS cluster, a new VPC, subnets, and other resources required to make Ocean Spark functional. - -### Using Terraform - -**Option 1**: Deploy Ocean Spark cluster in an existing VPC. -Follow [this example on AWS](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/from-private-vpc) or [this example on GCP](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/gcp-from-vpc) from the [ocean-spark Terraform module](https://registry.terraform.io/modules/spotinst/ocean-spark/spotinst/latest). - -**Option 2**: Deploy Ocean Spark cluster in a new VPC. -Follow [this example on AWS](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/from-scratch) or [this example on GCP](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/gcp-from-scratch) from the [ocean-spark Terraform module](https://registry.terraform.io/modules/spotinst/ocean-spark/spotinst/latest). - -### Additional Method - -You can also follow the documentation on how to get started with Ocean, and then use the method described below to import an existing Ocean cluster into Ocean Spark. - -## Import an Existing Kubernetes Cluster to Ocean Spark - -### Using Terraform - -Follow [this example](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/import-eks-cluster) from the [ocean-spark Terraform module](https://registry.terraform.io/modules/spotinst/ocean-spark/spotinst/latest) to import an existing EKS cluster (AWS) into Ocean Spark. To import an existing GKE cluster (GCP), use [this example](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/gcp-import-gke-cluster). - -### Additional Method - -You can also follow the documentation on [how to get started with Ocean](ocean/getting-started/), and then use the method described below to import an existing Ocean cluster into Ocean Spark. - -## Import an Existing Ocean Cluster to Ocean Spark - -Ocean Spark leverages Ocean under the hood, so it’s easy to import an existing Ocean cluster into Ocean Spark. Running this step will install a few additional pods on your Ocean cluster. These pods will enable the features related to monitoring and optimization specific to Apache Spark. - -### Using spotctl (AWS only) - -1. Make sure you can connect to the target Kubernetes cluster with the Kubernetes [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) utility. On AWS, install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) (and configure it for your AWS account). -2. Install the [spotctl command-line tool](https://github.com/spotinst/spotctl#installation). -3. Create a cluster by running this command, where the Ocean cluster ID is of the format o-XXXXXXXX: - -``` -$ spotctl ocean spark create cluster --ocean-cluster-id $YOUR_OCEAN_CLUSTER_ID -``` - -### Using Terraform - -Follow [this example](https://github.com/spotinst/terraform-spotinst-ocean-spark/tree/main/examples/import-ocean-cluster) from the [ocean-spark Terraform module](https://registry.terraform.io/modules/spotinst/ocean-spark/spotinst/latest). - -## Monitor your Ocean Spark Cluster Deployment - -When you start running the script or command to create the cluster, the following major events take place: - -* Kubernetes cluster creation (if creating a cluster from scratch). The duration of this step varies depending on the cloud provider, but this can take 20 minutes or more. You may be able to track progress from your cloud provider console. -* Ocean controller installation. The Ocean controller is installed on the cluster. The cluster is then registered with Spot and will be visible in the Spot console (under the Ocean UI). -* Ocean Spark controller installation. The Ocean Spark components are then installed, and the cluster will be visible in the Spot console (under the Ocean Spark UI). - -You can view the status of the newly created cluster on the Cluster page of the Ocean Spark console. The cluster status should move from Progressing to Available as the creation completes. Other statuses indicate an error. You can troubleshoot in the list of common issues below. - -## Requirements for a Functioning Ocean Spark Cluster - -This section provides a list of requirements for an Ocean Spark cluster deployment. - -**General Availability** versions are fully enabled for customer usage, ready for production use, and have no restrictions on support. These versions are recommended for the cluster underlying new applications as they provide customers with the most complete range of features and fixes. There is no set time on how long versions remain in the General Availability state; they will be moved to the Deprecated state according to the timetable of the cloud provider tables that are referenced below. - -**Deprecated versions** are fully supported and tested but are not our recommended choice for the cluster underlying new applications. There is no set time on how long versions remain in the Deprecated state; they will be moved to the Retired state according to the timetable of the cloud provider tables that are referenced below. - -**Retired versions** are no longer supported. We require that these clusters are upgraded or replaced. Versions will be moved to the Retired state according to the timetable of the cloud provider tables that are referenced below. Any issues encountered with a Retired cluster underlying your application will not be supported by us, but we will advise you during your update of the cluster to a more suitable version. - -### AWS - -- The Kubernetes cluster should use a [version supported by Amazon](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Spot will begin supporting, with General Availability, a version two months after the Amazon EKS release column date listed in that linked table. Spot will treat the version as Deprecated at the End of standard support column date listed in that linked table. Spot will treat the version as Retired at the End of extended support column date listed in that linked table. -- The VPC subnets should have the [proper tags](https://aws.amazon.com/premiumsupport/knowledge-center/eks-vpc-subnet-discovery/) to be discoverable by Kubernetes: - - On all subnets: `kubernetes.io/cluster/: shared` - - On public subnets: `kubernetes.io/role/elb: 1` -- The instance profile assumed by cluster nodes should have: - - The [required permissions](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) for EKS -- The permission to create security groups within the VPC -- The cluster nodes should be in a security group that allows them: - - To connect to one another - - To reach the Internet - - To connect to the Kubernetes API (which is in the cluster security group) -- If nodes are run in private subnets, make sure a NAT gateway is available in the cluster to enable egress to the Internet. -- All the Ocean Spark Virtual Node Groups (VNGs) should have access to the same subnets, or at least to the same availability zones (AZs). - -### GCP - -- The Kubernetes cluster should use a [version supported by GCP](https://cloud.google.com/kubernetes-engine/docs/release-schedule). Spot will begin supporting, with General Availability, a version two months after the Stable - Available column date listed in that linked table. Spot will treat the version as Deprecated six months before the End of life column date listed in that linked table. Spot will treat the version as Retired at the End of life column date listed in that linked table. -- The service account assumed by cluster nodes should have at least the following roles: `monitoring.viewer`, `monitoring.metricWriter`, `logging.logWriter`, and `stackdriver.resourceMetadata.writer`. More details in [this section of GCP doc](https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#use_least_privilege_sa) -- If Spark applications use custom Docker images stored in Container Registry, the node service account should also have `objectViewer` access to the GCS bucket where the Docker images are stored. -- The cluster nodes should be allowed: - - To connect to one another - - To reach the Internet - - To connect to the Kubernetes API -- If the cluster nodes are private, make sure a NAT service is installed in the Cloud Router of the VPC. -- All the Ocean Spark Virtual Node Groups (VNGs) should have access to the same subnets, or at least to the same locations (also called availability zones by analogy with AWS). - -### Azure - -- The Kubernetes cluster should use a [version supported by Azure](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli.). Spot will begin supporting, with General Availability, a version two months after the AKS GA column date listed in that linked table. Spot will treat the version as Deprecated at the End of life column date listed in that linked table. Spot will treat the version as Retired at the Platform support column date listed in that linked table. - -## What’s Next? - -Learn how to [submit your first Spark application](ocean-spark/getting-started/run-your-first-app). diff --git a/src/docs/ocean-spark/getting-started/run-your-first-app.md b/src/docs/ocean-spark/getting-started/run-your-first-app.md deleted file mode 100644 index 7de3ce4e8e..0000000000 --- a/src/docs/ocean-spark/getting-started/run-your-first-app.md +++ /dev/null @@ -1,151 +0,0 @@ - - -# Run Your First App - -Now that you have created your first [Ocean Spark cluster](ocean-spark/getting-started/create-cluster), you are ready to run your first app. - -## Prerequisites - -To run your first app, you will need to have: -- The Ocean Spark cluster ID of the cluster you just created (of the format osc-e4089a00). You can find this in the console in the [list of clusters](ocean-spark/product-tour/manage-clusters), or by using the Get Cluster List in the API. -- A [Spot token](administration/api/create-api-token) to interact with [Spot API](https://docs.spot.io/api/). -- A Spot Account ID, this can be found in the same menu location as the API key - -Using the [Ocean Spark API](https://docs.spot.io/api/#tag/Ocean-Spark), you can run, configure, and monitor applications using the different endpoints available. - -To know more about the API endpoints and parameters, check out the API reference. - -## Run Pi App - -The command below will run the classic Monte-Carlo Pi computation contained in all Spark distributions: - -``` -curl -k -X POST \ -'https://api.spotinst.io/ocean/spark/cluster//app?accountId=' \ --H 'Content-Type: application/json' \ --H 'Authorization: Bearer ...' \ --d '{ - "jobId": "spark-pi", - "configOverrides": - { - "type": "Scala", - "sparkVersion": "3.2.0", - "mainApplicationFile": "local:///opt/spark/examples/jars/examples.jar", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "mainClass": "org.apache.spark.examples.SparkPi", - "arguments": ["1000"], - "executor": { - "instances": 1 - } - } -}' -``` - -Here's a breakdown of the payload: -- We assign the job ID "spark-pi" to the application. A job is a logical grouping of applications. It is typically a scheduled workload that runs every day or every hour. Every run of a job is called an application in Ocean Spark. In the console, the Jobs view lets you track performance of jobs over time. A unique app ID will be generated from the job ID (although you can specify one yourself). -- Default configurations are overridden in configOverrides: - - This is a Scala application running Spark 3.2.0. - - The command to run is specified by mainApplicationFile, mainClass, and arguments. - -The API then returns something like: - -```json -{ - "request":{ - "id":"39e2b4a4-46c9-4ff3-bc3a-e5d3f2432549", - "url":"/ocean/spark/cluster/osc-e4089a00/app", - "method":"POST", - "timestamp":"2021-11-14T21:28:35.546Z" - }, - "response":{ - "status":{ - "code":200, - "message":"OK" - }, - "kind":"spotinst:ocean:spark:application", - "items":[ - { - "internalId":"8ec73ba0-c7df-4b25-b21e-efaeb7c4bfe2", - "id":"spark-pi-2a201099-4ce9-4220-805e-049363174528", - "displayName":"spark-pi-2a201099-4ce9-4220-805e-049363174528", - "accountId":"act-27419163", - "organizationId":606079874885, - "userId":42, - "clusterId":"osc-e4089a00", - "controllerClusterId":"arnar-rokkar-111", - "appState":"NEW", - "submissionSource":"public-api", - "createdAt":"2021-11-14T21:28:35.546Z", - "updatedAt":"2021-11-14T21:28:35.546Z", - "job":{ - "id":"spark-pi", - "displayName":"spark-pi" - }, - "config":{ - "type":"Scala", - "sparkVersion":"3.2.0", - "image":"gcr.io/ocean-spark/spark:platform-3.2.0-dm15", - "mainApplicationFile":"local:///opt/spark/examples/jars/examples.jar", - "mainClass":"org.apache.spark.examples.SparkPi", - "arguments":[ - "1000" - ], - "sparkConf":{ - "spark.kubernetes.allocation.batch.size":"10", - "spark.sql.execution.arrow.enabled":"true", - "spark.kubernetes.allocation.driver.readinessTimeout":"120s", - "spark.sql.execution.arrow.pyspark.enabled":"true", - "spark.sql.execution.arrow.sparkr.enabled":"true", - "spark.sql.adaptive.enabled":"true", - "spark.storage.decommission.shuffleBlocks.enabled":"true", - "spark.storage.decommission.rddBlocks.enabled":"true", - "spark.storage.decommission.enabled":"true", - "spark.decommission.enabled":"true", - "spark.dynamicAllocation.enabled":"false", - "spark.dynamicAllocation.shuffleTracking.enabled":"true", - "spark.dynamicAllocation.executorAllocationRatio":"0.33", - "spark.dynamicAllocation.sustainedSchedulerBacklogTimeout":"30", - "spark.cleaner.periodicGC.interval":"1min" - }, - "driver":{ - "cores":4, - "coreRequest":"3460m", - "memory":"8192m", - "envVars":{ - "KUBERNETES_REQUEST_TIMEOUT":"30000", - "KUBERNETES_CONNECTION_TIMEOUT":"30000" - }, - "instanceType":"m5.xlarge", - "spot":false - }, - "executor":{ - "cores":4, - "instances":1, - "coreRequest":"3460m", - "memory":"8192m", - "instanceType":"m5.xlarge", - "spot":true - }, - "priority":"normal" - } - } - ] -``` - -Note that some additional configurations are automatically set by Ocean Spark. In particular, the appId is a unique identifier of this Spark application on your cluster. Here it has been generated automatically from the jobId, but you can set it yourself in the payload of your request to launch an app. - -Beside the appId, Ocean Spark also set some defaults to increase the stability and performance of the app. Learn more about [configuration management](ocean-spark/configure-spark-apps/) and auto-tuning. - -The application you just created should appear in the Ocean Spark console: - - - -Clicking on the application name opens the [application details page](ocean-spark/product-tour/view-application-details). At this point, you can open the Spark UI, follow the live log stream, or kill the app. - - - -This example uses a JAR embedded in the Spark Docker image and neither reads nor writes data. For a more real-world use case, learn how to [access your own data](ocean-spark/configure-spark-apps/access-your-data). - -## What’s Next - -Take the Ocean Spark [Product Tour](ocean-spark/product-tour/) where you will learn how to manage your clusters, view cluster details, and much more. diff --git a/src/docs/ocean-spark/getting-started/troubleshoot-cluster-deployment.md b/src/docs/ocean-spark/getting-started/troubleshoot-cluster-deployment.md deleted file mode 100644 index 463560343f..0000000000 --- a/src/docs/ocean-spark/getting-started/troubleshoot-cluster-deployment.md +++ /dev/null @@ -1,180 +0,0 @@ - - -# Troubleshoot Cluster Deployment - -This page describes a list of common issues specific to the cluster deployment phase that you could encounter and solutions to fix them. - -## Ocean cluster is unreachable -### Identify the issue - -In the Spot console, browse to the Ocean cluster list. Check if your cluster is marked as unreachable. - - - -### Troubleshoot - -- Ensure that the Ocean controller pod is running. Run ```kubectl get pods -n kube-system```. There should be a “spotinst-kubernetes-cluster-controller-...” pod in Running state. -- If the Ocean controller pod is in Pending state, ensure that there is a node in the cluster that can run the pod. Run ```kubectl get nodes```. There should be at least one node. -- If there are no nodes in the cluster, it is likely that EC2 instances can't join the cluster. See the corresponding section below. -- If the Ocean controller is in the Terminating/CrashLoopBackOff state, it means that the pod can't reach the Internet and call the Spot API at https://api.spotinst.io, or that the Spot credentials used by the Ocean controller are wrong. See the Ocean controller troubleshooting guide and the corresponding section below. - -## EC2 instances can't join the cluster -### Identify the issue - -1. List the EC2 instances in the AWS console of your AWS account. -2. Look for EC2 instances belonging to the EKS cluster. Their name usually contains the EKS cluster name as a prefix. -3. Tail the system log of the EC2 instance (under Actions > Monitor and troubleshoot > Get system log) -4. Look for any mentions of the words “eks”, “bootstrap”, “kubelet”. - -### Troubleshoot - -- If you find an access management error stating that the instance can't list EKS clusters, ensure that the cluster nodes assume an instance profile that grants them the required permissions. -- If the system log seems to be in progress (i.e., a command has just started, no success nor failure message), ensure that the cluster nodes can talk to each other and to the Kubernetes API. Check your configuration and ensure that: - - Cluster nodes are in a security group that allows traffic within it. - - The security group of the cluster nodes is allowed to reach the cluster security group. - -## Ocean controller can’t reach the Internet -### Identify the issue - -1. Tail the logs of the Ocean controller and look for any errors regarding connectivity: - -``` -kubectl logs -n kube-system -l 'k8s-app=spotinst-kubernetes-cluster-controller' -``` - -2. Run a pod on the same node as the Ocean controller pod, exec into it, and call the Spot API: - -``` -curl https://api.spotinst.io -``` - -### Troubleshoot - -- Follow the Ocean controller troubleshooting guide. -- Ensure that the nodes within the cluster are in a security group granting them access to the Internet. -- If the cluster is in a private VPC, ensure it contains a NAT gateway to enable egress to the Internet. - -## Ocean Spark cluster is in a degraded or progressing state -### Identify the issue - -Go to the Ocean Spark cluster list and look for your Ocean Spark cluster. - - - -### Troubleshoot - -- Ensure that no Ocean Spark pod is stuck in a pending state with ```kubectl get pods -n spot-system```. If an Ocean Spark pod is stuck in pending, then the Ocean cluster probably can’t scale up. See the corresponding section below. -- Ensure that a load balancer can be created by Ocean Spark with ```kubectl get svc -n spot-system```. If this command shows a service whose EXTERNAL-IP is stuck in pending, this means that the load balancer can't be created. See the corresponding section below. - -## Ocean-managed nodes can't join the cluster -### Identify the issue - -1. Go to the Nodes tab in your cluster’s page in Ocean. -2. Look for nodes failing to join the cluster. - - - -### Troubleshoot - -- As a rule of thumb, ensure that your Virtual Node Groups (VNGs) are configured like the EKS-managed cluster nodes, i.e., they have the same security group and the same instance profile. -- Please refer to “EC2 instances can't join the cluster”. All instructions apply. - -## Ocean Spark load balancer can’t be created -### Identify the issue - -1. Run ```kubectl get svc -n spot-system``` -2. If this command shows a service whose EXTERNAL-IP is stuck in pending, this means that the Ocean Spark load balancer can’t be created. - -### Troubleshoot - -- Ensure your VPC subnets have the proper tags to be discoverable by Kubernetes - - On all subnets: ```kubernetes.io/cluster/: shared``` - - On public subnets: ```kubernetes.io/role/elb: 1``` -- Ensure that the instance profile assumed by the cluster nodes grants them the permission to create a new security group within the VPC. - -## Spark application can’t acquire executor pods -### Identify the issue - -1. In the driver log of your Spark application, a message like this appears: ```Initial job has not accepted any resources. Check your cluster UI to ensure that workers are registered and have sufficient resources``` -2. There are pending executor pods in the cluster: ```kubectl get pods -n spark-apps``` - -### Troubleshoot - -- Ensure that all your Virtual Node Groups (VNGs) have access to the same subnets. This is required because Ocean Spark puts the driver and executors of a given Spark application in the same availability zone to reduce network costs. -- Have a look at the Ocean log tab of your cluster in the Ocean section of the Spot console. Look for messages stating why the cluster is not scaling up. - -## Kubernetes logs in the Spot console cannot be displayed - -### Identify the issue - -The following message appears in the Kubernetes logs view in the application page. - -``` -An unexpected error happened while fetching the logs, please refresh the page. -``` - -### Troubleshoot - -This issue is likely caused by restrictions on node-to-node communication over the network. -Please go to the "node-to-node communication not allowed" below. - -## Spark driver logs cannot be displayed although pod is running - -* The following message appears in the Driver logs view in the application page. -``` -The driver pod is not running yet, please wait a few seconds... -``` -* Check whether the driver pod of the Spark application is running on the Kubernetes cluster with ```kubectl get pods -n spark-apps```. If it is running, driver logs should be available in the Driver logs view. - -Optional check: -* Tail the logs of the driver pod with ```kubectl logs -n spark-apps -driver``` -* You may see name resolution errors like ```kubernetes.default.svc: Temporary failure in name resolution``` in the driver log. This would be a further indication that node-to-node communication is the culprit (see the troubleshoot section below). -* Not seeing name resolution errors in the driver log does not completely rule out node-to-node communication as the culprit however. It may be that a security group rule specifically allows DNS traffic (port 53), while other types of traffic are restricted. - -### Troubleshoot - -This issue is likely caused by restrictions on node-to-node communication over the network. -Please go to the "node-to-node communication not allowed" below. - -## Node-to-node communication not allowed - -### Identify the issue - -Ocean Spark pods must be able to communicate with one another and with Spark applications. -Any one of the following issues detailed above may be a sign that node-to-node communication is not configured properly: -* Kubernetes logs in the Spot console cannot be displayed -* Spark driver logs cannot be displayed although pod is running - -### Troubleshoot - -Ensure node-to-node communication is possible in the cluster. - -The security group(s) used by the cluster nodes must contain an inbound rule like: - -| Direction | Type | Protocol | Port | Source / Destination | -| :-------- | :---------- | :------- | :--- | :------------------- | -| Inbound | All traffic | All | All | Self | - -## App submission by spark-operator fails - -### Identify the issue - -* Inspect the Kubernetes logs view of your Spark application in the Spot console -* In the Kubernetes logs view, the spark-operator reports that it can't submit the application because it fails to create a driver pod: -``` -Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-apps] failed. -``` - -**Explanation:** The spark-operator registers a [mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to customize Spark pods at submission time (for example to mount volumes on old versions of Spark). -This means that when a pod is launched, Kubernetes requests the spark-operator webhook to give it an opportunity to mutate the pod. -As a result, if communication from the Kubernetes control plane to the nodes on port 443 is restricted, this request will fail. - -### Troubleshoot - -Ensure cluster-to-node communication is possible in the cluster. - -The security group(s) used by the cluster nodes must contain an inbound rule allowing HTTPS traffic from the Kubernetes control plane: - -| Direction | Type | Protocol | Port | Source / Destination | -| :-------- | :---- | :------- | :--- | :----------------------------------- | -| Inbound | HTTPS | TCP | 443 | Cluster control plane security group | diff --git a/src/docs/ocean-spark/product-tour/README.md b/src/docs/ocean-spark/product-tour/README.md deleted file mode 100644 index 2261955c13..0000000000 --- a/src/docs/ocean-spark/product-tour/README.md +++ /dev/null @@ -1,38 +0,0 @@ - - -# Product Tour - -This section takes you through a tour of the main pages of Ocean for Apache Spark (also referred to as Ocean Spark): Clusters, Applications, and Jobs. Let’s first define these major concepts. - - - -## Clusters - -[Clusters](ocean-spark/product-tour/manage-clusters) make up the compute infrastructure on which your Spark applications run. These Kubernetes clusters are typically long-running clusters. After initially provisioning them, most people let them run 24x7. Clusters do not have a fixed size. Ocean takes care of automatically scaling them up and down (by adding or removing nodes, based on the load), so that your costs are minimized if few or no Spark applications are running. - -Distinct Spark applications running on the same cluster are isolated from each other. Each Spark application can run its own Docker image with its own Spark version. For this reason, many customers choose to use a single cluster to run all their Spark applications. Another common setup is to have one cluster per cloud account and environment (e.g., dev, staging, prod), or one cluster per cloud region (e.g., us-west, us-east). - -Ocean Spark lets you manage clusters in a self-service way. - -## Applications - -A Spark application is the runtime execution of Spark code, submitted interactively through a notebook or as a file to execute through an API call. When you submit a Spark application on a cluster, Kubernetes first needs to schedule the Spark driver pod, which means placing it on a Kubernetes node by reusing existing capacity or provisioning a new node. Kubernetes then downloads the application’s Docker image, and runs it. Once the Spark driver has started, it will request Spark executors, which will be scheduled on the same cluster. A Spark application is therefore made of one Spark driver (one pod), and a variable number of Spark executors (one executor = one pod). - -Even if you have multiple clusters (hosted in the same cloud account), Ocean for Apache Spark lets you [monitor the applications](ocean-spark/product-tour/monitor-applications) running on these clusters on a single dashboard. - -## Jobs - -A [Spark job](https://docs.spot.io/api/#tag/Ocean-Spark) is a logical grouping of Spark applications which you explicitly define when you submit Spark applications, as the Spark job identifier is a required field in our [REST API](). Applications belonging to the same job are also called executions of the job. - -For example you could define a job with the ID `daily-etl-ingestion`. This job would be scheduled on a daily basis from Airflow, and then applications within this job could be `daily-etl-ingestion-2022-01-01`, `daily-etl-ingestion-2022-01-02`, `daily-etl-ingestion-2022-01-03`, etc. - -The interest of grouping Spark applications together within a job is that you get a dedicated dashboard where you can track all the executions of your job and view relevant metrics and trends (e.g., duration, cloud costs, and volume of data read and written). Ocean Spark can automatically tune specific infrastructure parameters and Spark configurations at the level of a job by learning from the historical performance characteristics of previous executions of the job. - -A job belongs to a specific cluster, but the main Jobs dashboard gives you visibility over all the jobs you have defined across all your clusters (as long as these clusters are hosted in the same cloud account). - -> **Tip**: A notebook is a special type of Spark application which does not belong to any job. - -## What’s Next? - -- Continue the Product Tour and learn how to [manage your clusters](ocean-spark/product-tour/manage-clusters). -- [Get started](ocean-spark/getting-started/) with your first cluster in Ocean for Apache Spark. diff --git a/src/docs/ocean-spark/product-tour/analyze-costs.md b/src/docs/ocean-spark/product-tour/analyze-costs.md deleted file mode 100644 index 2f193d2cdc..0000000000 --- a/src/docs/ocean-spark/product-tour/analyze-costs.md +++ /dev/null @@ -1,130 +0,0 @@ - - -# Analyze Costs - -Ocean for Apache Spark (Ocean Spark) provides full visibility into the cost of your Ocean Spark cluster, including a view of your total cloud compute costs over time. In addition, detailed information about your cloud consumption is provided in cost breakdown views showing core and application hours used. - -## Get Started - -To get to the Cluster Cost Analysis tab, do the following: - -1. In the Spot console, go to Ocean for Spark in the menu tree and click Clusters. -2. In the [list of Clusters](ocean-spark/product-tour/manage-clusters), click on a Cluster Name and click the Cost Analysis tab. - -The Cost Analysis tab opens and shows the cluster name at the top. A graph of Spark App Cost over Time appears with Cloud Compute Cost as the default view. - - - -## Spark App Cost over Time - -The top part of the cost analysis page is a bar chart showing your Spark application usage metrics on the cluster over time. - -Cloud compute costs, Core Hours, and App Hours are the usage metrics presented. The metrics are calculated for each Spark application that runs on the cluster. Cloud compute costs correspond to your cloud provider bill, while the Core Hours and App Hours metrics can influence your Ocean Spark service fee. These metrics are then aggregated per day, based on the timestamp (in UTC time zone) at which the application finished running in your cloud account. - - - -### Filtering and Grouping - -You can adjust your view of the data by setting the following parameters: -- Filter by: Set the time range for displaying data in the chart. - - - -- Group by: Group the data according to job or user. - - - -### Narrow down Information in the Chart - -Each bar shows the total cost of the cluster for that time increment and is broken down to show the cost of each job or user in the cluster. -- You can include or exclude jobs or users from the display by clicking on the names in the key above the bars. -- To see a subtotal for a particular job or user, click on its color in a bar. - -## Cloud Cost View - -*Cloud cost* is defined as the sum of compute (instances) and storage (volumes) costs incurred by running your Spark application in your cloud account. This information is estimated by Spot based on the cloud provider billing data, and based on your application resource requirements. You can also find similar information in the Ocean console. - -Cloud Cost is the default view (as shown in the example [above](ocean-spark/product-tour/analyze-costs?id=get-started)). Each bar in this chart shows the cost broken down into jobs. Each color in the bar represents a job. You can hover your mouse over the different colors in a bar to see information about each job. - -> **Tip**: It can take up to two days after an application’s completion for the cost data to be accurate. - - - -## Core Hours View - -Your Spark applications define the number of cores allocated to the driver pod and the number of cores allocated to each executor pod. The *core hours* used by an application correspond to the sum over each of these pods of the uptime duration of the pod multiplied by the number of cores allocated to it. - -Sample calculation: -- A driver pod with one core runs for 30 minutes -- An executor pod with four cores runs for 30 minutes -- A second executor pod with four cores runs for 10 minutes and 15 seconds - -This application used 1 * 30/60 + 4 * 30/60 + 4 * 615 / 3600 = 3.18 core hours. - -In the Core Hours view, each bar shows the total number of core hours, and the different colors show the breakdown of core hours used by each job. - - - -## App Hours View - -The *App Hours* metric is defined as the sum of the runtime durations of your application, where the runtime duration corresponds to the difference between the application’s completion time, and the time when the application entered the Running state (meaning the Spark driver has started running). - -In the App Hours view, each bar shows the total number of application hours, and the different colors show the breakdown of core hours used by each application. - - - -## Download - -Click Download to download the data that forms any of the above cost analysis bar charts to a CSV file. - - - -## Top Spend Summary - -Below the bar chart, a table summarizes your top spend jobs or users in the cluster. The jobs or users appear in order from highest to lowest cost proportion in your cluster. - - - -The table includes the following information for each application: - -- Job or User -- Core Hours -- App Hours -- Cloud Compute Costs -- % of Total Costs - -The total cost for all the jobs or users in the cluster appears at the bottom of the table. - -### Drill Down - -In the Top Spend Summary, you can see a breakdown of costs per application for each job or user. Just click on the job or user name. - - - -## Allocate Cloud Costs Using Custom Labels - -You can define custom cost labels for each application you run on Ocean Spark. - -Here is an example configuration which you can insert in your configuration templates, in your job configuration, or directly in your API calls as configOverrides. - -```json -{ - "labels": { - "team": "data-engineering", - "project": "etl", - "environment": "production", - } -} -``` - -You will then be able to view your cloud costs, grouped by these custom cost labels, using the Cost Analysis page of Ocean (and not Ocean Spark). To find this page: - -1. From your Ocean Spark cluster page, click "View cluster in Ocean". -2. Then click "Cost Analysis". -3. In the "Group By" dropdown, select "Label (Resource)", and then pick your label key ("team", "project", or "environment"). - -Visit the Ocean documentation to learn more about [how Ocean estimates your cloud costs](/ocean/features/cost-analysis). - -## What’s Next? - -Learn more about [monitoring your applications](ocean-spark/product-tour/monitor-applications). diff --git a/src/docs/ocean-spark/product-tour/manage-clusters.md b/src/docs/ocean-spark/product-tour/manage-clusters.md deleted file mode 100644 index b98149cdca..0000000000 --- a/src/docs/ocean-spark/product-tour/manage-clusters.md +++ /dev/null @@ -1,46 +0,0 @@ - - -# Manage Clusters - -Ocean for Apache Spark (also referred to as Ocean Spark) enables you to see an overview of all your Ocean Spark clusters, get status at a glance, perform tasks such as adding and removing clusters, and drill down to more detailed cluster information when you need to. - -To manage your clusters in the Spot console, go to Ocean for Spark in the menu tree and click Clusters. - - - -## View List of Clusters - -The list of Ocean Spark clusters gives you a quick view of your clusters and basic information including: -Cluster Name: The user-given name of the cluster. A colored icon next to the cluster name indicates the current health status. - -- Cluster ID: The ID that Ocean Spark assigned to the cluster upon creation. -- Region: The cloud provider region where the cluster is located. -- Nodes: The number of nodes in the cluster. -- Running Apps: The number of applications running in the cluster. -- Creation Date: The date the cluster was created. -- Action: An option to remove the cluster from Wave. - -## View Cluster Details - -To get detailed information, statistics, and operational information about a cluster, click on the Cluster Name. This will open the [Cluster Overview](ocean-spark/product-tour/view-cluster-details) tab for that cluster which serves as your operational dashboard for the cluster. - -## Filter Cluster List - -If you have a long list of clusters, you can use the filter above the list to find one or multiple clusters. You can filter by cluster name, cluster ID, or region. Alternatively, you can enter a tag, an attribute, a keyword, or simply a string of text into the filter box and type Enter. - -## Add Cluster - -To add a cluster, click Add Cluster above the cluster list and complete the procedures described in [Get Started](ocean-spark/getting-started/) with Ocean for Apache Spark. - -## Remove Cluster - -To disconnect a cluster you don’t need any more, do the following: - -1. Click Remove in the Action column. -2. Enter the name of the cluster to confirm, and click Remove Cluster. - - - -## What’s Next? - -Learn how to get detailed cluster statistics and trends in the [Cluster Overview](ocean-spark/product-tour/view-cluster-details) tab. diff --git a/src/docs/ocean-spark/product-tour/monitor-applications.md b/src/docs/ocean-spark/product-tour/monitor-applications.md deleted file mode 100644 index be282ebe01..0000000000 --- a/src/docs/ocean-spark/product-tour/monitor-applications.md +++ /dev/null @@ -1,35 +0,0 @@ - - -# Monitor Applications - -Ocean for Apache Spark (also referred to as Ocean Spark) enables you to see an overview of all your Spark applications, get status at a glance, drill down to more detailed application information, and kill your application when you need to. - -To monitor your applications in the Spot console, go to Ocean for Spark in the menu tree and click Applications. - - - -## View Application List - -The list of Ocean Spark applications gives you a quick view of your applications and basic information including: - -- Application: The name of the application and under it the name of the job it belongs to. To see more details about the application or the job, click on the application or job name. -- Cluster: The Ocean Spark cluster where the application is running. To see more information about the cluster, click on the cluster name. -- Started at: Start date and time. -- Duration: Amount of time of the application run. -- User: Name of the user running the application. -- Cores: Number of Spark cores currently allocated to a running application, counting both the cores allocated to the Spark driver and cores allocated to Spark executors. -- Core Hours: The total core resources used by the application. This metric is calculated as the sum over each container (driver or executor) of its uptime duration multiplied by the number of cores allocated to it. -- Cloud Cost: The cloud cost of the app incurred so far. This metric is updated each hour, so it may not be available for recent applications. -- Action: If you want to stop the application, click Kill. - -## Filter Application List - -If you want to monitor a specific subset of applications, use the filters at the top to create a shortened list. - -## View Application Details - -To get detailed information, metrics, and operational information about an application, click the application name. This will open the [Application Overview](ocean-spark/product-tour/view-application-details) tab for that application which serves as your operational dashboard for the application. - -## What’s Next? - -Learn more about how you can see [detailed application information](ocean-spark/product-tour/view-application-details). diff --git a/src/docs/ocean-spark/product-tour/monitor-jobs.md b/src/docs/ocean-spark/product-tour/monitor-jobs.md deleted file mode 100644 index 422c40b3c6..0000000000 --- a/src/docs/ocean-spark/product-tour/monitor-jobs.md +++ /dev/null @@ -1,38 +0,0 @@ - - -# Monitor Jobs - -Ocean for Apache Spark (also referred to as Ocean Spark) defines “jobs” as a logical grouping of applications. You decide on this grouping by providing a job identifier (job-id) when you submit a Spark application to our REST API. - -The Jobs section of the console gives you an overview of all your Spark jobs so you can see the status at a glance of the most recent execution of the job, and drill down to more detailed information. - -To monitor your jobs in the Spot console, go to Ocean for Spark in the menu tree and click Jobs. - - - -## View Job List - -The list of Ocean Spark jobs gives you a quick view of your jobs and basic information including: - -- Job Info - - Job: Name of the job. Click here to see job details. - - Cluster: Cluster that the job belongs to. -- Last Application Info - - Application: Name or the app. - - Status - - Started at - - Duration - - User - - Cloud Cost: These are the cloud provider costs incurred by the application. This information becomes available within the hour after the application finishes. The computation takes into account whether your application was running on spot or on-demand nodes. - -## Filter Job List - -If you want to monitor a specific subset of jobs, use the filters at the top to create a shortened list. - -## View Job Details - -To get detailed information, metrics, and operational information about a job, click the job name. This will open the [Job details page](ocean-spark/product-tour/view-job-details) for that job which serves as your operational dashboard for the job. - -## What’s Next? - -Learn more about [viewing job details](ocean-spark/product-tour/view-job-details). diff --git a/src/docs/ocean-spark/product-tour/use-vngs.md b/src/docs/ocean-spark/product-tour/use-vngs.md deleted file mode 100644 index 462e8c6f90..0000000000 --- a/src/docs/ocean-spark/product-tour/use-vngs.md +++ /dev/null @@ -1,42 +0,0 @@ - - -# Use Virtual Node Groups with Ocean Spark - -Ocean for Apache Spark (also referred to as Ocean Spark) is built on top of [Ocean](ocean/), the engine automating cloud infrastructure management for containers. As a result, when Ocean Spark customers run Spark applications, they benefit from all the features of Ocean at no additional cost. For example, your Ocean Spark cluster is also visible from the Ocean user interface, giving you visibility over its nodes, pods, and cloud provider costs in real time. - -You can use the same cluster to run both Spark and non-Spark workloads. In this situation: -- The Spark workloads will benefit from the features of Ocean Spark and will be charged a fee according to the Ocean Spark pricing, and nothing else. -- The non-Spark workloads will benefit from the features of Ocean only and will be charged a fee according to the Ocean pricing, and nothing else. - -To implement this mechanism and to have a clear distinction between Spark and non-Spark workloads, Ocean for Apache Spark applications run on dedicated [Virtual Node Groups](ocean/features/vngs/) (VNGs). This means the set of nodes on which Spark workloads run is separate from the nodes used by non-Spark workloads. - -## Automatically created VNGs - -When you create an Ocean Spark cluster, two VNGs called ocean-spark-on-demand and ocean-spark-spot are automatically created. You can see them in the Ocean interface under the Virtual Node Groups tab. - - - -These VNGs are dedicated to Ocean Spark, and they have certain best practice configurations automatically set in them (root volume size and a user data script to mount local SSDs when they are available). - -The on-demand VNG is configured to use on-demand nodes, while the spot VNG is configured to use spot nodes. Note that this is a best-effort attempt, but that in some cases on-demand instances can be launched on the spot VNG, as Ocean will fall back to using on-demand instances if it failed to launch spot ones. If this occurs, you can track it in the Log tab. - -## Adding, Modifying, and Deleting VNGs - -You can add, modify, or delete VNGs as long as these changes maintain the requirement to have two VNGs dedicated to Ocean Spark at all times (one configured to use on-demand nodes, and the other configured to use spot nodes). - -Additional VNGs can be created in the console, using Terraform, or using the Spot API. In the console, a checkbox shown in the example below gives you the option to dedicate a VNG to Ocean Spark. - - - -## VNG Applications - -VNGs are a powerful abstraction, you can use them to: -- Control the instance profile (IAM role) or service account. This is a convenient way to give your Spark applications access to the data they need. -- Define headroom and reduce your Spark application startup time. -- Control scaling by setting minimum and maximum sizes on a VNG, restricting the list of allowed instance types, or setting the percentage of nodes that can be scaled down in a single operation (we recommend 100% for fast scale down). -- Control networking by configuring the security groups and subnets in use. -- Define custom cloud tags, node labels, and taints. - -## What’s Next? - -Learn more about these features in the Ocean documentation about [Virtual Node Groups](ocean/features/vngs/). diff --git a/src/docs/ocean-spark/product-tour/view-application-details.md b/src/docs/ocean-spark/product-tour/view-application-details.md deleted file mode 100644 index 533e155246..0000000000 --- a/src/docs/ocean-spark/product-tour/view-application-details.md +++ /dev/null @@ -1,94 +0,0 @@ - - -# View Application Details - -To drill down into the details about your Ocean for Apache Spark application, start with the Overview tab, which gives you quick access to insights and summary data about the application. You can obtain an overview of your current cost, efficiency status, app metrics, and access to logs. You can view more details about the app in additional tabs including its [configuration](ocean-spark/product-tour/view-application-details?id=view-configuration) and a listing of [Spark issues](ocean-spark/product-tour/view-application-details?id=view-spark-issues). - -To get to the App Overview tab, do the following: - -1. In the Spot console, go to Ocean for Spark in the menu tree and click Applications. -2. In the [list of applications](ocean-spark/product-tour/monitor-applications), click an app name. - - - -The App page opens with the Overview tab open and the app name at the top. Next to the App name, a status icon indicates the App status. - -The App Overview includes the following main areas: - -- Metrics -- App Info -- Logs - -## Metrics - -Application Metrics is a summary line providing data about your app usage. The following information is presented: - -- Cloud Compute Cost: The cloud provider’s compute costs incurred by this application. -- Core Hours: The core resources used by the application. This metric is calculated as the sum over each container (driver or executor) of its uptime duration multiplied by the number of cores allocated to it. -- Data Read: Amount of data read by this application. -- Data Written: Amount of data written by this application. -- Duration: Amount of time this application has run. -- Efficiency Score: The fraction of the time that Spark executor cores are running Spark tasks. - -## App Info - -The App Info area gives you a quick point of reference for vital information about the application. - - - -You can edit the App Name by clicking the edit icon by the name. - -## Insights - -The Insights area gives information about the resource usage of the application -over time. The first tab shows executor CPU usage, broken down by categories -(CPU, I/O, shuffle, GC, Spark internals). This graph aligns with a timeline of -your Spark jobs and stages, so that it's easy to correlate CPU metrics with the -code of your Spark application. - - - -The second tab provides a report of the memory usage of your spark executors -over the application's job and stages timeline. On the left hand side, you can -see the peak memory usage over the total available physical memory for each -executor, broken down by category (JVM, Python, Other). This graph helps -you tune your container memory sizes - so that memory usage stays in the 70-90% -range. Click the executor list to view detailed memory usage for that executor -in the bottom graph. - -> **Note:** The memory usage depicted in this graph is different from the memory reported in the Spark UI. -> -> The graphs in this tab report the Resident Set Size (RSS) memory used by Spark and its child processes. -> RSS (Resident Set Size) refers to the amount of physical memory (RAM) that a process is currently using. This -> memory is what a process needs to quickly perform its operations without having to swap data in and out of -> disk storage, which is much slower. -> -> In the context of Apache Spark, the processTreeRSS metric is broken down into three categories: Java, Python, -> and Other (like R). This is because Spark applications can run tasks using different programming languages, -> and each of these languages manages memory in its own way. -> -> RSS is a measure of all the memory used by a process, while on-heap and off-heap memory are specific types of -> memory used within that process. RSS also includes other memory types like the program's code, stack -> memory, mapped memory, shared libraries, etc. - - - -## Logs - -You can view the Driver Logs or the Kubernetes Logs while the application is running. You can also download the logs once the application has finished running. - -> **Tip**: If you want to change the severity level of your Driver logs, you can do this easily from your Spark application code, for example by setting `sc.setLogLevel("DEBUG")`. - -## View Configuration - -To view the configuration, click the Configuration tab. - -## View Spark Issues - -Click the Spark Issues tab to see a list of all the issues with error messages. Click on an issue to expand the card and view more detailed information about the error or warning. - - - -## What’s Next? - -Learn more about [monitoring jobs](ocean-spark/product-tour/monitor-jobs). diff --git a/src/docs/ocean-spark/product-tour/view-cluster-details.md b/src/docs/ocean-spark/product-tour/view-cluster-details.md deleted file mode 100644 index 2a129a5ee7..0000000000 --- a/src/docs/ocean-spark/product-tour/view-cluster-details.md +++ /dev/null @@ -1,94 +0,0 @@ - - -# View Cluster Details - -To drill down into the details about your Ocean for Apache Spark cluster, start with the Overview tab, which gives you quick access to insights and summary data over the entire cluster. You can obtain an overview of your current cost, efficiency status, detailed cluster information, a Spark application overview, and tracking of cluster analytics. You can view more details about the cluster in additional tabs including a [cost analysis](ocean-spark/product-tour/analyze-costs) and the [Ocean Controller logs](ocean-spark/product-tour/view-cluster-details?id=view-ocean-controller-log). - -To get to the Cluster Overview tab, do the following: - -1. In the Spot console, go to Ocean for Spark in the menu tree and click Clusters. -2. In the list of Clusters, click a Cluster Name. - - - -The Cluster page opens with the Overview tab open and the cluster name at the top. Next to the cluster name, a status icon indicates the cluster status as one of the following: - -- Progressing - The cluster resources are being created. -- Deleting - The cluster resources are being deleted. -- Available - The cluster is available. -- Unreachable - The cluster stopped sending heartbeat. This can be caused by the cluster going down or by a networking issue between the cluster and Spot's backend. -- Degraded - One of the cluster components is unhealthy. Some features may not be available. -- Failing - A critical cluster component is unhealthy. -- Unknown - The cluster status API has an unexpected, internal error. - - - -The Cluster Overview includes the following main areas: - -- Metrics -- Cluster Info -- Applications -- Analytics - -## Metrics - -Cluster Metrics is a summary line providing insights into your cluster usage. The default display shows data from the last 24 hours. You can also see the numbers for the last seven days and the last 30 days. The following information is presented: - -- Cloud Compute Cost: The cloud provider’s compute costs incurred by all applications in this cluster during the selected period. -- Efficiency Score: The fraction of the time that Spark executor cores are running Spark tasks. -- Core Hours: Total core resources used by all your Spark applications in this cluster during the selected time period. -- Data Read: Amount of data read during the selected time period. -- Data Written: Amount of data written during the selected time period. - -## Cluster Info - -The Cluster Info area gives you a quick point of reference for vital information about the cluster. - - - -The following details and references are provided: - -- View Cluster in Ocean: Link to the cluster in Ocean. -- Region: Cloud provider region -- Date Created -- Last Heartbeat: The last heartbeat or the Ocean Spark controller. -- Kubernetes Version -- Ocean Spark Operator Version -- Spark Nodes: The current number of Kubernetes nodes dedicated to Spark in your cluster. -- Status - -## Applications - -The Applications line gives you a quick rundown of the status of your Spark applications on the cluster. Each tile shows the number of applications in each status during the selected time period. The following statuses are shown: - -- Pending -- Running -- Completed -- Killed -- Failed - -If you want to go directly to the Applications view, click View Apps or click directly on one of the Application status cards. - - - -## Cluster Analytics - -Ocean Spark provides detailed analytics about the cluster. You can view the following graphs and charts: - -- App Completion Trend: This graph is a histogram showing the number of completed, failed, killed, and timed out application runs over time. -- App Load History: This graph shows the number of running and pending apps over the selected time and the number of Spark cores used over the same time period. -- App Last Completion Runtime: A bar chart showing the amount of time to run the last completion of a Spark application. The chart shows the last runs of the last eight applications run. - -You can set the time span shown in each graph to one, seven, or 30 days. - -## View Ocean Controller Log - -To view the Ocean controller log for the cluster, click the Ocean Controller Logs tab at the top of the Cluster page. - -## Remove Cluster - -You can remove a cluster directly from the Cluster page. Just click Remove in the upper right. - -## What’s Next? - -Learn more about Ocean Spark’s [Cost Analysis](ocean-spark/product-tour/analyze-costs) of your Spark cluster. diff --git a/src/docs/ocean-spark/product-tour/view-job-details.md b/src/docs/ocean-spark/product-tour/view-job-details.md deleted file mode 100644 index d334864bcf..0000000000 --- a/src/docs/ocean-spark/product-tour/view-job-details.md +++ /dev/null @@ -1,51 +0,0 @@ - - -# View Job Details - -To drill down into the details about your Ocean for Apache Spark job, start with the Overview tab, which gives you quick access to insights and summary data about the job. You can obtain an overview of your cost trend and other related trends, job analytics, and status of applications relevant to this job. - -To get to the Job overview page, do the following: - -1. In the Spot console, go to Ocean for Spark in the menu tree and click Jobs. -2. In the [list of jobs](ocean-spark/product-tour/monitor-jobs), click a job name. - - - -The Job page opens showing the job name at the top. - -The App overview includes the following main areas: - -- Job Info -- Job trends -- Job Analytics -- Applications - -## Job Info - -The Job Info area gives you a quick point of reference for vital information about the job. - - - -## Job Trends - -The trends displayed include the following median metrics (computed over the last 10 applications): - -- Median Cloud Cost -- Median Duration -- Median Data Read -- Median Data Written -- Median Efficiency Score - -## Job Analytics - -In the Job Analytics graph, you can choose two metrics to plot on the same graph, as shown in the example below, so that it is easy to see the trends and correlations between two metrics. - - - -## Applications - -In the Applications list at the bottom, you can see the list of the executions of this job. - -## What’s Next? - -Learn more about [configuring Spark applications](ocean-spark/configure-spark-apps/). diff --git a/src/docs/ocean-spark/support/README.md b/src/docs/ocean-spark/support/README.md deleted file mode 100644 index 8a2ef301af..0000000000 --- a/src/docs/ocean-spark/support/README.md +++ /dev/null @@ -1,5 +0,0 @@ - - -# Get Support - -[Contact Support in the Spot console online chat or by email](https://spot.io/support/). diff --git a/src/docs/ocean-spark/tools-integrations/README.md b/src/docs/ocean-spark/tools-integrations/README.md deleted file mode 100644 index 0ccab20e9e..0000000000 --- a/src/docs/ocean-spark/tools-integrations/README.md +++ /dev/null @@ -1,11 +0,0 @@ - - -# Tools and Integrations - -Ocean for Apache Spark (Ocean Spark) is a managed cloud-native Spark service built on top of [Ocean’s](ocean/) serverless engine, dedicated to making Apache Spark developer-friendly and cost effective. Ocean Spark is deployed on a Kubernetes cluster inside your cloud account and gives you a serverless experience when working with Apache Spark. - -Ocean Spark comes built-in with integrations with popular data tools such as [Jupyter Notebooks](ocean-spark/tools-integrations/connect-jupyter-notebooks) and scheduling solutions like Airflow. The procedures in this section detail how to configure and use them with an Ocean Spark cluster. - -## What’s Next? - -To learn more about the tools and integrations available, choose a topic in the sidebar on the left. diff --git a/src/docs/ocean-spark/tools-integrations/aws-glue-catalog.md b/src/docs/ocean-spark/tools-integrations/aws-glue-catalog.md deleted file mode 100644 index 09acb08155..0000000000 --- a/src/docs/ocean-spark/tools-integrations/aws-glue-catalog.md +++ /dev/null @@ -1,141 +0,0 @@ - - -# AWS Glue Data Catalog - -You can use the [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html) as a metastore to persist metadata about your Spark tables, such as definition, location, and statistics. This is an alternative to using a [Hive Metastore](https://docs.spot.io/ocean-spark/tools-integrations/hive-metastore). The main benefit of Glue is that it natively allows querying from other AWS services such as Athena and Redshift. - -The [Spark docker images](https://docs.spot.io/ocean-spark/configure-spark-apps/docker-images) (Spark 3.0 and later since dm18) support connecting to Glue as the metastore since May 2022. Once you use a compatible image, you will need to configure your Spark applications to use Glue. - -The procedures below differ depending on whether Ocean Spark is deployed in the same AWS account as Glue, or whether they are in separate accounts. - -## Ocean Spark in Same AWS Account - -The first step is to create an IAM policy granting your Spark applications access to Glue. You can do this in the AWS console, under IAM > Policies > Create policy, by entering the following JSON block. -You should replace `` with your actual account ID. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "glue:*", - "Resource": [ - "arn:aws:glue:*::catalog", - "arn:aws:glue:*::database/*", - "arn:aws:glue:*::table/*/*" - ] - } - ] -} -``` - -You should then attach this policy to the IAM role used by your Spark applications. Identify the virtual node groups used by your Spark applications and the IAM role they are using. -Refer to our documentation on [how to configure data access](ocean-spark/configure-spark-apps/access-your-data?id=your-data-is-in-the-same-aws-account-as-the-ocean-spark-cluster) to get a better understanding of this. - -The final step is to pass the following configuration to your Spark applications. You can put the configuration in a configuration template or pass it directly in your API calls as configOverrides. In the example below, replace with your AWS account ID. - -```json -{ - "sparkConf": { - "spark.sql.catalogImplementation": "hive" - }, - "hadoopConf": { - "hive.metastore.glue.catalogid": "", - "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" - } -} -``` - -## Ocean Spark in Different AWS Account - -The procedures below are based on the official [AWS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/cross-account-access.html) - -In this example, we assume that Ocean Spark is deployed in `` and that Glue is deployed in ``. Glue is deployed in an AWS region ``, which could be, for example, us-west-2. - -You should first identify the IAM role(s) used by your Spark applications. Let’s assume that there is a single IAM role called ``, but there could be more than one role. We will grant this IAM role access to Glue. We need to make changes in two places. - -In the AWS console in account B, go to Glue > Settings, and add the following permissions: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "glue:*", - "Principal": { - "AWS": [ "" ] - }, - "Resource": [ - "arn:aws:glue:::database/*", - "arn:aws:glue:::catalog", - "arn:aws:glue:::table/*/*" - ] - } - ] -} -``` - - - -Then in the AWS console in account A, create the following IAM policy: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "glue:*", - "Resource": [ - "arn:aws:glue:::database/*", - "arn:aws:glue:::catalog", - "arn:aws:glue:::table/*/*" - ] - } - ] -} -``` - -Attach this policy to the IAM role(s) used by your Spark applications. - -The final step is to pass the following configuration to your Spark applications. You can use a configuration template or pass this directly in your API calls as configOverrides: - -```json -{ - "sparkConf": { - "spark.sql.catalogImplementation": "hive" - }, - "hadoopConf": { - "hive.metastore.glue.catalogid": "", - "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" - } -} -``` - -## Test Glue Functionality - -To test querying the Glue catalog, you can [start a Jupyter notebook](https://docs.spot.io/ocean-spark/tools-integrations/connect-jupyter-notebooks) using a configuration template with the above configurations. - -In this example, we will use the database `db_film` of the Glue Catalog. - - - -This database has an S3 bucket location (using S3A protocol) and tables in parquet format. - - - -You can show the available database by running `spark.sql("SHOW DATABASES")` - -You can describe a database by running `spark.sql("DESCRIBE DATABASE db_film")` - - - -You can list the tables within a database with `spark.sql("SHOW TABLES db_film")` -You can then query these tables, as well as create new ones, or create a new database. - - -## What's Next? - -Learn more about the Ocean Spark features in the [Product Tour](ocean-spark/product-tour/). diff --git a/src/docs/ocean-spark/tools-integrations/connect-jupyter-notebooks.md b/src/docs/ocean-spark/tools-integrations/connect-jupyter-notebooks.md deleted file mode 100644 index 4f282510ea..0000000000 --- a/src/docs/ocean-spark/tools-integrations/connect-jupyter-notebooks.md +++ /dev/null @@ -1,260 +0,0 @@ - - -# Connect Jupyter Notebooks - -Ocean Spark’s integration with Jupyter Notebooks enables you to run Jupyter kernels with Spark support on an Ocean Spark cluster. You can connect your notebooks from a Jupyter or Jupyterlab server running locally or from a hosted JupyterHub. - -Assumption: You already know how to [create and manage Config templates](ocean-spark/configure-spark-apps/?id=configuration-templates) for Ocean Spark. - - - -## Connect a Local Jupyter Server - -The Jupyter notebook server has an option to specify a gateway service in charge of running kernels on its behalf. Ocean Spark can fill this role and enables you to run Jupyter Spark kernels on the platform. - -Install the Jupyter notebook Python package locally. Be sure to use the latest version (or at least 6.0.0) with: - -``` -pip install notebook --upgrade -``` - -Launch a local Jupyter notebook server configured to interact with an Ocean Spark cluster: - -``` -jupyter notebook \ - --GatewayClient.url=https://api.spotinst.io/ocean/spark/cluster//notebook/ \ - --GatewayClient.auth_token= \ - --GatewayClient.request_timeout=600 - -# With Notebook v7+, add this option : - --GatewayWebSocketConnection.kernel_ws_protocol="" -``` - -- The GatewayClient.url points to an Ocean Spark cluster, with an Ocean Spark cluster ID of the format *osc-xxxxxxxx* that you can find on the [Clusters](https://console.spotinst.com/ocean/spark/clusters) list in the Spot console. -- The GatewayClient.auth_token is a [Spot API token](administration/api/create-api-token). -- The GatewayClient.request_timeout parameter specifies the maximum amount of time Jupyter will wait until the Spark driver starts. If you have capacity available in your cluster, the waiting time should be very short. If there isn't capacity, the Kubernetes cluster will get a new node from the cloud provider, which usually takes a couple of minutes. *You should set the request_timeout to 10 minutes to give you a security margin.* Omitting this parameter prevents you from starting a notebook. -- The GatewayWebSocketConnection.kernel_ws_protocol specifies we want to use the legacy websocket subprotocol for compatibility reason. - -> **Tip**: If you run into issues starting the Jupyter notebook server, ensure that your Ocean for Apache Spark cluster is marked as available in the Spot console. - -Ocean Spark is also compatible with JupyterLab. Install with: - -``` -pip install jupyterlab --upgrade -``` - -and run with: - -``` -jupyter lab \ - --GatewayClient.url=https://api.spotinst.io/ocean/spark/cluster//notebook/ \ - --GatewayClient.request_timeout=600 \ - --GatewayClient.auth_token= - -# With JupyterLab v4+, add this option : - --GatewayWebSocketConnection.kernel_ws_protocol="" -``` - -## Define Jupyter kernels with configuration templates - -Jupyter uses [kernels](https://jupyter.readthedocs.io/en/latest/glossary.html#term-kernel) to provide support for different languages and to configure notebook behavior. When a Jupyter server is connected to Ocean Spark, any Configuration template can be used as a kernel. - -You can use the Spot console or the API to create a Configuration template. Here’s a configuration template example to help you get started: - -```json -{ - "type": "Python", - "sparkVersion": "3.2.1", - "sparkConf": { - "spark.dynamicAllocation.enabled": "true", - "spark.dynamicAllocation.minExecutors": "0", - "spark.dynamicAllocation.maxExecutors": "10", - "spark.dynamicAllocation.initialExecutors": "1" - } - } -``` - -After creating it in the Spot console: - - - -The Configuration Template “notebook-template” appears in the list of kernels in the Jupyter dashboard: - - - -## Scala Kernels - -Ocean Spark also supports Jupyter Scala kernels. To open up a Scala kernel, all you need is to change the `type` field -in your configuration template. Here's an example configuration for a Scala kernel: - -```json -{ - "type": "Scala", - "sparkVersion": "3.2.1", - "sparkConf": { - "spark.dynamicAllocation.enabled": "true", - "spark.dynamicAllocation.minExecutors": "0", - "spark.dynamicAllocation.maxExecutors": "10", - "spark.dynamicAllocation.initialExecutors": "1" - } - } -``` - -**Warning**: Adding external JAR dependencies to Scala Notebooks -The `deps.jars` field in the application configuration does not work with Scala Notebooks and **should not be set**. The JARs specified in this field are not available on the driver Java classpath. - -Instead, you can add external JARs to the Spark context from the notebook with these magic commands (once the Spark session is up): - -- Add a JAR with URL: `%AddJar ` - ``` - %AddJar https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.20/postgresql-42.2.20.jar - ``` -- Add a dependency from maven repo: `%AddDeps ` - ``` - %AddDeps org.postgresql postgresql 42.2.20 - ``` - If the dependency has transitive dependencies, you can add the `--transitive` flag to add those dependencies. - -More documentation for these magic commands is available in the [Toree documentation](https://toree.incubator.apache.org/docs/current/user/faq/). - -## Use a notebook - -When you open a notebook, you need to wait for the kernel (i.e., the Spark driver) to be ready. As long as the kernel is marked as "busy" in the top right corner of the page, it means it has not started yet. This can take a few minutes. You can track the progress by looking at your [Spark application page](ocean-spark/product-tour/view-application-details) in the Spot console. - -Here are the objects you can use to interact with Spark: -- The Spark context in variable sc -- The Spark SQL context in variable sqlContext - -If those objects are not ready yet, you should see something like this upon invocation: - -``` -<__main__.WaitingForSparkSessionToBeInitialized at 0x7f8c15f4f240> -``` - -After a few seconds, they should be ready and you can use them to run Spark commands: - - - -You can install your own libraries by running: - -``` -!pip3 install -``` - -> **Tip**: Installing the libraries this way makes them available only for the driver. If the libraries need to be available for both driver and executors, install directly in the Docker image. - -If you are new to Jupyter notebooks, you can use this [tutorial](https://www.dataquest.io/blog/jupyter-notebook-tutorial/) as a starting point. - -## Close a Notebook - -To close a notebook application, you should not use the "Kill" action from the Spot console, because Jupyter interprets this as a kernel failure and it restarts your kernel, causing a new notebook application to appear. - -Close your notebooks from Jupyter (File > Close & Halt). This terminates the Spark app in the Ocean Spark cluster. - -### Important Note - -In some cases, a notebook may be "leaked", for example, if the Jupyter server (running on your laptop) quits abruptly or loses internet connection. This may leave a notebook application running on Ocean Spark without being linked to a Jupyter server. In this scenario, use the Kill action to terminate it. If no action is taken, the inactive kernel will be culled in 60 minutes. - -## Inject environment variables - -If you need to transfer information from your local environment to the notebook kernel in a dynamic way, you can use environment variables. - -The environment variables are only injected into the notebook kernel if they are prefixed with KERNEL_VAR_. In the kernel, the environment variables are stripped of their prefix, for instance, if you run the following command and open a notebook: - -``` -KERNEL_VAR_FOO=bar jupyter notebook \ - --GatewayClient.url=https://api.spotinst.io/ocean/spark/cluster//notebook/ \ - --GatewayClient.auth_token= \ - --GatewayClient.request_timeout=600 -``` - -The env variable FOO=bar is available in the notebook: - - - -## Notebooks are regular Spark applications - -Ocean for Spark essentially makes no distinction between notebooks and Spark applications launched by API requests. All options and features of Spark applications are available to notebooks. - -Notebooks appear in the [Applications](ocean-spark/product-tour/monitor-applications) list, so you can see their logs and configurations, access the Spark UI, and more: - - - -You can use the Type dropdown, as shown above, to filter on notebooks. -Additionally, any configuration option for Spark applications can be applied to notebooks via the Config template mechanism. - -## Launch Jupyter Notebook with VS Code IDE - -Ocean for Apache Spark (OfAS) provides a continuously optimized and autoscaled infrastructure and has featured support for integration with Jupyter notebooks. You can have this interactive notebook within your familiar IDE, such as VS Code, to benefit from other IDE built-in features including Git integration. - -This procedure describes how to use VS Code to run Jupyter notebooks, while the code runs on an Ocean for Apache Spark cluster. - -### Step 1: Install VS Code Editor - -Click the following link to find information on how to install VS Code locally: https://code.visualstudio.com/download. - -Verify that you have the latest version. - -If this is already installed, proceed to the next step. - -### Step 2: Install OfAS Jupyter Extension - -The OfAS extension to the VS Code IDE enables you to interactively launch your work as an OfAS Spark application, while writing the Notebook’s code, without needing to switch to a different tool. - -If you already have the VS Code editor installed, connect a notebook to your cluster as instructed below. - -1. Click [Ocean for Apache Spark Jupyter extension](https://marketplace.visualstudio.com/items?itemName=spot-by-netapp.spot-jupyter-extension). - -2. Click Install. When the VS Code editor is installed, the VS Code window opens. - - - -3. Click Install. - -### Step 3: Install Microsoft Jupyter Extension - -Repeat the steps from Step 2 in the following link to install the Microsoft Jupyter extension: https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter . - -#### Connect Notebook to your OfAS Cluster - -1. Create or open a Jupyter notebook file. -2. In the VS Code window, click "Select Kernel" in the top-right corner. - - - -3. Select Spot Ocean for Apache Spark item in the list. -4. Enter your account ID, token and select the cluster you want to use from your account that appears in the dropdown menu. -5. Select the config-template you want to use. (Config-templates can take few seconds to appear in the list) -6. Run the code in your notebook. The first execution can take approximately 1-5 minutes as a Spark application needs to be started in your cluster. - -> **Tip**: Closing your notebook may not result in the termination of the notebook application. You may have to do so from the Spot console. You can also shutdown kernels without leaving VSCode in the [Jupyter PowerToys](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.vscode-jupyter-powertoys) extension. - -#### Troubleshooting - -* If Spot Ocean for Apache Spark in the Jupyter Connection options doesn’t appear, ensure that the VSCode and Jupyter extensions are updated to their latest version. - -* If your cluster doesn't appear in the list, check if it appears as `AVAILABLE` in the Spot console - -* If config-templates in the kernel picker doesn’t appear, follow these steps : - - 1. Close your notebook files. - 2. Open the Command Palette (Cmd+Shift+P) and select `Python: Clear Cache and Reload Window`. - 3. Open the file again and connect to cluster again. - 4. Your config-templates should appear in the kernel picker. - -## Connecting to JupyterHub - -If you prefer to run your Jupyter Notebooks in a hosted environment that can be shared across teams and developers, JupyterHub is an excellent solution. JupyterHub will give you the same developer experience that you are familiar with using local notebooks, but with added features and functionality for managing authentication, user access, and multiple configuration environments and templates. - -To help you get started, we've built a simple Docker image that can get you running notebooks in minutes. Follow these steps to deploy JupyterHub on your local machine: -1. Pull this repo - https://github.com/spotinst/ocean-spark-examples/tree/master/jupyterhub-in-docker -2. Add your Ocean Spark cluster-id to the `jupyterhub_config.py` -3. Run the command `make run` -4. Navigate to `localhost:8000` and input your Ocean Spark API key when prompted. -5. Select a kernel (i.e., an Ocean Spark configuration template), and begin executing Spark code. - -This template is a great starting place, but feel free to adapt and alter the logic to meet your team's configuration requirements. - -## What's Next? - -Learn how to [run apps from Airflow](ocean-spark/tools-integrations/run-apps-from-airflow). diff --git a/src/docs/ocean-spark/tools-integrations/hive-metastore.md b/src/docs/ocean-spark/tools-integrations/hive-metastore.md deleted file mode 100644 index 7c7db46198..0000000000 --- a/src/docs/ocean-spark/tools-integrations/hive-metastore.md +++ /dev/null @@ -1,217 +0,0 @@ - - -# Hive Metastore - -Configuring a Hive Metastore makes your table metadata persistent across your Apache Spark applications and enables the sharing of Spark tables across multiple Spark infrastructures. - - -We recommend the Hive local mode in which the Spark driver of your application communicates directly to a remote database, as it is easy to set up and does not require any maintenance. - -The first section below explains how to create a Hive Metastore database. If you already have one, you can skip to the next section, Connect to a Hive Metastore. - -## Create a Hive Metastore database - -To create a Hive Metastore database, complete the following procedures: -1. Create a database service -2. Configure Connectivity -3. Create the database -4. Create the Hive schema - -These steps are described below. - -### Create a database service - -In this step, you will create a database service in your cloud provider. - -On AWS, RDS allows you to create managed services for Aurora, MySQL, MariaDB, PostgreSQL, Oracle, and Microsoft SQL Server. You should consider availability, backups, security, and maintenance in this choice. - -This document provides an example assuming an RDS running PostgreSQL. Pleas see the relevant [AWS instructions](https://aws.amazon.com/getting-started/hands-on/create-connect-postgresql-db/). - -### Configure Connectivity - -In this step, you ensure the database can be accessed by your Ocean Spark applications. - -One way to ensure connectivity is to create an inbound rule for TCP on port 5432 and to configure the database to be publicly accessible. - -#### Inbound rule - -Go to the VPC security group of the database service and create an inbound rule similar to the following: - - - -#### Public Access - -Please see the relevant [AWS instructions](https://aws.amazon.com/premiumsupport/knowledge-center/rds-connectivity-instance-subnet-vpc/). The connectivity of your RDS service should look like the screen capture below: - - - -### Create the database - -You can create the Hive database directly in SQL or through the UI by using a database client tool like [pgAdmin](https://www.pgadmin.org/) or [Postico](https://eggerapps.at/postico/). From the database client, create a connection to your database service and create a database that will store all the Hive metadata. - -### Create the Hive schema - -The last step to prepare the Hive metastore is to create the Hive schema. -Retrieve the following SQL scripts: -- [Hive Schema](https://github.com/apache/hive/blob/rel/release-2.3.4/metastore/scripts/upgrade/postgres/hive-schema-2.3.0.postgres.sql) -- [Hive Transactional Schema](https://github.com/apache/hive/blob/rel/release-2.3.4/metastore/scripts/upgrade/postgres/hive-txn-schema-2.3.0.postgres.sql) - -Look for `\i hive-txn-schema-2.3.0.postgres.sql;` in the first script and replace it with the content of the second script. Then run the whole script in your database client tool. - -The above scripts are for Hive 2.3 and PostgreSQL. You can find scripts for other Hive versions and databases in the [Apache Hive repository](https://github.com/apache/hive/tree/rel/release-2.3.4/metastore/scripts/upgrade). - -Hive version 2.3 is the Spark default. Unless you have specific constraints, you should choose this version. - -## Connect to a Hive Metastore database - -To connect to a Hive Metastore database, complete the following procedures: -1. Add the JDBC Driver JAR file -2. Configure your Spark applications to connect to the metastore - -### Add the JDBC Driver JAR file - -In this step, you will add the JAR file containing the JDBC driver of your database to your Spark applications. - -For PostgreSQL the JAR file can be found in the [MVN repository](https://mvnrepository.com/artifact/org.postgresql/postgresql/42.2.20). - -#### Option 1: Download and Copy the JDBC driver JAR file to your Spark image - -Add the JAR file directly to your Spark docker images: - -``` -FROM gcr.io/ocean-spark/spark:platform-3.0.2-latest -COPY ./jars/postgresql-42.2.20.jar /opt/spark/jars/postgresql-42.2.20.jar -``` - -#### Option 2: Define the JAR file as a dependency in your Spark applications - -The JAR will then be downloaded at runtime when each Spark application starts: - -```json -"deps": { - "jars": [ - "https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.20/postgresql-42.2.20.jar" - ] - } - ``` - -For a complete reference on template attributes see the Spot [API reference](https://docs.spot.io/api/#operation/OceanSparkClusterApplicationSubmit). - -### Configure your Spark applications to connect to the metastore - -You can use either of the options below. - -#### Option 1: Add the configurations to your sparkConf - -Define or modify a configuration template to include the following flags: - -```json -"sparkConf": { - "spark.sql.catalogImplementation": "hive", - "spark.sql.hive.metastore.sharedPrefixes": "org.postgresql", - "spark.hadoop.javax.jdo.option.ConnectionURL": "jdbc:postgresql://hive1...xxx...amazonaws.com:5432/hive1", - "spark.hadoop.javax.jdo.option.ConnectionPassword": "xxx", - "spark.hadoop.javax.jdo.option.ConnectionUserName": "xxx", - "spark.hadoop.javax.jdo.option.ConnectionDriverName": "org.postgresql.Driver", - "spark.hadoop.hive.metastore.warehouse.dir": "... Accessible Cloud Storage ..." - } - ``` - -Additionally if you use an older version of Hive, you can add: - -```json - "spark.sql.hive.metastore.version": "... version number ...", - "spark.sql.legacy.timeParserPolicy": "LEGACY" -``` - -#### Option 2: Configure the connection in the core-site.xml file - -You can also specify these configurations in the core-site.xml file, which has the added benefit of hiding the database credentials. The following flags should be set in the core-site.xml file. Other flags can still be passed using the sparkConf. - -```XML - - - javax.jdo.option.ConnectionURL - jdbc:postgresql://hive1......amazonaws.com:5432/hive1 - - - javax.jdo.option.ConnectionPassword - xxx - - - javax.jdo.option.ConnectionUserName - xxx - - - javax.jdo.option.ConnectionDriverName - org.postgresql.Driver - - -``` - -This file should be added in a folder defined by an environment variable ($HADOOP_CONF_DIR) read by Spark. Read our documentation on how to [configure environment variables](ocean-spark/configure-spark-apps/secrets-environment-variables). - -The hive-site.xml file can be added directly into your docker image by following this example: - -``` -FROM gcr.io/ocean-spark/spark:platform-3.1.1-latest` - -COPY requirements.txt . -RUN pip3 install -r requirements.txt - -EXPORT HADOOP_CONF_DIR=/opt/spark/hive -COPY hive-site.xml /opt/spark/hive - -COPY src/ src/ -COPY main.py . -``` - -Alternatively, you can define the hive-site.xml file as a Kubernetes secret and mount it into your Spark pods. - -To do this, you will need [kubectl](https://kubernetes.io/docs/tasks/tools/) access to your cluster. Write the desired hive-site.xml file locally, and then run the following command: - -``` kubectl create secret generic --from-file=/path/to/hive-site.xml -n spark-apps ``` - -You should then make a few edits to your Spark application configurations: -1. Add a `volumes` key with a reference to the Kubernetes secret you created and a name -2. Add a `volumeMounts` section to your driver and executor configurations. Its `name` field references the volume name defined above, while the mountPath tells Kubernetes where to mount the file. -3. Do not forget to set the HADOOP_CONF_DIR environment variable so Spark knows to look for the hive-site.xml file at the right location. - -Here’s what your Spark application configuration should look like: - -```json - "driver": { - "envVars": { - "HADOOP_CONF_DIR": "/opt/spark/hive" - }, - "volumeMounts": [ - { - "name": "hive-credentials", - "mountPath": "/opt/spark/hive" - } - ], - }, - "executor": { - "envVars": { - "HADOOP_CONF_DIR": "/opt/spark/hive" - }, - "volumeMounts": [ - { - "name": "hive-auth", - "mountPath": "/opt/spark/hive" - } - ] - }, - "volumes": [ - { - "name": "hive-credentials", - "secret": { - "secretName": "hive-credentials" - } - } - ] -``` - -## What's Next? - -Learn more about the Ocean Spark features in the [Product Tour](ocean-spark/product-tour/). diff --git a/src/docs/ocean-spark/tools-integrations/jdbc.md b/src/docs/ocean-spark/tools-integrations/jdbc.md deleted file mode 100644 index 0ce5b73b0a..0000000000 --- a/src/docs/ocean-spark/tools-integrations/jdbc.md +++ /dev/null @@ -1,76 +0,0 @@ - - -# JDBC - -JDBC, or Java Database Connectivity, is an API used in Java programming to interact with databases. It provides a standard abstraction for Java applications to communicate with various databases. JDBC allows applications to send requests made by users to the specified database. It is used to write programs required to access databases. Apache Spark provides a JDBC interface through the HiveThriftServer. - -You can execute SQL queries directly by using the JDBC driver in code, a database tool or from another Spark session. Here is a Java example: - -```Java -var prop = new Properties(); -var query = "select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count"; -var jdbcUrl = "jdbc:ofas://api.spotinst.io/"+clusterId+"/"+appId+"?profile=default"; - -try (var conn = DriverManager.getConnection(jdbcUrl, prop); var stmt = conn.createStatement(); var rs = stmt.executeQuery(query)) { - var metadata = rs.getMetaData(); - int columnCount = metadata.getColumnCount(); - while(rs.next()) { - var row = new StringBuilder(); - int i; - for (i = 1; i < columnCount; i++) { - row.append(rs.getString(i)).append(", "); - } - row.append(rs.getString(i)); - System.out.println(row); - } -} -``` - -## Server Side - -To enable JDBC connections to the Spark Application, start the HiveThriftServer. - -### Launch a JDBC Server - -```json -"mainClass": "com.netapp.spark.HiveThriftServer", -"deps": { - "packages": ["com.netapp.spark:hive:1.2.1"], - "repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"] -} -``` - -## Client Side - -### Ocean Spark JDBC Driver - -Use the Ocean Spark JDBC driver with a database tool or in your code project. The driver is available at the following maven coordinates: - -``` -com.netapp.spark:ofas-jdbc:1.2.2 -``` - -Use the following public maven repository - -``` -https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters -``` - -The jdbc url looks like - -``` -jdbc:ofas://api.spotinst.io/{clusterId}/{appId}?profile=default -``` - -additional url parameters are token, account and mode. -Mode is the thrift transport mode and can be 'http' or 'thrift'. - -### Spotctl - -Use the spotctl command line tool with the port option, --port 10000 or --port hive - -``` -spotctl ocean spark connect --cluster-id osc-cluster --app-id appid --endpoint hive -``` - -You can now connect to the interactive Spark application through a Hive Thrift library or the Hive JDBC driver. diff --git a/src/docs/ocean-spark/tools-integrations/notebook-filesystem-plugins.md b/src/docs/ocean-spark/tools-integrations/notebook-filesystem-plugins.md deleted file mode 100644 index 027fb82270..0000000000 --- a/src/docs/ocean-spark/tools-integrations/notebook-filesystem-plugins.md +++ /dev/null @@ -1,29 +0,0 @@ - - -# Jupyter Notebook Filesystem Plugins - -With the current setup of launching a local notebook connecting to Ofas through the Jupyter Enterprise Gateway, you can launch notebooks stored on their machines. These notebooks can be shared via cloud storage services like Google Drive, Microsoft OneDrive and Dropbox installed on the desktop machine. You can also leverage Git for sharing notebooks, providing the added advantage of not storing any notebook data or results on OfAS' side. This relieves you from the responsibility of ensuring the data's protection. - -This approach is also applicable to Spark Connect, allowing the notebook using Ofas to be executed from any location, including platforms like Google Colab. - -Hosted notebooks offer a different scenario where everything is run within the cloud. You no longer have direct access to their local notebook files and credentials required for remote storage services must be sent over the wire. - -Jupyter Notebook Filesystem Plugins provide few Jupyter plugins that enable access to remote file systems. - -## jupyter-filesystem-access (https://github.com/jupyterlab-contrib/jupyterlab-filesystem-access) - -Born out of the JupyterLite project, the still expoerimenal jupyter-filesystem-access plugin taps into the recently added feature added to the Chrome and Edge browser enabling access to the local filesystem. With the plugin enabled, the user can select a directory on their machine. The content will be accessible in Jupyter lab as a file tree on the left. This enables you to launch your existing notebooks stored on their macines. - -## jupyterlab-git (https://github.com/jupyterlab/jupyterlab-git) - -The Git extension for jupyter is an offical extension provided be the jupyter team. When enabled, a Git menu opens in the top menubar. The plugin allows you to check your notebook projects from Git repositories with the relevant credentials. The python implementation of Git is user, so there is no need for Git to be installed on the image. - -## jupyterlab-github (https://github.com/jupyterlab/jupyterlab-github) - -Similar to the jupyter-git extension, but more specific to Github, the Github plugin offers mount-like experience to accessing notebooks in Github, as opposed to the checkout, push and pull process with the Git plugin. - -## jupyter-fs (https://github.com/jpmorganchase/jupyter-fs) - -The jupyter-fs backend is build on top of PyFilesystem, while the frontend is built on top of the JupyterLab FileTree. The plugin lets you set up and use as many filebrowsers as they like, connected to whatever local and/or remote filesystem-like resources they want, such as s3://, smb:// etc. - -The Git related plugins are official and are part of our notebooks images. The others can be manually installed through the Extension Manager. diff --git a/src/docs/ocean-spark/tools-integrations/run-apps-from-airflow.md b/src/docs/ocean-spark/tools-integrations/run-apps-from-airflow.md deleted file mode 100644 index 3f30d210a8..0000000000 --- a/src/docs/ocean-spark/tools-integrations/run-apps-from-airflow.md +++ /dev/null @@ -1,229 +0,0 @@ - - -# Run Apps from Airflow - -This page describes how to configure Airflow to trigger Spark applications on Ocean for Apache Spark (also referred to as Ocean Spark). Our Airflow plugin is compatible with Airflow 1 and Airflow 2. - -Assumption: You already have access to a running Airflow environment. You could deploy and manage Airflow yourself, or use a managed service like [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) (see the tutorial on the [Spot blog page](https://spot.io/blog/orchestrate-spark-pipelines-with-airflow-on-ocean-for-apache-spark).) or [Astronomer](https://www.astronomer.io/). If you don’t have access to Airflow, we show you how to set up an Airflow sandbox with Docker. If you don’t need this, you can skip this optional section below. -Optional: Spin up an Airflow sandbox with Docker -Run a local Airflow server with Airflow 2 or 1. - -#### Airflow 2 - -```shell -mkdir -p dags/ -docker run -d -p 8080:8080 -e FERNET_KEY=`openssl rand -base64 32` \ - --mount type=bind,source="$(pwd)"/dags/,target=/opt/airflow/dags/ \ - --name test-airflow apache/airflow:2.2.3-python3.7 webserver -``` - -#### Airflow 1 - -```shell -mkdir -p dags/ -docker run -d -p 8080:8080 -e FERNET_KEY=`openssl rand -base64 32` \ - --mount type=bind,source="$(pwd)"/dags/,target=/usr/local/airflow/dags/ \ - --name test-airflow puckel/docker-airflow:1.10.9 webserver -``` - -The Airflow UI is now available at http://localhost:8080/. -Connect to the container with the following command: - -```docker exec -ti test-airflow /bin/bash``` - -When you are finished with the tutorial, kill the docker image with this command: - -```docker kill test-airflow``` - -## Install the Ocean Spark Airflow Provider -### Docker Sandbox - -If you use the Docker sandbox, run all the commands in this section from within the Docker container. Connect to it with the following command: - -```docker exec -ti test-airflow /bin/bash``` - -Install the Ocean Spark [Airflow Provider](https://pypi.org/project/ocean-spark-airflow-provider/) using pip: - -```pip install ocean-spark-airflow-provider``` - -Configure a connection to Ocean Spark in Airflow: - -#### Airflow 2 - -Run the following commands to use the built-in SQLite database. (Please do not use this for production environments). - -```shell -cd $AIRFLOW_HOME - -airflow db init - -airflow scheduler -D - -airflow users create --role Admin \ - --username admin \ - --email admin \ - --firstname admin \ - --lastname admin \ ---password admin -``` - -Run the following command to create the connection: - -```shell -airflow connections add --conn-type ocean-spark-airflow-provider \ - --conn-host \ - –conn-login \ - --conn-password ocean_spark_default -``` - -Use the username `admin` and password `admin` when you bring up the Airflow UI at http://localhost:8080/. - -#### Airflow 1 - -```shell -cd $AIRFLOW_HOME -airflow connections -a --conn_id ocean_spark_default \ - --conn_type ocean-spark-airflow-provider \ - --conn_host \ - –conn_login \ - --conn_password -``` - -Your Ocean Spark cluster ID is visible in the Clusters list in the Spot console (format osc-xxxxxxxx). See [How to create an API token](administration/api/create-api-token). - -### Airflow UI - -The Airflow UI is available at http://localhost:8080/. - -#### Airflow 2 - -You can also configure the connection using the Airflow UI. -1. Go to Admin -> Connections -> Add a new record (+ sign). -2. Select “Ocean For Apache Spark” in the Connection Type dropdown. - - - -3. Enter the following details in the connection window, and then click Save. - - Connection ID: Use ocean_spark_default by default. You may use a different name. - - Connection Type: Select “Ocean For Apache Spark” from the dropdown - - Description: Enter any optional text to describe the connection. - - Cluster ID: The ID of your Ocean Spark cluster (format osc-xxxxxxxx) - - Account ID: The Spot Account ID the cluster belongs to, which corresponds to a cloud provider account. - - API token: Your Spot API token (see How to create an API token) - -4. If you do not see Ocean for Apache Spark in the connection types, restart the Airflow web server using the following command: - -```docker restart test-airflow``` - -#### Airflow 1 - -The Ocean for Apache Spark connection type is not available for Airflow 1. Instead, create an HTTP connection and add your cluster ID as Host, account ID as Login, and your API token as password. - - - -## Create an example DAG - -The example file below defines an Airflow DAG with a single task that runs the canonical Spark Pi on Ocean for Apache Spark. - -``` -from airflow import DAG, utils - -from airflow import __version__ as airflow_version -if airflow_version.startswith("1."): - # Airflow 1, import as plugin - from airflow.operators.ocean_spark import OceanSparkOperator -else: - # Airflow 2 - from ocean_spark.operators import OceanSparkOperator - -args = { - "owner": "airflow", - "email": [], # ["airflow@example.com"], - "depends_on_past": False, - "start_date": utils.dates.days_ago(0, second=1), -} - -dag = DAG(dag_id="single-task", default_args=args, schedule_interval=None) - -spark_pi_task = OceanSparkOperator( - task_id="spark-pi", - dag=dag, - config_overrides={ - "type": "Scala", - "sparkVersion": "3.2.0", - "image": "gcr.io/ocean-spark/spark:platform-3.2-latest", - "imagePullPolicy": "IfNotPresent", - "mainClass": "org.apache.spark.examples.SparkPi", - "mainApplicationFile": "local:///opt/spark/examples/jars/examples.jar", - "arguments": ["1000"], - "driver": { - "cores": 1, - }, - "executor": { - "cores": 1, - }, - }, -) -``` - -Since the OceanSparkOperator is a thin wrapper around the Ocean Spark API, its arguments should be familiar if you have already submitted an app through the API. -Please see the [API reference](https://docs.spot.io/api/#tag/Ocean-Spark) for Ocean Spark. - -If you omit the job_id argument, the Airflow argument task_id will be used as the Ocean Spark job ID. More complex examples are available in the Ocean for Apache Spark Airflow [plugin repository](https://github.com/spotinst/ocean-spark-airflow-provider/tree/main/deploy/airflow2/dags). - -Copy the file to your Airflow DAGs storage location, usually $AIRFLOW_HOME/dags. - -### Docker Sandbox - -If you use the Docker sandbox, copy the file into the local folder dags/. This folder is mounted into the container's $AIRFLOW_HOME/dags/ path, and the file will then be available to Airflow. - -Depending on your configuration, you may need to restart the Airflow webserver for the DAG to appear in the DAG list. - -```docker restart test-airflow``` - -## Run the DAG - -Connect to your Airflow webserver (http://localhost:8080/ on the Docker sandbox). - -#### Airflow 2 - -1. To run the DAG, click the Play button on the right and Trigger DAG. - - - -2. Click the DAG name and get to the tree view. Click the green square to open a popup with more options. - - - -3. In the pop-up, click View log. The log shows that Airflow interacts with the Ocean Spark API to track the status of the Spark application. - -A URL to Ocean Spark Application Page can also be found in the log. It brings you directly to the page of the application. - - - -#### Airflow 1 - -1. If you do not see the DAGs, restart the Airflow webserver. - -```docker restart test-airflow``` - -2. To run the DAG, toggle the switch on the left to On and click the Play button on the right. - - - -3. Click on the DAG name and get to the tree view. Click on the green square to open a pop-up with more options. -4. If the square is not green yet, click the Refresh action button. - - - -5. In the pop-up, click View log. The log shows that Airflow interacts with the Ocean Spark API to track the status of the Spark application. - -A URL to Ocean Spark Application page can also be found in the log. It brings you directly to the [Application Details](ocean-spark/product-tour/view-application-details) page. - - - -You are now ready to schedule Spark applications on Ocean Spark using Airflow. - -## What’s Next? - -Learn more about creating and configuring a [Hive Metastore](ocean-spark/tools-integrations/hive-metastore). diff --git a/src/docs/ocean-spark/tools-integrations/shuffle-plugin.md b/src/docs/ocean-spark/tools-integrations/shuffle-plugin.md deleted file mode 100644 index 26cb27f8b3..0000000000 --- a/src/docs/ocean-spark/tools-integrations/shuffle-plugin.md +++ /dev/null @@ -1,146 +0,0 @@ - - -# External Shuffle Storage - -When External Shuffle Storage is turned on, Spark writes shuffle data to a shared remote filesystem, such as S3. -This enables recovering shuffle data written by failed Spark kubernetes pods, avoiding task retries. -External Shuffle Storage is also useful with dynamic allocation enabled, as it allows scaling down Spark executors that are kept running to serve shuffle data for other tasks. -Storing shuffle data on a remote drive accessible from all executors can save time and resources. - -## Configuration - -To turn on External Shuffle Storage, add the following configuration in your Spark application: - -```json -{ - "shuffle": { - "enabled": "true", - "rootDir": "s3a:///path/to/shuffle" - } -} -``` - -The `shuffle.rootdir` configuration is the location where the shuffle data will be written. -The shuffle reuse feature writes the shuffle data to the Hadoop filesystem and, as such, supports any filesystem that Hadoop supports. -The root dir option can be a local path, HDFS path, or any other Hadoop-supported filesystem. -A shared remote drive such as S3 CSI must be mounted on all the executors in the cluster when using a local path. - -For instance - -```json -{ - "shuffle": { - "rootDir": "/opt/spark/work-dir/shuffle" - }, - "volumes": [ - { - "name": "spark-data", - "persistentVolumeClaim": { - "claimName": "s3-claim" - } - } - ], - "driver": { - "volumeMounts": [ - { - "mountPath": "/opt/spark/work-dir/shuffle", - "name": "spark-data" - } - ] - }, - "executor": { - "volumeMounts": [ - { - "mountPath": "/opt/spark/work-dir/shuffle", - "name": "spark-data" - } - ] - } -} -``` - -## Optimizations - -The External Shuffle Storage plugin shards the shuffle files on different S3 folder prefixes for better performance. -The configuration key `spark.shuffle.s3.folderPrefixes` can be used to control the number of partitions, with the default of 10. - -```json -{ - "shuffle": { - "rootDir": "/shuffle" - }, - "sparkConf": { - "spark.shuffle.s3.folderPrefixes": "2" - }, - "volumes": [ - { - "name": "spark-vol1", - "persistentVolumeClaim": { - "claimName": "s3-claim-1" - } - }, - { - "name": "spark-vol2", - "persistentVolumeClaim": { - "claimName": "s3-claim-2" - } - } - ], - "driver": { - "volumeMounts": [ - { - "mountPath": "/shuffle/0", - "name": "spark-vol1" - }, - { - "mountPath": "/shuffle/1", - "name": "spark-vol2" - } - ] - }, - "executor": { - "volumeMounts": [ - { - "mountPath": "/shuffle/0", - "name": "spark-vol1" - }, - { - "mountPath": "/shuffle/1", - "name": "spark-vol2" - } - ] - } -} -``` - -The above configuration will shard the shuffle data across two different PVC volumes defined in kubernetes, such as - -```json -{ - "apiVersion": "v1", - "kind": "PersistentVolumeClaim", - "metadata": { - "name": "s3-claim-1" - }, - "spec": { - "accessModes": ["ReadWriteMany"], - "resources": { - "requests": { - "storage": "200Gi" - } - }, - "storageClassName": "sc-ontap-nas" - } -} -``` - -When using S3 as the shuffle storage medium, adjusting the `spark.hadoop.fs.s3a.block.size` and `spark.hadoop.fs.s3a.multipart.size` configurations can also improve performance. - -## Limitations - -- Shuffle Data reuse is only available for Spark 3.2 and later. -- Preferably set the `spark.dynamicAllocation.shuffleTracking.enabled` to false when using External Shuffle Storage. - - - - diff --git a/src/docs/ocean-spark/tools-integrations/spark-connect.md b/src/docs/ocean-spark/tools-integrations/spark-connect.md deleted file mode 100644 index b99005c88d..0000000000 --- a/src/docs/ocean-spark/tools-integrations/spark-connect.md +++ /dev/null @@ -1,113 +0,0 @@ - - -# Spark Connect - -In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. The separation between client and server allows Spark and its open ecosystem to be leveraged from everywhere. It can be embedded in modern data applications, in IDEs, Notebooks and programming languages. Ocean Spark support Spark Connect, and that can be especially useful for the direct execution of Spark SQL. - -Once you are connected to the remote application (as described in the Client side section below) you can execute SQL queries directly from code, notebook or pyspark shell. - -```Python -df = spark.sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count") -df.write.save("s3://results_bucket/fruits.parquet") -``` - -## Server Side - -To start a Spark application with SparkConnect server, either run the mainClass SparkConnectServer or enable the SparkConnect plugin. Using the Spark Connect plugin, the application can run other tasks or services while enabling Spark Connect. - -### Spark Connect Launch using the SparkConnectServer - -```json -"mainClass": "com.netapp.spark.SparkConnectServer", -"deps": { - "packages": ["org.apache.spark:spark-connect_2.12:3.4.3", "com.netapp.spark:spark-connect:1.2.9"], - "repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"] -} -``` - -### Spark Connect Launch using the Spark Connect plugin - -```json -"sparkConf": { - "spark.plugins": "org.apache.spark.sql.connect.SparkConnectPlugin" -}, -"deps": { - "packages": ["org.apache.spark:spark-connect_2.12:3.4.3"] -} -``` - -## Client Side - -### Python Library - -On the client side use the ocean-spark-connect (https://pypi.org/project/ocean-spark-connect) python library to interact with the Spark Connect session. - -```python -from ocean_spark_connect.ocean_spark_session import OceanSparkSession - -spark = OceanSparkSession.Builder().cluster_id("osc-cluster").appid("appid").profile("default").getOrCreate() -spark.sql("select random()").show() -spark.stop() -``` - -The profile is read from ~/.spotinst/credentials with the following format: - -``` -[default] -token = MYTOKEN -account = act-xxx -``` - -Instead of using a profile you can specify the token and account directly as builder options. - -```python -spark = OceanSparkSession.Builder().cluster_id("osc-cluster").appid("appid").account("acc-xxx").token("MYTOKEN") -``` - -### Spotctl - -Use the spotctl command line tool to open a websocket proxy to the interactive Spark application. - -```sh -brew install spotinst/tap/spotctl -spotctl configure -``` - -```sh -spotctl ocean spark connect --cluster-id osc-cluster --app-id appid -``` - -spotctl will start a service on port 15002 (the default Spark Connect port) - -```sh -pyspark --remote sc://localhost -``` - -## Example - -Start the application using Postman, from the console or the command line. - -```sh -curl -k -X POST 'https://api.spotinst.io/ocean/spark/cluster/{clusterId}/app?accountId={accountId}' -H 'Content-Type: application/json' -H 'Authorization: Bearer {token}' -d ' -``` - -```json -{ - "jobId": "spark-connect", - "configOverrides": { - "type": "Scala", - "sparkVersion": "3.4.3", - "mainApplicationFile": "local:///opt/spark/jars/spark-core_2.12-3.4.3.jar", - "mainClass": "com.netapp.spark.SparkConnectServer", - "deps": { - "packages": [ - "org.apache.spark:spark-connect_2.12:3.4.3", - "com.netapp.spark:spark-connect:1.2.9" - ], - "repositories": [ - "https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters" - ] - } - } -} -``` diff --git a/src/docs/paths.json b/src/docs/paths.json index e0e23cb02e..ff8fe9fb18 100644 --- a/src/docs/paths.json +++ b/src/docs/paths.json @@ -46,6 +46,7 @@ "/administration/policies/README.md", "/administration/README.md", "/administration/release-notes-core/README.md", + "/administration/savings.md", "/administration/sso-access-control/account-level-sso.md", "/administration/sso-access-control/organization-level-sso-OLD.md", "/administration/sso-access-control/organization-level-sso.md", @@ -59,42 +60,9 @@ "/administration/users-a/create-new-user.md", "/administration/users-a/edit-user-details.md", "/administration/users-a/README.md", + "/administration/vcpu.md", "/administration/_sidebar.md", "/ai-bot/disclaimer-page.md", - "/billing-engine/get-started/connect-aws.md", - "/billing-engine/get-started/connect-azure.md", - "/billing-engine/get-started/connect-google.md", - "/billing-engine/get-started/README.md", - "/billing-engine/README.md", - "/billing-engine/release-notes/README.md", - "/billing-engine/tutorials/allocation-assignments.md", - "/billing-engine/tutorials/analysis.md", - "/billing-engine/tutorials/billing-engine-policy.md", - "/billing-engine/tutorials/dashboard/README.md", - "/billing-engine/tutorials/families.md", - "/billing-engine/tutorials/invoicegenerator.md", - "/billing-engine/tutorials/plans.md", - "/billing-engine/tutorials/README.md", - "/billing-engine/tutorials/strategies.md", - "/billing-engine/_media/README.md", - "/billing-engine/_sidebar.md", - "/cloud-analyzer/getting-started/connect-account-customer-working-with-msp.md", - "/cloud-analyzer/getting-started/connect-account-multiple-organizations-to-single-master-payer.md", - "/cloud-analyzer/getting-started/connect-master-payer-account-first-registration.md", - "/cloud-analyzer/getting-started/connect-your-aws-master-payer-account-existing-customer.md", - "/cloud-analyzer/getting-started/README.md", - "/cloud-analyzer/README.md", - "/cloud-analyzer/tutorials/analyze-your-costs.md", - "/cloud-analyzer/tutorials/cloud-analyzer-policy/create-cloud-analyzer-policy-with-cloudformation.md", - "/cloud-analyzer/tutorials/cloud-analyzer-policy/README.md", - "/cloud-analyzer/tutorials/cloud-analyzer-policy/z-README.md", - "/cloud-analyzer/tutorials/README.md", - "/cloud-analyzer/tutorials/save-analysis-reports.md", - "/cloud-analyzer/tutorials/use-optimization-dashboard/containers.md", - "/cloud-analyzer/tutorials/use-optimization-dashboard/elastic-applications.md", - "/cloud-analyzer/tutorials/use-optimization-dashboard/README.md", - "/cloud-analyzer/tutorials/use-optimization-dashboard/reservations.md", - "/cloud-analyzer/z_sidebar.md", "/connect-your-cloud-provider/additional-account.md", "/connect-your-cloud-provider/aws-account.md", "/connect-your-cloud-provider/azure-account.md", @@ -114,43 +82,7 @@ "/connect-your-cloud-provider/README.md", "/connect-your-cloud-provider/release-notes-connecting/README.md", "/connect-your-cloud-provider/release-notes-dashboard/README.md", - "/cost-intelligence/get-started/connect-aws-stacksets.md", - "/cost-intelligence/get-started/connect-aws.md", - "/cost-intelligence/get-started/connect-azure.md", - "/cost-intelligence/get-started/connect-with-azure-cli.md", - "/cost-intelligence/get-started/README.md", - "/cost-intelligence/README.md", - "/cost-intelligence/release-notes/README.md", - "/cost-intelligence/tutorials/administration/policies.md", - "/cost-intelligence/tutorials/administration/README.md", - "/cost-intelligence/tutorials/best-practice-checks/README.md", - "/cost-intelligence/tutorials/charts/README.md", - "/cost-intelligence/tutorials/charts/REAMDE.md", - "/cost-intelligence/tutorials/cost-intelligence-policy/README.md", - "/cost-intelligence/tutorials/dashboard/ci-dashbords-data-joins.md", - "/cost-intelligence/tutorials/dashboard/derived-values.md", - "/cost-intelligence/tutorials/dashboard/forecasting.md", - "/cost-intelligence/tutorials/dashboard/README.md", - "/cost-intelligence/tutorials/integrations/databricks.md", - "/cost-intelligence/tutorials/integrations/datadog.md", - "/cost-intelligence/tutorials/integrations/mongodb.md", - "/cost-intelligence/tutorials/integrations/new-relic.md", - "/cost-intelligence/tutorials/integrations/ocean.md", - "/cost-intelligence/tutorials/integrations/openai.md", - "/cost-intelligence/tutorials/integrations/README.md", - "/cost-intelligence/tutorials/integrations/snowflake.md", - "/cost-intelligence/tutorials/integrations/splunk.md", - "/cost-intelligence/tutorials/inventory.md", - "/cost-intelligence/tutorials/README.md", - "/cost-intelligence/tutorials/workflow-builder/anomaly-detection.md", - "/cost-intelligence/tutorials/workflow-builder/configuring-and-sending-emails.md", - "/cost-intelligence/tutorials/workflow-builder/configuring-budgets-thresholds.md", - "/cost-intelligence/tutorials/workflow-builder/README.md", - "/cost-intelligence/_media/README.md", - "/cost-intelligence/_sidebar.md", - "/design-documents/msp/msp-enrollment.md", - "/design-documents/spot-pc/README.md", - "/design-documents/spot-pc/spot-pc-test.md", + "/connect-your-cloud-provider/update-azure-credentials.md", "/eco/azure-tutorials/access-roles-read-only.md", "/eco/azure-tutorials/choose-ri-strategy.md", "/eco/azure-tutorials/README.md", @@ -158,6 +90,10 @@ "/eco/eco-in-cloudcheckr/connect-your-aws-account-in-cloudcheckr.md", "/eco/eco-in-cloudcheckr/README.md", "/eco/eco-in-cloudcheckr/view-savings-in-dashboard.md", + "/eco/eco-in-flexera-one/README.md", + "/eco/gcp-tutorials/access-roles-read-only.md", + "/eco/gcp-tutorials/README.md", + "/eco/gcp-tutorials/view-your-savings.md", "/eco/getting-started/connect-account-customer-working-with-msp.md", "/eco/getting-started/connect-account-multiple-organizations-to-single-master-payer.md", "/eco/getting-started/connect-azure-account.md", @@ -168,6 +104,8 @@ "/eco/getting-started/connect-your-aws-account.md", "/eco/getting-started/eco-for-aws.md", "/eco/getting-started/eco-savings-blox.md", + "/eco/getting-started/gettingstarted-azure.md", + "/eco/getting-started/gettingstarted-gc.md", "/eco/getting-started/README.md", "/eco/README.md", "/eco/release-notes/README.md", @@ -250,6 +188,8 @@ "/elastigroup/features/z-stateful-instance/stateful-elastigroup-flow.md", "/elastigroup/features/z-stateful-instance/stateful-instance-actions.md", "/elastigroup/features/z-stateful-instance/z-stateful-instances.md", + "/elastigroup/features-azure/commitments-elast-aks.md", + "/elastigroup/features-azure/commitments-setup-aks.md", "/elastigroup/features-azure/dns.md", "/elastigroup/features-azure/networking.md", "/elastigroup/features-azure/od-spotvm.md", @@ -372,9 +312,6 @@ "/faqs/faqs-ocean.md", "/faqs/README.md", "/faqs/_sidebar.md", - "/kb/demo.md", - "/kb/README.md", - "/kb/_faqs-for-review.md", "/managed-instance/application-persistency/README.md", "/managed-instance/azure/features/actions.md", "/managed-instance/azure/features/persist-network.md", @@ -393,6 +330,9 @@ "/managed-instance/azure/tutorials/README.md", "/managed-instance/azure/tutorials/set-health-checks-and-autohealing.md", "/managed-instance/azure/tutorials/view-details.md", + "/managed-instance/features/block-device-mapping-stateful-node.md", + "/managed-instance/features/commitments-setup-stateful-aks.md", + "/managed-instance/features/commitments-stateful-aks.md", "/managed-instance/features/data-volume-persistence.md", "/managed-instance/features/managed-instance-actions.md", "/managed-instance/features/network-persistence.md", @@ -420,6 +360,8 @@ "/ocean/features/auto-upgrade-aks-patch-version.md", "/ocean/features/avail-zones-scores.md", "/ocean/features/cluster-orientation.md", + "/ocean/features/commitments-aks.md", + "/ocean/features/commitments-setup-aks.md", "/ocean/features/committed-use-discount.md", "/ocean/features/connect-eks-cluster-test.md", "/ocean/features/cost-analysis.md", @@ -427,6 +369,7 @@ "/ocean/features/distribute-vcpu.md", "/ocean/features/dynamic-commitments-aws.md", "/ocean/features/eks-auto-ami.md", + "/ocean/features/eks-bottlerocket-ami-conf.md", "/ocean/features/elastic-ip.md", "/ocean/features/gke-cluster-vng-orientation.md", "/ocean/features/headroom.md", @@ -445,6 +388,7 @@ "/ocean/features/revert-to-lower-cost-node.md", "/ocean/features/right-sizing.md", "/ocean/features/roll-gen.md", + "/ocean/features/roll-gke.md", "/ocean/features/roll.md", "/ocean/features/running-hours.md", "/ocean/features/scaling-ecs.md", @@ -478,6 +422,7 @@ "/ocean/README.md", "/ocean/release-notes/README.md", "/ocean/test.md", + "/ocean/testimages.md", "/ocean/tips-and-best-practices/manage-machine-types.md", "/ocean/tips-and-best-practices/README.md", "/ocean/tips-and-best-practices/upgrade-aks-cluster.md", @@ -550,53 +495,9 @@ "/ocean/tutorials/use-right-sizing.md", "/ocean/tutorials/view-scaling-constraints.md", "/ocean/_sidebar.md", - "/ocean-spark/configure-clusters/README.md", - "/ocean-spark/configure-permissions/README.md", - "/ocean-spark/configure-spark-apps/access-your-data.md", - "/ocean-spark/configure-spark-apps/common-spark-configs.md", - "/ocean-spark/configure-spark-apps/docker-images.md", - "/ocean-spark/configure-spark-apps/memory-&-cores.md", - "/ocean-spark/configure-spark-apps/mount-volumes.md", - "/ocean-spark/configure-spark-apps/package-spark-code.md", - "/ocean-spark/configure-spark-apps/README.md", - "/ocean-spark/configure-spark-apps/secrets-environment-variables.md", - "/ocean-spark/data-plane-release-notes/README.md", - "/ocean-spark/docker-images-release-notes/gen18.md", - "/ocean-spark/docker-images-release-notes/gen19.md", - "/ocean-spark/docker-images-release-notes/gen20.md", - "/ocean-spark/docker-images-release-notes/gen21.md", - "/ocean-spark/docker-images-release-notes/gen22.md", - "/ocean-spark/docker-images-release-notes/gen23.md", - "/ocean-spark/docker-images-release-notes/gen24.md", - "/ocean-spark/docker-images-release-notes/gen25.md", - "/ocean-spark/docker-images-release-notes/legacy-dm-images.md", - "/ocean-spark/docker-images-release-notes/README.md", - "/ocean-spark/getting-started/create-cluster.md", - "/ocean-spark/getting-started/README.md", - "/ocean-spark/getting-started/run-your-first-app.md", - "/ocean-spark/getting-started/troubleshoot-cluster-deployment.md", - "/ocean-spark/product-tour/analyze-costs.md", - "/ocean-spark/product-tour/manage-clusters.md", - "/ocean-spark/product-tour/monitor-applications.md", - "/ocean-spark/product-tour/monitor-jobs.md", - "/ocean-spark/product-tour/README.md", - "/ocean-spark/product-tour/use-vngs.md", - "/ocean-spark/product-tour/view-application-details.md", - "/ocean-spark/product-tour/view-cluster-details.md", - "/ocean-spark/product-tour/view-job-details.md", - "/ocean-spark/README.md", - "/ocean-spark/support/README.md", - "/ocean-spark/tools-integrations/aws-glue-catalog.md", - "/ocean-spark/tools-integrations/connect-jupyter-notebooks.md", - "/ocean-spark/tools-integrations/hive-metastore.md", - "/ocean-spark/tools-integrations/jdbc.md", - "/ocean-spark/tools-integrations/notebook-filesystem-plugins.md", - "/ocean-spark/tools-integrations/README.md", - "/ocean-spark/tools-integrations/run-apps-from-airflow.md", - "/ocean-spark/tools-integrations/shuffle-plugin.md", - "/ocean-spark/tools-integrations/spark-connect.md", - "/ocean-spark/_sidebar.md", "/over-test-22-08-24-01.md", + "/reusables/_README.md", + "/reusables/_reusable-sample.md", "/spot-connect/actions/aws_system_status.md", "/spot-connect/actions/cloudwatch.md", "/spot-connect/actions/conditional.md", @@ -674,57 +575,6 @@ "/spot-connect/sample-workflows/rightsize-ec2-instances.md", "/spot-connect/sample-workflows/stop-underutilized-resources.md", "/spot-connect/_sidebar.md", - "/spot-pc/features/concepts/avd-primer.md", - "/spot-pc/features/concepts/backup.md", - "/spot-pc/features/concepts/os-patching.md", - "/spot-pc/features/concepts/README.md", - "/spot-pc/features/concepts/security.md", - "/spot-pc/features/concepts/session-inactivity-timeout.md", - "/spot-pc/features/concepts/spot-groups.md", - "/spot-pc/features/concepts/spot-pc-and-cloud-pc.md", - "/spot-pc/features/concepts/spot-pc-manager.md", - "/spot-pc/features/README.md", - "/spot-pc/features/spot-pc-console/admins.md", - "/spot-pc/features/spot-pc-console/overview.md", - "/spot-pc/features/spot-pc-console/README.md", - "/spot-pc/features/spot-pc-console/tenant/config-actions.md", - "/spot-pc/features/spot-pc-console/tenant/data-volumes.md", - "/spot-pc/features/spot-pc-console/tenant/logs.md", - "/spot-pc/features/spot-pc-console/tenant/machines.md", - "/spot-pc/features/spot-pc-console/tenant/overview.md", - "/spot-pc/features/spot-pc-console/tenant/README.md", - "/spot-pc/features/spot-pc-console/tenant/security.md", - "/spot-pc/features/spot-pc-console/tenant/user-sessions.md", - "/spot-pc/getting-started/onboarding-workflow.md", - "/spot-pc/getting-started/onboarding-workflow_v2.md", - "/spot-pc/getting-started/prerequisites/end-user-prerequisites.md", - "/spot-pc/getting-started/prerequisites/README.md", - "/spot-pc/getting-started/README.md", - "/spot-pc/getting-started/release-lifecycle.md", - "/spot-pc/getting-started/release-notes.md", - "/spot-pc/getting-started/roles-and-responsibilities.md", - "/spot-pc/README.md", - "/spot-pc/tips-and-best-practices/README.md", - "/spot-pc/tools-and-integrations/README.md", - "/spot-pc/troubleshooting/getting-support.md", - "/spot-pc/troubleshooting/monitored-metrics.md", - "/spot-pc/troubleshooting/README.md", - "/spot-pc/tutorials/add-tenant.md", - "/spot-pc/tutorials/connect-to-desktop.md", - "/spot-pc/tutorials/deploy-image-update.md", - "/spot-pc/tutorials/deploy-spot-pc.md", - "/spot-pc/tutorials/deploy-windows-365-cloud-pc.md", - "/spot-pc/tutorials/edit-spot-group.md", - "/spot-pc/tutorials/edit-w365.md", - "/spot-pc/tutorials/enable-support-access.md", - "/spot-pc/tutorials/install-ad-connect.md", - "/spot-pc/tutorials/manage-admins.md", - "/spot-pc/tutorials/manage-users-and-groups.md", - "/spot-pc/tutorials/README.md", - "/spot-pc/tutorials/README0.md", - "/spot-pc/tutorials/setup-company-share.md", - "/spot-pc/tutorials/setup-mfa-conditional-access.md", - "/spot-pc/_sidebar.md", "/spot-security/features/administration.md", "/spot-security/features/analyze-risks/README.md", "/spot-security/features/analyze-risks/remediate.md", @@ -759,6 +609,7 @@ "/spot-storage/README.md", "/spot-storage/recommendations-log.md", "/spot-storage/_sidebar.md", + "/testpage.md", "/tools-and-provisioning/ansible.md", "/tools-and-provisioning/ci-cd/chef.md", "/tools-and-provisioning/ci-cd/gitlab.md", @@ -814,39 +665,7 @@ "/tools-and-provisioning/terraform/tools/resources.md", "/tools-and-provisioning/terraform/tools/upgrade-to-terraform-v012.md", "/tools-and-provisioning/_sidebar.md", - "/wave/features/cluster-management.md", - "/wave/features/cost-analysis.md", - "/wave/features/README.md", - "/wave/features/wave-cluster-overview.md", - "/wave/getting-started/README.md", - "/wave/overview.md", - "/wave/README.md", - "/wave/_sidebar.md", - "/wnew/readme.md", - "/z-managed-instance/features/data-volume-persistence.md", - "/z-managed-instance/features/managed-instance-actions.md", - "/z-managed-instance/features/network-persistence.md", - "/z-managed-instance/features/README.md", - "/z-managed-instance/features/replacement-process.md", - "/z-managed-instance/features/root-volume-persistence.md", - "/z-managed-instance/features/stateful-managed-instances.md", - "/z-managed-instance/features/third-party-integrations.md", - "/z-managed-instance/getting-started/create-a-new-managed-instance.md", - "/z-managed-instance/getting-started/join-an-existing-managed-instance.md", - "/z-managed-instance/getting-started/README.md", - "/z-managed-instance/README.md", - "/z-managed-instance/tutorials/README.md", - "/z-managed-instance/tutorials/upgrade-an-existing-elastigroup-to-managed-instance.md", - "/z-managed-instance/_sidebar.md", - "/z-ocean-cd/features/external-verifications.md", - "/z-ocean-cd/features/granular-visibility/detailed-rollout-view.md", - "/z-ocean-cd/features/granular-visibility/README.md", - "/z-ocean-cd/features/README.md", - "/z-ocean-cd/features/rollback.md", - "/z-ocean-cd/features/webhook-notifications.md", - "/z-ocean-cd/getting-started/README.md", - "/z-ocean-cd/ocean-cd-overview.md", - "/z-ocean-cd/README.md", "/_404.md", + "/_media/reusable-sample-media.md", "/_sidebar.md" ] diff --git a/src/scripts/aliases.js b/src/scripts/aliases.js index 26a8b36234..489d37cbce 100644 --- a/src/scripts/aliases.js +++ b/src/scripts/aliases.js @@ -1788,24 +1788,6 @@ export const aliases = (() => { "/tools-and-provisioning/cloudfoundry-bosh": "/connect-your-cloud-provider/first-account/", - - "/wave/": "/ocean-spark/", - - "/wave/features/": "/ocean-spark/product-tour/", - - "/wave/features/cluster-management": - "/ocean-spark/product-tour/manage-clusters", - - "/wave/features/cost-analysis": "/ocean-spark/product-tour/analyze-costs", - - "/wave/features/wave-cluster-overview": - "/ocean-spark/product-tour/view-cluster-details", - - "/wave/getting-started/": "/ocean-spark/getting-started/", - - "/wave/overview": "/ocean-spark/", - - "/wave/wave-overview": "/ocean-spark/", }; for (let [key, value] of Object.entries(map)) {