How to Integrate Mist API with AWS EMR?

[pmiyandada]

Hi Guys,
We have a requirement to integrate mist with AWS EMR to run multiple jobs. Could you please suggest to us how to integrate Mist API with AWS EMR.
Thanks in advance.

[blvp]

Hi there.
We have a feature-preview support for AWS EMR dynamic cluster creation and documentation for this is located here: https://hydrosphere.io/mist-docs/aws-emr.html

Be careful, this version is not released yet and experimental. I wouldn't recommend using it in production.

We neither have official documentation or guide about Mist and AWS EMR integration, but I have some practical experience about it. You can try to follow it, but I can't guarantee it will work in your case.

In practice mist instance is always deployed alongside with your EMR cluster so it has a default access to EMR's yarn master service and has a spark EMR distribution. To achieve the default behaviour you either need to modify spark-defaults or add it in your Context configuration of a function. There are several spark configurations that you need to override:

spark.executor.instances = <max num instances for a function execution>
spark.master = yarn
spark.submit.deployMode = "cluster"
spark.executor.memory = <depends on your workload and instance types>
spark.executor.cores = <depends on your workload and instance types>
spark.yarn.executor.memoryOverhead = <depends on your workload and instance types>

It should assign any available elasticIP when deployed so you can access it directly by this IP.

By setting above options you will be able to:

1. start driver application (mist worker) in EMR in cluster mode
2. function context configuration will account for yarn's specific configuration
3. you don't need explicitly set spark.master address it will be derived from spark on emr distribution.

After mist deployment and Context configuration is set properly you should be able to invoke function through mist ui or through any other source (Http, Mqtt, Kafka ).

[pmiyandada]

Hi Blvp,

Does mist have to be installed on the EMR master node? if yes, then Mist would be tightly coupled with a single spark cluster? what if I want to run multiple jobs using the same mist.

and I am unable to create a Mist setup by using a cloud formation template. It's throwing an error image that does not exist (ami-027583e616ca104df).

So, If we are going with the other AMI, How to start the mist service without spark home directory ?
and How to apply cloned hello. mist conf file through Mist-CLI without Nginx.
ex:- mist-cli --host $MistLogin:$MistPassword@$public-dns-name --port 80 apply -f conf
If you run the above command with port 80 (without Nginx) throwing some error. Please find the below error log.

Error: 404 Client Error: Not Found for url: http://$MistLogin:$MistPassword@$public-dns-name:80/v2/api/contexts:

<title>404 Not Found</title>

Not Found

The requested URL was not found on this server.

Request body: {"maxJobs": 1, "streamingDuration": "1s", "downtime": "1200s", "runOptions": "", "maxConnFailures": 1, "precreated": false, "sparkConf": {"spark.submit.deployMode": "cluster", "spark.master": "spark://holy-excalibur:7077"}, "workerMode": "exclusive", "name": "root_standalone"}
updating Function root_hello-mist-scala
Error: Function root_hello-mist-scala is not valid. Please check: Context root_10_emr_ctx.conf should exists remotely.
Thanks in advance.

Best Regards,
Miyandada.P