DSP-19969 allow spark executor log to separate file

[alexliu68]

No description provided.

[RussellSpitzer]

I'm not sure I understand this, the Jira is about a directory which is being created, but this pr is to add a command which changes the local executor Directory for logging?

Is it that the default "LoggingOptions" env variable is creating the wrong directory? Shouldn't we just change it then?

[RussellSpitzer]

Looking at the original

# Override logging options to provide per-context logging
LOGGING_OPTS="$LOGGING_OPTS_FILE
              -DLOG_DIR=$1"

Log dir should be set to the first arg, checking our dse start script we just pass through args

                if [ "$DSERM" == "v1" ]; then
                    SJS_LAUNCHER="$SPARK_HOME/bin/spark-submit" run_with_framework "$SPARK_JOBSERVER_HOME"/manager_start.sh "$@"
                else
                    SJS_LAUNCHER="$DSE_COMPONENTS_ROOT/spark-launcher/bin/spark-submit" "$SPARK_JOBSERVER_HOME"/manager_start.sh "$@"
                fi

[alexliu68]

Spark job server by default sets executor logging to spark-job-server.log which is spark job server (app driver) log. This ticket removes spark job default logging to executor, and use default spark executor logging. It keeps spark job server to spark-job-server.log. We can't just change LoggingOptions for both, we have to keep LoggingOptions for spark job server and logging executor separately

[alexliu68]

@RussellSpitzer I replied your comment

[RussellSpitzer]

So now the default behavior is logging to SPARK_WORK/app/executor?
And what does -s do?

[alexliu68]

default doesn't change. -s option changes the executor log to separate spark log instead of default spark job server.