Connecting to a Spark Kubernetes Cluster

Corridor can run jobs on a Spark cluster deployed on Kubernetes using PYSPARK_SUBMIT_ARGS. Set the spark submit arguments in your Corridor API configuration so notebooks/jobs use the desired Kubernetes Spark runtime settings.

Prerequisites

Before configuring Corridor:

• Spark cluster is reachable from the Corridor API/worker hosts.
• A Kubernetes namespace exists for Spark driver and executor pods.
• A Kubernetes service account exists with permissions to create/list/delete pods in that namespace.
• A Spark-compatible container image is available and can be pulled by the cluster.

Example configuration

Set PYSPARK_SUBMIT_ARGS in api_config.py with k8s cluster details. See the Sample config below, use your own values for endpoint, image, network, and sizing:

os.environ['PYSPARK_SUBMIT_ARGS'] = (
    '--master k8s://https://<k8s-api-server> '
    '--deploy-mode client '
    '--conf "spark.kubernetes.container.image=<spark-image>:<tag>" '
    '--conf "spark.kubernetes.namespace=<spark-namespace>" '
    '--conf "spark.kubernetes.authenticate.driver.serviceAccountName=<spark-service-account>" '
    '--conf "spark.jars.packages=org.apache.hadoop:hadoop-aws:<version>,com.amazonaws:aws-java-sdk-bundle:<version>" '
    '--conf "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" '
    '--conf "spark.hadoop.fs.s3a.path.style.access=true" '
    '--conf "spark.driver.host=<driver-host-ip>" '
    '--conf "spark.driver.port=<driver-port>" '
    '--conf "spark.blockManager.port=<blockmanager-port>" '
    '--conf "spark.driver.bindAddress=0.0.0.0" '
    '--conf "spark.executor.instances=<num-executors>" '
    '--conf "spark.executor.memory=<executor-memory>" '
    '--conf "spark.executor.cores=<executor-cores>" '
    '--conf "spark.driver.extraJavaOptions=-Xss10m -XX:+UseG1GC" '
    '--conf "spark.executor.extraJavaOptions=-Xss10m -XX:+UseG1GC" '
    '--driver-memory <driver-memory> '
    'pyspark-shell'
)

This configuration tells PySpark to submit workloads to a Kubernetes Spark cluster in client mode. The spark.kubernetes.* settings select the target namespace, container image, and service account used for driver and executor pods. The spark.driver.* and executor resource settings define runtime networking and sizing. Replace all placeholder values with environment-specific values before use.

Related

• Spark UI: configure links from a Corridor job to the Spark Web UI and History Server for a cluster set up this way
• Job Specific Configurations: run individual jobs with their own Spark session settings instead of a cluster-wide configuration