---
title: "Connecting to a Spark Kubernetes Cluster"
description: "How to configure PYSPARK_SUBMIT_ARGS so Corridor jobs run on a Spark cluster deployed on Kubernetes."
---

Corridor can run jobs on a Spark cluster deployed on Kubernetes using `PYSPARK_SUBMIT_ARGS`.
Set the spark submit arguments in your Corridor API configuration so notebooks/jobs use the desired
Kubernetes Spark runtime settings.

## Prerequisites

Before configuring Corridor:

- Spark cluster is reachable from the Corridor API/worker hosts.
- A Kubernetes namespace exists for Spark driver and executor pods.
- A Kubernetes service account exists with permissions to create/list/delete pods in that namespace.
- A Spark-compatible container image is available and can be pulled by the cluster.

## Example configuration

Set `PYSPARK_SUBMIT_ARGS` in [`api_config.py`](/technology/installation/configuration-setup/api/) with k8s cluster details.
See the Sample config below, use your own values for endpoint, image, network, and sizing:

```python
os.environ['PYSPARK_SUBMIT_ARGS'] = (
    '--master k8s://https://<k8s-api-server> '
    '--deploy-mode client '
    '--conf "spark.kubernetes.container.image=<spark-image>:<tag>" '
    '--conf "spark.kubernetes.namespace=<spark-namespace>" '
    '--conf "spark.kubernetes.authenticate.driver.serviceAccountName=<spark-service-account>" '
    '--conf "spark.jars.packages=org.apache.hadoop:hadoop-aws:<version>,com.amazonaws:aws-java-sdk-bundle:<version>" '
    '--conf "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" '
    '--conf "spark.hadoop.fs.s3a.path.style.access=true" '
    '--conf "spark.driver.host=<driver-host-ip>" '
    '--conf "spark.driver.port=<driver-port>" '
    '--conf "spark.blockManager.port=<blockmanager-port>" '
    '--conf "spark.driver.bindAddress=0.0.0.0" '
    '--conf "spark.executor.instances=<num-executors>" '
    '--conf "spark.executor.memory=<executor-memory>" '
    '--conf "spark.executor.cores=<executor-cores>" '
    '--conf "spark.driver.extraJavaOptions=-Xss10m -XX:+UseG1GC" '
    '--conf "spark.executor.extraJavaOptions=-Xss10m -XX:+UseG1GC" '
    '--driver-memory <driver-memory> '
    'pyspark-shell'
)
```

This configuration tells PySpark to submit workloads to a Kubernetes Spark cluster in `client` mode.
The `spark.kubernetes.*` settings select the target namespace, container image, and service account
used for driver and executor pods. The `spark.driver.*` and executor resource settings define runtime
networking and sizing.
Replace all placeholder values with environment-specific values before use.

## Related

- [Spark UI](/technology/customization/spark-ui/): configure links from a Corridor job to the Spark Web UI and History Server for a cluster set up this way
- [Job Specific Configurations](/technology/customization/job-configs/): run individual jobs with their own Spark session settings instead of a cluster-wide configuration