Scalability and Sizing

If the platform seems slow, there may be an infrastructure constraint in terms of resources. This page describes how the different aspects of scalability were considered when designing the platform.

The way to think about scalability in the platform is:

• Production
• Simulation and other jobs
• User-initiated analytics
• Application

Production

Production scalability can be thought of in two parts:

• Time to execute the artifact bundle or policy
• Throughput for the production orchestration

The time taken for the artifact bundle or policy depends on the complexity of the logic written in the platform. This can be sped up by writing more efficient code or simplifying the logic.

The throughput for the orchestration depends on the method of deployment of the Production layer, whether that is an HTTP API, a serverless architecture, Python runtime execution, or something else.

Simulation and Other Jobs

Simulations and other jobs such as Comparison and Validation use the Celery workers to perform tasks. There are two parts to consider:

• Number of parallel jobs to run
• Cluster nodes available for use per job

If the number of parallel jobs is the concern, for example a large job blocking the queue of later jobs, this can be handled by adding more Spark - Celery Workers as needed, or by running multiple named workers on the same server. The Celery workers are stateless, all state is maintained in the task messaging queue (Redis), and removing or adding more is seamless.

Caution

Adding too many Spark - Celery workers can cause resource contention between the workers themselves and may require an increase in cluster nodes.

If the Spark jobs themselves are slow, the issue is more likely the cluster nodes available per job, that is, the number of Spark workers or YARN containers allocated per job. To make this faster, provide more Spark nodes.

User-Initiated Analytics

The platform’s Notebook section allows users to initiate their own jobs in a free-form manner using Python or PySpark. As these jobs are user-dependent, they can use a large number of cluster resources if not used carefully. If this causes issues with Simulation jobs, use separate YARN queues to manage resource allocation appropriately.

Application

Application scalability becomes important as the number of concurrent users grows over time. This can affect the usability of the application itself, and is addressed by identifying which portion of the application is causing the bottleneck. The Metadata Database or the API Server could be the cause. Either can be rectified by adding more resources to these components, or by parallelizing them with database read replicas or additional API servers.

Note

The API servers are stateless and can be scaled up or down as needed.