Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/getting_started/index.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Getting started
:description: Getting started with Apache Spark on Kubernetes, covering the Stackable operator and its dependencies and a first Spark job and its result.

This guide gets you started with Spark using the Stackable operator for Apache Spark.
It guides you through the installation of the operator and its dependencies, executing your first Spark job and reviewing its result.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Command line parameters
:description: Command-line parameters the Stackable operator for Apache Spark accepts, with their default values and usage examples.

This operator accepts the following command line parameters:

Expand Down
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/reference/crds.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= CRD Reference
:description: CRD reference for the Stackable operator for Apache Spark on the Stackable Hub, covering every field, default and API version.

Find all CRD reference for the Stackable Operator for Apache Spark at: {crd-ref}{crd-ref-release}&operator=spark-k8s-operator.
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Environment variables
:description: Environment variables the Stackable operator for Apache Spark accepts, with their default values and what each one controls.

This operator accepts the following environment variables:

Expand Down
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/reference/index.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Reference
:description: Reference for the Stackable operator for Apache Spark, covering CRD fields, command-line parameters and environment variables.

Consult the reference documentation section to find exhaustive information on:

Expand Down
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/usage-guide/index.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Usage guide
:description: Guides for the Stackable operator for Apache Spark beyond the basics, covering job dependencies, S3 connections and examples of usage scenarios.

Learn how to load your own xref:usage-guide/job-dependencies.adoc[] or configure an xref:usage-guide/s3.adoc[S3 connection] to access data. Have a look at the xref:usage-guide/examples.adoc[] to learn more about different usage scenarios.
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/usage-guide/logging.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Logging
:description: Logging for Apache Spark applications and the history server, covering the Vector agent the operator adds and forwarding logs to an aggregator.

The Spark operator installs a https://vector.dev/docs/setup/deployment/roles/#agent[vector agent] as a side-car container in every application Pod except the `job` Pod that runs `spark-submit`.
It also configures the logging framework to output logs in XML format.
Expand Down
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/usage-guide/openlineage.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= OpenLineage
:description: OpenLineage events from Apache Spark applications to a backend such as Marquez, covering authentication and job naming without manual sparkConf wiring.

https://openlineage.io/[OpenLineage] is an open standard for data lineage collection.
The Spark operator can automatically emit lineage events for a `SparkApplication` to an OpenLineage-compatible backend such as https://marquezproject.github.io/marquez/[Marquez], without any manual `sparkConf` wiring.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Spark applications
:description: The lifecycle of Apache Spark applications submitted as SparkApplication resources, covering resubmission and their Prometheus metrics.

Spark applications are submitted to the Spark Operator as SparkApplication resources.
These resources are used to define the configuration of the Spark job, including the image to use, the main application file, and the number of executors to start.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Graceful shutdown
:description: Graceful shutdown for Apache Spark, which the Stackable operator has no settings for, with Spark itself creating and stopping driver and executor Pods.

Normally you can configure the graceful shutdown as described in xref:concepts:operations/graceful_shutdown.adoc[].

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Operations
:description: Operating Apache Spark with the Stackable operator, covering Spark applications, graceful shutdown, Pod disruptions and Pod placement.

This section of the documentation is intended for the operations teams that maintain a Stackable Data Platform installation.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Allowed Pod disruptions
:description: PodDisruptionBudgets for the Apache Spark history server, and why the Stackable operator writes none for driver and executor Pods.

== Spark applications

Expand Down
1 change: 1 addition & 0 deletions docs/modules/spark-k8s/pages/usage-guide/overrides.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Configuration & Environment Overrides
:description: Overrides for Apache Spark, covering configuration, spark-env.sh, security.properties, environment variables, Pod specs and JVM arguments.

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).

Expand Down
3 changes: 2 additions & 1 deletion docs/modules/spark-k8s/pages/usage-guide/resources.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
= Resource Requests
:description: CPU and memory for Apache Spark jobs, drivers, executors and the history server, with examples of how to size each of them.

include::home:concepts:stackable_resource_requests.adoc[]
include::concepts:partial$stackable_resource_requests.adoc[]

If no resources are configured explicitly, the operator uses the following defaults for `SparkApplication` resources:

Expand Down
Loading