Skip to main content
Dependencies:
  • ProphecyWarehousePython 0.0.1+
Cluster requirements:
  • UC dedicated clusters not supported
  • UC standard clusters not supported
  • Livy clusters not supported
The azure-cosmos-spark connector is an integration between Azure CosmosDB and Apache Spark, which allows you to read and write data from and to CosmosDB using Spark.

Installation

For Spark 3.3, connect a dependency that has the following Maven coordinates on your Databricks or OnPrem Execution Cluster:
For other versions of Spark, see Azure Cosmos DB OLTP Spark 3 connector.

Usage

Parameters

Master key authentication

Service principle based authentication

Source

The Source gem reads data from CosmosDB and allows you to optionally specify the following additional properties.

Source properties

Throughput control configs

Having throughput controls help isolate the performance needs of applications running against a container by limiting the amount of request units a Spark client consumes. Several advanced scenarios benefit from client-side throughput control:
  • Different operations and tasks have different priorities
You can prevent normal transactions from throttling due to data ingestion or copy activities. Some operations or tasks aren’t sensitive to latency and are more tolerant to being throttled than others.
  • Provide fairness and isolation to different end users
Applications usually have many end users, where some may send too many requests. This consumes all available throughput and causes others to throttle.
  • Load balancing of throughput between different Azure Cosmos DB clients
Assure all the clients get an equal share of the throughput. To learn more about throughput control, see Azure Cosmos DB Spark connector: Throughput control.

Target

The Target gem writes data to CosmosDB and allows you to optionally specify the following additional properties.

Target properties

Supported write modes

Supported write strategies

ItemPatch properties

These configurations are valid only when you select ItemPatch as your Write Strategy.