Dependencies & cluster requirements
Dependencies & cluster requirements
Dependencies:
- ProphecyWarehousePython 0.0.1+
- UC dedicated clusters not supported
- UC standard clusters not supported
- Livy clusters not supported
azure-cosmos-spark connector is an integration between Azure CosmosDB and Apache Spark, which allows you to read and write data from and to CosmosDB using Spark.
Installation
For Spark 3.3, connect a dependency that has the following Maven coordinates on your Databricks or OnPrem Execution Cluster:Usage
Parameters
Master key authentication
Service principle based authentication
Source
The Source gem reads data from CosmosDB and allows you to optionally specify the following additional properties.Source properties
Throughput control configs
Having throughput controls help isolate the performance needs of applications running against a container by limiting the amount of request units a Spark client consumes. Several advanced scenarios benefit from client-side throughput control:- Different operations and tasks have different priorities
- Provide fairness and isolation to different end users
- Load balancing of throughput between different Azure Cosmos DB clients
To learn more about throughput control, see Azure Cosmos DB Spark connector: Throughput control.
Target
The Target gem writes data to CosmosDB and allows you to optionally specify the following additional properties.Target properties
Supported write modes
Supported write strategies
ItemPatch properties
These configurations are valid only when you selectItemPatch as your Write Strategy.

