Pipeline Components
Below is a list of DataStage workflow components that are supported by Prophecy and have corresponding Prophecy gems. These components are core part of the workflow as they perform data operations like reading data from source table, performing transformations, and writing data into the destination table.| DataStage Component | Equivalent Prophecy Gems |
|---|---|
| CTransformerStage (for cyclic stage variables) | Script |
| CTransformerStage (for non-cyclic stage variables) | SchemaTransform + Filter + Reformat |
| DB2ConnectorPX | Source/Target with DB2 warehouse type |
| Hierarchical Data stage | Script with only XML output |
| ODBCConnector | Source/Target with JDBC warehouse type |
| ODBCConnectorPX | Source/Target with JDBC warehouse type |
| OracleConnectorPX | Source/Target with the Oracle warehouse type |
| PxAggregator | Aggregate |
| PxChangeCapture | Script |
| PxColumnExport | SchemaTransform |
| PxColumnImport | SchemaTransform |
| PxCopy | Source/Target |
| PxDataSet | Source/Target with the CSV file type |
| PxExternalSource | Script to call external APIs |
| PxExternalTarget | Script to call external APIs |
| PxFilter | Filter |
| PxFunnel | SetOperation |
| PxJoin | Join |
| PxLookup | Join |
| PxMerge | Join |
| PxModify | Reformat |
| PxPeek | Target with the CSV file type |
| PxPivot | Script |
| PxRemDup | Deduplicate |
| PxRowGenerator | Source with the Seed file type |
| PxSequentialFile | Source/Target with the CSV file type |
| PxSort | Order By |
| TeradataConnectorPX, TeradataConnector | Source/Target |
| UnstructuredDataConnectorPX | Source with the XLSX file type |
| XMLInputPX | Reformat + Join+ Source with the XML file type |
| XMLOutputPX | Aggregate + Target gem with XML file type |
Orchestration Components
The following table describes components of DataStage sequence jobs that can be migrated to Prophecy. Note that all jobs are converted into Airflow jobs in Prophecy.| DataStage Component | Equivalent Prophecy Component |
|---|---|
| CCondition | Python Branch Component |
| CEndLoopActivity | Loop Component |
| CExecCommandActivity | Python Component |
| CExceptionHandler | Python Component |
| CJobActivity | Trigger DAG if you have a sequence job, or Pipeline Component if you have a parallel/server job |
| CNotificationActivity | Email Component |
| CSequencer | Databricks Job |
| CStartLoopActivity | Loop Component |
| CTerminatorActivity | Python Component |
| CUserVarsActivity | Python Component with Airflow XComs |
If you don’t see a particular component on this list, you can include custom abilities beyond
custom scripting. To learn more, see Gem Builder for
Spark.