| Ab Initio Component | Equivalent Prophecy Gems |
|---|---|
| Fixed File | Fixed File |
| INPUT TABLE | [Source] |
| READ MULTIPLE FILES | Read Multiple Files |
| READ SEPARATED VALUES | Read Separated Values |
| READ RAW | Read Raw |
| READ JSON | [Source] |
| READ EXCEL SPREADSHEET | [Source] |
| LOOKUP FILE | Lookup |
| OUTPUT FILE | [Target] |
| WRITE MULTIPLE FILES | Write Multiple Files |
| WRITE LOOKUP | Lookup + [Target] |
| Write Separated Values | Write Separated Values |
| JOIN | Join |
| REFORMAT | Reformat |
| NORMALIZE | Normalize |
| FILTER BY EXPRESSION | Filter |
| GATHER | Set Operation |
| ROLLUP | Aggregate |
| GATHER LOGS | [Target] |
| FUSE | Join |
| VALIDATE RECORDS | Filter or Row Distributor |
| LOCAL SORT, SORT | OrderBy |
| CHECKPOINTED SORT | OrderBy |
| SORT WITHIN GROUPS | OrderBy |
| DEDUP SORTED | Deduplicate |
| DENORMALIZE SORTED | WindowFunction |
| PARTIALSORT | OrderBy |
| SCAN | WindowFunction + Reformat |
| PARTITION BY EXPRESSION | Filter |
| PARTITION BY KEY AND SORT | OrderBy |
| PARTITION BY KEY | Repartition |
| PARTITION BY ROUND-ROBIN | RoundRobinPartition |
| HASH | Repartition |
| CREATE DATA | Create Data |
| META PIVOT | Meta Pivot |
| CONCATENATE | Set Operation |
| ASSIGN NEW KEYS | Assign Keys |
| LEADING RECORDS | Limit |
| JOIN WITH DB JDBC | [Source] + Join |
| GENERATE RECORDS | CreateData + Reformat |
| GET FILE MTIME | MTime |
| COMPARE RECORDS | Compare Record |
| META PIVOT | Meta Pivot |
| MULTI REFORMAT | Reformat |
| XML REFORMAT | Reformat |
| SFTP FROM | [Source] |
| FTP TO | [Target] |
| FTP FROM | [Source] |
| SUBGRAPH | Subgraph |
| Sequence | Sequence |
| Log | Log |
| Error Detection | Error Detection |
| Broadcast | Script |
| Procedural Transform | Script |
| Send Mail | Script |
| Sftp | Script |
| Std Out | Script |
| Trash | Script |
| Predicate Push Down | Script |
| Run Sql | Script |
| Select Procedure With Param | Script |
| Scan With Rollup | Script |
| File Intermediate | Script |
| Intermediate Lookup | Script |
Components
Ab Initio component mapping
List of supported Ab Initio components
For transpiling Ab Initio graphs to Prophecy pipelines, Transpiler first reads the uploaded files and then internally maps the Ab Initio components with Prophecy-supported gems to create the visual graph and open-source Spark code.
These Ab Initio components are core part of the graph as they perform data operations like reading data from source table, performing transformations and writing data into the destination table.
Below is a list of all the Ab Initio components that are supported by Prophecy and the corresponding Prophecy Spark gems:
If you are looking for a particular component and don’t see it on this list, fear not! Prophecy is extensible to allow inclusion of custom abilities way beyond custom scripting. Read this to learn more.