Manual migration challenge
Manually migrating from legacy ETL systems often takes up too much time and resources, and the process of breaking down and rebuilding Informatica workflows comes with significant risks, including the chance of data loss or disruption. Additionally, the costs can quickly add up, making it clear that you need a better solution.Spark vs. Informatica
Before we dive into what the Transpiler can do, it’s important to understand how it measures up against Informatica. While Informatica has been valued for its reliability and versatility, Apache Spark and Spark SQL provide a fresh perspective that delivers unmatched scalability, performance, and flexibility.| Capability | Apache Spark and Spark SQL | Informatica |
|---|---|---|
| Licensing | Open-source | Proprietary |
| Cost | Free to use | Requires licensing fees |
| Development Language | Scala, Java, Python, SQL | Proprietary language |
| Scalability | Highly scalable, supports large-scale data processing | Scalable, designed for enterprise-level data integration |
| Performance | In-memory processing for speed | Optimized for ETL operations and batch processing |
| Flexibility | Open-source ecosystem, vast libraries | Proprietary framework and components |
| Vendor Lock-In | No vendor lock-in, open standards | Vendor lock-in due to proprietary nature |
| Customization | Highly customizable, allows for custom development and extensions | Limited customization through extensions |
How Transpiler works
Transpiler is a powerful tool that transforms the often complex process of ETL migration into something more manageable by:-
Workflow Parsing
Transpiler naturally parses Informatica’s XML files. It deciphers its complex components and relationships to recreate the equivalent workflow in Prophecy. A Prophecy workflow is called a pipeline. Prophecy components contain all the business logic from the Informatica components.
-
Transformation Logic
By examining Informatica’s XML files, Transpiler identifies the embedded transformation logic within each component. Then, it generates highly optimized open-source Spark code that seamlessly integrates into any Spark environment.
For example, the following shows how a Join Gem looks in both the Visual and Code views:
-
Schema Mapping
Transpiler effectively maps the data schema from Informatica’s XML files, ensuring a smooth transition to Prophecy. This guarantees that the input and output schemas in Prophecy gems align.
For example, the following shows the schema inside the Join gem:
