Metadata-Driven ETL: delivering new data assets faster

We believe it's time to reveal how we work, why our metadata-driven approach allows us to deliver data integration projects more efficiently than traditional solutions. While traditional ETL (Extract, Transform, Load) has evolved significantly during the years, it remains a critical topic for enterprise data integration projects. This blog reveals the data integration philosophy that drives our minds and our solutions. The principles explained later in this blog are already implemented in the SAS-based software solution - MDIS Data Management. Customers in public, insurance, and banking sectors successfully use MDIS for enterprise data integration purposes for many years. The blog is intended for our existing customers, our future clients, and for everyone interested in modern data integration methodologies.

The Maintenance Challenge Behind Enterprise ETL

‍

 Organizations that rely on traditional data integration tools often face a familiar challenge: ETL systems become increasingly difficult to maintain as data volumes, business rules, and regulatory requirements continue to grow. What initially starts as a manageable collection of ETL jobs gradually evolves into thousands of lines of code, hundreds of workflows, and a growing dependency on a small number of experienced developers.

The Problem Lies in the Traditional Way ETL Solutions Are Designed

‍

A metadata-driven approach is a game changer - instead of building static ETL workflows, developers define metadata describing what should happen, while the platform determines how it should be executed. This significantly reduces development work, simplifies maintenance, and minimizes implementation risks.

Why Traditional ETL Becomes Expensive

‍

Traditional ETL projects are usually based on manually developed programs, graphical ETL workflows, manually maintained execution order, custom scheduling logic, and extensive testing after every change. As projects grow, implementation time and maintenance costs increase, while even small requirement changes require big modifications across multiple jobs.

Metadata Instead of Static ETL Flows

‍

Instead of programming complete ETL processes, developers configure metadata describing target tables, validation rules, transformations, historization logic, process dependencies, and execution parameters. The execution engine automatically generates the required processing logic.

Automatic Detection of Processing Order

‍

One of the most difficult tasks in large ETL environments is maintaining the correct processing order. Rather than hardcoding execution sequences, a metadata-driven platform analyzes validation rules, table dependencies, and process relationships to determine the optimal execution order automatically.

Record-Level Processing Improves Performance

‍

A metadata-driven approach can generate processing logic where a single SAS DATA step reads source records, performs joins, executes transformations, executes validation rules, and creates output records. So, the table is scanned only once and target table is created in a single step. In order to perform the same actions, traditional DBMS has to use several SQL steps and/or database cursors that are slow. This doesn’t allow to reach the same performance level as MDIS provides.

Parallel Processing Without Manual Optimization

‍

Historical execution statistics can be used to determine whether parallel execution is beneficial, the optimal number of parallel sessions, and dependency-aware scheduling. This allows the platform to optimize execution automatically.

Simplifying System Maintenance

‍

Business requirements continuously evolve. In regulated sectors, this may include new validation rules, audit expectations, reporting changes, regulatory updates, and new source systems. A metadata-driven architecture isolates business logic into configurable metadata, making changes localized, reducing regression testing effort, and minimizing deployment risks.

Built-in Operational Intelligence

‍

Execution registers, validation registers, change registers, historization registers, automatic recovery after failures, and execution simulation provide administrators with a centralized operational view of the entire ETL environment. These registers help administrators understand what was executed, which validations were applied, what changed, and where recovery is required after a failure.

Supporting the Entire Data Integration Lifecycle

‍

Metadata-driven architectures simplify development, testing, initial data loading, production operations, and future enhancements through reusable metadata rather than project-specific implementations.

Conclusion

‍

As enterprise environments continue to grow in size and complexity, maintaining traditional ETL implementations becomes increasingly expensive. A metadata-driven architecture replaces static workflows with configurable metadata and automated execution logic, reducing implementation effort, simplifying maintenance, and enabling organizations to focus on business requirements instead of ETL mechanics and deliver new data assets faster.

‍

… we will provide more details and unveil more in future blogs.

‍

‍