Insights

Data Pipelining is evolving. Gone are the days where teams of engineers are required to maintain the movement of datasets to your data warehouse. We are now living in a post Extract, Transform, and Load (ETL) paradigm and have entered anExtract, Load, and Transform (ELT)future. Now, instead of having a team of engineers work to extract data from all your apps, a single data analyst can manage the extraction and loading of hundreds of datasets with the use of Fivetran.

Managing the “T” in ELT

While extracting and loading have been simplified, we are still left with the complexity of transformations. Moving transformations to the end of the pipeline has eased a few pains, but there are still hurdles to climb.

There are a plethora of transformation tools out on the market, and it can be difficult to choose or justify adding another one to your toolset. Similarly, these tools aren’t always uniform or simple: drag and drop, scala, and python are all options out there. Lastly, these solutions can be resource heavy for engineering teams whose time and resources are already stretched thin.

A new way to Transform with Fivetran

We are proud to be partnering with Fivetran for their release of Fivetran Transformations. Transformations is an innovative new feature which allows users to manage in-warehouse transformations within Fivetran, utilizing the universality of SQL to open the doors for analysts to manage the full ELT pipeline and free up engineers for bigger projects.

A Single Point for ELT

In a world filled with thousands of apps, datasets, and complex transformations, the only path forward is through simplifying. For the first time, Fivetran Transformations allows the ELT process to be managed in a single application. Now the entire pipeline can be automated from start to finish, fully managed by Fivetran.

Fivetran Transformations allow the user to trigger transformations based on events or time. It also sends notifications when transformations fail, allowing for quick resolution with Fivetran’s robust logging and notification system.

SQL is the Language of Data

In Stack Overflow 2019 Programmer survey, SQL was the most widely known data language. Whether your data team is comprised of engineers, project managers, or data analysts, SQL is the universal language between them all.

Managing transformations in SQL promotes transparency and ease of use in a way that isn’t possible with other transformation languages. SQL is also easy to learn and hire for, making pipelines independent from hard to hire and expensive engineers.

“Fivetran Transformations empowers data analysts to own the entire data stack.”

Empower your Analysts

Lastly, Fivetran Transformations empowers data analysts to own the entire data stack: from all the disparate data sources and apps, to the data warehouse, and to the BI tool, an analyst can own the whole process.

The gap between a business’s goals and its technical skills can be vast. Engineers and business users seem to speak different languages, making the perceived divide between them even greater. Data analysts are well suited to bridge this gap, living with one foot in both the technical and business worlds. Empowering data analysts, who can speak both the technical and business languages, can bridge this divide and propel a company towards being data driven.

We Have a Plan

Transformations are difficult, but they can be simplified with robust planning. DAS42 can work with you to understand your business goals and implement a technical path towards achieving them.