Introduction

Damon: Hello, everyone, and welcome to another live webinar hosted by Silk. Today’s topic is Modern ETL Pipelines on AWS, Azure, and Google Cloud with Silk.

Damon: I’m pleased to introduce today’s speakers: Dave Larson and Esteban Cordero, both members of the Silk solution architecture team, which I’m pleased to lead. Thank you to everyone who registered and joined us live.

Damon: We’ll spend about 20 to 25 minutes on the main content and demonstration, then leave a few minutes for Q&A depending on the flow. Dave will share his screen, and Esteban will kick us off.

Esteban Cordero: Thank you, Damon. We’ll start by talking about why ETL jobs are challenging, especially in cloud environments.

Why ETL Jobs Are Challenging in the Cloud

Esteban Cordero: First and foremost, transactional databases are designed to run many small, concurrent transactions. ETL jobs do something completely different from what those databases were originally designed to handle. They often run very large SELECT queries that result in full table scans, and that has a real impact on how the database operates.

Esteban Cordero: Queries slow down, jobs take longer than expected, and most importantly, systems become unpredictable. That is never something you want in a production environment. The impact is especially significant when you have locked rows, updates in progress, or long-running queries.

Esteban Cordero: Traditional ETL pipelines are also commonly batch-oriented. They may run hourly or daily, which means data freshness can become a concern. You may not always have access to the latest data set when the business needs it.

Esteban Cordero: As data grows into multiple terabytes or even petabytes, those jobs become slower and heavier. Data duplication also increases infrastructure cost and complexity because every run copies data across multiple systems.

Esteban Cordero: In short, traditional ETL pipelines often force transactional systems, analytics systems, and networks to do work they were not really designed for. That is what creates performance issues, delays, and cost.

Esteban Cordero: The most important part of what we’re showing today is how Silk can refresh downstream copies without moving bytes across the network. In a traditional environment, refreshing copies means copying data, paying full storage costs for each copy, and waiting for the movement to complete.

Esteban Cordero: With Silk Echo, those refreshes can happen as metadata operations backed by Silk technology. They can complete in seconds, and the downstream clones consume near-zero incremental storage. Because ETL relies so heavily on copying and moving data, this approach helps eliminate one of the core challenges entirely.

Demo Architecture

Esteban Cordero: For the architecture we’ll show today, we’ll initiate the workflow from our developer portal, Backstage. Backstage will make API calls into the Silk Data Platform. Those calls generate Echo copies that refresh lower-level environments.

Esteban Cordero: Dave, do you want to take it from here?

Dave Larson: Absolutely. With the API integration, we’re going to take the extraction step away from the source database. Instead of pulling data across the network, we’ll make a copy of the database and mount it to the enterprise data warehouse server as a database called Silk DB Stage.

Dave Larson: That stage database is an exact duplicate of the source database, Silk DB. You’ll see how quickly this process runs. After that, we’ll run a transformation step. For the sake of the demo, we’ll simulate a very fast transformation, but you can use whatever transformation or ETL tool you already use.

Dave Larson: Once the transformation finishes loading the Silk EDW database, we’ll also show how to use snapshots and thin clones to make copies for downstream use cases such as reporting, AI, BI, or other analytics environments.

Dave Larson: Several Silk customers use this same kind of architecture today. They load the EDW, but they do not have end users connected directly to it. That allows the EDW to be continuously updated hourly, every four hours, twice a day, once a day, or on whatever schedule makes sense.

Dave Larson: Then Silk can make instant clones, often in less than two minutes, and refresh environments such as an AI database. One healthcare customer uses a green-gold approach: they refresh the green copy, promote it to gold, and allow users to connect to gold while the other environment is refreshed.

Dave Larson: At the end of the demo, we’ll run a validation check so you can see the result.

Initial Validation

Dave Larson: This screen is Backstage, and I’ll show you the Silk user interface in a moment. First, I want to show the databases we have in this demo.

Dave Larson: We’re running queries against the databases we mentioned earlier. The source database is Silk DB, and the table we’ll be reading from is the Sales table. It currently has roughly 272,000 records.

Dave Larson: The AI and reporting clone copies currently have zero rows in the Fact Sales table. That Fact Sales table is what we’ll update during the ETL process. We’ll come back to this screen as the workflow progresses.

Dave Larson: Now I’m going to load 10,000 new records into the source database. That brings the source to roughly 282,000 records.

Step 1: Extraction with Silk Echo

Dave Larson: In the workflow, step one is extraction. With Silk, that extraction step is a snapshot and thin clone operation. We’ll take an application-consistent snapshot of Silk DB, then mount it to the Silk EDW SQL Server as the stage environment.

Dave Larson: I’ll run that step now. The first thing we do is use Echo, which integrates through a host agent on the SQL Server database. This can also be used with Oracle, and in the near future with Postgres, MySQL, and similar database platforms.

Dave Larson: Right now we are putting the database into backup mode so we can take an application-consistent snapshot. Once that completes, we mount the disk used by the database, change the database name, mount it to the host, and bring the database online.

Dave Larson: Because we’re using Backstage, you can see each step as it runs. The first step takes about 41 seconds, and that time is driven primarily by placing the database into an application-consistent state. The actual Silk snapshot process takes less than a second.

Dave Larson: Silk snapshots are zero-byte snapshots. They are metadata pointers back to the source data at the time the snapshot was taken. The mounting step is also a very fast operation. Bringing the database online and refreshing the stage copy takes the remaining time.

Dave Larson: End to end, this step completes in just under two minutes. In this run, the snapshot duration was 41 seconds, the refresh was 50 seconds, and we now see roughly 282,000 rows in the staging environment.

Step 2: Transformation

Dave Larson: The next step is transformation. For the demo, we’re taking five separate tables and running an INSERT SELECT to create a fact table with descriptions. This step will not take long because it is intentionally simple for demonstration purposes.

Dave Larson: You can see the new records that we loaded into the source now available for transformation.

Step 3: Publish to Downstream Environments

Dave Larson: The next step is the publish step. This is where we create copies of the EDW database and put them on the AI and reporting clone environments.

Dave Larson: This process follows the same pattern you saw earlier: Silk takes a snapshot and then publishes the downstream clones. I’ll start that now.

Dave Larson: While this runs, I’ll switch to the Silk Echo user interface. The publish step will take about two minutes.

Silk Echo and Constellation View

Dave Larson: Within Silk, we have a visualization capability called Constellation, which is part of the Echo product. This view shows the host relationships and data lineage in the environment.

Dave Larson: Here is the Silk EDW server, and here is the stage copy that we created through a snapshot from the Silk production server. You can see the database, the snapshot we created, and where that snapshot is mounted.

Dave Larson: On the other side, we’re creating a new snapshot that will update the Silk EDW, Silk EDW AI, and Silk EDW Report databases on the downstream hosts.

Dave Larson: Echo also lets us look at the data from a database-specific view. We can go directly to the Silk database and see the database that was updated with the current snapshot. Through either the UI or API, we can refresh a database, create a new snapshot, and create nested tiers of thin clones.

Dave Larson: Silk supports snapshots and thin clones of thin clones. That is useful, for example, if you need to make a copy of the EDW, mask sensitive data such as Social Security numbers, and then create additional snapshots from that masked copy for lower environments.

Dave Larson: At this point, the Silk EDW AI and Silk EDW Report clone processes have completed. Back in Backstage, both database servers are fully refreshed, and the workflow logs show the steps that were run.

Final Validation and Full Automation

Dave Larson: Now we’ll run a final validation step. This simply shows that the downstream environments have the expected records after the workflow completes.

Dave Larson: So far, we’ve walked through the ETL process step by step. In practice, all of this can be automated. You do not need a person clicking through each step.

Dave Larson: I’ll load another 10,000 records and then run the full workflow. Instead of manually stepping through the process, the workflow makes API calls to the Silk user interface and to SQL Server. This could be the nightly job you already run, or it could be scheduled more frequently.

Dave Larson: The point is that all of these steps can be orchestrated and fully automated. You do not have to use Backstage. You can use any portal, programming language, workflow tool, or automation framework that can call REST APIs. That includes tools such as Terraform, Chef, Puppet, or your existing ETL orchestration platform.

Dave Larson: Silk has a fully published API on GitHub, a Terraform provider, and examples showing how to integrate with different automation tools.

Damon: Excellent. Esteban, Damon here. Anything else you would add while this is running?

Esteban Cordero: Thank you, Dave. That was an excellent walkthrough with a lot of helpful detail. I’ll also remind the audience that if you have questions, please use the Q&A feature and I’ll relay them to the team.

Damon: One thing I’d like to highlight is the visualization Dave has been showing, called Constellation. I think it’s a really compelling capability. Dave, could you spend another moment on that?

A Closer Look at Constellation

Dave Larson: Of course. One thing we wanted to make sure customers could do is visualize what is happening across their data copies.

Dave Larson: I’m changing the view now to show the database snapshot view. This allows you to see snapshots by host. For example, on the Silk EDW server, the EDW database has a couple of snapshots that it depends on. One is mounted to the host, and if I right-click, I can get properties and see details such as the source snapshot, database host, and creation time.

Dave Larson: Instead of looking only at tables, Constellation gives you a visual representation of everything connected to the Silk Data Pod. The Silk Data Pod is the virtual SAN that Silk deploys inside AWS, Azure, or GCP.

Dave Larson: Constellation is both a visualization layer and an interactive tool. In the hierarchical view, we can see the Silk DB server and the snapshots we created. If I need to create a new snapshot right now, I can do that through the API or directly through the UI.

Dave Larson: For this example, I’ll create a crash-consistent snapshot for speed. Silk looks at all databases on the host, and I can select one database or all databases. I’ll select the Silk DB snapshot and create it. In a few seconds, the new snapshot appears in the visualization.

Damon: That’s fantastic. It’s a useful way to show the lineage, or even the genealogy, of the data set.

Damon: The last point I want to make is about the developer portal, Backstage. We’ve heard from customers that Backstage is part of their infrastructure, so we chose to build this demo around that portal. But it is not required. Any interface, portal, programming language, or tool that can call REST APIs can integrate with Silk.

Dave Larson: Exactly. The full workflow just completed in about 225 seconds, a little under four minutes. The only step that will vary significantly is the transformation step. The Silk snapshot and clone operations remain consistent whether the database is 10 GB, 10 TB, or 50 TB.

Dave Larson: For example, one healthcare customer reduced a 15-hour end-to-end ETL process to about eight hours. The remaining time is the actual ETL transformation work. Silk removed the extraction and load overhead. That customer uses the green-gold copy approach, where the data warehouse is only offline for about 15 minutes around 6:00 a.m. while they refresh it. Users can continue working with the previous day’s data warehouse copy during the refresh window.

Q&A

Damon: Let’s open it up for questions.

Damon: The first question is about snapshot time. Dave, you touched on this earlier, but could you spend another moment explaining how Silk creates snapshots almost independently of volume size, and how that differs from native cloud storage approaches?

Dave Larson: When Silk takes a snapshot, we take a snapshot of the metadata that points to the individual blocks used by a volume. That gives us a point-in-time image. We can have tens of snapshots, whether they are hourly, every five minutes, or on demand.

Dave Larson: Once the snapshot is taken, it is a set of metadata pointers. We do not have to consume full additional capacity on the SAN at that moment. The database disk can continue to change, and we still have that point-in-time snapshot.

Dave Larson: We’re showing this in an ETL process, but customers also use snapshots and thin clones for dev/test copies, reporting environments, AI environments, and lower-level database environments.

Dave Larson: It does not matter whether the source is 10 GB or 10 TB. The time to take the snapshot is effectively the same. The only reason Echo snapshots can take longer is if we choose an application-consistent snapshot. In that case, Echo connects to the database and tells it to enter a quiesced state before the snapshot is taken.

Dave Larson: Some customers determine that crash-consistent snapshots are sufficient for their needs. In that case, the database behaves as if it lost power and then recovers. In our example, that would not have taken 36 or 41 seconds; it would have taken only two or three seconds.

Damon: Thank you. The next question is about ETL tools. What changes, if any, do customers need to make to existing ETL tools such as Informatica, ODI, or Airflow when moving to Silk? Related to that, what about ETL patterns where transformations run inside the database engine rather than in the ETL tool?

Dave Larson: If you look at ETL tools as they were first developed, they were designed to move data over the network. As Esteban mentioned earlier, there were different approaches: sometimes backup and restore, sometimes physically moving data with SELECT statements into staging tables.

Dave Larson: What Silk changes is the extraction step. Instead of pulling data across the network from the source, we make the source data available locally on the EDW server so the transformation can run there. We are shortening that window. A step that may have taken four or five hours can be reduced to 30 or 45 seconds because we are eliminating the bulk data movement.

Dave Larson: This is especially important in the cloud. If the source application and the data warehouse are in different resource groups, VNets, or regions, you may be paying for data movement. Silk can help eliminate that unnecessary movement.

Dave Larson: If transformations run directly inside the database, that does not really change. You still do the transformation work. In this demo, we ran a few SQL queries as the transformation step. With Informatica or another ETL tool, you would typically replace or augment the extraction pre-step with a Silk job. Most ETL tools can call external scripts or APIs, so the tool can trigger the Silk extraction workflow instead of using Backstage.

Dave Larson: The exact implementation depends on the ETL tool, but the concept is the same: Silk handles the data copy and refresh pattern, while your transformation logic continues to run where it makes sense.

Esteban Cordero: The important point is that transformation work can be resource intensive. Offloading that work to a secondary server, such as the EDW server in this demo, leaves the production database alone. Users do not experience the same service-level degradation that they might see if heavy ETL jobs were running directly against production.

Dave Larson: I’ll add one more point. Silk storage is typically 2x to 20x faster than cloud-based disk storage for many workloads because it provides very low-latency disk access and a scalable architecture. Customers moving from cloud disks such as Hyperdisk, Ultra Disk, Premium SSD v2, or EC2-attached storage often get more bandwidth and IOPS with Silk.

Dave Larson: That means even transformation steps that read and load data inside the database can run faster because they are not waiting on storage latency. We did not focus on that today, but it is an underlying benefit of the Silk platform.

Damon: That is an excellent point. Fundamentally, Silk provides a traditional enterprise storage experience in cloud environments. That includes pooled throughput, performance, data reduction, data duplication, replication, snapshots, and clones. Those benefits apply to ETL workloads and many other database workloads as well.

Live Product Confirmation and Closing

Damon: I also want to reiterate that what you saw today was live. This was not a click-through UI. Real database work was happening behind the scenes through Silk Echo and the relevant APIs.

Damon: Dave created an additional snapshot during the session, and it appeared in the product. There are live actions we can take directly in the Silk interface.

Dave Larson: That’s right. Rather than using the API call this time, I came into Silk directly and told it to do a refresh. I selected the most recent snapshot that I created manually a few minutes ago.

Dave Larson: The system is running through the refresh process now. It is unmapping the current drive, mounting the new thin clone from the new snapshot, and bringing the database online. Once I refresh the view, you can see that it is now connected to the new snapshot instead of the previous one.

Damon: Perfect. I love it. We’re just about at time, so we’ll wrap up here.

Damon: Our focus today was modern ETL pipelines across AWS, Azure, and Google Cloud. Silk supports all three major hyperscalers, including multi-cloud combinations. For example, some customers run production in AWS and disaster recovery in Azure.

Damon: Hopefully this demo illustrated another use case for Silk enterprise storage: accelerating workloads, reducing effort, integrating with existing tools, and ultimately helping customers save money.

Damon: If you’re interested in exploring the technology further, please reach out through email, LinkedIn, or the Silk website. We’ll publish the recording on the website as well. Thank you to Dave and Esteban for the focus and support, and thank you to everyone who attended.

Dave Larson: Thank you.

Esteban Cordero: Thank you.

Damon: Thanks, everyone. Goodbye.