How to Automate Snapshots and Run Orchestration with Open Source Tools

This blog talks about snapshots: What snapshots are, how snapshots are used, and how to use Silk to automate the creation of snapshots in the cloud while using open source tools, like Puppet or Chef, for orchestration.

What are Snapshots and How are Snapshots Used?

Snapshots are point-in-time clones of a database that are brought to non-production environments and used for data protection and backup, as well as for activities related to development, testing, reporting, and analytics. Snapshots can be taken manually, scheduled to run automatically, or triggered via API calls.

Snapshots are commonly used in workflows. Example use cases for using snapshots in workflows include:

  • Sub-environment refreshes
  • Operational recovery with redo logs
  • Snap and presenting snap to an ETL server that can get the data into a data warehouse (synapse)
  • Backup using ABS instances and replication with Silk for DB data
  • Backup environment using also Rubrik, CV or RMAN

Managing and consolidating snapshots is the practice of Copy Data Management (CDM).

How are Snapshots Taken on the Cloud?

Snapshots originated in the world of on-prem datacenters. Today, with cloud computing, there are new considerations to keep in mind when taking snapshots on cloud infrastructure.

Namely, that instantaneous, zero-footprint snapshots that were common on-prem are not a cloud native capability. Snapshots on the cloud are full copies of the dataset that take significant time to deploy and consume costly cloud resources each time they are taken. Database users must keep this in mind, especially when taking multiple copies of data.

Without the right tool for the job, developers can end up with a workflow that is much slower and more expensive to complete compared to on-prem. This can become a major problem for developers who need the agility the cloud provides, as well as the speed, power, and feature/functionality of rich data services they utilized on-prem.

Zero-Footprint Snapshots on the Cloud with Silk

The Silk Cloud Platform combines all of the great snapshotting capabilities of on-prem with the flexibility and scalability of the cloud.

The Silk Cloud Platform is a virtualization layer that sits between the customer’s databases and cloud infrastructure. Its patented technology offers a full suite of enterprise data services, including zero-footprint snapshots, replication, encryption, thin provisioning, data compression, and deduplication.

Silk’s snapshotting features enable the user to create enterprise-class, consistent, instantaneous, point-in-time replicas of data in the public cloud. The snapshots can be mounted to hosts and moved across zones, regions, and clouds. These are full read/write copies that the user can create and iterate against within seconds.

Instead of a complete data copy, which takes a considerable amount of time, costs money, and impacts performance, Silk copies the metadata and only stores the changes. The pointer-based snapshots run fast and there is no performance degradation, no matter how many copies or replications are made. They are also zero-footprint, meaning they do not consume additional cloud capacity or contribute to cloud footprint until changes are written to the copy.

While there are other Copy Data Management utilities that operate in the cloud and utilize redirect on write snapshot capabilities, those utilities require an intermediary server that manages the metadata for snapshots. This also results in a single point of failure and bottlenecks snapshot performance. Instead, Silk utilizes its native metadata technology to provide high performance snapshots without the need for an intermediary server and provides autonomous self-healing for the highest levels of availability in the cloud.

The key benefits of using Silk snapshots include:

  • The ability to create snapshots and run full production workloads
  • High performance copies of data
  • Cost-efficiency of cloud resources
  • Support for all public cloud infrastructure
  • Faster time to innovation and business outcomes

Read the Data Sheet: Zero-Footprint Snapshots on the Cloud with Silk

Customer Example of Using Snapshots in a Production Reporting Workflow

One of Silk’s customers in the healthcare industry utilizes snapshots in their production reporting workflow. They take instantaneous snapshots of their operational database Clarity db01 (SQL Server/ Oracle) and present the snapshot to an ETL server for analytics. The ETL process loads their analytics database Caboodle (SQL Server/Oracle non-prod) and once complete, additional snapshots of the ETL database are created for data validation, reporting, and other processes. All of this is accomplished with instantaneous snapshots at no additional storage costs for the customer. The customer has reduced ETL downtime from 7-10 hours to 15 minutes.

The workflow is depicted below.

Automating Snapshots with Silk and Using Open Source Tools for Orchestration

Many organizations on the cloud have adopted DevOps processes and are using configuration management tools, like Ansible, PowerShell, Puppet, or Chef, to standardize and automate/orchestrate the deployment and configuration of servers and applications.

Using Silk’s snapshot scheduler, the user can automate snapshots and integrate with open source tools to take care of the host-level activities – everything from mounting the volume to restarting the database.

There are various use cases where automated snapshots are used in workflows, such as:

  • Sub-production refreshes
  • Landscape refreshes
  • Data validation and reporting
  • Fast restore for operational recovery

Take a look at how Silk snapshots work and how to run a sub-production refresh in this 2-minute video.

Let’s take another example for a spin. This time it’s a customer who wants to automate the refreshing of Silk snapshot views to their reporting SQL database servers on a daily basis. The customer can use a script with the Silk Volume Shadow Services (VSS) provider to create consistent copies of their SQL databases. The script does the following:

  • Takes a snapshot once the freeze has been issued on the database. The snapshot code is plugged into the customer provided freeze/thaw script.
  • Target databases are gracefully brought offline before removing the view from the previous day.
  • Once the old view is removed, a view created from the newly taken snapshot is mapped to the host.
  • The script performs host-side operations to bring the refreshed database online.

Additional Snapshot Resources

 

Data Sheet: Zero-Footprint Snapshots on the Cloud with Silk

Webinar: Cloud Snapshots 101

Short demos:

To learn more about using Silk’s snapshots in the cloud, check out this Technical Demo of Silk and contact us to speak with a solution architect. We’re glad to share best practices with you and your team members.