Building a Hybrid Cloud Active Archive with Cloudian HyperStore

Summary

This article provides a comprehensive guide to building a hybrid cloud active archive using Cloudian HyperStore, inspired by the WGBH Boston case study. We will explore the steps involved in transitioning from traditional archives to a more modern, scalable, and cost-effective solution. By leveraging object storage and hybrid cloud capabilities, you can optimize media workflows, enhance data protection, and simplify archive management.

Scalable storage that keeps up with your ambitionsTrueNAS.

** Main Story**

Building a Hybrid Cloud Active Archive with Cloudian HyperStore: A Step-by-Step Guide

WGBH Boston, a prominent public broadcaster, faced challenges managing its vast media archive. Their legacy system, reliant on tape libraries and external hard drives, struggled to keep pace with increasing ingest rates, the transition to 4K video, and the need for faster media retrieval. Their solution? A hybrid cloud active archive built around Cloudian HyperStore. Follow these steps to build your own hybrid cloud active archive:

1. Assess Your Current Archive and Define Requirements

• Analyze your existing infrastructure: Evaluate the current size of your archive, the rate of data growth, and the types of data you store. Identify pain points, such as slow retrieval times, complex workflows, and escalating storage costs. WGBH, for example, faced long retrieval times, limited scalability, and inadequate data protection with their tape-based archive.
• Define your objectives: Outline specific goals for your new archive. These could include improving access speeds, enhancing scalability, strengthening data protection, and reducing costs. WGBH aimed to accelerate media workflows, simplify disaster recovery, and reduce storage footprint.
• Determine storage needs: Estimate the capacity required for your active archive, considering both current and future data growth. Consider factors like data retention policies and the adoption of higher-resolution media formats. WGBH initially deployed a three-petabyte Cloudian HyperStore cluster.

2. Choose a Hybrid Cloud Approach

• On-premises object storage: Select an on-premises object storage platform that offers scalability, high performance, and compatibility with cloud storage services. Cloudian HyperStore provides S3 compatibility, limitless scalability, and robust data management features, making it an ideal choice for active archives.
• Cloud storage for disaster recovery: Choose a cloud storage service for replicating your archive data for disaster recovery purposes. Consider factors such as cost, data durability, and ease of integration with your on-premises storage. WGBH utilized Amazon S3 and Glacier for their cloud storage needs.
• Integration and automation: Ensure seamless integration between your on-premises object storage and the chosen cloud service. Implement automated data replication and lifecycle management policies to streamline workflows and minimize manual intervention. WGBH automated data replication to Amazon Glacier using HyperStore’s policy-based management.

3. Implement Cloudian HyperStore

• Deployment options: Choose the appropriate HyperStore deployment model based on your specific requirements. Options include software-defined deployments on commodity hardware, pre-configured appliances, and virtualized deployments.
• Scalability and performance: Configure your HyperStore cluster to meet your performance and capacity needs. HyperStore’s scale-out architecture allows you to easily expand capacity by adding nodes to the cluster. Consider factors such as network bandwidth, storage media, and the expected workload.
• Data management features: Leverage HyperStore’s data management capabilities, such as data tiering, lifecycle management, and metadata tagging. These features help you optimize storage utilization, automate data migration, and improve search and retrieval. WGBH utilized metadata tagging to significantly enhance search capabilities.

4. Integrate with Cloud Storage

• Data replication: Configure automated data replication between HyperStore and your chosen cloud storage service. Establish replication schedules and policies to ensure data consistency and minimize the risk of data loss.
• Data lifecycle management: Implement data lifecycle management policies to automatically move data between different storage tiers based on access frequency and age. This helps optimize storage costs and ensures that frequently accessed data resides on the most performant storage tier.
• Disaster recovery planning: Develop a comprehensive disaster recovery plan that incorporates your hybrid cloud archive. Test your recovery procedures regularly to ensure that you can restore data quickly and efficiently in the event of a disaster.

5. Optimize Media Workflows

• Integration with media asset management (MAM) systems: Integrate HyperStore with your existing MAM system to streamline media workflows. This allows you to directly access archived content from within your MAM system, eliminating the need for manual file transfers.
• High-performance access: Leverage HyperStore’s high-performance access to accelerate media workflows. The fast retrieval times enable editors and producers to access archived content quickly, reducing production delays.
• Metadata-driven search: Utilize metadata tagging to enhance search and retrieval capabilities. Tagging media files with relevant metadata allows users to quickly locate specific content based on various criteria, improving productivity.

By following these steps, you can build a robust and scalable hybrid cloud active archive with Cloudian HyperStore. This approach offers significant advantages over traditional archive solutions, including improved scalability, enhanced data protection, and streamlined media workflows. The WGBH Boston case study serves as a compelling example of how this technology can transform archive management and drive operational efficiency.