Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[New] Using OBJ with CDN guides #7111

Merged
merged 9 commits into from
Oct 7, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
slug: optimizing-obj-bucket-architecture-for-akamai-cdn
title: "Optimizing Object Storage Bucket Architecture for Akamai CDN"
description: "This guide discusses design strategies and best practices for optimizing Linode Oject Storage bucket architecture for integrating with Akamai CDN."
authors: ["Akamai"]
contributors: ["Akamai"]
published: 2024-09-27
keywords: ['object storage','cdn','delivery','linode object storage','akamai cdn','akamai connected cloud','bucket architecture']
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
external_resources:
- '[Object Storage Product Documentation](https://techdocs.akamai.com/cloud-computing/docs/object-storage)'
- '[Akamai Content Delivery Documentation](https://techdocs.akamai.com/platform-basics/docs/content-delivery)'
- '[Using Object Storage With Akamai CDN](/docs/guides/using-object-storage-with-akamai-cdn/)'
---

Linode Object Storage can be an efficient, cost-effective solution for streaming and data delivery applications when used as an origin point for Akamai CDN. Since Object Storage is a part of Akamai Connected Cloud and uses the same backbone as Akamai CDN, egress can also be significantly reduced.

Your Object Storage bucket architecture is critical to performance success. Distributing content across buckets helps with load distribution, CDN optimization, and adds security benefits like segmentation and origin obfuscation. This guide walks through bucket design strategies using a commerce site example, including an optimal bucket architecture approach for Akamai CDN integration.

## How Object Storage Works

[Object Storage](https://techdocs.akamai.com/cloud-computing/docs/object-storage) stores files in a “flat” or unstructured file structure. This means that bucket contents do not exist in a hierarchy like traditional file structures; however, hierarchy can be emulated by creating folders within a bucket. Files (or “objects”) are stored alongside their rich metadata, and access can be granted on a per-object level, with each object receiving its own URL.

With this in mind, the bucket architecture you use is a crucial element for your application and general CDN integration. The more efficient your bucket design, the better the CDN will perform.

For up-to-date technical specifications of Linode Object Storage such as bucket and rate limits, see: [Object Storage: Technical Specifications and Considerations](https://techdocs.akamai.com/cloud-computing/docs/object-storage#technical-specifications-and-considerations)

## Bucket Architecture Strategies

### Example Scenario

For the example bucket architecture, consider the following scenario:

- An online commerce site serving audio, video, and image-based content
- All audio, video, and image files are stored on object storage as an origin point

### What "Not" To Do

In the first bucket example, the top-level, root directory `store` has paths to three lower-level directories containing different types of content: music, video, and images. `store` and its sub-directories are in a single bucket with a defined requests per second (RPS) limit, where the RPS limit is shared by all objects in the bucket.

Paths in this bucket are: `store/music`, `store/video`, and `store/images`.

![Single Bucket Arch Level 1](Single-bucket-arch-level1.png)

Continuing with this approach, additional content such as copyrighted music or free music may logically be placed in the `/music` directory and would continue on for each sub-directory.

Paths in this bucket now include: `store/music`, `store/music/copyright`, `store/music/free`, `store/video`, and `store/images`

![Single Bucket Arch Level 2](Single-bucket-arch-level2.png)

This architecture places content paths and object endpoints all within the same bucket, `store`. This may potentially result in poor performance with rate limits for the bucket being reached as the number of users and requests increases.

### What To Do & How To Leverage CDN Advantages

#### CDN Offloading

One way to overcome these issues is by utilizing Akamai CDN and its ability to cache and deliver content closer to end users. Once the CDN caches an object like a song, video, or image, it doesn’t need to be pulled from origin storage again until the object file changes or the cache timeout expires. For example, by offloading requests to the CDN, a single mp3 audio file downloaded 1 million times only requires a single request to the source bucket.

#### Distributed Bucket Architecture

While Akamai CDN can help get around some limitations of a single-bucket architecture, you can improve and optimize your object storage performance even further by **distributing content across multiple buckets**. Doing this results in:

- Overall load distribution
- Increased requests-per-second (RPS) capacity
- Segmentation of content across endpoints can act as a security measure in case of compromise
- Distribution of the number of single endpoint requests from the CDN

Rather than placing music, video, and image content folders within the same root directory, you can triple [RPS capacity](https://techdocs.akamai.com/cloud-computing/docs/object-storage#technical-specifications-and-considerations) by placing each type of content (music, video, and images) in their own buckets. In the example below, additional content is placed underneath each top-level directory (`store-music`, `store-video`, and `store-images`).

- Paths for the `store-music` bucket include: `store-music`, `store-music/copyright`, and `store-music/free`
- Paths for the `store-video` bucket include: `store-video`
- Paths for the `store-images` bucket include: `store-images`

![Multi-Bucket Arch 1](Multi-bucket-arch1.png)

As the site scales, you can create additional buckets and move sub-category content (such as copyrighted or free music) to their *own buckets*. This further increases RPS capacity while further distributing the number of endpoints from which the CDN can cache content. In the example below, RPS capacity is now four times the original bucket architecture.

- Paths for the `copyright` bucket include: `copyright`
- Paths for the `store-music` bucket now include: `store-music` and `store-music/free`
- Paths for the `store-video` bucket include: `store-video`
- Paths for the `store-images` bucket include: `store-images`

![Multi-Bucket Arch 2](Multi-bucket-arch2.png)

## CDN & the Relationship To Bucket Design

### CDN Considerations

Each bucket in your architecture has the ability to serve as a single origin endpoint from which Akamai CDN can pull content. This results in a distributed backend with less opportunities for RPS limits to be reached. The more distributed your backend architecture is, the higher the capacity for scaling.

### Relationship To Bucket Design

Bucket design is strictly architectural, and CDNs can often overcome flaws of poorly architected environments. However, when bucket architecture is designed well, the benefits can directly translate to the CDN. Object Storage bucket architecture should be designed to be functional, scalable, performant, resilient, and cost efficient so that Akamai CDN serves your content as effectively as possible.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
slug: using-object-storage-with-akamai-cdn
title: "Using Object Storage With Akamai CDN"
description: "This guide presents the benefits and strategies for pairing Linode Object Storage with Akamai CDN for data delivery."
authors: ["Akamai"]
contributors: ["Akamai"]
published: 2024-09-27
keywords: ['object storage','cdn','delivery','linode object storage','akamai cdn','akamai connected cloud']
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
external_resources:
- '[Object Storage Product Documentation](https://techdocs.akamai.com/cloud-computing/docs/object-storage)'
- '[Akamai Content Delivery Documentation](https://techdocs.akamai.com/platform-basics/docs/content-delivery)'
---

Object storage stores data in an unstructured, flat format where data is stored as “objects”. Object storage is cheaper than block storage, and it can be especially useful for large volumes of unstructured data like photos, videos, documents, web pages, sensor data, and audio files. Cloud-based object storage systems distribute this unstructured data across multiple physical devices for redundancy, while users are able to access the content from a single interface. Object storage is HTTP accessible and ideal for building cloud-native applications that require scale and flexibility, as well as storage for existing data like analytics, backups, and archives.

With its reliability and cost-effectiveness, Linode Object Storage can be coupled with Akamai’s content delivery network (CDN) to create a viable solution for applications that need to serve large files and large amounts of data. This includes video on demand (VOD) streaming, ecommerce, firmware updating, media-based website content, and other applications.

The solution presented in this guide focuses on the architectural best practices for designing and operating reliable, secure, efficient, and sustainable content storage and delivery systems using [Linode Object Storage](https://techdocs.akamai.com/cloud-computing/docs/object-storage) on Akamai Connected Cloud with [Akamai CDN](https://techdocs.akamai.com/platform-basics/docs/content-delivery).

## Object Storage For Data Delivery

With the right [bucket architecture](/docs/guides/optimizing-obj-bucket-architecture-for-akamai-cdn/), object storage can be used to house content for effective unstructured data delivery. Object storage supports file-critical features such as encryption, compression, deduplication, and versioning, and its accessibility via HTTP protocols offers instant access to objects.
jddocs marked this conversation as resolved.
Show resolved Hide resolved

For example, these factors make object storage ideal for storing unstructured data like video and audio files that don’t require frequent updating and services that require efficient input/output.

## Object Storage Specifications, Considerations, and Strategies for Streaming Data

### Technical Specifications

When considering applications -- like streaming video or audio -- that involve sourcing data from object storage, there are a few things to keep in mind, including content access, bandwidth limitations, latency sensitivity, scalability, egress costs, and other technical limitations:

- **Bandwidth and latency:** Object storage solutions are not always optimized for high-bandwidth streaming applications. This can lead to potential buffering issues and slower delivery speeds.

- **Scalability:** To preserve infrastructure integrity, object storage systems may have limitations in place for concurrent connections and simultaneous user streams. This can impact the ability to efficiently handle spikes in demand.

- **Retrieval speed:** Without CDN edge-based caching, content streamed directly from object storage origin servers may result in slower data retrieval speeds and higher latency.

- **Data transfer and egress costs:** Streaming directly from object storage can incur high egress costs since each user request may result in data transfer fees, whereas CDNs cache data at the edge to optimize delivery.

- **Resource limits:** Object storage systems may have limitations on the size of individual objects, potentially restricting the size of videos that can be efficiently streamed.

- **Rate limits:** Object storage solutions often have rate limits imposed on a requests per second (RPS) basis. If users are accessing Object Storage endpoints directly without the use of a CDN, there is a greater chance RPS limits are reached as more users access content.

- **Updating manifests:** Bucket contents can be determined by syncing with a manifest. This means that when source manifests are updated often, buckets may constantly have their contents changed.

### Internet Considerations
jddocs marked this conversation as resolved.
Show resolved Hide resolved

There are multiple internet-related factors to consider when building out a content delivery solution:

- **First mile:** The content provider connection point. Should content providers set up content to be accessible from a single physical location (i.e. an origin data center), user access is limited by “first mile” connectivity limitations.

- **Last mile:** The end-user connection point. Slow user internet connection speeds are uncontrollable and unpredictable. They can also mask or obfuscate problems related to the “first mile”, network peering, and bottleneck issues.

- **Peering points:** The connection points between two networks. These connections are not guaranteed and are difficult to troubleshoot. Networks benefit financially from connectivity with content providers and end users but not each other.

- **Backbone:** The infrastructure and physical connections that make up the internet. Hardware and network capacity limitations can cause performance issues.

### Data Center-Based Architectures

Consider the data center-based architecture below. Without the use of a CDN, object storage content distribution is limited to individual, origin data centers. Data center-based delivery relies heavily on public networks to pull and deliver content from regional origin points. At scale, this may result in bandwidth issues, high latency, and inefficient experiences for large numbers of users.

![DC-Based Architecture](DC-Based-Architecture.png)

## Advantages of Using Akamai CDN
jddocs marked this conversation as resolved.
Show resolved Hide resolved

CDNs are crucial for data delivery solutions because they cache content like images, videos, and other files on edge servers closer to end-users. This reduces latency and overall load on origin servers, improves delivery speed and efficiency, and saves money by reducing origin requests and lowering egress. For example, in use cases such as content streaming, CDNs can cache segments from streams to reduce startup times, limit stream interruptions, and eliminate potential buffering issues.

Object storage can’t scale linearly with the number of users, but using it as an origin point for CDN delivery overcomes this by allowing users to interact with the CDN instead of the object storage system. Applications can then store objects independently of user traffic, with the added benefit of CDN-based security. Using Linode Object Storage on Akamai Connected Cloud with Akamai CDN can also significantly reduce object storage-based egress costs.
jddocs marked this conversation as resolved.
Show resolved Hide resolved

### CDN-Based Distributed Architecture

Akamai’s distributed architecture brings content closer to end-users and offers several technical benefits for streaming applications, including:
jddocs marked this conversation as resolved.
Show resolved Hide resolved

- Audience and end-user geographic reach
- Programmability through well-defined APIs
- Flexibility to support structured and semi-structured data
- Scalable consumption models
- Efficient data recovery and retrieval based on detailed metadata
- Capacity advantages via distribution across multiple nodes
- Enhanced data durability and availability through replication and distribution mechanisms
- Limiting reliance on transit networks
- Origin offloading, content caching, and reduced latency

In the CDN-based architectures below, Akamai CDN lives between end users and the origin data point. As end users request information, Akamai CDN sources and caches data from Linode Object Storage on Akamai Connected Cloud, and then serves that data from an edge server close to the end user. In this scenario, latency is reduced, users never need to interact with the origin, and the object storage infrastructure remains protected and optimized for efficiency and growth.

![CDN-Based Architecture](CDN-Based-Architecture.png)

![Edge vs. DC Latency](Edge-vs-DC-latency.png)

## Using Akamai CDN with Object Storage for Data Delivery

Object storage is a standardized, key component in many content production tools, enabling seamless integration and organization of large amounts of files. Its interoperability across various applications, tools, and workflows ensures data is centralized and easily accessible. Object storage simplifies content distribution and backup processes, and it provides a secure, cost-effective method for storing and sharing large volumes of unstructured data.

When architected properly and paired with Akamai’s CDN, Linode Object Storage allows for fast, efficient access to vast libraries of content connected on the same backbone. This results in readily available content - a crucial factor for streaming and other data delivery workflows.

## Next Steps

To learn more about object storage bucket design for streaming and steps for CDN integration, see the following guide:

- [Optimizing Object Storage Bucket Architecture for Akamai CDN](/docs/guides/optimizing-obj-bucket-architecture-for-akamai-cdn/)
Loading