-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
9 changed files
with
183 additions
and
0 deletions.
There are no files selected for viewing
Binary file added
BIN
+165 KB
...ject-storage/how-to-optimize-bucket-architecture-for-cdn/Multi-bucket-arch1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+170 KB
...ject-storage/how-to-optimize-bucket-architecture-for-cdn/Multi-bucket-arch2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+142 KB
...orage/how-to-optimize-bucket-architecture-for-cdn/Single-bucket-arch-level1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+152 KB
...orage/how-to-optimize-bucket-architecture-for-cdn/Single-bucket-arch-level2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
88 changes: 88 additions & 0 deletions
88
...es/platform/object-storage/how-to-optimize-bucket-architecture-for-cdn/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
--- | ||
slug: how-to-optimize-bucket-architecture-for-cdn | ||
title: "How to Optimize Bucket Architecture for CDN" | ||
description: "Two to three sentences describing your guide." | ||
og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." | ||
authors: ["Akamai"] | ||
contributors: ["Akamai"] | ||
published: 2024-09-27 | ||
keywords: ['list','of','keywords','and key phrases'] | ||
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' | ||
external_resources: | ||
- '[Link Title 1](http://www.example.com)' | ||
- '[Link Title 2](http://www.example.net)' | ||
--- | ||
|
||
Linode Object Storage can be an efficient, cost-effective solution for streaming and data delivery applications when used as an origin point for Akamai CDN. Since Object Storage is a part of Akamai Connected Cloud, egress can also be significantly reduced. | ||
|
||
Your Object Storage bucket architecture is critical to performance success. Distributing content across buckets helps with load distribution, CDN optimization, and adds security benefits like segmentation and origin obfuscation. This guide walks through bucket design strategies using a commerce site example, including optimal bucket architecture for Akamai CDN integration. | ||
|
||
## How Object Storage Works | ||
|
||
[Object Storage](https://techdocs.akamai.com/cloud-computing/docs/object-storage) stores files in a “flat” or unstructured file structure. This means that bucket contents do not exist in a hierarchy like traditional file structures, however, hierarchy can be emulated by creating folders within a bucket. Files (or “objects”) are stored alongside their rich metadata, and access can be granted on a per-object level, with each object receiving its own URL. | ||
|
||
With this in mind, the bucket architecture you use is a crucial element for your application and general CDN integration. | ||
|
||
For up-to-date technical specifications of Linode Object Storage such as bucket and rate limits, see: [Object Storage: Technical Specifications and Considerations](https://techdocs.akamai.com/cloud-computing/docs/object-storage#technical-specifications-and-considerations) | ||
|
||
## Bucket Architecture Strategies | ||
|
||
### Example Scenario | ||
|
||
For the example bucket architecture, consider a commerce site serving audio, video, and image-based content. | ||
|
||
### What "Not" To Do | ||
|
||
In the first bucket example, the top-level, root directory “store” has paths to three lower-level directories containing different types of content: music, video, and images. | ||
|
||
Paths in this bucket are: `store/music`, `store/video`, and `store/images` | ||
|
||
![Single Bucket Arch Level 1](Single-bucket-arch-level1.png) | ||
|
||
Continuing with this approach, additional content such as copyrighted music or free music may logically be placed in the music directory and would continue on for each sub-directory. | ||
|
||
Paths in this bucket now include: `store/music`, `store/music/copyright`, `store/music/free`, `store/video`, and `store/images` | ||
|
||
![Single Bucket Arch Level 2](Single-bucket-arch-level2.png) | ||
|
||
This architecture places content paths all within the same bucket (“store”). This may potentially result in poor performance with rate limits for the bucket being reached as the number of users and requests increases. | ||
|
||
One way to overcome these issues is by utilizing Akamai CDN and its ability to cache and deliver content closer to end users. Once the CDN caches an object (like a song, video, or image), it doesn’t need to be pulled from origin storage again until the object file changes or the cache timeout expires. | ||
|
||
- **For example:** This offloading means that a single mp3 audio file downloaded 1 million times would only require a single request to the bucket. | ||
|
||
### What To Do & How To Leverage CDN Advantages | ||
|
||
The better way to architect your bucket is to **distribute content across multiple buckets** to improve performance. Doing this results in: | ||
|
||
- Overall load distribution | ||
- Increased requests-per-second (RPS) capacity | ||
- Segmentation of content across endpoints can act as a security measure in case of compromise | ||
- Distribution of the number of single endpoint requests from the CDN | ||
|
||
Rather than placing music, video, and image content folders within the same root directory, you can triple [RPS capacity](https://techdocs.akamai.com/cloud-computing/docs/object-storage#technical-specifications-and-considerations) by placing each type of content (music, video, and images) in their own buckets. In the example below, additional content can be placed underneath each top-level directory (store-music, store-video, and store-images). | ||
|
||
- Paths for the store-music bucket include: `store-music`, `store-music/copyright`, and `store-music/free` | ||
- Paths for the store-video bucket include: `store-video` | ||
- Paths for the store-images bucket include: `store-images` | ||
|
||
![Multi-Bucket Arch 1](Multi-bucket-arch1.png) | ||
|
||
As the site scales, you can create additional buckets and move sub-category content (such as copyrighted or free music) to their own buckets. This further increases RPS capacity while further distributing the number of endpoints from which the CDN can cache content. In the example below, RPS capacity is now 4x the original bucket architecture. | ||
|
||
- Paths for the copyright bucket include: `copyright` | ||
- Paths for the store-music bucket now include: `store-music` and `store-music/free` | ||
- Paths for the store-video bucket include: `store-video` | ||
- Paths for the store-images bucket include: `store-images` | ||
|
||
![Multi-Bucket Arch 2](Multi-bucket-arch2.png) | ||
|
||
## CDN & the Relationship To Bucket Design | ||
|
||
### CDN Considerations | ||
|
||
Each bucket in your architecture has the ability to serve as a single origin endpoint from which Akamai CDN can pull content. This results in a distributed backend with less opportunities for RPS limits to be reached. The more distributed your backend architecture is, the higher the capacity for scaling. | ||
|
||
### Relationship To Bucket Design | ||
|
||
Bucket design is strictly architectural, and CDNs can often overcome flaws of poorly architected environments. However, when bucket architecture is designed well, the benefits can directly translate to the CDN. Object Storage bucket architecture should be designed to be functional, scalable, performant, resilient, and cost efficient so that Akamai CDN serves your content as effectively as possible. |
Binary file added
BIN
+130 KB
.../object-storage/using-object-storage-with-akamai-cdn/CDN-Based-Architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+122 KB
...m/object-storage/using-object-storage-with-akamai-cdn/DC-Based-Architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+121 KB
...form/object-storage/using-object-storage-with-akamai-cdn/Edge-vs-DC-latency.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
95 changes: 95 additions & 0 deletions
95
docs/guides/platform/object-storage/using-object-storage-with-akamai-cdn/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
--- | ||
slug: using-object-storage-with-akamai-cdn | ||
title: "Using Object Storage With Akamai CDN" | ||
description: "Two to three sentences describing your guide." | ||
authors: ["Akamai"] | ||
contributors: ["Akamai"] | ||
published: 2024-09-27 | ||
keywords: ['list','of','keywords','and key phrases'] | ||
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' | ||
external_resources: | ||
- '[Object Storage Product Documentation](https://techdocs.akamai.com/cloud-computing/docs/object-storage)' | ||
- '[Akamai Content Delivery Documentation](https://techdocs.akamai.com/platform-basics/docs/content-delivery)' | ||
--- | ||
|
||
Object storage stores data in an unstructured, flat format where data is stored as “objects”. Object storage is cheaper than block storage, and can be especially useful for large volumes of unstructured data like photos, videos, documents, web pages, sensor data, and audio files. Cloud-based object storage systems distribute this unstructured data across multiple physical devices for redundancy, while users are able to access the content from a single interface. Object storage is HTTP accessible and ideal for building cloud-native applications that require scale and flexibility, as well as storage for existing data like analytics, backups, archives, and more. | ||
|
||
With its reliability and cost-effectiveness, Linode Object Storage can be coupled with Akamai’s content delivery network (CDN) to create a viable solution for applications that need to serve large files and large amounts of data. This includes applications like video on demand (VOD) streaming, ecommerce, firmware updating, media-based website content, and more. | ||
|
||
The solution presented in this guide focuses on the architectural best practices for designing and operating reliable, secure, efficient, and sustainable content storage and delivery systems using Linode Object Storage on Akamai Connected Cloud with Akamai CDN. | ||
|
||
## Object Storage For Data Delivery | ||
|
||
With the right bucket architecture, object storage can be used to house content for effective unstructured data delivery. Object storage supports file-critical features such as encryption, compression, deduplication, and versioning, and its accessibility via HTTP protocols offers instant access to objects. | ||
|
||
For example, these factors make object storage ideal for storing unstructured data like video and audio files that don’t require frequent updating and services that require efficient input/output. | ||
|
||
## Object Storage Specifications, Considerations, and Strategies | ||
|
||
When considering applications like streaming that involve sourcing data from object storage, there are a few things to keep in mind, including content access, bandwidth limitations, latency sensitivity, scalability, egress costs, and other technical limitations: | ||
|
||
- **Bandwidth and latency:** Object storage solutions are not always optimized for high-bandwidth streaming applications. This can lead to potential buffering issues and slower delivery speeds. | ||
|
||
- **Scalability:** To preserve infrastructure integrity, object storage systems may have limitations in place for concurrent connections and simultaneous user streams. This can impact the ability to efficiently handle spikes in demand. | ||
|
||
- **Retrieval speed:** Without CDN edge-based caching, content streamed directly from object storage origin servers may result in slower data retrieval speeds and higher latency. | ||
|
||
- **Data transfer and egress costs:** Streaming directly from object storage can incur high egress costs since each user request may result in data transfer fees, whereas CDNs cache data at the edge to optimize delivery. | ||
|
||
- **Resource limits:** Object storage systems may have limitations on the size of individual objects, potentially restricting the size of videos that can be efficiently streamed. | ||
|
||
- **Rate limits:** Linode Object Storage currently has a per bucket rate limit of 750 requests per second (RPS). If users are accessing Object Storage endpoints directly without the use of a CDN, there is a greater chance RPS limits are reached. | ||
|
||
- **Updating manifests:** Bucket contents can be determined by syncing with a manifest. This means that when source manifests are updated often, buckets may constantly have their contents changed. | ||
|
||
![DC-Based Architecture](DC-Based-Architecture.png) | ||
|
||
Without the use of a CDN, object storage content distribution is limited to individual, origin data centers. As shown in the diagram above, data center-based delivery relies heavily on transit networks to pull and deliver content from regional origin points. At scale, this may result in bandwidth issues, high latency, and inefficient experiences for large numbers of users. | ||
|
||
### Internet Considerations | ||
|
||
There are multiple internet-related factors to consider when building out a content delivery solution: | ||
|
||
- **First mile:** The content provider connection point. Should content providers set up content to be accessible from a single physical location (i.e. origin), user access is limited by “first mile” connectivity limitations. | ||
|
||
- **Last mile:** The end-user connection point. Slow user internet connection speeds are uncontrollable and unpredictable, although they can mask problems related to the “first mile”, network peering, and bottleneck issues. | ||
|
||
- **Peering points:** The connection points between two networks. These connections are not guaranteed and are difficult to troubleshoot. Networks benefit financially from connectivity with content providers and end users but not each other. | ||
|
||
- **Backbone:** The infrastructure and physical connections that make up the internet. Hardware and network capacity limitations can cause performance issues. | ||
|
||
## Advantages of Using Akamai CDN | ||
|
||
CDNs are crucial for data delivery solutions because they cache content like images, videos, and other files on edge servers close to end-users. This reduces latency and overall load on origin servers, improves delivery speed and efficiency, and saves money by reducing origin requests and lowering egress. In use cases such as content streaming, CDNs can cache segments from streams to reduce startup times, limit stream interruptions, and eliminate potential buffering issues. | ||
|
||
Object storage can’t scale linearly with the number of users, but using it as an origin point for CDN delivery overcomes this by allowing users to interact with the CDN directly. Applications can then store objects independently of user traffic, with the added benefit of CDN security. | ||
|
||
### CDN-Based Distributed Architecture | ||
|
||
Akamai’s distributed architecture brings content closer to end-users and offers several technical benefits for streaming applications, including: | ||
|
||
- Audience and end-user geographic reach | ||
- Programmability through well-defined APIs | ||
- Flexibility to support structured and semi-structured data | ||
- Scalable consumption models | ||
- Efficient data recovery and retrieval based on detailed metadata | ||
- Capacity advantages via distribution across multiple nodes | ||
- Enhanced data durability and availability through replication and distribution mechanisms | ||
- Limiting reliance on transit networks | ||
- Origin offloading, content caching, and reduced latency | ||
|
||
![CDN-Based Architecture](CDN-Based-Architecture.png) | ||
|
||
![Edge vs. DC Latency](Edge-vs-DC-latency.png) | ||
|
||
## Using Akamai CDN with Object Storage for Data Delivery | ||
|
||
Object storage is a standardized, key component in many content production tools, enabling seamless integration and organization of large amounts of files. Its interoperability across various applications, tools, and workflows ensures data is centralized and easily accessible. Object storage simplifies content distribution and backup processes, and it provides a secure, efficient method for storing and sharing large volumes of media data. | ||
|
||
When architected properly and paired with Akamai’s CDN, object storage allows for fast, cost-effective access to vast libraries of content connected on the same backbone. This results in readily available content - a crucial factor for streaming and other data delivery workflows. | ||
|
||
## Next Steps | ||
|
||
To learn more about object storage bucket design for streaming and steps for CDN integration, see the following guides: | ||
|
||
- [How to Optimize Object Storage Bucket Architecture for CDN]() |