Welcome!

Industrial IoT Authors: Pat Romanski, William Schmarzo, Elizabeth White, Stackify Blog, Yeshim Deniz

Related Topics: @CloudExpo

@CloudExpo: Article

Amazon’s Elastic Block Store Opens Up S3 and The Cloud

The Big SAN in the Sky

By Thorsten von Eicken
Article Rating:
August 25, 2008 06:00 AM EDT
Reads:
 35,812
With EBS volumes up to 1TB in size and beyond in a file system is possible to mount quickly – and multiple volumes on the same instance, such that file systems of 10TB are practical. The volumes can further be backed-up to S3 using snapshots, and they can be replicated by creating new volumes from the snapshots. What is particularly nice is that a volume can be created in any availability zone (think datacenter) of a region from a snapshot, so copying a large volume across datacenters can be off-loaded to EBS and is done very efficiently.

Incremental snapshotting of volumes and freezing
I mentioned that snapshots were a very useful but also a complex, difficult to understand feature. I wanted to explain how snapshots of an EBS volume can be taken at any time, and why using RightScale’s scripts to freeze data are important -- and a great add-on to this important feature.

Taking a snapshot causes the data on the volume to be written to S3 where it is stored redundantly in multiple availability zones as all data in S3 is. It’s worth noting snapshots do not appear in your S3 buckets, thus you can't access them using the standard S3 API. You can only list the snapshots using the EC2 API and you can restore a snapshot by creating a new volume from it.

The second thing is that snapshots are incremental, which means that in order to create a snapshot, EBS saves only the disk blocks that have changed to S3.

Each volume is divided up into blocks. When the first snapshot of a volume is taken, all blocks of the volume that have ever been written are copied to S3, and then a snapshot table of contents is written to S3 that lists all these blocks. Now, when the second snapshot is taken of the same volume, only the blocks that have changed since the first snapshot are copied to S3. The table of contents for the second snapshot is then written to S3 and lists all the blocks on S3 that belong to the snapshot. Some are shared with the first snapshot, some are new. The third snapshot is created similarly and can contain blocks copied to S3 for the first, second and third snapshots.

There are two nice things about the incremental nature of the snapshots: it saves time and space. Taking subsequent snapshots can be very fast because only changed blocks need to be sent to S3, and it saves time because you're only paying for the storage in S3 of the incremental blocks. What is difficult to answer is how much space a snapshot uses. Or, to put it differently, how much space would be saved if a snapshot were deleted. If you delete a snapshot, only the blocks that are only used by that snapshot (i.e. are only referenced by that snapshot's table of contents) are deleted.

Something to be very careful about with snapshots is consistency. A snapshot is taken at a precise moment in time even though the blocks may trickle out to S3 over many minutes. But in most situations you will really want to control what's on disk vs. what's in-flight at the moment of the snapshot. This is particularly important when using a database. We recommend you freeze the database (or any application writing critical data to disk), freeze the file system, take the snapshot, then unfreeze everything. At the file system level we've been using xfs for all the large local drives and EBS volumes because it's fast to format and supports freezing. Thus when taking a snapshot we perform an xfs freeze, take the snapshot, and unfreeze. All this ensures that the snapshot doesn't contain partial updates that need to be recovered when the snapshot is mounted.

With support for large datasets, attachments, better throughput, snapshotting and more robust, incremental backups and redundancy, Amazon’s EBS should attract a lot more enterprise and on-demand customers, as well as Web 2.0 users with large database-driven applications.

Thorsten von Eicken is RightScale, Inc.’s Chief Technical Officer. To try out a free developer version of RightScale, visit http://www.rightscale.com/m/products.html#developer.

Published August 25, 2008 – Reads 35,812
Copyright © 2008 SYS-CON Media, Inc. — All Rights Reserved.
Syndicated stories and blog feeds, all rights reserved by the author.

More Stories By Thorsten von Eicken

Thorsten von Eicken is the CTO and a founder of RightScale and is responsible for the overall technology direction of the RightScale Cloud Management Platform. Previously, he was founder and chief architect at Expertcity (acquired by Citrix Online), where he directed the architecture of the company's online services, including the popular GoToMeeting service. von Eicken also managed the Expertcity/Citrix data center operations, where he acquired deep knowledge in deploying and running secure, scalable online services. He was a professor of Computer Science at Cornell University and received his Ph.D. from the University of California, Berkeley.

Comments (1)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

IoT & Smart Cities Stories
By Zakia Bouachraoui
Moroccanoil®, the global leader in oil-infused beauty, is thrilled to announce the NEW Moroccanoil Color Depositing Masks, a collection of dual-benefit hair masks that deposit pure pigments while providing the treatment benefits of a deep conditioning mask. The collection consists of seven curated shades for commitment-free, beautifully-colored hair that looks and feels healthy.
Sep. 6, 2019 09:00 AM EDT
By Zakia Bouachraoui
The textured-hair category is inarguably the hottest in the haircare space today. This has been driven by the proliferation of founder brands started by curly and coily consumers and savvy consumers who increasingly want products specifically for their texture type. This trend is underscored by the latest insights from NaturallyCurly's 2018 TextureTrends report, released today. According to the 2018 TextureTrends Report, more than 80 percent of women with curly and coily hair say they purcha...
Sep. 5, 2019 07:00 PM EDT
By Zakia Bouachraoui
The textured-hair category is inarguably the hottest in the haircare space today. This has been driven by the proliferation of founder brands started by curly and coily consumers and savvy consumers who increasingly want products specifically for their texture type. This trend is underscored by the latest insights from NaturallyCurly's 2018 TextureTrends report, released today. According to the 2018 TextureTrends Report, more than 80 percent of women with curly and coily hair say they purcha...
Sep. 4, 2019 11:15 PM EDT
By Zakia Bouachraoui
We all love the many benefits of natural plant oils, used as a deap treatment before shampooing, at home or at the beach, but is there an all-in-one solution for everyday intensive nutrition and modern styling?I am passionate about the benefits of natural extracts with tried-and-tested results, which I have used to develop my own brand (lemon for its acid ph, wheat germ for its fortifying action…). I wanted a product which combined caring and styling effects, and which could be used after shampo...
Sep. 4, 2019 11:00 PM EDT  Reads: 312
By Elizabeth White
The platform combines the strengths of Singtel's extensive, intelligent network capabilities with Microsoft's cloud expertise to create a unique solution that sets new standards for IoT applications," said Mr Diomedes Kastanis, Head of IoT at Singtel. "Our solution provides speed, transparency and flexibility, paving the way for a more pervasive use of IoT to accelerate enterprises' digitalisation efforts. AI-powered intelligent connectivity over Microsoft Azure will be the fastest connected pat...
Jul. 1, 2019 07:30 AM EDT
By Zakia Bouachraoui
There are many examples of disruption in consumer space – Uber disrupting the cab industry, Airbnb disrupting the hospitality industry and so on; but have you wondered who is disrupting support and operations? AISERA helps make businesses and customers successful by offering consumer-like user experience for support and operations. We have built the world’s first AI-driven IT / HR / Cloud / Customer Support and Operations solution.
Jun. 27, 2019 08:00 AM EDT
By Pat Romanski
Codete accelerates their clients growth through technological expertise and experience. Codite team works with organizations to meet the challenges that digitalization presents. Their clients include digital start-ups as well as established enterprises in the IT industry. To stay competitive in a highly innovative IT industry, strong R&D departments and bold spin-off initiatives is a must. Codete Data Science and Software Architects teams help corporate clients to stay up to date with the mod...
Jun. 24, 2019 06:00 AM EDT
By Zakia Bouachraoui
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Jun. 17, 2019 01:00 PM EDT
By Pat Romanski
Druva is the global leader in Cloud Data Protection and Management, delivering the industry's first data management-as-a-service solution that aggregates data from endpoints, servers and cloud applications and leverages the public cloud to offer a single pane of glass to enable data protection, governance and intelligence-dramatically increasing the availability and visibility of business critical information, while reducing the risk, cost and complexity of managing and protecting it. Druva's...
Jun. 15, 2019 08:15 PM EDT
By Pat Romanski
BMC has unmatched experience in IT management, supporting 92 of the Forbes Global 100, and earning recognition as an ITSM Gartner Magic Quadrant Leader for five years running. Our solutions offer speed, agility, and efficiency to tackle business challenges in the areas of service management, automation, operations, and the mainframe.
Jun. 15, 2019 01:00 PM EDT
MORE »