Skip to content

Commit

Permalink
Update Readme.
Browse files Browse the repository at this point in the history
  • Loading branch information
rlmcpherson committed Feb 18, 2014
1 parent e20162f commit eb4a375
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 5 deletions.
15 changes: 11 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# s3gof3r #

s3gof3r provides fast, concurrent, streaming access to Amazon S3. It includes a command-line interface: `gof3r`.
s3gof3r provides fast, parallelized, pipelined streaming access to Amazon S3. It includes a command-line interface: `gof3r`.

It is tuned for high speed transfer of large objects into and out of Amazon S3. Streaming support allows for usage like:
It is optimized for high speed transfer of large objects into and out of Amazon S3. Streaming support allows for usage like:

```
$ tar -czf - <my_dir/> | gof3r put -b <s3_bucket> -k <s3_object>
Expand Down Expand Up @@ -30,6 +30,10 @@ These tests were performed on an m1.xlarge EC2 instance with a virtualized 1 Gig

**Features**

- *Speed:* Especially for larger s3 objects where parallelism can be exploited, s3gof3r will saturate the bandwidth of an EC2 instance. See the Benchmarks above.

- *Streaming Uploads and Downloads:* As the above examples illustrate, streaming allows the gof3r command-line tool to be used with linux/unix pipes. This allows transformation of the data in parallel as it is uploaded or downloaded from S3.

- *End-to-end Integrity Checking:* s3gof3r calculates the md5 hash of the stream in parallel while uploading and downloading. On upload, a file containing the md5 hash is saved in s3. This is checked against the calculated md5 on download. On upload, the content-md5 of each part is calculated and sent with the header to be checked by AWS. s3gof3r also checks the 'hash of hashes' returned by S3 in the `Etag` field on completion of a multipart upload. See the [S3 API Reference](http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html) for details.

- *Retry Everything:* All http requests and every part is retried on both uploads and downloads. Requests to S3 frequently time out, especially under high load, so this is essential to complete large uploads or downloads.
Expand Down Expand Up @@ -130,8 +134,11 @@ To install just the package for use in other Go programs:
Help Options:
-h, --help Show this help message
```



**See godoc.org for more documentation, including the s3gof3r package api:**

See godoc.org for more documentation:
s3gof3r package: [http://godoc.org/github.com/rlmcpherson/s3gof3r](http://godoc.org/github.com/rlmcpherson/s3gof3r)

package: [http://godoc.org/github.com/rlmcpherson/s3gof3r](http://godoc.org/github.com/rlmcpherson/s3gof3r)
command-line interface: [http://godoc.org/github.com/rlmcpherson/s3gof3r/gof3r](http://godoc.org/github.com/rlmcpherson/s3gof3r/gof3r)
3 changes: 2 additions & 1 deletion s3gof3r.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// Package s3gof3r provides fast, concurrent, streaming access to Amazon S3. Includes a CLI.
// Package s3gof3r provides fast, parallelized, streaming access to Amazon S3. It includes a command-line interface: `gof3r`.

package s3gof3r

import (
Expand Down

0 comments on commit eb4a375

Please sign in to comment.