mirror of
https://github.com/donnemartin/data-science-ipython-notebooks.git
synced 2024-03-22 13:30:56 +08:00
Fixed #9: Add notebook info for Boto, the official AWS SDK for Python.
This commit is contained in:
parent
3ab740904f
commit
49cf71e672
|
@ -74,6 +74,7 @@ IPython Notebook(s) demonstrating Amazon Web Services (AWS) and AWS tools functi
|
|||
|
||||
| Notebook | Description |
|
||||
|------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [boto](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/aws/aws.ipynb#boto) | Official AWS SDK for Python. |
|
||||
| [s3cmd](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/aws/aws.ipynb#s3cmd) | Interacts with S3 through the command line. |
|
||||
| [s3distcp](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/aws/aws.ipynb#s3distcp) | Combines smaller files and aggregates them together by taking in a pattern and target file. S3DistCp can also be used to transfer large volumes of data from S3 to your Hadoop cluster. |
|
||||
| [s3-parallel-put](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/aws/aws.ipynb#s3-parallel-put) | Uploads multiple files to S3 in parallel. |
|
||||
|
|
130
aws/aws.ipynb
130
aws/aws.ipynb
|
@ -14,6 +14,7 @@
|
|||
"# Amazon Web Services (AWS)\n",
|
||||
"\n",
|
||||
"* SSH to EC2\n",
|
||||
"* Boto\n",
|
||||
"* S3cmd\n",
|
||||
"* s3-parallel-put\n",
|
||||
"* S3DistCp\n",
|
||||
|
@ -65,6 +66,135 @@
|
|||
"!ssh -i key.pem ec2-user@ipaddress"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Boto\n",
|
||||
"\n",
|
||||
"[Boto](https://github.com/boto/boto) is the official AWS SDK for Python.\n",
|
||||
"\n",
|
||||
"Install Boto:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install Boto"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Configure boto by creating a ~/.boto file with the following:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aws_access_key_id = YOURACCESSKEY\n",
|
||||
"aws_secret_access_key = YOURSECRETKEY"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Work with S3:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto\n",
|
||||
"s3 = boto.connect_s3()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Work with EC2:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto.ec2\n",
|
||||
"ec2 = boto.ec2.connect_to_region('us-east-1')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create a bucket and put an object in that bucket:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto\n",
|
||||
"import time\n",
|
||||
"s3 = boto.connect_s3()\n",
|
||||
"\n",
|
||||
"# Create a new bucket. Buckets must have a globally unique name (not just\n",
|
||||
"# unique to your account).\n",
|
||||
"bucket = s3.create_bucket('boto-demo-%s' % int(time.time()))\n",
|
||||
"\n",
|
||||
"# Create a new key/value pair.\n",
|
||||
"key = bucket.new_key('mykey')\n",
|
||||
"key.set_contents_from_string(\"Hello World!\")\n",
|
||||
"\n",
|
||||
"# Sleep to ensure the data is eventually there.\n",
|
||||
"# This is often referred to as \"S3 eventual consistency\".\n",
|
||||
"time.sleep(2)\n",
|
||||
"\n",
|
||||
"# Retrieve the contents of ``mykey``.\n",
|
||||
"print key.get_contents_as_string()\n",
|
||||
"\n",
|
||||
"# Delete the key.\n",
|
||||
"key.delete()\n",
|
||||
"\n",
|
||||
"# Delete the bucket.\n",
|
||||
"bucket.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Each service supports a different set of commands. Refer to the following for more details:\n",
|
||||
"* [AWS Docs](https://aws.amazon.com/documentation/)\n",
|
||||
"* [Boto Docs](http://boto.readthedocs.org/en/latest/index.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
|
|
Loading…
Reference in New Issue
Block a user