StarMirror/data-science-ipython-notebooks
1
0
Fork 0
mirror of https://github.com/donnemartin/data-science-ipython-notebooks.git synced 2024-03-22 13:30:56 +08:00
Code Issues Projects Releases Wiki Activity

Added Kinesis commands.

Browse Source
This commit is contained in:
Donne Martin 2015-03-09 16:10:54 -04:00
parent 1815c9a122
commit cd84ffb2f0
1 changed files with 113 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

115
aws/aws.ipynb
View File

@ -1,7 +1,7 @@
{
"metadata": {
"name": "",
"signature": "sha256:22b98dd51ff479ba7e10ae0a2f0c4e85b11642fe49dfacfa45c7c4137881b3b7"
"signature": "sha256:6bd9f74232672cb4b50a0a1b5a5abd778aefd1ee3da04b9643d730eb97896b73"
},
"nbformat": 3,
"nbformat_minor": 0,
@ -19,7 +19,8 @@
"* s3-parallel-put\n",
"* S3DistCp\n",
"* mrjob\n",
"* Redshift"
"* Redshift\n",
"* Kinesis"
]
},
{
@ -659,6 +660,116 @@
"\n",
"The rows are distributed across the slices in a round-robin fashion, regardless of the values in any particular column. EVEN distribution is appropriate when a table does not participate in joins or when there is not a clear choice between KEY distribution and ALL distribution. EVEN distribution is the default distribution style."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Kinesis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a stream:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"aws kinesis create-stream --stream-name Foo --shard-count 1 --profile adminuser"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"List all streams:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"aws kinesis list-streams --profile adminuser"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get info about the stream:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"aws kinesis describe-stream --stream-name Foo --profile adminuser"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Put a record to the stream:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"aws kinesis put-record --stream-name Foo --data \"SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=\" --partition-key shardId-000000000000 --region us-east-1 --profile adminuser"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get records from a given shard:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo --query 'ShardIterator' --profile adminuser)\n",
"aws kinesis get-records --shard-iterator $SHARD_ITERATOR"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Delete a stream:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"aws kinesis delete-stream --stream-name Foo --profile adminuser"
],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}