StarMirror/data-science-ipython-notebooks
1
0
Fork 0
mirror of https://github.com/donnemartin/data-science-ipython-notebooks.git synced 2024-03-22 13:30:56 +08:00
Code Issues Projects Releases Wiki Activity

Added Redshift snippets.

Browse Source
This commit is contained in:
Donne Martin 2015-02-24 07:17:26 -05:00
parent 89a7bf4b93
commit 06b1d3bf2a
1 changed files with 135 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

138
commands/aws.ipynb
View File

@ -1,7 +1,7 @@
{
"metadata": {
"name": "",
"signature": "sha256:44b8b97435ef131ae163887a097426e7d9818e0394168d960e892917df05417c"
"signature": "sha256:db6ce15c2920169c4db02cf87f3c86fba56a73d96b8b710f037c3304434f9071"
},
"nbformat": 3,
"nbformat_minor": 0,
@ -60,7 +60,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup S3cmd\n",
"## S3cmd\n",
"\n",
"Before I discovered [S3cmd](http://s3tools.org/s3cmd), I had been using the [S3 console](http://aws.amazon.com/console/) to do basic operations and [boto](https://boto.readthedocs.org/en/latest/) to do more of the heavy lifting. However, sometimes I just want to hack away at a command line to do my work.\n",
"\n",
@ -167,7 +167,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup s3-parallel-put\n",
"## s3-parallel-put\n",
"\n",
"[s3-parallel-put](https://github.com/twpayne/s3-parallel-put.git) is a great tool for uploading multiple files to S3 in parallel."
]
@ -374,6 +374,138 @@
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Redshift"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy values from the given S3 location containing CSV files to a Redshift cluster:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"copy table_name from 's3://source/part'\n",
"credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX'\n",
"csv;"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy values from the given location containing TSV files to a Redshift cluster:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"copy table_name from 's3://source/part'\n",
"credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX'\n",
"csv delimiter '\\t';"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"View Redshift errors:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"select * from stl_load_errors;"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Vacuum Redshift in full:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"VACUUM FULL;"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Analyze the compression of a table:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"analyze compression table_name;"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cancel the query with the specified id:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cancel 18764;"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The CANCEL command will not abort a transaction. To abort or roll back a transaction, you must use the ABORT or ROLLBACK command. To cancel a query associated with a transaction, first cancel the query then abort the transaction.\n",
"\n",
"If the query that you canceled is associated with a transaction, use the ABORT or ROLLBACK. command to cancel the transaction and discard any changes made to the data:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"abort;"
],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}