Added AWS command line snippet to run S3DisctCp. Added snippet to control compression.

commands/aws.ipynb

@@ -1,7 +1,7 @@
 {
  "metadata": {
   "name": "",
-  "signature": "sha256:b65124c49718f35b0c88555bf8a8a9c76c41dc50c2b980ec01ebe55e39688d3e"
+  "signature": "sha256:cbe332e49ee1a97b7e73961a5b49d8f32b4b13da36145a34bb8ecb72d5843320"
  },
  "nbformat": 3,
  "nbformat_minor": 0,
@@ -55,6 +55,79 @@
      "language": "python",
      "metadata": {},
      "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "## S3DistCp"
+     ]
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "To run S3DistCp with the EMR command line, ensure you are using the proper version of Ruby:"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "rvm --default ruby-1.8.7-p374"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "The EMR command line below executes the following:\n",
+      "* Create a master node and slave nodes of type m1.small\n",
+      "* Runs S3DistCp on the source bucket location and concatenates files that match the date regular expression, resulting in files that are roughly 1024 MB or 1 GB\n",
+      "* Places the results in the destination bucket"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "./elastic-mapreduce --create --instance-group master --instance-count 1 \\\n",
+      "--instance-type m1.small --instance-group core --instance-count 4 \\\n",
+      "--instance-type m1.small --jar /home/hadoop/lib/emr-s3distcp-1.0.jar \\\n",
+      "--args \"--src,s3://my-bucket-source/,--groupBy,.*([0-9]{4}-01).*,\\\n",
+      "--dest,s3://my-bucket-dest/,--targetSize,1024\""
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "For further optimization, compression can be helpful to save on AWS storage and bandwidth costs, to speed up the S3 to/from EMR transfer, and to reduce disk I/O. Note that compressed files are not easy to split for Hadoop. For example, Hadoop uses a single mapper per GZIP file, as it does not know about file boundaries.\n",
+      "\n",
+      "What type of compression should you use?\n",
+      "\n",
+      "* Time sensitive job: Snappy or LZO\n",
+      "* Large amounts of data: GZIP\n",
+      "* General purpose: GZIP, as it\u2019s supported by most platforms\n",
+      "\n",
+      "You can specify the compression codec (gzip, lzo, snappy, or none) to use for copied files with S3DistCp with \u2013outputCodec. If no value is specified, files are copied with no compression change. The code below sets the compression to lzo:"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "--outputCodec,lzo"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
     }
    ],
    "metadata": {}