Merge branch 'main' into atqy/refactor-ingest-data

Author: atqy

introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining.ipynb

@@ -124,7 +124,7 @@
    "source": [
     "## Training the ResNet model\n",
     "\n",
-    "In this demo, we are using [Caltech-256](http://www.vision.caltech.edu/Image_Datasets/Caltech256/) dataset, which contains 30608 images of 256 objects. For the training and validation data, we follow the splitting scheme in this MXNet [example](https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/data/caltech256.sh). In particular, it randomly selects 60 images per class for training, and uses the remaining data for validation. The algorithm takes `RecordIO` file as input. The user can also provide the image files as input, which will be converted into `RecordIO` format using MXNet's [im2rec](https://mxnet.incubator.apache.org/how_to/recordio.html?highlight=im2rec) tool. It takes around 50 seconds to converted the entire Caltech-256 dataset (~1.2GB) on a p2.xlarge instance. However, for this demo, we will use record io format. \n",
+    "In this demo, we are using [Caltech-256](http://www.vision.caltech.edu/Image_Datasets/Caltech256/) dataset, which contains 30608 images of 256 objects. For the training and validation data, we follow the splitting scheme in this MXNet [example](https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/data/caltech256.sh). In particular, it randomly selects 60 images per class for training, and uses the remaining data for validation. The algorithm takes `RecordIO` file as input. The user can also provide the image files as input, which will be converted into `RecordIO` format using MXNet's [im2rec](https://mxnet.incubator.apache.org/how_to/recordio.html?highlight=im2rec) tool. It takes under a minute to convert the entire Caltech-256 dataset (~1.2GB) on a p3.2xlarge instance. However, for this demo, we will use record io format. \n",
     "\n",
     "Once we have the data available in the correct format for training, the next step is to actually train the model using the data. After setting training parameters, we kick off training, and poll for status until training is completed.\n",
     "\n",
@@ -149,7 +149,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "After setting training parameters, we kick off training, and poll for status until training is completed, which in this example, takes between 10 to 12 minutes per epoch on a p2.xlarge machine. The network typically converges after 10 epochs. However, to save the training time, we set the epochs to 2 but please keep in mind that it may not be  sufficient to generate a good model. "
+    "After setting training parameters, we kick off training, and poll for status until training is completed, which in this example, takes between 3 to 5 minutes per epoch on a p3.2xlarge machine. The network typically converges after 10 epochs. However, to save the training time, we set the epochs to 2 but please keep in mind that it may not be  sufficient to generate a good model. "
    ]
   },
   {
@@ -207,7 +207,7 @@
     "    \"AlgorithmSpecification\": {\"TrainingImage\": training_image, \"TrainingInputMode\": \"File\"},\n",
     "    \"RoleArn\": role,\n",
     "    \"OutputDataConfig\": {\"S3OutputPath\": \"s3://{}/{}/output\".format(bucket, job_name_prefix)},\n",
-    "    \"ResourceConfig\": {\"InstanceCount\": 1, \"InstanceType\": \"ml.p2.xlarge\", \"VolumeSizeInGB\": 50},\n",
+    "    \"ResourceConfig\": {\"InstanceCount\": 1, \"InstanceType\": \"ml.p3.2xlarge\", \"VolumeSizeInGB\": 50},\n",
     "    \"TrainingJobName\": job_name,\n",
     "    \"HyperParameters\": {\n",
     "        \"image_shape\": image_shape,\n",
@@ -429,7 +429,7 @@
     "        \"SplitType\": \"None\",\n",
     "        \"CompressionType\": \"None\",\n",
     "    },\n",
-    "    \"TransformResources\": {\"InstanceType\": \"ml.p2.xlarge\", \"InstanceCount\": 1},\n",
+    "    \"TransformResources\": {\"InstanceType\": \"ml.c5.xlarge\", \"InstanceCount\": 1},\n",
     "}\n",
     "\n",
     "print(\"Transform job name: {}\".format(batch_job_name))\n",
@@ -821,7 +821,7 @@
    "metadata": {},
    "source": [
     "#### Create Endpoint\n",
-    "Next, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete."
+    "Next, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications."
    ]
   },
   {
@@ -967,265 +967,6 @@
     "# the result will output the probabilities for all classes\n",
     "# find the class with maximum probability and print the class index\n",
     "index = np.argmax(result)\n",
-    "object_categories = [\n",
-    "    \"ak47\",\n",
-    "    \"american-flag\",\n",
-    "    \"backpack\",\n",
-    "    \"baseball-bat\",\n",
-    "    \"baseball-glove\",\n",
-    "    \"basketball-hoop\",\n",
-    "    \"bat\",\n",
-    "    \"bathtub\",\n",
-    "    \"bear\",\n",
-    "    \"beer-mug\",\n",
-    "    \"billiards\",\n",
-    "    \"binoculars\",\n",
-    "    \"birdbath\",\n",
-    "    \"blimp\",\n",
-    "    \"bonsai-101\",\n",
-    "    \"boom-box\",\n",
-    "    \"bowling-ball\",\n",
-    "    \"bowling-pin\",\n",
-    "    \"boxing-glove\",\n",
-    "    \"brain-101\",\n",
-    "    \"breadmaker\",\n",
-    "    \"buddha-101\",\n",
-    "    \"bulldozer\",\n",
-    "    \"butterfly\",\n",
-    "    \"cactus\",\n",
-    "    \"cake\",\n",
-    "    \"calculator\",\n",
-    "    \"camel\",\n",
-    "    \"cannon\",\n",
-    "    \"canoe\",\n",
-    "    \"car-tire\",\n",
-    "    \"cartman\",\n",
-    "    \"cd\",\n",
-    "    \"centipede\",\n",
-    "    \"cereal-box\",\n",
-    "    \"chandelier-101\",\n",
-    "    \"chess-board\",\n",
-    "    \"chimp\",\n",
-    "    \"chopsticks\",\n",
-    "    \"cockroach\",\n",
-    "    \"coffee-mug\",\n",
-    "    \"coffin\",\n",
-    "    \"coin\",\n",
-    "    \"comet\",\n",
-    "    \"computer-keyboard\",\n",
-    "    \"computer-monitor\",\n",
-    "    \"computer-mouse\",\n",
-    "    \"conch\",\n",
-    "    \"cormorant\",\n",
-    "    \"covered-wagon\",\n",
-    "    \"cowboy-hat\",\n",
-    "    \"crab-101\",\n",
-    "    \"desk-globe\",\n",
-    "    \"diamond-ring\",\n",
-    "    \"dice\",\n",
-    "    \"dog\",\n",
-    "    \"dolphin-101\",\n",
-    "    \"doorknob\",\n",
-    "    \"drinking-straw\",\n",
-    "    \"duck\",\n",
-    "    \"dumb-bell\",\n",
-    "    \"eiffel-tower\",\n",
-    "    \"electric-guitar-101\",\n",
-    "    \"elephant-101\",\n",
-    "    \"elk\",\n",
-    "    \"ewer-101\",\n",
-    "    \"eyeglasses\",\n",
-    "    \"fern\",\n",
-    "    \"fighter-jet\",\n",
-    "    \"fire-extinguisher\",\n",
-    "    \"fire-hydrant\",\n",
-    "    \"fire-truck\",\n",
-    "    \"fireworks\",\n",
-    "    \"flashlight\",\n",
-    "    \"floppy-disk\",\n",
-    "    \"football-helmet\",\n",
-    "    \"french-horn\",\n",
-    "    \"fried-egg\",\n",
-    "    \"frisbee\",\n",
-    "    \"frog\",\n",
-    "    \"frying-pan\",\n",
-    "    \"galaxy\",\n",
-    "    \"gas-pump\",\n",
-    "    \"giraffe\",\n",
-    "    \"goat\",\n",
-    "    \"golden-gate-bridge\",\n",
-    "    \"goldfish\",\n",
-    "    \"golf-ball\",\n",
-    "    \"goose\",\n",
-    "    \"gorilla\",\n",
-    "    \"grand-piano-101\",\n",
-    "    \"grapes\",\n",
-    "    \"grasshopper\",\n",
-    "    \"guitar-pick\",\n",
-    "    \"hamburger\",\n",
-    "    \"hammock\",\n",
-    "    \"harmonica\",\n",
-    "    \"harp\",\n",
-    "    \"harpsichord\",\n",
-    "    \"hawksbill-101\",\n",
-    "    \"head-phones\",\n",
-    "    \"helicopter-101\",\n",
-    "    \"hibiscus\",\n",
-    "    \"homer-simpson\",\n",
-    "    \"horse\",\n",
-    "    \"horseshoe-crab\",\n",
-    "    \"hot-air-balloon\",\n",
-    "    \"hot-dog\",\n",
-    "    \"hot-tub\",\n",
-    "    \"hourglass\",\n",
-    "    \"house-fly\",\n",
-    "    \"human-skeleton\",\n",
-    "    \"hummingbird\",\n",
-    "    \"ibis-101\",\n",
-    "    \"ice-cream-cone\",\n",
-    "    \"iguana\",\n",
-    "    \"ipod\",\n",
-    "    \"iris\",\n",
-    "    \"jesus-christ\",\n",
-    "    \"joy-stick\",\n",
-    "    \"kangaroo-101\",\n",
-    "    \"kayak\",\n",
-    "    \"ketch-101\",\n",
-    "    \"killer-whale\",\n",
-    "    \"knife\",\n",
-    "    \"ladder\",\n",
-    "    \"laptop-101\",\n",
-    "    \"lathe\",\n",
-    "    \"leopards-101\",\n",
-    "    \"license-plate\",\n",
-    "    \"lightbulb\",\n",
-    "    \"light-house\",\n",
-    "    \"lightning\",\n",
-    "    \"llama-101\",\n",
-    "    \"mailbox\",\n",
-    "    \"mandolin\",\n",
-    "    \"mars\",\n",
-    "    \"mattress\",\n",
-    "    \"megaphone\",\n",
-    "    \"menorah-101\",\n",
-    "    \"microscope\",\n",
-    "    \"microwave\",\n",
-    "    \"minaret\",\n",
-    "    \"minotaur\",\n",
-    "    \"motorbikes-101\",\n",
-    "    \"mountain-bike\",\n",
-    "    \"mushroom\",\n",
-    "    \"mussels\",\n",
-    "    \"necktie\",\n",
-    "    \"octopus\",\n",
-    "    \"ostrich\",\n",
-    "    \"owl\",\n",
-    "    \"palm-pilot\",\n",
-    "    \"palm-tree\",\n",
-    "    \"paperclip\",\n",
-    "    \"paper-shredder\",\n",
-    "    \"pci-card\",\n",
-    "    \"penguin\",\n",
-    "    \"people\",\n",
-    "    \"pez-dispenser\",\n",
-    "    \"photocopier\",\n",
-    "    \"picnic-table\",\n",
-    "    \"playing-card\",\n",
-    "    \"porcupine\",\n",
-    "    \"pram\",\n",
-    "    \"praying-mantis\",\n",
-    "    \"pyramid\",\n",
-    "    \"raccoon\",\n",
-    "    \"radio-telescope\",\n",
-    "    \"rainbow\",\n",
-    "    \"refrigerator\",\n",
-    "    \"revolver-101\",\n",
-    "    \"rifle\",\n",
-    "    \"rotary-phone\",\n",
-    "    \"roulette-wheel\",\n",
-    "    \"saddle\",\n",
-    "    \"saturn\",\n",
-    "    \"school-bus\",\n",
-    "    \"scorpion-101\",\n",
-    "    \"screwdriver\",\n",
-    "    \"segway\",\n",
-    "    \"self-propelled-lawn-mower\",\n",
-    "    \"sextant\",\n",
-    "    \"sheet-music\",\n",
-    "    \"skateboard\",\n",
-    "    \"skunk\",\n",
-    "    \"skyscraper\",\n",
-    "    \"smokestack\",\n",
-    "    \"snail\",\n",
-    "    \"snake\",\n",
-    "    \"sneaker\",\n",
-    "    \"snowmobile\",\n",
-    "    \"soccer-ball\",\n",
-    "    \"socks\",\n",
-    "    \"soda-can\",\n",
-    "    \"spaghetti\",\n",
-    "    \"speed-boat\",\n",
-    "    \"spider\",\n",
-    "    \"spoon\",\n",
-    "    \"stained-glass\",\n",
-    "    \"starfish-101\",\n",
-    "    \"steering-wheel\",\n",
-    "    \"stirrups\",\n",
-    "    \"sunflower-101\",\n",
-    "    \"superman\",\n",
-    "    \"sushi\",\n",
-    "    \"swan\",\n",
-    "    \"swiss-army-knife\",\n",
-    "    \"sword\",\n",
-    "    \"syringe\",\n",
-    "    \"tambourine\",\n",
-    "    \"teapot\",\n",
-    "    \"teddy-bear\",\n",
-    "    \"teepee\",\n",
-    "    \"telephone-box\",\n",
-    "    \"tennis-ball\",\n",
-    "    \"tennis-court\",\n",
-    "    \"tennis-racket\",\n",
-    "    \"theodolite\",\n",
-    "    \"toaster\",\n",
-    "    \"tomato\",\n",
-    "    \"tombstone\",\n",
-    "    \"top-hat\",\n",
-    "    \"touring-bike\",\n",
-    "    \"tower-pisa\",\n",
-    "    \"traffic-light\",\n",
-    "    \"treadmill\",\n",
-    "    \"triceratops\",\n",
-    "    \"tricycle\",\n",
-    "    \"trilobite-101\",\n",
-    "    \"tripod\",\n",
-    "    \"t-shirt\",\n",
-    "    \"tuning-fork\",\n",
-    "    \"tweezer\",\n",
-    "    \"umbrella-101\",\n",
-    "    \"unicorn\",\n",
-    "    \"vcr\",\n",
-    "    \"video-projector\",\n",
-    "    \"washing-machine\",\n",
-    "    \"watch-101\",\n",
-    "    \"waterfall\",\n",
-    "    \"watermelon\",\n",
-    "    \"welding-mask\",\n",
-    "    \"wheelbarrow\",\n",
-    "    \"windmill\",\n",
-    "    \"wine-bottle\",\n",
-    "    \"xylophone\",\n",
-    "    \"yarmulke\",\n",
-    "    \"yo-yo\",\n",
-    "    \"zebra\",\n",
-    "    \"airplanes-101\",\n",
-    "    \"car-side-101\",\n",
-    "    \"faces-easy-101\",\n",
-    "    \"greyhound\",\n",
-    "    \"tennis-shoes\",\n",
-    "    \"toad\",\n",
-    "    \"clutter\",\n",
-    "]\n",
     "print(\"Result: label - \" + object_categories[index] + \", probability - \" + str(result[index]))"
    ]
   },

introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-incremental-training-highlevel.ipynb

@@ -75,7 +75,7 @@
    "metadata": {},
    "source": [
     "### Data preparation\n",
-    "Download the data and transfer to S3 for use in training. In this demo, we are using [Caltech-256](http://www.vision.caltech.edu/Image_Datasets/Caltech256/) dataset, which contains 30608 images of 256 objects. For the training and validation data, we follow the splitting scheme in this MXNet [example](https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/data/caltech256.sh). In particular, it randomly selects 60 images per class for training, and uses the remaining data for validation. The algorithm takes `RecordIO` file as input. The user can also provide the image files as input, which will be converted into `RecordIO` format using MXNet's [im2rec](https://mxnet.incubator.apache.org/how_to/recordio.html?highlight=im2rec) tool. It takes around 50 seconds to converted the entire Caltech-256 dataset (~1.2GB) on a p2.xlarge instance. However, for this demo, we will use record io format. "
+    "Download the data and transfer to S3 for use in training. In this demo, we are using [Caltech-256](http://www.vision.caltech.edu/Image_Datasets/Caltech256/) dataset, which contains 30608 images of 256 objects. For the training and validation data, we follow the splitting scheme in this MXNet [example](https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/data/caltech256.sh). In particular, it randomly selects 60 images per class for training, and uses the remaining data for validation. The algorithm takes `RecordIO` file as input. The user can also provide the image files as input, which will be converted into `RecordIO` format using MXNet's [im2rec](https://mxnet.incubator.apache.org/how_to/recordio.html?highlight=im2rec) tool. It takes under a minute to convert the entire Caltech-256 dataset (~1.2GB) on a p3.2xlarge instance. However, for this demo, we will use record io format. "
    ]
   },
   {
@@ -166,7 +166,7 @@
     "    training_image,\n",
     "    role,\n",
     "    train_instance_count=1,\n",
-    "    train_instance_type=\"ml.p2.xlarge\",\n",
+    "    train_instance_type=\"ml.p3.2xlarge\",\n",
     "    train_volume_size=50,\n",
     "    train_max_run=360000,\n",
     "    input_mode=\"File\",\n",
@@ -304,7 +304,7 @@
     "    training_image,\n",
     "    role,\n",
     "    train_instance_count=1,\n",
-    "    train_instance_type=\"ml.p2.xlarge\",\n",
+    "    train_instance_type=\"ml.p3.2xlarge\",\n",
     "    train_volume_size=50,\n",
     "    train_max_run=360000,\n",
     "    input_mode=\"File\",\n",

introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-transfer-learning-highlevel.ipynb

@@ -155,7 +155,7 @@
     "    training_image,\n",
     "    role,\n",
     "    instance_count=1,\n",
-    "    instance_type=\"ml.p2.xlarge\",\n",
+    "    instance_type=\"ml.p3.2xlarge\",\n",
     "    train_volume_size=50,\n",
     "    train_max_run=360000,\n",
     "    input_mode=\"File\",\n",