Streaming binary file from http response up to AWS S3 with Node.js

Recently, I found myself needing to take images from a remote source and copying them to an Amazon S3 bucket to be used by a client website. Initially, the task seems simple enough... for each image:

1. Send GET request to remote source for image
2. Write response onto webserver as file
3. Read file and upload to S3

Although each image would simply overwrite the previous one on the webserver, there really isn't a need for any additional disk writing to the webserver, since the end goal is only to copy the images up to S3:

1. Send GET request to remote source for image
2. Forward response to S3

The concept is fairly trivial, but still took me a good amount of time to nail down due to a bit of a quirk I wasn't aware of with the Node.js Request module.  The modules used:

• request
• aws-sdk

The biggest gotcha here was that the request module expects a String in the response body by default. To make sure the response stream is kept as raw bytes, the request should be considered to use null as the encoding . Related code:

I've only tried this so far with images, but it should work with just about any small file. Unfortunately, I am uncertain about how well this works with a large file, where memory issues may arise. Obviously, the code will need to be tweaked to fit your needs.