Are you happy with your application's upload speed?

Does your application suffer from slow file uploads? Do the uploads seem to get slower as the file size gets larger? Do some users experience even slower uploads based on their geographic location? Are you looking for a better, stronger, faster way to speed up file uploads? Well, I have a potential solution for you! Let me introduce Amazon Web Service (AWS) S3 presigned URLs...

What is the problem?

In many traditional applications a client (browser, desktop, mobile, CLI) uploads files to an application server such as Apache, Nginx, or perhaps a custom application written in Java, C#, Node.JS, etc... In any of these scenarios we will likely see the application rely on writing to a local filesystem, or perhaps it already writes the file to an existing S3 bucket. In either case, the upload speed may suffer for a variety of reasons. It could be that the application is simply undersized or written inefficiently for large file uploads. Maybe the underlying filesystem is slow or resource constrained. Or perhaps you are just trying to move bytes from one geographic location to another on the other side of the planet. Here is where leveraging presigned S3 URLs for uploads can save the day...

What is a presigned S3 URL?

A presigned S3 URL is simply a pre-authenticated URL with an pre determined expiration that can be used to read or write a file directly from an S3 bucket. The purpose of the presigning is that an authorized backend application with appropriate IAM permissions will build a temporary presigned URL to an otherwise private S3 bucket. Working with a presigned URL is usually a two-step process. The client must first communicate with a trusted backend application to obtain a presigned URL. Next, the client can then use the temporary URL before the expiration time to perform the direct S3 operation such as a PUT in our example below.

How does a presigned URL help?

There are at least three main benefits to using a presigned URL for uploads as opposed to a traditional application.

1. No application overhead or resource constraints - Directly uploading to S3 eliminates any compute resources a traditional application would require
2. Leverage the AWS infrastructure - AWS S3 is one of the first cloud services and likely one of the most widely used service over the years. The speed and reliability of S3 is probably better than anything else we could build with a more than reasonable budget.
3. S3 Transfer Acceleration - When transfer acceleration is enabled and the presigned URL endpoints are used we can ensure a client in any location around the globe will see the lowest latency and fastest transfer speeds possible. In a traditional application we are usually bound by an endpoint that resides in a single geo-location, meaning, as a client becomes further and further away the transfer speeds decrease. We can even see this effect with S3 itself when transfer acceleration is not enabled. When this happens the S3 endpoint will be in whatever AWS region that bucket was defined in. Below is a simple table showing the differences:

All operations were performed with the same ~415MB ZIP file.

What we can see is that being further away from the upload endpoint makes a difference when we do not use the transfer acceleration endpoints. When possible, try to use the transfer acceleration URLs. Note: This must be explicitly enabled on your S3 bucket(s) before you can use this feature.

For comparison's sake, at a recent client engagement I was tasked with investigating these exact improvements as compared to the existing application. Shockingly, the client's application would perform progressively slower as the file size increased. So how does it compare?

Existing Client Application: ~3.8 minutes!

It is important for me to mention that the application is running within the AWS us-east-1 region. Although this is further than the us-east-2 S3 test bucket above, it does not account for the nearly 3x increase in time. Given these findings and subsequent implementation I was able to significantly improve the user's experience when uploading these large files.

Example

Below is an example using the AWS SDK for Java 2. Note: There are similar ways to accomplish this with the other AWS SDKs.

Our example (available on GitHub) uses SpringBoot MVC to build a simple demo.

@SpringBootApplication
@Controller
@Slf4j
public class App {
	public static void main(String[] args) {
		SpringApplication.run(App.class, args);
	}


	@Value("${aws_access_key}")
	String awsAccessKey;
	@Value("${aws_secret_key}")
	String awsSecretKey;
	@Value("${s3_bucket}")
	String bucketName;
	@Value("${s3_region}")
	String bucketRegion;


	/** Serves the src/main/resources/templates/index.html template */
	@GetMapping("/")
	public String appEntryPoint() {
		return "index";
	}

	/** REST endpoint to obtain a presigned url to S3 */
	@GetMapping(value = "/signedUploadUrls", produces = APPLICATION_JSON_VALUE)
	@ResponseBody
	public PresignedUrlResponse getPresignedUrls(@RequestParam String fileName) throws Exception {
		PresignedUrlResponse response = PresignedUrlResponse.builder()
				.urlWithoutAcceleration(presignedUploadUrl(fileName, config -> config.accelerateModeEnabled(false)))
				.urlWithAcceleration(presignedUploadUrl(fileName, config -> config.accelerateModeEnabled(true)))
				.build();
		log.info("Response: {}", response);
		return response;
	}

    /** Helper function to build presigned URL with customized `S3Configuration` */
	private PresignedUrlResponse.PresignedUrl presignedUploadUrl(String objectKey, Consumer<S3Configuration.Builder> s3Configuration) {
		S3Configuration.Builder builder = S3Configuration.builder();
		s3Configuration.accept(builder);
		S3Configuration configuration = builder.build();

		AwsCredentialsProvider credentialsProvider =
				StaticCredentialsProvider.create(AwsBasicCredentials.create(awsAccessKey, awsSecretKey));

		S3Presigner s3Presigner =
				S3Presigner.builder()
						.credentialsProvider(credentialsProvider)
						.serviceConfiguration(configuration)
						.region(Region.of(bucketRegion))
						.build();

		PresignedPutObjectRequest presignedRequest = s3Presigner.presignPutObject(x -> x.signatureDuration(Duration.ofMinutes(1))
				.putObjectRequest(r -> r.bucket(bucketName).key(objectKey)));

		// It is recommended to close the S3Presigner when it is done being used, because some credential
		// providers (e.g. if your AWS profile is configured to assume an STS role) require system resources
		// that need to be freed. If you are using one S3Presigner per application (as recommended), this
		// usually is not needed.
		s3Presigner.close();

		return PresignedUrlResponse.PresignedUrl.builder()
				.url(presignedRequest.url().toString())
				.method(presignedRequest.httpRequest().method().name())
				.headers(presignedRequest.httpRequest().headers())
				.expiration(presignedRequest.expiration())
				.build();
	}

	/** Simple Response Object */
	@lombok.Value
	@Builder
	public static class PresignedUrlResponse {
		private PresignedUrl urlWithoutAcceleration;
		private PresignedUrl urlWithAcceleration;

		@lombok.Value
		@Builder
		public static class PresignedUrl {
			private String url;
			private String method;
			private Map<String, List<String>> headers;
			private Instant expiration;
		}
	}
}

In this simple web application we have two endpoints. The first is the @GetMapping("/") that returns a very simple HTML page with an upload form. The second endpoint @GetMapping(value = "/signedUploadUrls", produces = APPLICATION_JSON_VALUE) is where we build some presigned URLS. This example actually builds two presigned URLs. One uses transfer acceleration and the other does not. This is so we can demonstrate subsequent uploads from the browser to see how the speed differs depending on location.

Special Note About CORS

If you intend to use presigned URLS from a browser you must ensure you enable the appropriate Cross-origin resource sharing (CORS) policy on your S3 bucket. Be sure to restrict the AllowedOrigin to only those domains your app should trust. For more details please see the official AWS documentation.

Until next time.. Enjoy your new fast uploads! :)