"Resumable Upload" with "Signed URL" on Google Cloud Storage using Java
Based on the documentation on how to create objects in google-cloud-storage (see the "create" method in https://googleapis.github.io/google-cloud-java/google-cloud-clients/apidocs/index.html ) ), when trying to upload large files, we should use the blob.writer(...) method, as it presumably handles resumable uploads automatically somehow. Is this correct?
But if we want resumable uploads of SIGNED urls, how can we do this in Java? (Any sample code or pointers would be greatly appreciated; my research so far leads me to believe that it is not possible to use the beautifully created Java library, and instead requires the use of "PUT" and "POST in Java" statements after generating the signed url. Is this the "best" approach so far?)
Regarding the first point, yes, the blob.writer(...)
method handles resumable uploads automatically. Unfortunately, this method cannot be called from a signed URL, but can only upload files directly from the byte stream.
However, as you mentioned, there are other ways to create resumable uploads from signed URLs, e.g. using PUT
method seems like a good workaround.
What I do is:
Create a signed URL using the "PUT" method . You can do this by specifying SignUrlOption , and I specified a service account in the bucket with the required permissions.
Use URLFetch to throw an HTTP request to this signed URL. I believe e.g. you can't use the curl command directly, and the UrlFetch API does the trick.
Add the uploadType=resumable header to the
urlFetch
HTTP request. See this documentation for how it works and other parameters and information.I configured
URLFetch
to make an asynchronous call to the signed URL as I thought it would be more convenient when uploading large files.
Example code used in App Engine handler:
package com.example.storage;
import java.io.IOException;
import java.io.FileInputStream;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import java.util.HashMap;
import java.util.Map;
import java.nio.charset.StandardCharsets;
import java.net.URL;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
// Cloud Storage Imports
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage.SignUrlOption;
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.google.cloud.storage.HttpMethod;
// Url Fetch imports
import com.google.appengine.api.urlfetch.HTTPMethod;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;
import com.google.appengine.api.urlfetch.HTTPHeader;
@WebServlet(name = "MainStorage", value = "/")
public class MainStorage extends HttpServlet {
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
// Bucket parameters
String bucketName = "MY-BUCKET-NAME";
String blobName = "MY-BLOB-NAME";
String keyPath = "/PATH-TO-SERVICE-ACCOUNT-KEY/key.json";
BlobId blobId = BlobId.of(bucketName, blobName);
Storage storage = StorageOptions.getDefaultInstance().getService();
// Create signed URL with SignUrlOptions
URL signedUrl = storage.signUrl(BlobInfo.newBuilder(bucketName, blobName).build(), 14, TimeUnit.DAYS,
SignUrlOption.signWith(ServiceAccountCredentials.fromStream(new FileInputStream(keyPath))),
SignUrlOption.httpMethod(HttpMethod.PUT));
// Contents to upload to the Blob
String content = "My-File-contents";
// Build UrlFetch request
HTTPRequest upload_request = new HTTPRequest(signedUrl, HTTPMethod.PUT);
upload_request.setPayload(content.getBytes(StandardCharsets.UTF_8));
// Set request to have an uploadType=resumable
HTTPHeader set_resumable = new HTTPHeader("uploadType", "resumable");
upload_request.setHeader(set_resumable);
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
// Do an asynchronous call to the signed URL with the contents
fetcher.fetchAsync(upload_request);
// Return response to App Engine handler call
response.setContentType("text/plain");
response.getWriter().println("Hello Storage");
}
}
There may be a better way to do this, but I believe it gives an idea of how to do this type of application.