Signed URL for GCS PUT request on GAE with Python
I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error:SignatureDoesNotMatch
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>PUT
123456789
/mybucket/mycat.jpg</StringToSign></Error>
According to the documentation on creating signed URLs using the program and GCP sample Python code , I'm doing the following:
- build my signature string
- sign
- base64 encoding
- url encodes the result (though the python example doesn't do this...
Since this program is running on a Google App Engine (GAE) application, I don't need to get a JSON key file for my service account user, but use App Identity Services to sign it. Here is the code in the Flask project:
google_access_id = app_identity.get_service_account_name()
expires = arrow.utcnow().replace(minutes=+10).replace(microseconds=0).timestamp
resource = '/mybucket/mycat.jpg'
args = self.get_parser.parse_args()
signature_string = 'PUT\n'
# take MD5 of file being uploaded and its content type, if provided
content_md5 = args.get('md5') or ''
content_type = args.get('contenttype') or ''
signature_string = ('PUT\n'
'{md5}\n'
'{content_type}\n'
'{expires}\n'
'{resource}\n').format(
md5=content_md5,
content_type=content_type,
expires=expires,
resource=resource)
log.debug('signature string:\n{}'.format(signature_string))
_, signature_bytes = app_identity.sign_blob(signature_string)
signature = base64.b64encode(signature_bytes)
# URL encode signature
signature = urllib.quote(signature)
media_url = 'https://storage.googleapis.com{}'.format(resource)
return dict(GoogleAccessId=google_access_id,
Expires=expires,
Signature=signature,
bucket='mybucket',
media_url='{}?GoogleAccessId={}&Expires={}&Signature={}'.format(media_url, google_access_id, expires, signature))
This log.debug
statement prints a signature file that exactly matches the signature in the GCS XML error response above. If it matches, why can't I upload?
Using gsutil
, I can create a signed URL using the same GAE service account and it works fine in Postman. I see that gsutil
the URL encodes the signature, but neither seems to matter when creating my own signed URL: GCS gets my PUT request and complains that the signature doesn't match, even though the signature it shows matches mine . Logged debug messages. I've also tried the case with the trailing \n in the original signature string, or without the trailing \n.
edit:POST
I followed the Base64 example to encode the policy before the policy and encode it again after it's signed . I tried this approach when creating the PUT signature and it made no difference
The answer is very close to other answers found on SO and elsewhere, and points Content-Type
to the need to use headers. This is true to an extent, but is overshadowed by my main problem: I'm relying on the default GAE service account, which has "editor" permissions, which I think can read and write GCS. I created a key file from that account, used it gsutil
, and gave the following prompt:
[email protected] does not have permissions on gs://mybucket/cat.jpg, using this link will likely result in a 403 error until at least READ permissions are granted
Correct, you get an error from GCS when trying to put a file here , but it's not a 404 error, but the one shown in the question .SignatureDoesNotMatch
The solution is divided into 2 parts:
- Give the default GAE service account more permissions from the "Storage" permission set:
- storage object creator
- Storage Object Viewer
Content-Type
Be sure to use the Header when putting the file into GCS because it defaults to the requirement in the Header even if I don't specify it in the Signature string to be signedtext/plain
.
Final observation : Even after I've added the correct permissions to the account, I'm still getting warnings from gsutil before creating a new JSON key file from the IAM console. Also, signed URLs created with gsutil and old key files are not working ( error) SignatureDoesNotMatch
even with permissions set on the IAM console . The Python code running on GAE works fine without any updates - it just needs to set permissions in IAM before matching the Content-Header.