Signed URL for GCS PUT request on GAE with Python


hamx0r

I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error:SignatureDoesNotMatch

<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>PUT


123456789
/mybucket/mycat.jpg</StringToSign></Error>

According to the documentation on creating signed URLs using the program and GCP sample Python code , I'm doing the following:

  1. build my signature string
  2. sign
  3. base64 encoding
  4. url encodes the result (though the python example doesn't do this...

Since this program is running on a Google App Engine (GAE) application, I don't need to get a JSON key file for my service account user, but use App Identity Services to sign it. Here is the code in the Flask project:

google_access_id = app_identity.get_service_account_name()
expires = arrow.utcnow().replace(minutes=+10).replace(microseconds=0).timestamp
resource = '/mybucket/mycat.jpg'

args = self.get_parser.parse_args()
signature_string = 'PUT\n'
# take MD5 of file being uploaded and its content type, if provided
content_md5 = args.get('md5') or ''
content_type = args.get('contenttype') or ''
signature_string = ('PUT\n'
                    '{md5}\n'
                    '{content_type}\n'
                    '{expires}\n'
                    '{resource}\n').format(
                    md5=content_md5,
                    content_type=content_type,
                    expires=expires,
                    resource=resource)
log.debug('signature string:\n{}'.format(signature_string))
_, signature_bytes = app_identity.sign_blob(signature_string)
signature = base64.b64encode(signature_bytes)
# URL encode signature
signature = urllib.quote(signature)
media_url = 'https://storage.googleapis.com{}'.format(resource)
return dict(GoogleAccessId=google_access_id,
            Expires=expires,
            Signature=signature,
            bucket='mybucket',
            media_url='{}?GoogleAccessId={}&Expires={}&Signature={}'.format(media_url, google_access_id, expires, signature))

This log.debugstatement prints a signature file that exactly matches the signature in the GCS XML error response above. If it matches, why can't I upload?

Using gsutil, I can create a signed URL using the same GAE service account and it works fine in Postman. I see that gsutilthe URL encodes the signature, but neither seems to matter when creating my own signed URL: GCS gets my PUT request and complains that the signature doesn't match, even though the signature it shows matches mine . Logged debug messages. I've also tried the case with the trailing \n in the original signature string, or without the trailing \n.

edit:POSTI followed the Base64 example to encode the policy before the policy and encode it again after it's signed . I tried this approach when creating the PUT signature and it made no difference

hamx0r

The answer is very close to other answers found on SO and elsewhere, and points Content-Typeto the need to use headers. This is true to an extent, but is overshadowed by my main problem: I'm relying on the default GAE service account, which has "editor" permissions, which I think can read and write GCS. I created a key file from that account, used it gsutil, and gave the following prompt:

[email protected] does not have permissions on gs://mybucket/cat.jpg, using this link will likely result in a 403 error until at least READ permissions are granted

Correct, you get an error from GCS when trying to put a file here , but it's not a 404 error, but the one shown in the question .SignatureDoesNotMatch

The solution is divided into 2 parts:

  1. Give the default GAE service account more permissions from the "Storage" permission set:
    1. storage object creator
    2. Storage Object Viewer
  2. Content-TypeBe sure to use the Header when putting the file into GCS because it defaults to the requirement in the Header even if I don't specify it in the Signature string to be signed text/plain.

Final observation : Even after I've added the correct permissions to the account, I'm still getting warnings from gsutil before creating a new JSON key file from the IAM console. Also, signed URLs created with gsutil and old key files are not working ( error) SignatureDoesNotMatcheven with permissions set on the IAM console . The Python code running on GAE works fine without any updates - it just needs to set permissions in IAM before matching the Content-Header.

Related


Signed URL for GCS PUT request on GAE with Python

hamx0r I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error

Signed URL for GCS PUT request on GAE with Python

hamx0r I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error

Signed URL for GCS PUT request on GAE with Python

hamx0r I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error

Signed URL for GCS PUT request on GAE with Python

hamx0r I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error

Signed URL for GCS PUT request on GAE with Python

hamx0r I'm trying to create a signed URL for uploading files directly to Google Cloud Storage (GCS). I got it working using POST using this Github example , which makes use of policies. Following best practices, I'm refactoring to use PUT and getting the error

Put files into Google Cloud Storage (GCS) via signed URL

kctang: I'm trying to use a "signed URL" to access/upload files to Google Cloud Storage (GCS). Follow the instructions at https://developers.google.com/storage/docs/accesscontrol#Signing-Strings what did I do: "Web Application" is registered in Google Cloud Co

Put files into Google Cloud Storage (GCS) via signed URL

tea soup I'm trying to use a "signed URL" to access/upload files to Google Cloud Storage (GCS). Follow the instructions at https://developers.google.com/storage/docs/accesscontrol#Signing-Strings what did I do: "Web Application" is registered in Google Cloud C

Put files into Google Cloud Storage (GCS) via signed URL

kctang: I'm trying to use a "signed URL" to access/upload files to Google Cloud Storage (GCS). Follow the instructions at https://developers.google.com/storage/docs/accesscontrol#Signing-Strings what did I do: "Web Application" is registered in Google Cloud Co

Put files into Google Cloud Storage (GCS) via signed URL

kctang: I'm trying to use a "signed URL" to access/upload files to Google Cloud Storage (GCS). Follow the instructions at https://developers.google.com/storage/docs/accesscontrol#Signing-Strings what did I do: "Web Application" is registered in Google Cloud Co

Upload file to GCS in GAE (Python)

Unaiherran I upload files into my GCS bucket via GAE upload_url = blobstore.create_upload_url('/upload', gs_bucket_name='my_bucket') as described in the documentation and in this question Everything works fine, however, when I try to read the content, I see t

Upload file to GCS in GAE (Python)

Unaiherran I upload files into my GCS bucket via GAE upload_url = blobstore.create_upload_url('/upload', gs_bucket_name='my_bucket') as described in the documentation and in this question Everything works fine, however, when I try to read the content, I see t

GAE Python GCS filename access old files

Lindsay In my GAE Python app, I'm writing code to store images in GCS. I wrote the image as follows: bucket_name = os.environ.get(u'BUCKET_NAME', app_identity.get_default_gcs_bucket_name()) filename = u'/{}/{}/image'.format(bucket_name, str(object.key.id())) m

GAE Python GCS filename access old files

Lindsay In my GAE Python app, I'm writing code to store images in GCS. I wrote the image as follows: bucket_name = os.environ.get(u'BUCKET_NAME', app_identity.get_default_gcs_bucket_name()) filename = u'/{}/{}/image'.format(bucket_name, str(object.key.id())) m

HMAC signed request in Python

Rudo I'm trying to create HMAC-SHA512 signed requests for API calls in Python 3.4 using the requests library. I'm trying to follow the documentation and I'm getting this error: AttributeError: '_hashlib.HASH' object has no attribute 'new' Here is some code. I

url put request OkHTTPClient

Bobby I am trying to achieve this This is/delete/{id} How can i do this in java? String id = "123"; OkHttpClient client = new OkHttpClient(); RequestBody form = new FormBody.Builder() .add("id", id) .build(); Request request = new Request.Buil

Maximum expiry date for signed URL expiry date (GAE)

Mahdi I'm using Google App Engine (Java based) to create a signed url for my file in Google Cloud Storage. According to the description of google cloud , we need to define the validity period. For example, is it possible to create a long maturity period of 50

Maximum expiry date for signed URL expiry date (GAE)

Mahdi I'm using Google App Engine (Java based) to create a signed url for my file in Google Cloud Storage. According to the description of google cloud , we need to define the validity period. For example, is it possible to create a long maturity period of 50

Maximum expiry date for signed URL expiry date (GAE)

Mahdi I'm using Google App Engine (Java based) to create a signed url for my file in Google Cloud Storage. According to the description of google cloud , we need to define the validity period. For example, is it possible to create a long maturity period of 50

Python delete request for other API (on GAE)

bayanka After searching for a while, I found the following solution for api calls that require Delete method. First try: (httplib library) url = '/v1/users/'+ spotify_user_id +'/playlists/'+ playlist_id +'/tracks' data = json.dumps({"tracks": [{ "uri"

Python delete request for other API (on GAE)

bayanka After searching for a while, I found the following solution for api calls that require Delete method. First try: (httplib library) url = '/v1/users/'+ spotify_user_id +'/playlists/'+ playlist_id +'/tracks' data = json.dumps({"tracks": [{ "uri"