Google Cloud Storage

CloudBD disks require access to a "storage remote" for storing all disk data blocks and metadata. This page will guide you through the creation of a Google storage remote that uses Google Cloud Storage (GCS) as well as any extra steps for improved security and performance.

Each Google storage remote requires its own GCS bucket for storing disks. Google storage remotes belong to the same Google region as their GCS bucket (chosen during bucket creation). GCS buckets have additional costs and higher latency when accessed by services from outside of their region. When using CloudBD disks from Compute Engine instances your Compute Engine instances must be located in the same region as your CloudBD storage remote. To create CloudBD disks in different Google regions a separate Google storage remote (and GCS bucket) is required for each region.

A Google storage remote can store any number of CloudBD disks so a single storage remote per Google region may be sufficient.

Quick Start Guide

In this Quick Start guide we will create a remote for a Google region that is secured by a new Google service account.

Select Project

Select a project or create a new project that will contain the remote storage bucket and the service account.

Create Service Account

The CloudBD disk driver will require a Google service account to access the remote storage bucket. Any service account with admin access to the GCS bucket will work but for best security we recommend creating a new service account for CloudBD disks and then restrict its access to only the CloudBD buckets.

When creating a service account make sure you download or save the private key file in JSON format. It is only available at creation time of the key and is required to authenticate to your GCS storage remote.

• Guide to creating a service account

  1. Go to IAM & Admin -> Service Accounts -> Create Service Account
  2. Enter a name for your CloudBD service account
  3. Leave the Project role empty (no permissions)
  4. Select Furnish a new private key -> Key type JSON
  5. Save and Download the private key json file

Create GCS Bucket

All disk data and metadata is stored in a GCS bucket. When creating a GCS bucket make sure to use the same location region as the Compute Engine VM instances that will use the CloudBD disks.

Note: CloudBD only supports buckets with Regional Storage Class.

• Guide to creating a Cloud Storage bucket

  1. Go to Storage -> Create Bucket
  2. Enter a name for your bucket
  3. Select Storage Class: Regional
  4. Select the location
  5. Select Create

Grant Bucket Access

The service account created above will need storage admin access to the GCS bucket created for the disk data. This access can be provided to the service account on a per bucket basis.

1. Go to Storage -> Browser
2. Select the '⋮' icon on the far right of your GCS bucket
3. Select Edit Bucket Permissions
4. In Add Members enter the client email of your service account created above
5. Select the Storage Admin role
6. Click the Add button

Deploy Config File

Now that your Google storage remote has been created you need to create a storage remote config file and deploy it to the servers that will manage and use CloudBD disks. Storage remote config files are placed in the /etc/cloudbd/remotes.d/ directory. These config files must end in ".conf" and use the basic INI config format of key = value. Below are the required fields for a Google storage remote.

Required Fields

• type

  Declares the type of remote connection. For Google Cloud Storage only OAuth 2.0 is supported.

  type = gcs_oauth_v2
  

• bucket

  The name of the bucket set up as a CloudBD remote.

  bucket = <Your bucket name>
  

• client_email

  The client email of a Google service account with the necessary read and write permissions to the remote storage bucket.

  client_email = <Your Google service account client email>
  

• private_key

  The private key of the Google service account with the given client_email.

  private_key = <Your Google service account private key>
  

Optional Fields

• protocol

  Whether to use http or https when communicating with GCS. We recommend using the 'http' protocol for GCS when attaching disks to Google virtual machine instances and using the 'https' protocol when attaching disks to servers located outside of Google's cloud environment. The protocol is optional and will default to using 'http' if not set. Using the http protocol is less CPU intensive but requires trust of the local network. The https protocol is more secure because it encrypts all network data but requires more CPU overhead when reading and writing to the CloudBD disks.

  protocol = <http | https>
  

Example GCS Remote Config File

type = gcs_oauth_v2
bucket = testdata1.example.com
client_email = cloudbd-secure@example.iam.gserviceaccount.com
private_key = -----BEGIN PRIVATE KEY-----\nMIIEvwEXAMPLEgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCijFkuX7a1FNDH\nufLiSutoZWzl25yhHrFbuuKkSIoyBPnYkyG2T2otZ7aIqxyIPIuJL91XEaxqqUQu\n9s4JSKeh6YI60yN/5k8DsXm6jIl/r5K0BBd8Slq2d50igzHbh8Ye7gX5UVjbAPXm\nypP3mY491GuBLHdVjcjLK2rT2P3tZ7HqKnQunKHefqlPwLecdDKeDPgzL7tr8s0C\nQ7PUCb5wDcFFP0abgfk9o2SleqdTtflYW/o5/Kf9swCgoVYsfCuuQSSLnUdsZO4D\n6x8J9QkyxHUcYTqgfriTgeM7hRKL6NHg1VGaMEJE4zqBSKqrWqIweGNDkpBNyXrW\nN/SuMkM1AgMBAAECggEAAN9LhIKwF6+qWZ0yn+bjjBZsp+hm/J47Vv//HQ4oH+wt\nUynAIi3A+ZsbA/isCCQuFycLdCpNnQ1DiMjdQpZCExVhIxSsNBCk4IxI38qZzIWE\n/jfFo26al/PkTmlAdoEGkoW9k5Vrm2EC2jRXUc8kDCxcXvgb6OItcxnjGgNGKkLP\nrFWyGoBpFHxuGbd0wtno7ar/w/xxSAiOOChE48J55hn8QO0aHCdoWhMl9a/b38sQ\nhf7uBgZOM99rVNBo3Z0m4g5JcZ7yu4hutRCiSFKJTT02RKdugzYc3/r+83gp3NoL\nzR5dNel35sxv7/6Slx+x3BdS938wWkTg37Gr3xuPDwKBgQDWGFXxsMVlf4YCKhUM\nR0a2OhXAs5+ZoM7ChbapxDtQUHT1mt3Uxe46c3oBtW6+FU4EgKufS7NKpR65LV7N\n36YSUqNIzAPhoPvovonBrYR8WUlogCcgOJ5rZl+qtbPKOmj9/JJa4QJ5SkUbQGYf\nidm668e2FHkoPqmo6FbZMyWM6wKBgQDCXSe2GJUa4BaFbTQGUNtYmbcDe47H+XZf\nDRk9ro1xPE5v1oa288uZkgY/tao3IT9TPQOP3kSTGvq6QJl/ZdoVZUx6oqVSmn8i\nMqezYCsc2IHaM5Q5qtdafV2TlWKFpaSbGW5LKARSMcJC/mY9P3vywAcx+JU+qUef\n11gvZyroXwKBgQC6EBRBJBuphhZ0sxXvwdIWdIyLGJBP9405SPPivoO93O6ItLqt\nPAc6DTmvWnqP//v5tclxe1yqNq5oyKFn0kKGEgOONJ/66umC8HTQOXi2iPJfO2DF\ng+9J74Kc3aUOz4yHj4WyLXL07jL46BLYvfjXlinOeCSDcVIJVUKFKQotWQKBgQCY\nxpSCZAw/1cieFFqCZGgl2p2qlSKEOag7ypX+kAgJ6/dTnXLCuBtfmRJkXa1wLCgU\ni68TFQkeAGT4pfKB21Ny4e8drl748Qvhm6OKMZtxzxUyqLxoH8urC0PUz+VjW/Jp\nH7I8TTYoTYh/FGzU0+PfyeVd+CGH4ZO1JpXgnN1zTQKBgQCzViab2psVMNhudYn8\nO4e2T/WufmhSL8jxn9qJ7FzXa/Ki6aGVvIFTTzLsrKRMZaE93PxPHzd2iSx7HAi6\nSSKPV2bcK2ZLc3x4f4erkzFE91MD8mFp8opk2euUVqpZAdBnE0zm82cnEoQSboJi\nQQFxWEXAMPLEA9fKxsbnVkwuBA==\n-----END PRIVATE KEY-----\n