We are excited to announce that we have added preview support for client-side encryption of data in Azure Storage .NET client library for Blob, Table and Queue data. We have also added support for integrating with Azure Key Vault to manage keys. The process of encryption and decryption follow the “envelope” technique.
Envelope Technique
Encryption using the envelope technique works in the following way:
- The Azure Storage client SDK will generate a content encryption key (CEK) which is a one-time-use symmetric key.
- User data is encrypted using this CEK.
- The CEK is then wrapped (encrypted) using the key encryption key KEK. The KEK is identified by a key identifier and can be an asymmetric key pair or a symmetric key and can be managed locally or stored in Azure Key Vaults. The Storage client itself never has access to KEK. It just invokes the key wrapping algorithm that is provided by Key Vault. Users can choose to use custom providers for key wrapping/unwrapping if desired.
- The encrypted data is then uploaded to the Azure Storage service. The wrapped key along with some additional encryption metadata is either stored as metadata (on a blob) or interpolated with the encrypted data (queue messages and table entities).
The decryption process works in the following way:
- It is assumed that the user has the key encryption key (KEK) either managed locally or in Azure Key Vaults. The user does not need to know the specific key that was used for encryption. Instead, a key resolver which resolves different key identifiers to keys can be set up and used.
- The client SDK downloads the encrypted data along with any encryption material that is stored on the service.
- The wrapped content encryption key (CEK) is then unwrapped (decrypted) using the key encryption key (KEK). Here again, the Storage client does not have access to KEK. It just invokes the custom or Key Vault provider’s unwrapping algorithm.
- The content encryption key (CEK) is then used to decrypt the encrypted user data.
Encryption Mechanism
The client library uses AES in order to encrypt user data. Specifically, Cipher Block Chaining (CBC) mode with AES. Each service works somewhat differently, so we will discuss each of them here.
Blobs
In the current version, the client library supports encryption of entire blobs only - specifically encryption is supported when users use UploadFrom* methods or BlobWriteStream. For downloads, both full and range downloads are supported.
During encryption, the client library will generate a random Initialization Vector (IV) of 16 bytes along with a random content encryption key (CEK) of 32 bytes and do envelope encryption of the blob data using this information. The wrapped CEK and some additional encryption metadata are then stored as blob metadata along with the encrypted blob on the service. Important – it is important that if you are editing or uploading your own metadata for the blob, you need to make sure that this metadata is preserved. If you upload new metadata without this metadata, the wrapped CEK, IV and other metadata will be lost and the blob content will never be retrievable again.
Downloading an encrypted blob in this case involves getting the entire blob content using the DownloadTo*/BlobReadStream convenience methods. The wrapped CEK is unwrapped and used along with the IV (stored as blob metadata in this case) to return the decrypted data to the users.
Downloading an arbitrary range (DownloadRange* methods) in the encrypted blob involves adjusting the range provided by users in order to get a small amount of additional data that can be used to successfully decrypt the requested range.
All blob types (Block blobs and page blobs) can be encrypted/decrypted using this scheme.
Queues
Since queue messages can be of any format, the client library defines a custom format that includes the Initialization Vector (IV) and the encrypted content encryption key (CEK) in the message text.
During encryption, the client library will generate a random IV of 16 bytes along with a random CEK of 32 bytes and do envelope encryption of the queue message text using this information. The wrapped CEK and some additional encryption metadata are then added to the encrypted queue message. This modified message (shown below) is stored on the service.
<MessageText>{"EncryptedMessageContents":"6kOu8Rq1C3+M1QO4alKLmWthWXSmHV3mEfxBAgP9QGTU++MKn2uPq3t2UjF1DO6w","EncryptionData":{…}}</MessageText>During decryption, the wrapped key is extracted from the queue message and unwrapped. The IV is also extracted from the queue message and used along with the unwrapped key to decrypt the queue message data. Note that the encryption metadata is small (under 500 bytes), so while it does count toward the 64KB limit for a queue message, the impact should be manageable.
Tables
In the current version, the client SDK supports encryption of entity properties for Insert / Replace. Merge is not currently supported due to some limitations - Since a subset of properties may have been encrypted previously using a different key, just merging the new properties and updating the metadata will result in data loss. This will either require extra service calls to read the pre-existing entity from the service or using a new key per property both of which are not suitable for performance reasons.
Table data encryption works as follows -
Users specify the properties that should be encrypted.
- The client library will generate a random Initialization Vector (IV) of 16 bytes along with a random content encryption key (CEK) of 32 bytes for every entity and do envelope encryption on the individual properties that should be encrypted by deriving a new IV per property.
- The wrapped CEK and some additional encryption metadata are then stored as 2 additional reserved properties. The first reserved property (_ClientEncryptionMetadata1) is a string property that holds the information about IV, version, wrapped key etc and the other reserved property (_ClientEncryptionMetadata2) is a binary property that holds the information about the properties that are encrypted.
- Due to these additional reserved properties required for encryption, users can now only have 250 custom properties instead of 252 and the overall size of the entity data allowed is less than 1MB.
Only string properties can be encrypted. If other types of properties have to be encrypted, users have to convert them to strings.
For tables, in addition to the encryption policy, users have to specify the properties that should be encrypted. This can be done by either specifying an [EncryptProperty] attribute (for POCO entities that derive from TableEntity) or an encryption resolver in request options. Encryption Resolver is a delegate that takes in PK, RK and a property name and returns a Boolean that indicates whether that property should be encrypted. During encryption, the client library will use this information to decide whether a property should be encrypted while writing to the wire. It also provides the advantage of letting users have some smart logic around, if X, then encrypt property A, else, encrypt properties A and B etc. Note it is not necessary to provide this information while reading/querying entities.
Batch Operations
In batch operations, the same KEK will be used across all the rows in that batch operation since the client library only allows one options object (and hence one policy/KEK) per batch operation. However, the client library internally will generate a new random IV and random CEK per row in the batch. Users can also choose to encrypt different properties for every operation in the batch by defining this behavior in the EncryptionResolver.
Queries
When users wish to perform query operations, they will have to specify a key resolver that is able to resolve all the keys in the result set. If an entity contained in the query result cannot be resolved to a provider, the client library will throw an error. For any query that does server side projections, the client library will add the special encryption metadata properties (_ClientEncryptionMetadata1 and _ClientEncryptionMetadata2) by default to the selected columns.
Azure Key Vault
Azure Key Vault—currently in Preview—helps safeguard cryptographic keys and secrets used by cloud applications and services. By using Azure Key Vault, users can encrypt keys and secrets (such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords) by using keys that are protected by hardware security modules (HSMs). More information about Key Vault and Getting Started documents can be found here.
The Storage client library uses the Key Vault core library in order to provide a common framework across Azure for managing keys. Users also get the additional benefit of using the Key Vault extensions library that provides a lot of useful functionality around simple and seamless Symmetric/RSA local and cloud key providers along with aggregation and caching.
Interface and Dependencies
There are three Key Vault packages:
- Microsoft.Azure.KeyVault.Core: This has IKey and IKeyResolver. It is a very small package and has no dependencies. The Storage Client Desktop and Phone libraries define this as a dependency.
- Microsoft.Azure.KeyVault: This is the Key Vault REST client.
- Microsoft.Azure.KeyVault.Extensions: This is extension code that includes implementations of cryptographic algorithms along with an RsaKey and a SymmetricKey. This depends on Core and KeyVault and provides functionality to define an aggregate resolver (when users want to use multiple key providers) and a caching key resolver. Although the Storage client library does not directly depend on this, if users wish to use Azure Key Vault to store their keys or just use the Key Vault extensions to consume the local and cloud crypto providers, they will need this package.
Key Vault is designed for high value master keys, and throttling limits per Vault are designed with this in mind. When doing client-side encryption with Key Vault, the preferred model is to use symmetric master keys stored as Secrets in Key Vault and cached locally. Users have to do the following –
- Create a secret offline and upload it to Key Vault.
- Use the secrets’ base identifier as a parameter to resolve the current version of the secret for encryption and cache this information locally (Using CachingKeyResolver takes care of this and users are not expected to implement their own caching logic).
- Use the caching resolver as an input while creating the Encryption Policy.
More information regarding Key Vault usage can be found in the code samples here.
Note
Encryption support is available only on Windows Desktop and Windows Phone. Windows Runtime does not have support for encryption. Additionally, Key Vault extensions are not supported for Windows phone yet. So if users want to use storage client encryption on phone, they will have to implement their own key providers. Also, due to a limitation in the Windows Phone .NET platform, page blob encryption is currently not supported on Windows Phone.
As noted in the announcement blog, please be aware that –
- This is a preview! It should not be used for production data.
- It is easy to corrupt data on the blob service or make it un-readable – If blob update operations like WritePages/ClearPages, PutBlock etc are done once an encrypted blob is written to the service, it will end up corrupting the encrypted blob making it unreadable. For encryption, only full blob upload methods and range/full blob download methods should be used.
- For tables, a similar constraint exists – Do not update encrypted properties without updating the encryption metadata.
- Also, if a SetMetadata is done on an encrypted blob (since set metadata is not additive), it can wipe out all encryption related metadata required for decryption. Same with snapshots - specifying metadata while creating a snapshot of an encrypted blob.
Client API / Interface
While creating an EncryptionPolicy object, users can provide only a Key (implementing IKey) or a resolver (implementing IKeyResolver) or both. IKey is the basic key type that is identified using a key identifier and provides the logic for wrapping/unwrapping. IKeyResolver is used to resolve a key during the decryption process. It defines a ResolveKey method that returns an IKey given a key identifier. This is used to provide users the ability to choose between multiple keys that are managed in multiple locations.
- For encryption, the key is used always and the absence of a key will result in an error.
- For decryption,
- The key resolver is invoked if specified to get the key. If the resolver is specified but does not have a mapping for the key identifier, an error is thrown.
- If resolver is not specified but a key is specified, key identifier on the key is matched with what is stored on the service and used.
GettingStartedSamples in the Storage client’s Github repo will demonstrate a more detailed end-to-end scenario for blobs, queues and tables along with Key Vault integration.
Blobs
Users will create a BlobEncryptionPolicy object and set it in the request options (per API or at a client level by using DefaultRequestOptions). Everything else will be handled by the client library internally.
// Create the IKey used for encryption.
RsaKey key = new RsaKey("private:key1"/* key identifier */);
// Create the encryption policy to be used for upload and download.
BlobEncryptionPolicy policy = new BlobEncryptionPolicy(key, null);
// Set the encryption policy on the request options.
BlobRequestOptions options = new BlobRequestOptions() { EncryptionPolicy = policy };
// Upload the encrypted contents to the blob.
blob.UploadFromStream(stream, size, null, options, null);
// Download and decrypt the encrypted contents from the blob.
MemoryStream outputStream = new MemoryStream();
blob.DownloadToStream(outputStream, null, options, null);
Queues
Users will create a QueueEncryptionPolicy object and set it in the request options (per API or at a client level by using DefaultRequestOptions). Everything else will be handled by the client library internally.
// Create the IKey used for encryption.
RsaKey key = new RsaKey("private:key1"/* key identifier */);
// Create the encryption policy to be used for upload and download.
QueueEncryptionPolicy policy = new QueueEncryptionPolicy(key, null);
// Add message
QueueRequestOptions options = new QueueRequestOptions() { EncryptionPolicy = policy };queue.AddMessage(message, null, null, options, null);
// Retrieve message
CloudQueueMessage retrMessage = queue.GetMessage(null, options, null);
Tables
In addition to creating an encryption policy and setting it on request options, users will have to specify an EncryptionResolver in TableRequestOptions or set attributes on the entity.
Using Resolver
// Create the IKey used for encryption.
RsaKey key = new RsaKey("private:key1"/* key identifier */);
// Create the encryption policy to be used for upload and download.
TableEncryptionPolicy policy = new TableEncryptionPolicy(key, null);
TableRequestOptions options = new TableRequestOptions()
{ EncryptionResolver = (pk, rk, propName) =>
{if (propName == "foo")
{returntrue;
}
returnfalse;
},
EncryptionPolicy = policy
};
// Insert Entity
currentTable.Execute(TableOperation.Insert(ent), options, null);
// Retrieve Entity
// No need to specify an encryption resolver for retrieve
TableRequestOptions retrieveOptions = new TableRequestOptions()
{ EncryptionPolicy = policy
};
TableOperation operation = TableOperation.Retrieve(ent.PartitionKey, ent.RowKey);
TableResult result = currentTable.Execute(operation, retrieveOptions, null);
Using Attributes
As mentioned above, if the entity implements TableEntity, then the properties can be decorated with the [EncryptProperty] attribute instead of specifying the EncryptionResolver.
[EncryptProperty]
publicstring EncryptedProperty1 { get; set; }Conclusion
We want feedback on the ease of use, security, or any other scenarios you would like to tell us about. This will enable us to use that feedback in shaping the final library.
Client-Side Encryption for Microsoft Azure Storage – Preview
Download the Azure Storage Client Library for .NET NuGet package
Download the Azure Storage Client Library for .NET Source Code from GitHub
Download the Azure Key Vault NuGet Core, Client, and Extensions packages
Visit the KV Documentation here
Veena Udayabhanu
Microsoft Azure Storage Team
![]()
![clip_image001[1]](http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-36-55-metablogapi/2335.clip_5F00_image0011_5F00_thumb_5F00_26DB56DB.png "clip_image001[1]")
