Secure Data Storage
Secure Data Storage (SDS) is a service that helps you securely control access to your data. SDS provides storage for sensitive data and lets you control who is authenticated (signed in) and authorized (has permissions) to use data..
This document explains how you can use SDS to store sensitive data and control access to the stored data. It also explains client-end functionality, i.e., how to decrypt the data post-reading.
- Secure Storage: Ensures that sensitive data is stored securely and access is controlled.
- Access Control: Provides comprehensive mechanisms to authenticate and authorize user access.
- Flexibility: Supports various data types including files and configurations, adaptable to different use cases.
- Scalability: Capable of handling large volumes of data and numerous applications.
To store App data in SDS for security purposes, register the App by calling the SDS CreateApp endpoint. You must specify a name that is unique within the Tenant and a base64 encoded RSA public key of any size.
A Tenant can register as many Apps with SDS as is required. On successful registration of the App, SDS returns a unique App ID in the response. You can use the App ID to read the application configuration or update the RSA public key.
🚧 It is the application's responsibility to create an RSA key pair and maintain the private key with itself. SDS only takes the public key to encrypt the data.
To register an App in the SDS service, call the SDS CreateApp endpoint.
Request bodies are specified in JSON format. The following example shows a request body to register an App:
Member | Description | Validation | Data type |
|---|---|---|---|
name | Name of the application which has to be registered. | pattern: ^[a-zA-Z0-9-_]{3,16}$ example: Participant | string |
key | The public key generated by the application. | | string |
To update an App in the SDS service, call the SDS UpdateApp endpoint.
Request bodies are specified in JSON format. The following example shows a request body to Update an App:
This section walks through generating an RSA key pair and obtaining a base64 encoded public key using OpenSSL. These keys are commonly used in various applications, such as JWT, SSH, and other scenarios that require RSA encryption.
Prerequisites
OpenSSL: Ensure that OpenSSL is installed on your system.
- macOS: Most macOS systems have OpenSSL pre-installed.
- Windows: If OpenSSL is not already available, you may need to install it separately.
Steps
- Open the Terminal (macOS) or Command Prompt (Windows):
- macOS: You can find the Terminal in Applications > Utilities.
- Windows: Search for "Command Prompt" in the Start menu.
- Generate an RSA Private Key:
This command generates a 2048-bit RSA private key and saves it to a file named private_key.pem.
3. Extract the Public Key from the Private Key:
This command takes the RSA private key from private_key.pem, extracts the corresponding public key, and saves it to public_key.pem
4. Convert the Public Key to Base64 (DER Format):
Result
- The private key is stored in the private_key.pem file.
- The public key is stored in the public_key.pem file.
- The Base64 encoded public key is displayed in the terminal output.
The X-SDS-SIGNATURE signature identifies which SDS application is making the request and is used for the authentication and authorization of that application.
For example:
- Only the owner application can modify an existing vault.
- Verify that the application has permission to store/read data to or from a vault.
The format of the signature is: <base64encoded_signature>.<base64encoded_appName>
Generate the signature by using the RSA private key which was generated by the App and SHA1WITHRSA as the algorithm. An RSA signature can only be verified using the public key of that pair and the same algorithm which was used to generate it.
Encode the signature in base64 and appended the base64 encoded App name with the dot sign.
❗CAUTION
The keys and signature might include some non-printable characters. We recommend that you generate it programmatically rather than using an online utility.
This header captures the AAA user ID, user name, and application ID of the entity accessing the data, adding an additional layer of traceability and security. Example
Note: The values above are examples. Replace them with your name, ID, and appID.
Automatic Header Hydration:
If the x-coreos-user-info header is not provided on the API call, the OS1 platform automatically populates using the information from the authentication token provided with the request. This means the platform derives the user information from the token if it is not supplied by the developer making the request, whether the request is made through the UI or a service.
Header Overwrite: If the x-coreos-user-info header is manually set by the developer, the service mesh will not modify this header. There are two key scenarios to consider:
- Requests Initiated from the UI: When a request originates from the UI (e.g., a web application or mobile app), the x-coreos-user-info header is automatically included in the request by the platform. In this case, the developer must forward this header exactly as it is received, without making any modifications, when interfacing with audit logs.
- Requests Initiated by Backend Services: For requests initiated by backend services (such as those sending scheduled SMS, emails, or push notifications), developers have the option to include the x-coreos-user-info header or let the platform handle it. If included, this header will be reflected in the audit logs.
Caution on Data Alteration
Modifying or overwriting header information can lead to incorrect data being logged in the audit stream. To prevent discrepancies, avoid altering or overwriting header information.
A Vault is a secure storage mechanism that is designed to protect your confidential data against unauthorized access. Tenants can create as many Vaults as needed to satisfy their business needs.
On successful creation of the Vault, the SDS service returns a unique ID for the vault which can be used to modify the Vault or read the existing configurations. After the Vault is created only the owner application can read or modify its configuration. The owner of the Vault has access to modify permissions for any other application to that Vault.
Vault permission is set by setting the App name and including permission flags. To set the Vault permissions the value of the appName member by specifying the name of the application for which the permission is required and setting the permission1 member to the three-digit binary string permission string. For example: appName= “Application123” and permission= “101”.
The three characters in the permission specify the write_permssion, decrypted_read_permission, encrypted_read_permission respectively. The allowed permissions for apps are:
- 110: Can read decrypted data from the vault and write data in the vault.
- 101: Can read data in encrypted form from the vault and write data in the vault.
- 100: Can write data in the vault.
- 010: Can read decrypted data from the vault.
- 001: Can read data in encrypted form from the vault.
- 000: No permissions assigned.
Only the applications which have permission to a Vault are able to write or read the data from that Vault. The owner of the Vault is assigned with permission 101 by default which can be later updated through the update-vault API.
📘 NOTE
Owner permission cannot be changed from default while creating the Vault but can be updated later.
Other than App permissions, Vaults have a read limit associated with them. This value specifies the maximum number of records that are returned in a single request. The minimum number of records is 1 and the maximum is 50.
Vault names must be unique to the tenant.
To create a Vault, call the CreateVault endpoint.
Request bodies are specified in JSON format. The following example shows a request body to create a Vault.
Member | Description | Validation | Data type |
|---|---|---|---|
name* | Vault name. | pattern: ^[a-zA-Z0-9-_]{3,16}$ example: phone-number | string |
owner* | Vault owner. | | string |
readLimit (Optional) | Read limit. | default: 1 minimum: 1 maximum: 50 | integer($int64) |
uniqueData | | default: false | boolean |
permissions (Optional) | List of applications allowed to read/write data in the vault according to the permission specified. | | |
app* | | example: Participant | string |
permission* | Application permission. | Valid values: [110, 101, 100, 010, 001, 000] 110: application with this permission can read decrypted data from the vault and write data in the vault. 101: application with this permission can read data from the vault and write data in the vault. 100: application with this permission can write data in the vault. 010: application with this permission can read decrypted data from the vault. 001: application with this permission can read data from the vault. 000: the application doesn't have any permission. | string |
enable (Optional) | | | boolean |
📘NOTE
The items marked (*) are mandatory attributes required while creating the Participants via batch processing.
To disable a Vault, call the UpdateVault endpoint and set the enable member of the request body to False. A Vault can be disabled only if no data exists in that Vault.
Data is confidential information that is stored in a Vault to protect it. This data can be stored, updated, deleted, or read depending on the permissions that the application has:
- The data can be stored in any of the Vaults by the applications which have write permission to that Vault.
- The existing data in a Vault can be updated by any application which has write permission to that Vault.
- The existing data can be read by any application which has any of the read permissions to that Vault.
The audit logging feature captures every API request made to the SDS data API (core/api/v1/sds/data). The captured log is included in an audit log stream for compliance, security, and traceability. This feature ensures a secure and compliant data environment by capturing detailed logs of every interaction within the SDS service. It enables tenant organizations to monitor and audit access and modifications made to the secure data.
Following are the key capabilities of the audit logging feature:
- Captures user activities
- Records user IDs, app IDs, timestamps, action details, and event types like creation or deletion.
- Tracks data accessed or modified.
- User and application IDs for verification.
- Provides security and compliance
- Handles PII data that ensures compliance with standards like GDPR and CCPA.
- Identifies unauthorized access attempts.
- Enhanced user and application identification
- Improved logging mechanisms now utilize the x-coreos-user-info header for detailed user and application identification.
- Configurable data access logging
- Vaults can be configured to log PII data access. You can configure which vaults log access to PII data, allowing for customized logging settings based on the sensitivity of the data.
The SDS Data API incorporates detailed audit logging to track and record API requests. This process involves two main steps:
- Request logging: Every API request that is sent to the SDS data API is automatically logged. This includes the endpoint accessed, the time of access, and the data requested or sent.
- User and application Identification: The x-coreos-user-info header is utilized to capture and log identification details related to each request such as:
- AAA User ID: The unique identifier for the user making the request.
- User Name: The name of the user making the request.
- Application ID: The ID of the application initiating the request.
To enable and utilize the audit logging features, app developers must ensure that the x-coreos-user-info header is included in API requests. This header is crucial for logging user and application identification of data accurately. To learn more, refer to the X-Coreos-User-Info Header section.
Audit logs are stored securely in S3 buckets as Parquet files, which are an efficient format for handling large volumes of data. You can access and analyze these logs using AWS Athena, which allows you to run SQL queries directly on the data. This setup ensures that log data is stored efficiently and is easily accessible for compliance audits or security reviews.
Below are the list of attributes that will be logged as part of the audit logs.
- Tenant ID: The tenant is trying to access the data. (Partition Key)
- Timestamp: When the access occurred. (Partition Key) in INT64 Mills
- Initiator ID: The identity of the user/app/ system who accessed the data.
- Request ID: To correlate actions during a session.
- Event Type: read, write, update, delete
- Resource Accessed: Specific data or resource that was accessed. In this case, the SDS data-id.
- Outcome: Success or failure of the action. This is decided based on the access policy of the data.
- Additional Metadata: This includes the reason for access, API call, tenant, module, and so on.
An application can store data in a Vault if it has the write permission, i.e., any of the following: 100, 101, or 110. The data should be base64 encoded while storing it in the vault.
Along with the data, you can include metadata information. The metadata can be of any type (string/number/list/JSON). On successfully storing the data in the vault, SDS returns a unique data ID in the response which you can use to read or update the data.
To write data to a Vault, call the CreateData endpoint.
Request bodies are specified in JSON format. The following example shows a request body to write data to a Vault.
Member | Description | Data type |
|---|---|---|
data | Data to be stored. | string |
meta | Metadata for data stored in the vault. It is a key value pair. | |
vault | Data vault. | string |
An application can update data in a Vault by its ID if it has the write permission, i.e., any of the following: 100, 101, or 110. With this permission, the data, the metadata stored with it, or even the Vault can be updated.
To update the Vault, the requesting app must have the write permission to both the existing and the modified Vault.
To update data in a Vault, call the UpdateData endpoint.
Request bodies are specified in JSON format. The following example shows a request body to update data in a Vault.
An application can read data from a Vault by its ID if it has any of the read permissions, i.e., any of the following: 101, 110, 001, or 010.
Apps that have any of the decrypted read permissions (110 or 010) retrieve the data in the same form as it was stored.
Apps that have any of the encrypted read permissions (101 or 001) retrieve the data in the encrypted form. SDS uses hybrid encryption to provide a high level of security to data. Also, it allows for storing more data as RSA is limited to encrypting the data of a maximum of 512 bytes if RSA-2048 is used to generate the key pair.
The data from the Vault is encrypted with a secret (AES-128 bit key) of length 16 which is different every time and initialization-vector of 16 bytes ({0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}. The algorithm used to encrypt the data is AES/CBC/PKCS5Padding. The same should be used to decrypt the data.
The secret is encrypted with the application’s public key which was shared at the time of registration using the algorithm RSA/ECB/PKCS1Padding.
Both the encrypted secret and the encrypted data are sent in the response by appending with the dot sign. For example: <encrypted_secret>.<encrypted_data>
📘 INFO
Both the secret and the data are base64 encoded after encryption is applied. Base64 decoding has to be applied before decrypting them.
To decrypt the encrypted secret, first, apply base64 decoding to the secret and then use the private key of the same RSA key-pair generated at the time of registration.
The algorithm used to encrypt the secret is RSA/ECB/PKCS1Padding. The same should be used to decrypt the secret. The decrypted secret should have a length of 16 which is actually a 128-bit AES key.
To decrypt the data part, first, apply base64 decoding to the data string and then use the decrypted secret generated in the above step as the AES key. The algorithm used to encrypt the data is AES/CBC/PKCS5Padding. The same should be used to decrypt the data.
📘 NOTE
Since the keys may include some nonprintable characters. Hence, it is recommended to decrypt it programmatically rather than using any online utility. Also, do verify the length of the secret after decryption.
Apps can retrieve data from a Vault for more than one data ID at a time. When reading data using multiple data IDs, the rules around permissions and encryption-decryption of data are followed in the same way as for the single read. Note that the data read limit still applies when retrieving data for multiple data IDs.
The SDS service allows you to store confidential files within Vaults. These files can be read, downloaded, or searched based on the permission(s) that the application has.
You can store files or URLs of files which can be any file format: e.g. text, pdf, doc, audio, and video. Within SDS, you can also store tags with files that allow you to search through them. A tag consists of a tag key and a tag value. Tag keys and tag values are both required, but tag values can be empty (null) strings. For example, The key might be CONSIGNEE, and the value might be ADDRESS.
The SDS service allows you to upload files in two ways:
- File upload: By sending the file in the multipart-form-request or providing file data in binary format.
- URL upload: By sending a URL where the document is originally stored.
To upload a file, call the SDS UploadFile endpoint.
Request bodies are specified in JSON format. The following example shows a request body to upload a file to a Vault.
Member | Description | Data type |
|---|---|---|
name | Name of the file. | string |
url | URL for the file. | string |
description | Description of the file. | string |
tags | Includes key and value. | string |
vault | Data vault | string |
type | File types. Valid values: FILE, URL. | string |
callback | CallbackURL to send file upload status. | string($uri) |
The download of files from the SDS service is performed in two ways:
- If the file is stored in the SDS service then you are sent a pre-signed link that has a configurable expiration time.
- If only the file URL is stored then the store URL is returned.
When you upload a file, you have the option to add a tag along with it. These tags can be used to search and filter uploaded files. During the search, matching files or URL metadata is exposed to you. After that, you can download the file based on your ID.
For one tag there can be multiple docs available. During the search, the metadata for all matching documents is sent. After searching based on a tag, you are sent the metadata but you'll not be able to view the file content yet.
To view the content of the file, you need to make a separate request to download the file. After making a download request, you are checked for download access. If you have access, the file is downloaded and can be viewed.