Secure Data Storage

overview secure data storage (sds) is a service that helps you securely control access to your data sds provides storage for sensitive data and lets you control who is authenticated (signed in) and authorized (has permissions) to use data this document explains how you can use sds to store sensitive data and control access to the stored data it also explains client end functionality, i e , how to decrypt the data post reading secure storage ensures that sensitive data is stored securely and access is controlled access control provides comprehensive mechanisms to authenticate and authorize user access flexibility supports various data types including files and configurations, adaptable to different use cases scalability capable of handling large volumes of data and numerous applications getting started with secure data storage (sds) service​ to store app data in sds for security purposes, register the app by calling the sds createapp endpoint you must specify a name that is unique within the tenant and a base64 encoded rsa public key of any size a tenant can register as many apps with sds as is required on successful registration of the app, sds returns a unique app id in the response you can use the app id to read the application configuration or update the rsa public key 🚧 it is the application's responsibility to create an rsa key pair and maintain the private key with itself sds only takes the public key to encrypt the data to register an app in the sds service, call the sds createapp endpoint request bodies are specified in json format the following example shows a request body to register an app member description validation data type name name of the application which has to be registered pattern ^\[a za z0 9 ]{3,16}$ example participant string key the public key generated by the application string create app request body { "name" "sds app", "key" "miibijanbgkqhkig9w0baqefaaocaq8amiibcgkcaqeaji2fnzv1vvs9xq6ztghym9eyc8pz/+nr53v9ejxzbnoerokjwrbrhdxg/mnsr/xbk7tsyjk44ugrzwpcv+gkkurkkxuy4dtmkcacifpjolgvge4ildw31dkxrtihgkweduhskdgaux3480blgjaq560lhkzb50bpfmv+3avjofbqfkyylarxhzfgu/fxdqgbfjsfzgjjqqengu9mwsybkwjsjotyivgsxlt4x1wvwspdkq6i56ikrm1buwitgziodmbgoxqammeqgp4+bg6qodoehnfpzpytoj1oz6bkpprxi0jue5kntxtd3mynutqj3euedenozcswlwb7tu2tsqidaqab" } to update an app in the sds service, call the sds updateapp endpoint request bodies are specified in json format the following example shows a request body to update an app update app request body { "key" "miibijanbgkqhkig9w0baqefaaocaq8amiibcgkcaqeaji2fnzv1vvs9xq6ztghym9eyc8pz/+nr53v9ejxzbnoerokjwrbrhdxg/mnsr/xbk7tsyjk44ugrzwpcv+gkkurkkxuy4dtmkcacifpjolgvge4ildw31dkxrtihgkweduhskdgaux3480blgjaq560lhkzb50bpfmv+3avjofbqfkyylarxhzfgu/fxdqgbfjsfzgjjqqengu9mwsybkwjsjotyivgsxlt4x1wvwspdkq6i56ikrm1buwitgziodmbgoxqammeqgp4+bg6qodoehnfpzpytoj1oz6bkpprxi0jue5kntxtd3mynutqj3euedenozcswlwb7tu2tsqidaqab" } generate rsa key pair and base64 encoded public key using openssl this section walks through generating an rsa key pair and obtaining a base64 encoded public key using openssl these keys are commonly used in various applications, such as jwt, ssh, and other scenarios that require rsa encryption prerequisites openssl ensure that openssl is installed on your system macos most macos systems have openssl pre installed windows if openssl is not already available, you may need to install it separately steps open the terminal (macos) or command prompt (windows) macos you can find the terminal in applications > utilities windows search for "command prompt" in the start menu generate an rsa private key terminal openssl genpkey algorithm rsa out private key pem pkeyopt rsa keygen bits 2048 this command generates a 2048 bit rsa private key and saves it to a file named private key pem 3\ extract the public key from the private key terminal openssl rsa pubout in private key pem out public key pem this command takes the rsa private key from private key pem, extracts the corresponding public key, and saves it to public key pem 4\ convert the public key to base64 (der format) terminal openssl rsa pubin in public key pem outform der | base64 result the private key is stored in the private key pem file the public key is stored in the public key pem file the base64 encoded public key is displayed in the terminal output x sds signature the x sds signature signature identifies which sds application is making the request and is used for the authentication and authorization of that application for example only the owner application can modify an existing vault verify that the application has permission to store/read data to or from a vault the format of the signature is \<base64encoded signature> \<base64encoded appname> g enerate the signature generate the signature by using the rsa private key which was generated by the app and sha1withrsa as the algorithm an rsa signature can only be verified using the public key of that pair and the same algorithm which was used to generate it encode the signature in base64 and appended the base64 encoded app name with the dot sign ❗ caution caution the keys and signature might include some non printable characters we recommend that you generate it programmatically rather than using an online utility x coreos user info header this header captures the aaa user id, user name, and application id of the entity accessing the data, adding an additional layer of traceability and security example json { x coreos user info { id "c895429a 1e36 4b9e 8f44 c24f7fc63fcf", name "diwas kumar", appid "platform\ app\ dispatch fpa api" } } note the values above are examples replace them with your name, id, and appid automatic header hydration if the x coreos user info header is not provided on the api call, the os1 platform automatically populates using the information from the authentication token provided with the request this means the platform derives the user information from the token if it is not supplied by the developer making the request, whether the request is made through the ui or a service header overwrite if the x coreos user info header is manually set by the developer, the service mesh will not modify this header there are two key scenarios to consider requests initiated from the ui when a request originates from the ui (e g , a web application or mobile app), the x coreos user info header is automatically included in the request by the platform in this case, the developer must forward this header exactly as it is received, without making any modifications, when interfacing with audit logs requests initiated by backend services for requests initiated by backend services (such as those sending scheduled sms, emails, or push notifications), developers have the option to include the x coreos user info header or let the platform handle it if included, this header will be reflected in the audit logs caution on data alteration modifying or overwriting header information can lead to incorrect data being logged in the audit stream to prevent discrepancies, avoid altering or overwriting header information vaults a vault is a secure storage mechanism that is designed to protect your confidential data against unauthorized access tenants can create as many vaults as needed to satisfy their business needs vault creation on successful creation of the vault, the sds service returns a unique id for the vault which can be used to modify the vault or read the existing configurations after the vault is created only the owner application can read or modify its configuration the owner of the vault has access to modify permissions for any other application to that vault vault permissions vault permission is set by setting the app name and including permission flags to set the vault permissions the value of the appname member by specifying the name of the application for which the permission is required and setting the permission1 member to the three digit binary string permission string for example appname = “application123” and permission= “101” the three characters in the permission specify the write permssion , decrypted read permission , encrypted read permission respectively the allowed permissions for apps are 110 can read decrypted data from the vault and write data in the vault 101 can read data in encrypted form from the vault and write data in the vault 100 can write data in the vault 010 can read decrypted data from the vault 001 can read data in encrypted form from the vault 000 no permissions assigned only the applications which have permission to a vault are able to write or read the data from that vault the owner of the vault is assigned with permission 101 by default which can be later updated through the update vault api 📘 note note owner permission cannot be changed from default while creating the vault but can be updated later other vault configurations other than app permissions, vaults have a read limit associated with them this value specifies the maximum number of records that are returned in a single request the minimum number of records is 1 and the maximum is 50 vault names must be unique to the tenant to create a vault, call the createvault endpoint request bodies are specified in json format the following example shows a request body to create a vault member description validation data type name vault name pattern ^\[a za z0 9 ]{3,16}$ example phone number string owner vault owner string readlimit (optional) read limit default 1 minimum 1 maximum 50 integer($int64) uniquedata default false boolean permissions (optional) list of applications allowed to read/write data in the vault according to the permission specified app example participant string permission application permission valid values \[110, 101, 100, 010, 001, 000] 110 application with this permission can read decrypted data from the vault and write data in the vault 101 application with this permission can read data from the vault and write data in the vault 100 application with this permission can write data in the vault 010 application with this permission can read decrypted data from the vault 001 application with this permission can read data from the vault 000 the application doesn't have any permission string enable (optional) boolean 📘 note note the items marked ( ) are mandatory attributes required while creating the participants via batch processing create vault request { "name" "vault", "owner" "sds", "readlimit" 49, "permission" \[{ "app" "vaultapp", "permission" "110" }] }{ "permissions" \[ { "app" "app207", "permission" "010" } ], "enable" true } disabling vault to disable a vault, call the updatevault endpoint and set the enable member of the request body to false a vault can be disabled only if no data exists in that vault data operations data is confidential information that is stored in a vault to protect it this data can be stored, updated, deleted, or read depending on the permissions that the application has the data can be stored in any of the vaults by the applications which have write permission to that vault the existing data in a vault can be updated by any application which has write permission to that vault the existing data can be read by any application which has any of the read permissions to that vault audit logging the audit logging feature captures every api request made to the sds data api (core/api/v1/sds/data) the captured log is included in an audit log stream for compliance, security, and traceability this feature ensures a secure and compliant data environment by capturing detailed logs of every interaction within the sds service it enables tenant organizations to monitor and audit access and modifications made to the secure data audit log capabilities following are the key capabilities of the audit logging feature captures user activities records user ids, app ids, timestamps, action details, and event types like creation or deletion tracks data accessed or modified user and application ids for verification provides security and compliance handles pii data that ensures compliance with standards like gdpr and ccpa identifies unauthorized access attempts enhanced user and application identification improved logging mechanisms now utilize the x coreos user info header for detailed user and application identification configurable data access logging vaults can be configured to log pii data access you can configure which vaults log access to pii data, allowing for customized logging settings based on the sensitivity of the data capturing audit log the sds data api incorporates detailed audit logging to track and record api requests this process involves two main steps request logging every api request that is sent to the sds data api is automatically logged this includes the endpoint accessed, the time of access, and the data requested or sent user and application identification the x coreos user info header is utilized to capture and log identification details related to each request such as aaa user id the unique identifier for the user making the request user name the name of the user making the request application id the id of the application initiating the request enabling audit log to enable and utilize the audit logging features, app developers must ensure that the x coreos user info header is included in api requests this header is crucial for logging user and application identification of data accurately to learn more, refer to the secure data storage docid\ xjyuymlr rnr5xpe6iazl section accessing audit log audit logs are stored securely in s3 buckets as parquet files, which are an efficient format for handling large volumes of data you can access and analyze these logs using aws athena, which allows you to run sql queries directly on the data this setup ensures that log data is stored efficiently and is easily accessible for compliance audits or security reviews audit attributes below are the list of attributes that will be logged as part of the audit logs tenant id the tenant is trying to access the data (partition key) timestamp when the access occurred (partition key) in int64 mills initiator id the identity of the user/app/ system who accessed the data request id to correlate actions during a session event type read, write, update, delete resource accessed specific data or resource that was accessed in this case, the sds data id outcome success or failure of the action this is decided based on the access policy of the data additional metadata this includes the reason for access, api call, tenant, module, and so on data storage in vault an application can store data in a vault if it has the write permission, i e , any of the following 100, 101, or 110 the data should be base64 encoded while storing it in the vault along with the data, you can include metadata information the metadata can be of any type (string/number/list/json) on successfully storing the data in the vault, sds returns a unique data id in the response which you can use to read or update the data to write data to a vault, call the createdata endpoint request bodies are specified in json format the following example shows a request body to write data to a vault member description data type data data to be stored string meta metadata for data stored in the vault it is a key value pair vault data vault string create data { "data" "eyjwag9uzsi6mtizndu2fq==", "meta" { "name" "verification" }, "vault" "vault" } update data in vault an application can update data in a vault by its id if it has the write permission, i e , any of the following 100, 101, or 110 with this permission, the data, the metadata stored with it, or even the vault can be updated to update the vault, the requesting app must have the write permission to both the existing and the modified vault to update data in a vault, call the updatedata endpoint request bodies are specified in json format the following example shows a request body to update data in a vault sample payload to update data { "data" "eyjwag9uzsi6mtizndu2fq==", "enable" true, "meta" { "key" { "key2" "value" }, "key3" "value3" }, "vault" "vault1" } read data from vault an application can read data from a vault by its id if it has any of the read permissions, i e , any of the following 101, 110, 001, or 010 apps that have any of the decrypted read permissions (110 or 010) retrieve the data in the same form as it was stored apps that have any of the encrypted read permissions (101 or 001) retrieve the data in the encrypted form sds uses hybrid encryption to provide a high level of security to data also, it allows for storing more data as rsa is limited to encrypting the data of a maximum of 512 bytes if rsa 2048 is used to generate the key pair the data from the vault is encrypted with a secret ( aes 128 bit key) of length 16 which is different every time and initialization vector of 16 bytes ({0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} the algorithm used to encrypt the data is aes/cbc/pkcs5padding the same should be used to decrypt the data the secret is encrypted with the application’s public key which was shared at the time of registration using the algorithm rsa/ecb/pkcs1padding both the encrypted secret and the encrypted data are sent in the response by appending with the dot sign for example \<encrypted secret> \<encrypted data> 📘 info info both the secret and the data are base64 encoded after encryption is applied base64 decoding has to be applied before decrypting them how to decrypt the secret? to decrypt the encrypted secret, first, apply base64 decoding to the secret and then use the private key of the same rsa key pair generated at the time of registration the algorithm used to encrypt the secret is rsa/ecb/pkcs1padding the same should be used to decrypt the secret the decrypted secret should have a length of 16 which is actually a 128 bit aes key how to decrypt data? to decrypt the data part, first, apply base64 decoding to the data string and then use the decrypted secret generated in the above step as the aes key the algorithm used to encrypt the data is aes/cbc/pkcs5padding the same should be used to decrypt the data 📘 note note since the keys may include some nonprintable characters hence, it is recommended to decrypt it programmatically rather than using any online utility also, do verify the length of the secret after decryption multiple data read from a vault apps can retrieve data from a vault for more than one data id at a time when reading data using multiple data ids, the rules around permissions and encryption decryption of data are followed in the same way as for the single read note that the data read limit still applies when retrieving data for multiple data ids file the sds service allows you to store confidential files within vaults these files can be read, downloaded, or searched based on the permission(s) that the application has you can store files or urls of files which can be any file format e g text, pdf, doc, audio, and video within sds, you can also store tags with files that allow you to search through them a tag consists of a tag key and a tag value tag keys and tag values are both required, but tag values can be empty (null) strings for example, the key might be consignee, and the value might be address upload file the sds service allows you to upload files in two ways file upload by sending the file in the multipart form request or providing file data in binary format url upload by sending a url where the document is originally stored to upload a file, call the sds uploadfile endpoint request bodies are specified in json format the following example shows a request body to upload a file to a vault member description data type name name of the file string url url for the file string description description of the file string tags includes key and value string vault data vault string type file types valid values file, url string callback callbackurl to send file upload status string($uri) { "name" "file4", "url" "", "description" "description of the file", "tags" \[{"key" "name" ,"value" "sample"}], "vault" "vault27", "type" "url", "callback" "https //examplecallbackurl com" } download file the download of files from the sds service is performed in two ways if the file is stored in the sds service then you are sent a pre signed link that has a configurable expiration time if only the file url is stored then the store url is returned search file when you upload a file, you have the option to add a tag along with it these tags can be used to search and filter uploaded files during the search, matching files or url metadata is exposed to you after that, you can download the file based on your id for one tag there can be multiple docs available during the search, the metadata for all matching documents is sent after searching based on a tag, you are sent the metadata but you'll not be able to view the file content yet to view the content of the file, you need to make a separate request to download the file after making a download request, you are checked for download access if you have access, the file is downloaded and can be viewed