GPFS encryption with multiple remote tenants

OBS: Not having access to all keys for a file system being mounted is maybe not supported. At a minimum, “mmhealth” will complain about the keys listed in the policy that it has no access to.

GPFS RFAC (Remote Fileset Access Control) makes it possible to give remote clusters different views into which filesets are accessible in a file system. Unfortunately this is not a strong security bondary, as it’s implemented client-side, and could potentially be overridden by a rouge client. To protect against rouge clients, we can utilize encryption at rest with separate keys for each remote cluster. Then a rough client will only be able to list the file metadata (which is not encrypted), and won’t be able to access any of the encrypted data. To demonstrate this, we create the “cryptofs” file system in our storage cluster:

# mmvdisk vs define --vs cryptofs-01 --rg rg_sss6k --code 8+2p --block-size 16M --set-size 1T
# mmvdisk vs create --vs cryptofs-01
# mmvdisk fs create --fs cryptofs --vs cryptofs-01

Then create a fileset for each tenant:

# mmmount cryptofs -a
# mmcrfileset cryptofs tenant1 --inode-space new
# mmcrfileset cryptofs tenant2 --inode-space new
# mmlinkfileset cryptofs tenant1 -J /gpfs/cryptofs/tenant1
# mmlinkfileset cryptofs tenant2 -J /gpfs/cryptofs/tenant2
# mmauth grant client1.cluster -f cryptofs:root,tenant1
# mmauth grant client2.cluster -f cryptofs:root,tenant2

At this point the remote clusters can mount this file system, and will only see the relevant fileset. Next we need to connect storage cluster to key server, create tenant, create client, register client with tenants, and create a key for each tenant:

# mmkeyserv server add sklm1.example.net --backup sklm2.example.net
# mmkeyserv tenant add tenant1 --server sklm1.example.net
# mmkeyserv tenant add tenant2 --server sklm1.example.net
# mmkeyserv client create sss6k --server sklm1.example.net
# mmkeyserv client register sss6k --tenant tenant1 --rkm-id rkm_tenant1
# mmkeyserv client register sss6k --tenant tenant2 --rkm-id rkm_tenant2
# mmkeyserv key create --server sklm1.example.net --tenant tenant1
KEY-f839db18-5a63-4a2b-a5e8-7ac0793c30e7
# mmkeyserv key create --server sklm1.example.net --tenant tenant2
KEY-24d72e66-ab50-4076-a16c-df3f177a830f

Now we can use these keys for encrypting data in each of the filesets:

# cat <<'EOF' > cryptofs.policy
RULE 'default' SET POOL 'system'
RULE 'tenant1rule' ENCRYPTION 'tenant1key' IS ALGO 'DEFAULTNISTSP800131A' KEYS('KEY-f839db18-5a63-4a2b-a5e8-7ac0793c30e7:rkm_tenant1')
RULE 'tenant2rule' ENCRYPTION 'tenant2key' IS ALGO 'DEFAULTNISTSP800131A' KEYS('KEY-24d72e66-ab50-4076-a16c-df3f177a830f:rkm_tenant2')
RULE 'encTenant1' SET ENCRYPTION 'tenant1key' FOR FILESET ('tenant1')
RULE 'encTenant2' SET ENCRYPTION 'tenant2key' FOR FILESET ('tenant2')
EOF

# mmchpolicy cryptofs cryptofs.policy

On the client clusters, we need to register each cluster as a client of these same tenants. So for client that’s supposed to have access to tenant1, we would do:

# mmkeyserv server add sklm1.example.net --backup sklm2.example.net
# mmkeyserv tenant add tenant1 --server sklm1.example.net
# mmkeyserv client create client1 --server sklm1.example.net
# mmkeyserv client register client1 --tenant tenant1 --rkm-id rkm_tenant1

And similar for client2 with access to tenant2:

# mmkeyserv server add sklm1.example.net --backup sklm2.example.net
# mmkeyserv tenant add tenant2 --server sklm1.example.net
# mmkeyserv client create client2 --server sklm1.example.net
# mmkeyserv client register client2 --tenant tenant2 --rkm-id rkm_tenant2

We can then test from the client clusters:

# mmremotefs add cryptofs -f cryptofs -C sss6k -A yes
# mmmount cryptofs
# ls -al /gpfs/cryptofs
total 258
drwxr-xr-x. 4 root root 262144 Sep 17 12:00 .
drwxr-xr-x. 4 root root     42 Sep  1 12:16 ..
dr-xr-xr-x. 2 root root  16384 Jan  1  1970 .snapshots
drwx------. 2 root root   4096 Sep 17 11:58 tenant1

# echo encrypted > /gpfs/cryptofs/tenant1/encrypted.file

# mmlsattr -L  /gpfs/cryptofs/tenant1/encrypted.file|grep -i encr
file name:            /gpfs/cryptofs/tenant1/encrypted.file
Encrypted:            yes

# cat /gpfs/cryptofs/tenant1/encrypted.file
encrypted

Also we can do a mis-configuration, and allow client2 RFAC access to both tenant1 and tenant2:

# mmauth grant client2.cluster -f cryptofs:root,tenant1,tenant2

Then on that client2.cluster we can list both tenant1 and tenant2, but any access to files in tenant1 will fail because we don’t have access to the encryption key for those files:

# mmremotefs add cryptofs -f cryptofs -C sss6k -A yes
# mmmount cryptofs


# ls -al /gpfs/cryptofs/
total 258
drwxr-xr-x. 4 root root 262144 Sep 17 12:16 .
drwxr-xr-x. 4 root root     42 Sep  1 12:30 ..
dr-xr-xr-x. 2 root root  16384 Jan  1  1970 .snapshots
drwx------. 2 root root   4096 Sep 17 12:16 tenant1
drwx------. 2 root root   4096 Sep 17 12:17 tenant2

# ls -la  /gpfs/cryptofs/tenant1/encrypted.file
-rw-r--r--. 1 root root 10 Sep 17 12:16 /gpfs/cryptofs/tenant1/encrypted.file

# cat /gpfs/cryptofs/tenant1/encrypted.file
cat: /gpfs/cryptofs/tenant1/encrypted.file: Operation not permitted