GPFS Policy notes
My ever growing list of GPFS policy tips and tricks.
common.include
I like to maintain a few useful macros in a file I call “common.include” and then include this in all my policies with:
include(/gpfs/gpfsmgmt/etc/common.include)
If used in a file system installed policy, make sure that the file system containing the “common.include” file is mounted before the file system using it. Otherwise we won’t be able to mount it. I usually put it in the cesSharedRoot file system, and control the mounting order with “mmlsfs all --mount-priority”.
Useful includes from common.include:
define( is_premigrated, (MISC_ATTRIBUTES LIKE '%M%' AND (MISC_ATTRIBUTES NOT LIKE '%V%')))
define( is_migrated, (MISC_ATTRIBUTES LIKE '%V%') )
define( is_resident, (MISC_ATTRIBUTES NOT LIKE '%M%') )
define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))
define( hours_from_modification, (CURRENT_TIMESTAMP - MODIFICATION_TIME))
define( mb_allocated, (INTEGER(KB_ALLOCATED / 1024)))
/* Useful TCT related attributes */
define(EA_FSIZE, XATTR_INTEGER('dmapi.MCEA', 129, 8, 'BIG_ENDIAN'))
define(MTIME_SEC, XATTR_INTEGER('dmapi.MCEA', 9, 8, 'BIG_ENDIAN'))
define(MTIME_NSEC, XATTR_INTEGER('dmapi.MCEA', 17, 8, 'BIG_ENDIAN'))
define(MTIME_XATTR, MTIME_SEC+DECIMAL(MTIME_NSEC/1000)/1000000)
define(MTIME_FILE, DECIMAL(((MODIFICATION_TIME-TIMESTAMP(0)) SECONDS(12,6))))
/* TCT needs timestamp comparison to decide if cloud version is same as local version: */
define( is_premigrated_different, ((XATTR('dmapi.MCEA', 5, 1) == 'C') AND (MTIME_FILE != MTIME_XATTR)))
define( is_premigrated_same, ((XATTR('dmapi.MCEA', 5, 1) == 'C') AND (MTIME_FILE == MTIME_XATTR)))
define(
exclude_list,
(
FALSE
OR PATH_NAME LIKE '%/.mcstore/%'
OR PATH_NAME LIKE '%/.mcstore.bak/%'
OR PATH_NAME LIKE '%/.snapshots/%'
)
)
define(
weight_mig_offline,
(CASE
/*=== The file is very young, the ranking is very low ===*/
WHEN access_age <= 1 THEN 0
/* File is not premigrated, prefer not to select these */
WHEN NOT(is_premigrated) THEN 1
/*=== The file is very small, the ranking is low ===*/
WHEN mb_allocated < 1 THEN access_age
/*=== The file is resident and large and old enough,
the ranking is standard ===*/
ELSE mb_allocated * access_age
END)
)
Placement rules
Only the following attributes are valid for placement rules:
- FILESET_NAME
- GROUP_ID
- NAME
- USER_ID
Sample placement rules:
# cat placement.policy
RULE 'important-rule' SET POOL 'system' WHERE LOWER(NAME) LIKE '%.xls'
RULE 'homes' SET POOL 'silver' FOR FILESET ('homefileset')
RULE 'db' SET POOL 'system' FOR FILESET ('dbfileset')
RULE 'default' SET POOL 'data'
Randomize placement between two pools:
At some point we had a file system with capacity from 2 different ESS models, with different number of disks and capacity. We wanted to spread to the GL2S data1 pool, and to the GL2 data2 pool. To do that, we used the SQL RAND() function in the placement policy. As an additional protection against running full, we also added a “LIMIT(90)” to stop filling up the data1 pool if it was already more than 90% full:
RULE 'GL2S' SET POOL 'data1' LIMIT(90) WHERE INTEGER(RAND()*30)<20
RULE 'GL2' SET POOL 'data2'
Hack for preventing storing a given filetype:
Create a descOnly NSD as only disk in a dummy-pool, and use rule like:
RULE 'do-not-place-mp3' SET POOL 'dummy' WHERE LOWER (NAME) LIKE '%.mp3%'
Maybe better to use EXCLUDE:
RULE 'no-mp3-allowed' SET POOL EXCLUDE WHERE NAME LIKE '%.mp3'
Overflow placement:
Default place new files in “flash” pool, and only place new files in “sas” pool if “flash” pool is more than 95% full. OBS: existing files can continue growing the the pool they’re already placed in, so make sure there’s enough buffer to not run out of capacity:
RULE 'new' SET POOL 'flash' LIMIT(95)
RULE 'spillover' SET POOL 'sas'
Protect some files with replication
With TCT setup for a University, we too often experienced corrupted TCT database when we had down disks or crashes. Therefore we implemented replication protection for the TCT database files. This was done by:
- creating a separate group account with gid = 2222
- chowning the .mcstore folder to “root:2222”
- adding sticky group (chmod g+s .mcstore) i
- and then following policy to add replication to these files:
RULE 'repMcstore' SET POOL 'system' REPLICATE(2) WHERE GROUP_ID = 2222
Per user usage summary
Since offline files are not counted in GPFS quota, here’s a sample policy for listing all files:
/*
* List files
*
* mmapplypolicy gpfs0 -P xxxxxx.policy -I defer -f ./filelist
*
*/
/* list resident files */
RULE EXTERNAL LIST 'files' EXEC ''
RULE LIST 'files'
SHOW( VARCHAR(USER_ID) || ' ' ||
VARCHAR(GROUP_ID) || ' ' ||
VARCHAR(FILESET_NAME) || ' ' ||
VARCHAR(FILE_SIZE) || ' ' ||
VARCHAR(KB_ALLOCATED) )
The output file from this can then be piped through awk to give per user summary with:
cat file | awk '{a[$4] += $7} END{ for (i in a) print i, a[i]}'
GROUP POOLs
GROUP POOLs can be used to balance data between pools. F.ex. if we want to balance the popol utilization so that “flash” pool if filled 75% full with small files, we might do:
RULE 'defineTiers' GROUP POOL 'Tiers' IS 'flash' LIMIT(75) then 'sas'
/* Migrate new files ingested to flash to sas if they are larger than 512MB */
RULE 'LargeToSAS' MIGRATE FROM POOL 'flash' to POOL 'sas' WHERE FILE_SIZE>524288
RULE 'Rebalance' MIGRATE FROM POOL 'Tiers' TO POOL 'Tiers' WEIGHT(-KB_ALLOCATED)
Or maybe utilize FILE_HEAT to keep hot files in flash pool:
/* Rebalance files smaller than 512 MB between pools */
RULE 'Rebalance' MIGRATE FROM POOL 'Tiers' TO POOL 'Tiers' WEIGHT(FILE_HEAT) WHERE FILE_SIZE<524288
Report on replication status:
Lit if there are illReplicated, Nonreplicated and Properly-replicated files:
# cat check-replication.policy
/*
*
* MISC_ATTRIBUTES:
*
* J -- illReplicated
* 2 -- Data blocks are replicated
*/
RULE EXTERNAL LIST 'A' EXEC ''
RULE 'illReplicated' LIST 'A' WHERE MISC_ATTRIBUTES LIKE '%J%'
RULE 'Nonreplicated' LIST 'A' WHERE NOT MISC_ATTRIBUTES LIKE '%2%'
RULE 'Properly-replicated' LIST 'A' WHERE MISC_ATTRIBUTES LIKE '%2%' AND NOT MISC_ATTRIBUTES LIKE '%J%'
Test searching for xattrs:
xattrs could be used in policies for f.ex. letting users tag files that they want to migrate to cloud or tape.
# mmchattr --set-attr user.sendtocloud=yes /gpfs/gpfsmgmt/fil2
# mmlsattr -L -d /gpfs/gpfsmgmt/fil2
<snip>
user.sendtocloud: "yes"
# cat policy.pol
RULE EXTERNAL LIST 'files' EXEC ''
RULE LIST 'files' WHERE xattr('user.sendtocloud') LIKE 'yes'
# mmapplypolicy /gpfs/gpfsmgmt/ -P policy.pol -I defer -f /tmp/pol
# cat /tmp/pol.list.files
37390 999490048 0 -- /gpfs/gpfsmgmt/fil2
Set extended attribute using policy:
RULE 'pa' SET POOL 'gold' WHERE (some-condition) AND SetXattr('some-attribute-name','attribute-value')
RULE 'pb' SET POOL 'silver' WHERE (some-other-condition) AND SetXattr('another-attribute-name','another-attribute-value')
Find files changed after timestamp:
RULE EXTERNAL LIST 'modfiles' EXEC ''
RULE 'findModified' LIST 'modfiles' WHERE ( MODIFICATION_TIME >= TIMESTAMP("2024-07-06 11:22:00") OR CHANGE_TIME >= TIMESTAMP("2024-07-06 11:22:00") )
Delete policy
Are you sure?
# cat delete.policy
RULE 'delete' DELETE
# mmapplypolicy /gpfs/gpfs0/testtest/ -P delete.policy -I test -L3
Migrate only given filesets
define(
filesetlist,
(
'fileset1',
'fileset2'
)
)
RULE 'migFilesets' MIGRATE TO POOL 'data' FOR FILESET filesetlist
List files in a given pool:
/*
*
* mmapplypolicy /ibm/fs2/folder/ -P list-data-files.policy -I defer -f /tmp/filelist
*
*/
RULE EXTERNAL LIST 'files' EXEC ''
RULE LIST 'files' FROM POOL 'data'
SHOW( VARCHAR(USER_ID) || ' ' ||
VARCHAR(GROUP_ID) || ' ' ||
VARCHAR(FILESET_NAME) || ' ' ||
VARCHAR(FILE_SIZE) || ' ' ||
VARCHAR(KB_ALLOCATED) )
Check for TCT offline blobs
# cat /root/check-offline-blob/check-block.sh
#! /bin/sh -
#
# Arguments to this script:
#
# $@ = LIST /mnt/gpfs01/.mmSharedTmpDir/mmPolicy.ix.19751.02F6A4F0.1 7
#
#
case $1 in
TEST )
exit 0
;;
LIST )
awk -F ' -- ' '{print $2}' $2 | while read -r filename ;
do
echo -n "Checking $filename: "
object=$(mmcloudgateway files list "$filename" | awk '/^Base Name/ {print $4 "/0000000000000001.DATA"}')
mmcloudgateway files checkCloudObject --container digitalpatologi --cloud-service-name cosservice --object-name "$object"
done
;;
esac
# cat /root/check-offline-blob/check-block.policy
/* # mmapplypolicy /scale/t4xray/Images/CUCCA/CLAUDIA/20509000/20150903 -P /root/check-offline-blob/check-block.policy -I yes -N scale1_site1 */
include(/scalemgmt/policy/common.include)
RULE EXTERNAL LIST 'checkBlob' EXEC '/root/check-offline-blob/check-block.sh'
RULE 'findOffline' LIST 'checkBlob' WHERE is_migrated
# mmapplypolicy /scale/digitalpatologi/ -P /root/check-offline-blob/check-block.policy -I yes -N all > /root/check-offline-blob/digitalpatologi.out 2>&1
Compress some stuff
/*
*
* mmapplypolicy datalake -P /root/compression.policy -I yes -N cesNodes
*
*/
define( compressed, (MISC_ATTRIBUTES LIKE '%c%'))
define( illcompressed, (MISC_ATTRIBUTES LIKE '%K%'))
define( days_since_creation, (DAYS(CURRENT_TIMESTAMP) - DAYS(CREATION_TIME)))
define( days_since_access, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))
define( days_since_modified, (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFIED_TIME)))
RULE 'compression' MIGRATE COMPRESS ('lz4')
WHERE
days_since_access > 30
AND KB_ALLOCATED > 48
AND (illcompressed OR NOT compressed)
AND NOT FILESET_NAME LIKE 'object_fileset'
AND NOT FILESET_NAME LIKE 'obj_unified'
Clear immutable…
define( days_since_retention, (DAYS(CURRENT_TIMESTAMP) - DAYS(EXPIRATION_TIME)))
define( isImmutable, (MISC_ATTRIBUTES LIKE '%X%')
RULE EXTERNAL LIST 'clearImmutable' EXEC '/root/set-retention/clear-immutable.sh' OPTS ''
RULE 'findImmutable' LIST 'clearImmutable' FOR FILESET retention31 WHERE isImmutable AND days_since_retention > 0
#! /bin/bash
case $1 in
TEST )
exit 0
;;
LIST )
perl -pe 's/.*? -- //' "$2" | while read -r filename
do
mmchattr -i no "$filename"
done
;;
esac
Check for ACL files
# cat /root/list-acl-files.policy
RULE EXTERNAL LIST 'files' EXEC ''
RULE LIST 'files'
WHERE MISC_ATTRIBUTES LIKE '%+%'
# mmapplypolicy fs1 -P /root/list-acl-files.policy -I defer -f /tmp/list-acls
# cat /tmp/list-acls.list.files
6582 1538356456 0 -- /ibm/fs1/testfile
7427 1587228471 0 -- /ibm/fs1/nfsacltest-/test/file2
7429 815710946 0 -- /ibm/fs1/nfsacltest-/test/file3
7430 1763917096 0 -- /ibm/fs1/nfsacltest-/test/file5
7439 220873565 0 -- /ibm/fs1/nfsacltest-/test/file1.touch
29952 1203511191 0 -- /ibm/fs1/acltest/source/file1
44544 283750887 0 -- /ibm/fs1/acltest/destination/file1
524289 1276685378 0 -- /ibm/fs1/smbshare1/target/file
530688 1966400534 0 -- /ibm/fs1/smbshare1/source/file
530689 1127981020 0 -- /ibm/fs1/smbshare1/file2
<snip>
Unreplicate
RULE 'unreplicate' MIGRATE REPLICATE(1)
Find file types:
# cat list-filetypes.policy
RULE LIST 'files' WHERE MODE like '-%'
RULE LIST 'directories' DIRECTORIES_PLUS WHERE MODE like 'd%'
RULE LIST 'links' DIRECTORIES_PLUS WHERE MODE like 'l%'
RULE LIST 'chardevs' DIRECTORIES_PLUS WHERE MODE like 'c%'
RULE LIST 'blockdevs' DIRECTORIES_PLUS WHERE MODE like 'b%'
RULE LIST 'pipes' DIRECTORIES_PLUS WHERE MODE like 'p%'
RULE LIST 'sockets' DIRECTORIES_PLUS WHERE MODE like 's%'
RULE LIST 'other' DIRECTORIES_PLUS WHERE MODE like '?%'
# mmapplypolicy /gpfs/gpfs1/directory -P list-filetypes.policy -I test
<snip>
[I] Summary of Rule Applicability and File Choices:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 6 0 6 0 0 RULE 'files' LIST 'files' WHERE(.)
1 7 256 7 256 0 RULE 'directories' LIST 'directories' DIRECTORIES_PLUS WHERE(.)
2 0 0 0 0 0 RULE 'links' LIST 'links' DIRECTORIES_PLUS WHERE(.)
3 0 0 0 0 0 RULE 'chardevs' LIST 'chardevs' DIRECTORIES_PLUS WHERE(.)
4 0 0 0 0 0 RULE 'blockdevs' LIST 'blockdevs' DIRECTORIES_PLUS WHERE(.)
5 0 0 0 0 0 RULE 'pipes' LIST 'pipes' DIRECTORIES_PLUS WHERE(.)
6 0 0 0 0 0 RULE 'sockets' LIST 'sockets' DIRECTORIES_PLUS WHERE(.)
7 0 0 0 0 0 RULE 'other' LIST 'other' DIRECTORIES_PLUS WHERE(.)
[I] Filesystem objects with no applicable rules: 0.
Premigrate everything:
RULE 'premigrate' MIGRATE TO POOL 'extpool'
THRESHOLD(0,100,0)
WHERE NOT is_premigrated
Bucket report on file ages:
# cat file-age.policy
/* define days */
define( access_days, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))
define( modified_days, (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME)))
define( creation_days, (DAYS(CURRENT_TIMESTAMP) - DAYS(CREATION_TIME)))
RULE LIST '0-30 days since last-access' WHERE access_days < 30
RULE LIST '30-60 days since last-access' WHERE access_days >= 30 AND access_days < 60
RULE LIST '60-90 days since last-access' WHERE access_days >= 60 AND access_days < 90
RULE LIST '90-180 days since last-access' WHERE access_days >= 90 AND access_days < 180
RULE LIST '180-365 days since last-access' WHERE access_days >= 180 AND access_days < 365
RULE LIST '1-2 years since last-access' WHERE access_days >= 365 AND access_days < 730
RULE LIST '2-3 years since last-access' WHERE access_days >= 730 AND access_days < 1095
RULE LIST '3-4 years since last-access' WHERE access_days >= 1095 AND access_days < 1460
RULE LIST '4-5 years since last-access' WHERE access_days >= 1460 AND access_days < 1825
RULE LIST '>5 years since last-access' WHERE access_days >= 1825
RULE LIST '0-30 days since last-write' WHERE modified_days < 30
RULE LIST '30-60 days since last-write' WHERE modified_days >= 30 AND modified_days < 60
RULE LIST '60-90 days since last-write' WHERE modified_days >= 60 AND modified_days < 90
RULE LIST '90-180 days since last-write' WHERE modified_days >= 90 AND modified_days < 180
RULE LIST '1-2 years since last-write' WHERE modified_days >= 365 AND modified_days < 730
RULE LIST '2-3 years since last-write' WHERE modified_days >= 730 AND modified_days < 1095
RULE LIST '3-4 years since last-write' WHERE modified_days >= 1095 AND modified_days < 1460
RULE LIST '4-5 years since last-write' WHERE modified_days >= 1460 AND modified_days < 1825
RULE LIST '>5 years since last-write' WHERE modified_days >= 1825
RULE LIST '0-30 days since creation' WHERE creation_days < 30
RULE LIST '30-60 days since creation' WHERE creation_days >= 30 AND creation_days < 60
RULE LIST '60-90 days since creation' WHERE creation_days >= 60 AND creation_days < 90
RULE LIST '90-180 days since creation' WHERE creation_days >= 90 AND creation_days < 180
RULE LIST '180-365 days since creation' WHERE creation_days >= 180 AND creation_days < 365
RULE LIST '1-2 years since creation' WHERE creation_days >= 365 AND creation_days < 730
RULE LIST '2-3 years since creation' WHERE creation_days >= 730 AND creation_days < 1095
RULE LIST '3-4 years since creation' WHERE creation_days >= 1095 AND creation_days < 1460
RULE LIST '4-5 years since creation' WHERE creation_days >= 1460 AND creation_days < 1825
RULE LIST '>5 years since creation' WHERE creation_days >= 1825