Using the Oracle ASM Cluster File System (Oracle ACFS) on Linux, Part Three

[oracle@racnode1 ~]$ /sbin/acfsutil info fs /documents3
/documents3
    ACFS Version: 11.2.0.1.0.0
    flags:        MountPoint,Available
    mount time:   Sat Nov 27 03:07:50 2010
    volumes:      1
    total size:   26843545600
    total free:   23191826432
    primary volume: /dev/asm/docsvol3-300
        label:                 DOCSVOL3
        flags:                 Primary,Available
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 153603
        size:                  26843545600
        free:                  23191826432
    number of snapshots:  1
    snapshot space usage: 560463872

From the example above, you can see that I have only one active snapshot that is consuming approximately 560MB of disk space. This coincides with the size of the file I removed earlier (/documents3/linux.x64_11gR2_examples.zip) which triggered a back up of all modified file image blocks.

To query all snapshots, simply list the directories under '<ACFS_MOUNT_POINT>/.ACFS/snaps'. Each directory under the snaps directory is an Oracle ACFS snapshot.

Another useful technique used to obtain information about Oracle ACFS snapshots is to query the view V$ASM_ACFSSNAPSHOTS from the Oracle ASM instance:

column snap_name    format a15  heading "Snapshot Name"
column fs_name      format a15  heading "File System"
column vol_device   format a25  heading "Volume Device"
column create_time  format a20  heading "Create Time"

======================================================================

SQL> select snap_name, fs_name, vol_device,
  2  to_char(create_time, 'DD-MON-YYYY HH24:MI:SS') as create_time
  3  from v$asm_acfssnapshots
  4  order by snap_name;

Snapshot Name   File System     Volume Device             Create Time
--------------- --------------- ------------------------- --------------------
snap1           /documents3     /dev/asm/docsvol3-300     27-NOV-2010 16:11:29

Delete Oracle ACFS Snapshot

Use the 'acfsutil snap delete' command to delete an existing Oracle ACFS snapshot:

[root@racnode1 ~]# /sbin/acfsutil snap delete snap1 /documents3
acfsutil snap delete: Snapshot operation is complete.

Managing ACFS

Oracle ACFS and Dismount or Shutdown Operations

If you take anything away from this article, know and understand the importance of dismounting any active file system configured with an Oracle ASM Dynamic Volume Manager (ADVM) volume device, BEFORE shutting down an Oracle ASM instance or dismounting a disk group! Failure to do so will result in I/O failures and very angry users!

After the file system(s) have been dismounted, all open references to Oracle ASM files are removed and associated disk groups can then be dismounted or the Oracle ASM instance shut down.

If the Oracle ASM instance or disk group is forcibly shut down or fails while an associated Oracle ACFS is active, the file system is placed into an offline error state. When the file system is placed in an offline error state, applications will start to encounter I/O failures and any Oracle ACFS user data and metadata being wrien at the time of the termination may not be flushed to ASM storage before it is fenced. If a SHUTDOWN ABORT operation on the Oracle ASM instance is required and you are not able to dismount the file system, issue two sync command to flush any cached file system data and metadata to persistent storage:

[root@racnode1 ~]# sync
[root@racnode1 ~]# sync

Using a two-node Oracle RAC, I forced an Oracle ASM instance shutdown on node 1 to simulate a failure: (Note: This should go without saying, but I'll say it anyway. DO NOT attempt the following on a production environment.)

SQL> shutdown abort
ASM instance shutdown

Any subsequent attempt to access an offline file system on that node will result in an I/O error:

[oracle@racnode1 ~]$ ls -l /documents3
ls: /documents3: Input/output error

[oracle@racnode1 ~]$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     145344992  22459396 115383364  17% /
/dev/sdb1            151351424    192072 143346948   1% /local
/dev/sda1               101086     12632     83235  14% /boot
tmpfs                  2019256         0   2019256   0% /dev/shm
df: `/documents1': Input/output error
df: `/documents2': Input/output error
df: `/documents3': Input/output error
domo:PUBLIC          4799457152 1901758592 2897698560  40% /domo

Recovering a file system from an offline error state requires dismounting and remounting the Oracle ACFS file system. Dismounting an active file system, even one that is offline, requires stopping all applications using the file system, including any shell references. For example, I had a shell session that previously changed directory (cd) into the /documents3 file system before the forced shutdown:

[root@racnode1 ~]# umount /documents1
[root@racnode1 ~]# umount /documents2
[root@racnode1 ~]# umount /documents3
umount: /documents3: device is busy
umount: /documents3: device is busy

Use the Linux fuser or lsof command to identify processes and kill if necessary:

[root@racnode1 ~]# fuser /documents3
/documents3:         16263c

[root@racnode1 ~]# kill -9 16263

[root@racnode1 ~]# umount /documents3

Restart the Oracle ASM instance (or in my case, all Oracle grid infrastructure services were stopped as a result of me terminating the Oracle ASM instance):

[root@racnode1 ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sdb1 on /local type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
oracleasmfs on /dev/oracleasm type oracleasmfs (rw)
domo:PUBLIC on /domo type nfs (rw,addr=192.168.1.121)
/dev/asm/docsvol1-300 on /documents1 type acfs (rw)
/dev/asm/docsvol2-300 on /documents2 type acfs (rw)
/dev/asm/docsvol3-300 on /documents3 type acfs (rw)

Resize File System

With Oracle ACFS, as long as there exists free space within the ASM disk group, any of the ASM volumes can be dynamically expanded which means the file system gets expanded as a result. Note that if you are using another file system other than Oracle ACFS, as long as that file system can support online resizing, they too can be dynamically re-sized. The one exception to 3rd party file systems is online shrinking. Ext3, for example, supports online resizing but does not support online shrinking.

Use the following syntax to add space to an Oracle ACFS on the fly without the need to take any type of outage.

First, verify there is enough space in the current Oracle ASM disk group to extend the volume:

SQL> select name, total_mb, free_mb, round((free_mb/total_mb)*100,2) pct_free
  2  from v$asm_diskgroup
  3  where total_mb != 0
  4  order by name;

Disk Group        Total (MB)    Free (MB)  % Free
--------------- ------------ ------------ -------
CRS                    2,205        1,809   82.04
DOCSDG1               98,303       12,187   12.40
FRA                   33,887       22,795   67.27
RACDB_DATA            33,887       30,584   90.25

The same task can be accomplished using the ASMCMD command-line utility:

[grid@racnode1 ~]$ asmcmd lsdg

From the 12GB of free space in the DOCSDG1 ASM disk group, let's extend the file system (volume) by another 5GB. Note that this can be performed while the file system is online and accessible by clients � no outage is required:

[root@racnode1 ~]# /sbin/acfsutil size +5G /documents3
acfsutil size: new file system size: 26843545600 (25600MB)

Verify the new size of the file system from all Oracle RAC nodes:

[root@racnode1 ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     145344992  21952712 115890048  16% /
/dev/sdb1            151351424    192072 143346948   1% /local
/dev/sda1               101086     12632     83235  14% /boot
tmpfs                  2019256   1135852    883404  57% /dev/shm
domo:PUBLIC          4799457152 1901103872 2898353280  40% /domo
/dev/asm/docsvol1-300
                      33554432    197668  33356764   1% /documents1
/dev/asm/docsvol2-300
                      33554432    197668  33356764   1% /documents2
/dev/asm/docsvol3-300
                      26214400    183108  26031292   1% /documents3


[root@racnode2 ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     145344992  13803084 124039676  11% /
/dev/sdb1            151351424    192072 143346948   1% /local
/dev/sda1               101086     12632     83235  14% /boot
tmpfs                  2019256   1135852    883404  57% /dev/shm
domo:Public          4799457152 1901103872 2898353280  40% /domo
/dev/asm/docsvol1-300
                      33554432    197668  33356764   1% /documents1
/dev/asm/docsvol2-300
                      33554432    197668  33356764   1% /documents2
/dev/asm/docsvol3-300
                      26214400    183108  26031292   1% /documents3

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/acfsload start -s

Unload the Oracle ASM volume driver: