Siempre he tenido mi amor odio con AFD, anteriormente misma situación con ASMLIB.
Es una capa adicional muy cómoda, muy interesante y con beneficios tanto en seguridad en acceso a información como rendimiento (o eso dicen). Pero esto cuando "va bien".
Tanto AFD como ACFS son muy delicados en cuanto a versionado de SO y cambio de paquetería. Seguro que hay muchas historias entre dba donde nos hemos visto en la situación de un downgrade (incluso upgrade después de un parche de la GI) de SO debido a problemas de compatibilidades o encontrarnos bugs relativos a esto.
Así que he tenido una situación donde, sin problema de configuración aparente tanto de SW como de HW y un despliegue limpio, a la hora de crear una BBDD en almacenamiento compartido (RAC) la máquina incurría en un crash.
Después de varias pruebas llego a la siguiente conclusión: AFD está provocando el crash cuando una instancia de BBDD intenta leer la información. Hay separación de roles y pasa exactamente lo mismo con la BBDD de la GIMR (grid) como con la BBDD de negocio de propietario Oracle.
Detalle del crash, atención al proceso (oracle_xxxx_or) donde confirmo mis sospechas:
crash /var/crash/127.0.0.1-2023-12-12-12:03:08/vmcore /usr/lib/debug/lib/modules/5.15.0-200.131.27.1.el8uek.x86_64/vmlinux
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/lib/modules/5.15.0-200.131.27.1.el8uek.x86_64/vmlinux [TAINTED]
DUMPFILE: /var/crash/127.0.0.1-2023-12-12-12:03:08/vmcore [PARTIAL DUMP]
CPUS: 8
DATE: Tue Dec 12 12:03:03 CET 2023
UPTIME: 00:18:31
LOAD AVERAGE: 0.75, 0.51, 0.33
TASKS: 946
NODENAME: ractest-001
RELEASE: 5.15.0-200.131.27.1.el8uek.x86_64
VERSION: #2 SMP Wed Nov 22 18:08:57 PST 2023
MACHINE: x86_64 (3192 Mhz)
MEMORY: 64 GB
PANIC: "Oops: 0003 [#1] SMP PTI" (check log for details)
PID: 19614
COMMAND: "oracle_19614_or"
TASK: ffff9a0180d64100 [THREAD_INFO: ffff9a0180d64100]
CPU: 1
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 19614 TASK: ffff9a0180d64100 CPU: 1 COMMAND: "oracle_19614_or"
#0 [ffffac34cb44f920] machine_kexec at ffffffffb748e6d4
#1 [ffffac34cb44f978] __crash_kexec at ffffffffb7599fc7
#2 [ffffac34cb44fa38] crash_kexec at ffffffffb759b33e
#3 [ffffac34cb44fa40] oops_end at ffffffffb744292b
#4 [ffffac34cb44fa60] page_fault_oops at ffffffffb74a1cf4
#5 [ffffac34cb44fab8] exc_page_fault at ffffffffb80ab59e
#6 [ffffac34cb44fae0] asm_exc_page_fault at ffffffffb8200ba2
[exception RIP: safd_rio_unmappages+99]
RIP: ffffffffc0bda623 RSP: ffffac34cb44fb90 RFLAGS: 00010246
RAX: fffff1ad78a90008 RBX: 0000000000000000 RCX: 0000000000000004
RDX: fffff1ad78a90007 RSI: 0000000000000000 RDI: fffff1ad75dc5000
RBP: ffff9a0199cfe780 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff9a11479e9b00
R13: 0000000000000000 R14: 0000000000000000 R15: ffff9a115b4c1278
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffac34cb44fba0] afdq_request_drop at ffffffffc0bd0ce7 [oracleafd]
#8 [ffffac34cb44fbb8] afdq_request_wait at ffffffffc0bd14ad [oracleafd]
#9 [ffffac34cb44fbe8] afdq_batch_submit at ffffffffc0bd1a06 [oracleafd]
#10 [ffffac34cb44fcb0] afdc_io at ffffffffc0bc3472 [oracleafd]
#11 [ffffac34cb44fd58] afdc_execute_ioctl at ffffffffc0bc4793 [oracleafd]
#12 [ffffac34cb44feb8] afd_ioctl at ffffffffc0bd8e51 [oracleafd]
#13 [ffffac34cb44fee8] block_ioctl at ffffffffb7920778
#14 [ffffac34cb44ff08] __x64_sys_ioctl at ffffffffb77c6c3f
#15 [ffffac34cb44ff38] do_syscall_64 at ffffffffb80a77f8
#16 [ffffac34cb44ff50] entry_SYSCALL_64_after_hwframe at ffffffffb82000dc
RIP: 00007f850f7fd7cb RSP: 00007ffc2faec2d8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 000000001952a6e0 RCX: 00007f850f7fd7cb
RDX: 000000001952a6e0 RSI: 0000000000507606 RDI: 000000000000000d
RBP: 00007ffc2faec350 R8: 00007f850e342000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000019543aa0 R15: 00007ffc2faec578
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
En este caso, al no haber Bug ni parche al respecto, decido prescindir de AFD y configurar UDEV.
Para ello detallo los pasos:
-- Identificar serial de los discos
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdf | egrep "ID_SERIAL="
E: ID_SERIAL=36000c291aee699ca9d4b61bcf714792e
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sde | egrep "ID_SERIAL="
E: ID_SERIAL=36000c29738b3610ad67419747dd8e1b4
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdg | egrep "ID_SERIAL="
E: ID_SERIAL=36000c293d74cbd9492afe0f0bdb00811
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdi | egrep "ID_SERIAL="
E: ID_SERIAL=36000c2908ccf35f1dd8ce252bf780397
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdd | egrep "ID_SERIAL="
E: ID_SERIAL=36000c2997087fdd99df4ce170295a6c4
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdc | egrep "ID_SERIAL="
E: ID_SERIAL=36000c2976346fbb62cab3fa6b23ae832
[oracle@ractest-001 ~]$ udevadm info --query=all --name=/dev/sdh | egrep "ID_SERIAL="
E: ID_SERIAL=36000c29b4ab7e09ec87cc1d5b6bfca7a
-- Añadir reglas UDEV. Ambos nodos
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c2997087fdd99df4ce170295a6c4", SYMLINK+="oracleasm/asmdisk_testfra01", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c2976346fbb62cab3fa6b23ae832", SYMLINK+="oracleasm/asmdisk_testmgmtdb01", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c29738b3610ad67419747dd8e1b4", SYMLINK+="oracleasm/asmdisk_testocr1", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c291aee699ca9d4b61bcf714792e", SYMLINK+="oracleasm/asmdisk_testocr2", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c293d74cbd9492afe0f0bdb00811", SYMLINK+="oracleasm/asmdisk_testocr3", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c2908ccf35f1dd8ce252bf780397", SYMLINK+="oracleasm/asmdisk_testocr4", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36000c29b4ab7e09ec87cc1d5b6bfca7a", SYMLINK+="oracleasm/asmdisk_testocr5", OWNER="grid", GROUP="asmadmin", MODE="0660" ,OPTIONS:="nowatch"
[root@ractest-001 ~]# udevadm control --reload-rules && udevadm trigger --type=devices --action=change
-- Revisiones
[oracle@ractest-001 ~]$ ls -lrt /dev/sd*
brw-rw---- 1 root disk 8, 0 Dec 18 10:41 /dev/sda
brw-rw---- 1 root disk 8, 3 Dec 18 10:41 /dev/sda3
brw-rw---- 1 root disk 8, 1 Dec 18 10:41 /dev/sda1
brw-rw---- 1 root disk 8, 2 Dec 18 10:41 /dev/sda2
brw-rw---- 1 grid asmadmin 8, 32 Dec 18 10:42 /dev/sdc
brw-rw---- 1 grid asmadmin 8, 48 Dec 26 14:49 /dev/sdd
brw-rw---- 1 grid asmadmin 8, 16 Dec 26 14:49 /dev/sdb
brw-rw---- 1 grid asmadmin 8, 128 Dec 26 14:49 /dev/sdi
brw-rw---- 1 grid asmadmin 8, 112 Dec 26 14:49 /dev/sdh
brw-rw---- 1 grid asmadmin 8, 96 Dec 26 14:49 /dev/sdg
brw-rw---- 1 grid asmadmin 8, 80 Dec 26 14:49 /dev/sdf
brw-rw---- 1 grid asmadmin 8, 64 Dec 26 14:49 /dev/sde
[root@ractest-001 ~]# ls -lrt /dev/oracleasm/
total 0
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testfra01 -> ../sdd
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testocr3 -> ../sdg
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testdata01 -> ../sdb
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testocr5 -> ../sdh
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testocr2 -> ../sdf
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testocr1 -> ../sde
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testocr4 -> ../sdi
lrwxrwxrwx 1 root root 6 Dec 18 10:41 asmdisk_testmgmtdb01 -> ../sdc
Ya estamos en disposición de utilizar los discos con ASM.
[oracle@ractest-001 ~]$ lsblk --paths -o NAME,KNAME,FSTYPE,LABEL,MOUNTPOINT,SIZE,OWNER,GROUP,MODE,ALIGNMENT,MIN-IO,OPT-IO,PHY-SEC,LOG-SEC,ROTA,SCHED,RQ-SIZE,WSAME
NAME KNAME FSTYPE LABEL MOUNTPOINT SIZE OWNER GROUP MODE ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE WSAME
/dev/sda /dev/sda 200G root disk brw-rw---- 0 512 0 512 512 1 mq-deadline 256 4M
├─/dev/sda1 /dev/sda1 vfat partefi /boot/efi 1G root disk brw-rw---- 0 512 0 512 512 1 mq-deadline 256 4M
├─/dev/sda2 /dev/sda2 xfs partboot /boot 2G root disk brw-rw---- 0 512 0 512 512 1 mq-deadline 256 4M
└─/dev/sda3 /dev/sda3 LVM2_member 192G root disk brw-rw---- 0 512 0 512 512 1 mq-deadline 256 4M
├─/dev/mapper/vgsistema-root /dev/dm-0 xfs lvroot / 20G root disk brw-rw---- 0 512 0 512 512 1 128 4M
├─/dev/mapper/vgsistema-swap /dev/dm-1 swap lvswap [SWAP] 32G root disk brw-rw---- 0 512 0 512 512 1 128 4M
├─/dev/mapper/vgsistema-u01 /dev/dm-2 xfs lvu01 /u01 100G root disk brw-rw---- 0 512 0 512 512 1 128 4M
├─/dev/mapper/vgsistema-home /dev/dm-3 xfs lvhome /home 20G root disk brw-rw---- 0 512 0 512 512 1 128 4M
└─/dev/mapper/vgsistema-var /dev/dm-4 xfs lvvar /var 20G root disk brw-rw---- 0 512 0 512 512 1 128 4M
/dev/sdb /dev/sdb oracleasm 2T grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdc /dev/sdc oracleasm 200G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdd /dev/sdd oracleasm 400G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sde /dev/sde oracleasm 5G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdf /dev/sdf oracleasm 5G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdg /dev/sdg oracleasm 5G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdh /dev/sdh oracleasm 5G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sdi /dev/sdi oracleasm 5G grid asmadmin brw-rw---- 0 512 0 512 512 0 mq-deadline 256 4M
/dev/sr0 /dev/sr0 1024M root cdrom brw-rw---- 0 512 0 512 512 1 mq-deadline 64 0B
En este caso tener en cuenta definir en ASM:
asm_diskstring string /dev/oracleasm/*
Listo!