Failed Operation was vetoed by lock_manager. Check the event log | NetApp Notes

 Hi Friend, This scenario i came across while doing takeover and giveback process of NetApp Cluster nodes. 

First we have rebooted one node and giveback process completed successfully, however when we do the same on another node, Node came back normally however services and resources are not giveback to same node.

ARKIT-NA::*> storage failover show-giveback
               Partner
Node           Aggregate         Giveback Status
-------------- ----------------- ---------------------------------------------
ARKIT-NA01
               CFO Aggregates    Done
               aggr1_SATA_SNY_NA02
                                 Failed: Operation was vetoed by lock_manager. Check the event log
ARKIT-NA02
                                 No aggregates to give back

Using below command we are going to give back the resources to node2

ARKIT-NA::*> storage failover giveback -ofnode ARKIT-NA02 -override-vetoes true

Warning: Initiating a giveback with vetoes overridden will result in giveback proceeding even if the node detects outstanding issues that
         would make a giveback dangerous or disruptive. Do you want to continue? {y|n}: y

Info: Run the storage failover show-giveback command to check giveback status.

 After give back process initiated

ARKIT-NA::*> storage failover show-giveback
               Partner
Node           Aggregate         Giveback Status
-------------- ----------------- ---------------------------------------------
ARKIT-NA01
               CFO Aggregates    Done
               aggr1_SATA_SNY_NA02
                                 In progress module: lock_manager
ARKIT-NA02
                                 No aggregates to give back

ARKIT-NA::*> storage failover show
                              Takeover
Node           Partner        Possible State Description
-------------- -------------- -------- -------------------------------------
ARKIT-NA01       ARKIT-NA02       true     Connected to ARKIT-NA02, Giveback of
                                       SFO aggregates in progress
ARKIT-NA02       ARKIT-NA01       true     Connected to ARKIT-NA01

Check if any services issues on NetApp storage.

ARKIT-NA::*> system health subsystem show
Subsystem         Health
----------------- ------------------
SAS-connect       ok
Environment       ok
Memory            ok
Service-Processor ok
Switch-Health     ok
CIFS-NDO          ok
Motherboard       ok
IO                ok
MetroCluster      ok
MetroCluster_Node ok
FHM-Switch        ok
FHM-Bridge        ok
12 entries were displayed.

ARKIT-NA::*> system health alert show
This table is currently empty.

That's about this post, if your node is struck with giveback process use above command to giveback without vetoed.

How To Collect NDMP Logs from Netapp Cluster mode

Using this simple method you can collect Netapp NDMP Logs and troubleshoot NDMP issues on Netapp

Check the Node scoped mode is enabled

::> system services ndmp node-scope-mode status
NDMP node-scope-mode is disabled.

if the status is disabled then enable by running below command

 ::> system services ndmp node-scope-mode on
NDMP node-scope-mode is enabled.

Check the status whether it is enabled

::> system services ndmp node-scope-mode status
NDMP node-scope-mode is enabled.

Disable node-scoped NDMP debug mode using command below
 ::> system services ndmp node-scope-mode off
NDMP node-scope-mode is disabled.

If you like to collect the NDMP logs on vserver level then enable debugging on vserver level first using below commands

Check which Vserver is enabled for NDMP service

::> vserver services ndmp show
::> vserver services ndmp status

::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y



Now enable debug mode on vserver using below command

::*> vserver services ndmp modify -vserver ARKIT-NA -debug-filter normal -debug-enable true

Disable NDMP Debug mode using command below

::*> vserver services ndmp modify -vserver ARKIT-NA -debug-filter normal -debug-enable false 

Now download ndmp logs from SPI by default spi is enabled for C-Mode 8.2 or earlier versions

Go to browser and login to SPI

 http://FILER-IP-ADDRESS/spi

 Go to /etc/logs/mlog and download ndmpd.log*

That's how you can collect NDMP Debug logs for troubleshooting Netapp NDMP issues.


Facebook | Twitter | YouTube

Data ONTAP API Failed :Loop detected in next() for table compliance_status.

I see an error when adding licenses to Netapp Cluster mode simulator as mentioned below.

"Data ONTAP API Failed :Loop detected in next() for table compliance_status. Next on "NFS 1-81-0000000000000004082368511" returned "NFS 1-81-0000000000000004082368511". (Error: 13001)"

To resolve above issue you have to do below steps on second cluster node.

When you see an VLOADER> prompt type below command

VLOADER> setenv SYS_SERIAL_NUM 4032368-50-8
VLOADER> setenv bootarg.nvram.sysid 4032368508

Verify that the information was saved correctly or not

VLOADER> printenv SYS_SERIAL_NUM
VLOADER> printenv bootarg.nvram.sysid
VLOADER> boot

clean configuration and re-configure the second node.

That's it. Enjoy practicing Netapp Storage Lab.