Failed Operation was vetoed by lock_manager. Check the event log | NetApp Notes

 Hi Friend, This scenario i came across while doing takeover and giveback process of NetApp Cluster nodes. 

First we have rebooted one node and giveback process completed successfully, however when we do the same on another node, Node came back normally however services and resources are not giveback to same node.

ARKIT-NA::*> storage failover show-giveback
               Partner
Node           Aggregate         Giveback Status
-------------- ----------------- ---------------------------------------------
ARKIT-NA01
               CFO Aggregates    Done
               aggr1_SATA_SNY_NA02
                                 Failed: Operation was vetoed by lock_manager. Check the event log
ARKIT-NA02
                                 No aggregates to give back

Using below command we are going to give back the resources to node2

ARKIT-NA::*> storage failover giveback -ofnode ARKIT-NA02 -override-vetoes true

Warning: Initiating a giveback with vetoes overridden will result in giveback proceeding even if the node detects outstanding issues that
         would make a giveback dangerous or disruptive. Do you want to continue? {y|n}: y

Info: Run the storage failover show-giveback command to check giveback status.

 After give back process initiated

ARKIT-NA::*> storage failover show-giveback
               Partner
Node           Aggregate         Giveback Status
-------------- ----------------- ---------------------------------------------
ARKIT-NA01
               CFO Aggregates    Done
               aggr1_SATA_SNY_NA02
                                 In progress module: lock_manager
ARKIT-NA02
                                 No aggregates to give back

ARKIT-NA::*> storage failover show
                              Takeover
Node           Partner        Possible State Description
-------------- -------------- -------- -------------------------------------
ARKIT-NA01       ARKIT-NA02       true     Connected to ARKIT-NA02, Giveback of
                                       SFO aggregates in progress
ARKIT-NA02       ARKIT-NA01       true     Connected to ARKIT-NA01

Check if any services issues on NetApp storage.

ARKIT-NA::*> system health subsystem show
Subsystem         Health
----------------- ------------------
SAS-connect       ok
Environment       ok
Memory            ok
Service-Processor ok
Switch-Health     ok
CIFS-NDO          ok
Motherboard       ok
IO                ok
MetroCluster      ok
MetroCluster_Node ok
FHM-Switch        ok
FHM-Bridge        ok
12 entries were displayed.

ARKIT-NA::*> system health alert show
This table is currently empty.

That's about this post, if your node is struck with giveback process use above command to giveback without vetoed.

How To Automate SnapCenter Snapshot Mounting & Unmounting using PowerShell

How To Automate SnapCenter Snapshot Mounting & Unmounting using PowerShell

In this post we are going to see how to mount SnapCenter snapshots using PowerShell Cmdlets.

Scenario: We would like to backup the snapshots (Created by SnapCenter) data using one of backup application. Before we were used to have SnapManager with SnapDrive Commands. In SnapCenter Snapdrive may not work as expected, frequently we get timeout error.

In order to do this we need atleast SnapCenter version 4.4.

Create a secure string out of user password

> read-host -assecurestring | convertfrom-securestring | out-file c:\mysecurestring.txt

After creating the Secure string use below powershell cmdlets to establish connection with SnapCenter

$pass = cat C:\mysecurestring.txt | convertto-securestring 
$mycred = new-object -typename System.Management.Automation.PSCredential -argumentlist "Domain\UserName",$pass

Open-SmConnection -Credential $mycred -SMSbaseUrl https://SnapCenterIP:8146

Now Mount the snapshot using below command

Mount-SdSnapshot -Path SVMNAME:/vol/Volume/Lun -Snapshot SnapShotName -MountPath E: -Host BackupServer.domain.com

You can Un-Mount the mounted snapshot using below command

Dismount-SdSnapshot -Path "E:" -Host BackupServer.domain.com

That's it. You can use the same scripts as Pre and Post for your backup application

Comment your feedback

How To Upgrade Netapp SP Firmware | Netapp Cluster Mode | Netapp Ark Blog

 How to Upgrade NetApp SP Firmware Version

 Prerequisites

To update the SP firmware image, you need to have access to the following devices:

• A web server on a network accessible to your storage controller
• A console that has an SSH client that can be used to reach the SP or a serial console to the storage controller
• SSH and an account that is set up for SP access

Before using the SP CLI to update the SP firmware, you must ensure the following:

• An SP update is not currently triggered by ONTAP
• No SP firmware package is being downloaded from the ONTAP CLI when the SP is being updated from the SP CLI

ATTENTION: To prepare the SP for a smooth firmware update, reboot the SP first.

 Note: Please contact Netapp support in-case if your not confident enough to upgrade SP firmware.

 This firmware upgrade is non-disruptive, this upgrade not impact any services in NetApp filer. 

 Login to Netapp using SSH and reboot the SP using below command.

ARKIT-NA::> system service-processor reboot-sp -node ARKIT-NA02 

Note: If your console connection is through the SP, it will be disconnected.
      Do you want to reboot the SP ? {y|n}: y

 After that watch the Service processor status using below command 

ARKIT-NA::> sp show
                               IP           Firmware
Node          Type Status      Configured   Version   IP Address
------------- ---- ----------- ------------ --------- -------------------------
ARKIT-NA01      SP   rebooting   true         5.1       192.168.2.153
ARKIT-NA02      SP   online      true         5.1P1     192.168.2.154
2 entries were displayed.

Download SP firmware version to your web server default path (/var/www/html)

https://mysupport.netapp.com/NOW/download/tools/serviceimage/30803991_sp/SP_FW_308-03991_5.8.tar.gz

 Login to SP using SSH as admin user 

login as: admin
admin@192.168.2.153's password:
SP ARKIT-NA01>
SP ARKIT-NA01> sp update http://192.168.2.165/SP_FW_308-03991_5.8.tar.gz

NOTE: In most cases, ONTAP will automatically update the SP firmware with a
compatible firmware version in the background and no action is required from
you.

Manually installing the SP firmware using this command may result in
incompatible combinations of ONTAP and SP firmware.

Use of this command is not recommended.

Do you wish to continue? (y/n): y
Downloading package...
Uncompressing package...(md5sum=ba1559b21e9799581e8e72cd8cb07b43)
Unarchiving package...
Inspecting firmware...
Installing package...
Reprocessing configuration state...
Cleaning up...
SP FW Update Successful.
Primary firmware version 5.8
Backup firmware version 5.1

You need to reboot the SP for the new image to take effect.
Do you want to reboot the SP now? (y/n): y

Rebooting SP... 

 After SP is online verify the status 

ARKIT-NA::> sp show
                               IP           Firmware
Node          Type Status      Configured   Version   IP Address
------------- ---- ----------- ------------ --------- -------------------------
ARKIT-NA01      SP   online      true         5.8       192.168.2.153
ARKIT-NA02      SP   rebooting   true         5.1P1     192.168.2.154
2 entries were displayed.  

That's it your SP firmware is upgraded successfully.

Please comment your feedback.

Facebook | Twitter | Reddit | LinkedIn | Blog | YouTube | Github

Join Telegram Group