手動安裝 Python 相關套件及設定 cx_Oracle

下載 Python 2.7.15

wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz
tar -zxvf Python-2.7.15.tgz -C /tmp/
cd /tmp/Python-2.7.15

自定安裝路徑: /usr/local/python2.7
./configure --prefix=/usr/local/python2.7 --enable-shared --enable-optimizations

執行 # /usr/local/python2.7/bin/python -V 時，會出現下列錯誤

# /usr/local/python2.7/bin/python -V
/usr/local/python2.7/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

會發生這錯誤，主要是因為 libpython2.7.so.1.0 這支 library 沒找到，因為我們使用了自定義安裝，所以要解決這個問題很簡單，先找看看目前系統有沒有這檔案，這邊我使用 find 這個指令

# find / -name libpython2.7.so.1.0 -type f
/tmp/Python-2.7.15/libpython2.7.so.1.0
/usr/local/python2.7/lib/libpython2.7.so.1.0

接著我們確認一下  LD_LIBRARY_PATH 的相關路徑有哪些，並且把我們剛剛找到的路徑也給加進去就行了。

# echo $LD_LIBRARY_PATH

會發現是空值，再來我們就加入到自已的 ~/.bashrc or ~/.bash_profile，這邊我自已的習慣是加入到 ~/.bashrc

# echo 'export LD_LIBRARY_PATH=/usr/local/python2.7/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
# source ~/.bashrc

加入後，再次執行 /usr/local/python2.7/bin/python -V

# /usr/local/python2.7/bin/python -V
Python 2.7.15

• 手動安裝 setuptools 、 pip 、 wheel 、 cx_Oracle

因為要用 Python 去連 Oracle DB，需要使用 cx_Oracle，測試一下目前的 python 有沒有這個 cx_Oracle 模組

# /usr/local/python2.7/bin/python -c "import cx_Oracle"
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named cx_Oracle

先下載這四個 Tarball 檔，請連至 PYPI 搜尋並下載

• setuptools
• pip
• wheel
• cx_Oracle

解壓縮，並請依序安裝

# unzip /tmp/setuptools-40.8.0.zip
# cd /tmp/setuptools-40.8.0
# /usr/local/python2.7/bin/python setup.py install

# tar -zxvf /tmp/pip-19.0.2.tar.gz
# cd /tmp/pip-19.0.2
# /usr/local/python2.7/bin/python setup.py install

# tar -zxvf /tmp/wheel-0.33.1.tar.gz
# cd /tmp/wheel-0.33.1
# /usr/local/python2.7/bin/python setup.py install

# tar -zxvf /tmp/cx_Oracle-7.1.1.tar.gz
# cd /tmp/cx_Oracle-7.1.1/
# /usr/local/python2.7/bin/python setup.py install

• 設定安裝 Oracle Client

參考 cx_Oracle 7 Installation 這官方文件來做安裝

由於我的伺服器沒有對外的連線，所以只好使用 Oracle Instant Client Zip Files 的安裝方法，請至 Oracle Instant Client 去下載 Instant Client for Linux x86-64，這邊案例我是下載這個檔 instantclient-basic-linux.x64-18.3.0.0.0dbru.zip，版本為 18.3.0.0.0

• 解壓縮，並放到指定路徑下

# cd /tmp
# unzip instantclient-basic-linux.x64-18.3.0.0.0dbru.zip
# cp -r /tmp/instantclient_18_3 /usr/local/

• 安裝套件

# yum install libaio

• 把 instantclient_18_3 加入到自已的 ~/.bashrc

# sed '/LD_LIBRARY_PATH/d' -i ~/.bashrc
# echo 'export LD_LIBRARY_PATH=/usr/local/python2.7/lib:/usr/local/instantclient_18_3:$LD_LIBRARY_PATH' >> ~/.bashrc
# source ~/.bashrc

• 再次執行

# /usr/local/python2.7/bin/python -c "import cx_Oracle"

完成上述步驟後，執行連接 Oracle 的程式，會出現下面的錯誤

Traceback (most recent call last):
  File "/tmp/Connect_Oracle.py", line 1, in 
    con = cx_Oracle.connect('Account/OraclePassWord')
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "/lib64/libc.so.6: version `GLIBC_2.9' not found (required by /usr/local/instantclient_18_3/libipc1.so)". See https://oracle.github.io/odpi/doc/installation.html#linux for help

上面這錯誤訊息是說，找不到 Oracle Client Library /lib64/libc.so.6 找不到版本 GLIBC_2.9

請執行 strings /lib64/libc.so.6 確認版本是否有支援

# strings /lib64/libc.so.6 | grep 'GLIBC_'
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_PRIVATE

觀察上面資訊後，確認沒有 `GLIBC_2.9 的版本，那就要去下載 Tarball 來安裝了。

• GLIBC Download: glibc

找到自已所需要的版本

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.7 (Tikanga)

確認 glibc 版本

# strings /lib64/libc.so.6 | grep 'GLIBC_'
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_PRIVATE

這裡要注意一下，要編譯前，請先執行 ls -la /lib64/libc.so.6

# ls -la /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Apr 21  2015 /lib64/libc.so.6 -> libc-2.5.so

先前不知道，就直接編輯下去了，結果重新要做軟連結時，會導致無法連進去伺服器，也不能執行任何指令，千萬不要重開機，如果真的不小心，執行指令有問題時，沒關係，請執行下面這個執行就可以了

# LD_PRELOAD=/lib64/libc-2.5.so ls

這樣就可以了，最後提醒一下，從這裡開始就要小心了~~~!!!!

先解壓縮

# cd /tmp/
# tar -zxvf /tmp/glibc-2.9.tar.gz
# cd glibc-2.9
# ./configure  --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
configure: error: you must configure in a separate build directory

這裡會出錯，請先在當前目錄中，建立一個 build 的目錄

# mkdir -p /tmp/glibc-2.9/build
# ../configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
...(省略)
checking LD_LIBRARY_PATH variable... contains current directory
configure: error:
*** LD_LIBRARY_PATH shouldn't contain the current directory when
*** building glibc. Please change the environment variable
*** and run configure again.

這邊又出一個錯誤，解法很簡單，執行 unset LD_LIBRARY_PATH，解法我是參考這篇 编译glibc(gcc)以及过程中遇到的一些错误

# unset LD_LIBRARY_PATH

再次編譯

# ../configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
# make
# make install
# ls -la /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Feb 23 15:52 /lib64/libc.so.6 -> libc-2.9.so

再次檢查 glibc 版本

# strings  /lib64/libc.so.6  | grep 'GLIBC_'
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_PRIVATE

再次執行連接 Oracle DB 的程式

# /usr/local/python2.7/bin/python /tmp/Connect_Oracle.py
/usr/local/python2.7/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

會出現這錯誤，主要是我們剛剛在編譯時，有 unset LD_LIBRARY_PATH，導致 python 無法讀取到 libpython2.7.so.1.0，所以我們只要把 export LD_LIBRARY_PATH=/usr/local/python2.7/lib:/usr/local/instantclient_18_3:$LD_LIBRARY_PATH 執行一次就行了，或是直接執行 source ~/.bashrc 即可，先前有加進去了。

以上就是踩雷的過程，完工。

回收 Veritas Storage Disk on RHEL 6.4

• 先 umount 掛載點
• 再執行 deport
• 刪除 path
• 徹底清空 path

---

• 先確認是否有 Veritas Cluster

# vxdctl -c mode
mode: enabled: cluster inactive

出現這個 cluster inactive 代表沒有啟用 Cluster，所以可以接續做下去

• 顯示準備要下線的 vxdg group 的掛載點

# df -h | grep -A 1 testdbdg
/dev/vx/dsk/testdbdg/testdb-archive01
                      100G  7.2G   93G   8% /DB_TEST/archive01
/dev/vx/dsk/testdbdg/testdb-archive02
                      750G  596G  153G  80% /DB_TEST/archive02
/dev/vx/dsk/testdbdg/testdb-archive03
                      750G  591G  158G  79% /DB_TEST/archive03
/dev/vx/dsk/testdbdg/AP_TEST
                      100G  299M   94G   1% /AP_TEST

• 確認目前 lsscsi 狀態

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb
[2:0:2:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdc
[2:0:3:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdd
[2:0:4:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sde
[2:0:5:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdf
[2:0:6:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdg
[2:0:8:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdh
[2:0:9:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdi
[2:0:10:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdj
[2:0:11:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdk
[2:0:12:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdl
[2:0:13:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdm
[2:0:14:0]   disk    VMware   Virtual disk     1.0   /dev/sdn
[2:0:15:0]   disk    VMware   Virtual disk     1.0   /dev/sdo
[3:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sdp
[3:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdq

這時候會看到有 LUN C-Mode 是從 NETAPP 來的，你也可以使用 lsscsi -c or cat /proc/scsi/scsi 也可以獲得類似相同的訊息

# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 02 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 03 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 04 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 05 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 06 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 08 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 09 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 10 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 11 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 12 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 13 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 14 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 15 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi3 Channel: 00 Id: 01 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02

依據上面的輸出顯示，可以發現都有固定的格式，參考如下:

[H:C:T:L]

          H == hostadapter id (first one being 0)
          C == SCSI channel on hostadapter (first one being 0)
          T == ID
          L == LUN (first one being 0)

依據上面的說明，可以得到下面這樣的資訊

Host: scsi2 Channel: 00 Id: 02 Lun: 00
這個就等於 [2:0:2:0]

為了要驗証是否相同，我們可以執行 lsscsi 2:00:02:00 or lsscsi 2:0:2:0，都會獲得上面相同的輸出訊息

# lsscsi 2:00:02:00
[2:0:2:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdc
# lsscsi 2:0:2:0
[2:0:2:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdc

• Umount Mount Point

# df -h | grep -A 1 testdbdg | grep '/DB_TEST' | awk '{print $NF}' | head -n 1 | xargs umount

• Remove Veritas Volume

請先使用 vxdisk -o alldgs list | grep testdbdg 觀察資訊

# vxdisk -o alldgs list | grep testdbdg
cl8080b_0_11 auto:cdsdisk    testdbdg13    testdbdg      online clone_disk
cl8080b_0_12 auto:cdsdisk    testdbdg01    testdbdg      online clone_disk
cl8080b_0_13 auto:cdsdisk    testdbdg03    testdbdg      online clone_disk
cl8080b_0_14 auto:cdsdisk    testdbdg12    testdbdg      online clone_disk
cl8080b_0_15 auto:cdsdisk    testdbdg10    testdbdg      online clone_disk
cl8080b_0_16 auto:cdsdisk    testdbdg11    testdbdg      online clone_disk
cl8080b_0_18 auto:cdsdisk    testdbdg15    testdbdg      online clone_disk

接著使用 vxprint -htg testdbdg 觀察資訊

# vxprint -htg testdbdg
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

dg testdbdg      default      default  46000    1383821912.75.luffy01p

dm testdbdg01    cl8080b_0_12 auto     65536    524103024 -
dm testdbdg03    cl8080b_0_13 auto     65536    524103024 -
dm testdbdg10    cl8080b_0_15 auto     65536    209534256 -
dm testdbdg11    cl8080b_0_16 auto     65536    209534256 -
dm testdbdg12    cl8080b_0_14 auto     65536    209534256 -
dm testdbdg13    cl8080b_0_11 auto     65536    524103024 -
dm testdbdg15    cl8080b_0_18 auto     65536    209534256 -

v  AP_TEST       -            ENABLED  ACTIVE   209532928 SELECT   -        fsgen
pl AP_TEST-02    AP_TEST       ENABLED  ACTIVE   209532928 CONCAT   -        RW
sd testdbdg10-01 AP_TEST-02    testdbdg10 0       209532928 0        cl8080b_0_15 ENA

v  testdb-archive-node2 -     ENABLED  ACTIVE   209532928 SELECT   -        fsgen
pl testdb-archive-node2-02 testdb-archive-node2 ENABLED ACTIVE 209532928 CONCAT - RW
sd testdbdg11-01 testdb-archive-node2-02 testdbdg11 0 209532928 0    cl8080b_0_16 ENA

v  testdb-archive-node3 -     ENABLED  ACTIVE   209532928 SELECT   -        fsgen
pl testdb-archive-node3-02 testdb-archive-node3 ENABLED ACTIVE 209532928 CONCAT - RW
sd testdbdg15-01 testdb-archive-node3-02 testdbdg15 0 209532928 0    cl8080b_0_18 ENA

v  testdb-archive-node4 -     ENABLED  ACTIVE   209532928 SELECT   -        fsgen
pl testdb-archive-node4-02 testdb-archive-node4 ENABLED ACTIVE 209532928 CONCAT - RW
sd testdbdg12-01 testdb-archive-node4-02 testdbdg12 0 209532928 0    cl8080b_0_14 ENA

v  testdb-archive03 -         ENABLED  ACTIVE   1572141056 SELECT  -        fsgen
pl testdb-archive03-02 testdb-archive03 ENABLED ACTIVE 1572141056 CONCAT -   RW
sd testdbdg13-02 testdb-archive03-02 testdbdg13 75136 523937792 0    cl8080b_0_11 ENA
sd testdbdg01-01 testdb-archive03-02 testdbdg01 0 524101632 523937792 cl8080b_0_12 ENA
sd testdbdg03-01 testdb-archive03-02 testdbdg03 0 524101632 1048039424 cl8080b_0_13 ENA

過濾出準備要回收的 Volume，並觀察資訊

# vxprint -htg testdbdg | grep testdb-archive03
v  testdb-archive03 -         ENABLED  ACTIVE   1572141056 SELECT  -        fsgen
pl testdb-archive03-02 testdb-archive03 ENABLED ACTIVE 1572141056 CONCAT -   RW
sd testdbdg13-02 testdb-archive03-02 testdbdg13 75136 523937792 0    cl8080b_0_11 ENA
sd testdbdg01-01 testdb-archive03-02 testdbdg01 0 524101632 523937792 cl8080b_0_12 ENA
sd testdbdg03-01 testdb-archive03-02 testdbdg03 0 524101632 1048039424 cl8080b_0_13 ENA

對應欄位的意義

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
v  testdb-archive03 -         ENABLED  ACTIVE   1572141056 SELECT  -        fsgen

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
pl testdb-archive03-02 testdb-archive03 ENABLED ACTIVE 1572141056 CONCAT -   RW

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
sd testdbdg13-02 testdb-archive03-02 testdbdg13 75136 523937792 0    cl8080b_0_11 ENA
sd testdbdg01-01 testdb-archive03-02 testdbdg01 0 524101632 523937792 cl8080b_0_12 ENA
sd testdbdg03-01 testdb-archive03-02 testdbdg03 0 524101632 1048039424 cl8080b_0_13 ENA

停止 VxVM 啟用中的 Volume

• 指令: vxvol [-g diskgroup] stop volume_name

# vxvol -g testdbdg stop testdb-archive03

執行 vxprint -htg testdbdg | grep testdb-archive03 再次觀察資訊

# vxprint -htg testdbdg | grep testdb-archive03
v  testdb-archive03 -         DISABLED CLEAN    1572141056 SELECT  -        fsgen
pl testdb-archive03-02 testdb-archive03 DISABLED CLEAN 1572141056 CONCAT -   RW
sd testdbdg13-02 testdb-archive03-02 testdbdg13 75136 523937792 0    cl8080b_0_11 ENA
sd testdbdg01-01 testdb-archive03-02 testdbdg01 0 524101632 523937792 cl8080b_0_12 ENA
sd testdbdg03-01 testdb-archive03-02 testdbdg03 0 524101632 1048039424 cl8080b_0_13 ENA

會發現開頭 v 行中的 KSTATE & STATE 被改成 DISABLED & CLEAN， 這代表 volume 已被停止了，接下來就是要刪除 Volume 了

刪除 VxVM 已停止的 Volume

• 指令: vxassist [-g diskgroup] remove volume volume_name

# vxassist -g testdbdg remove volume testdb-archive03

再次執行 vxprint -htg testdbdg | grep testdb-archive03，並觀察資訊

這樣就已完成 VxVM Remove Volume 了，接下來就是刪除 Disk

• Remove Disk From DiskGroup

請執行 vxdisk -o alldgs list | grep -Ev '^sd'，並觀察資訊

# vxdisk -o alldgs list | grep -Ev '^sd'
DEVICE       TYPE            DISK         GROUP        STATUS
cl8080b_0_11 auto:cdsdisk    testdbdg13    testdbdg      online clone_disk
cl8080b_0_12 auto:cdsdisk    testdbdg01    testdbdg      online clone_disk
cl8080b_0_13 auto:cdsdisk    testdbdg03    testdbdg      online clone_disk
cl8080b_0_14 auto:cdsdisk    testdbdg12    testdbdg      online clone_disk
cl8080b_0_15 auto:cdsdisk    testdbdg10    testdbdg      online clone_disk
cl8080b_0_16 auto:cdsdisk    testdbdg11    testdbdg      online clone_disk
cl8080b_0_17 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_18 auto:cdsdisk    testdbdg15    testdbdg      online clone_disk
cl8080b_0_19 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_20 auto:cdsdisk    -            -            online clone_disk

會發現有幾個 Disk 出現 - 這樣的符號，代表還沒加入過任何的 VxVM 中的 DiskGroup Volume， 接下來要進行刪除 Disk 了，因為先前我們有紀錄下 Disk，所以可以參考下面的資訊來進行刪除。

# vxprint -htg testdbdg | grep testdb-archive03
v  testdb-archive03 -         DISABLED CLEAN    1572141056 SELECT  -        fsgen
pl testdb-archive03-02 testdb-archive03 DISABLED CLEAN 1572141056 CONCAT -   RW
sd testdbdg13-02 testdb-archive03-02 testdbdg13 75136 523937792 0    cl8080b_0_11 ENA
sd testdbdg01-01 testdb-archive03-02 testdbdg01 0 524101632 523937792 cl8080b_0_12 ENA
sd testdbdg03-01 testdb-archive03-02 testdbdg03 0 524101632 1048039424 cl8080b_0_13 ENA

刪除 VxVM 中的 Disk

• 指令: # vxdg [-g diskgroup] rmdisk diskname

# vxdg -g testdbdg rmdisk testdbdg13
# vxdg -g testdbdg rmdisk testdbdg01
# vxdg -g testdbdg rmdisk testdbdg03

再次執行 vxdisk -eo alldgs list | grep -Ev '^sd*'，請觀察資訊

# vxdisk -o alldgs list | grep -Ev '^sd*'
DEVICE       TYPE            DISK         GROUP        STATUS
cl8080b_0_11 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_12 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_13 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_14 auto:cdsdisk    testdbdg12    testdbdg      online clone_disk
cl8080b_0_15 auto:cdsdisk    testdbdg10    testdbdg      online clone_disk
cl8080b_0_16 auto:cdsdisk    testdbdg11    testdbdg      online clone_disk
cl8080b_0_17 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_18 auto:cdsdisk    testdbdg15    testdbdg      online clone_disk
cl8080b_0_19 auto:cdsdisk    -            -            online clone_disk
cl8080b_0_20 auto:cdsdisk    -            -            online clone_disk

• 徹底清空 PATH

請先執行 vxdisk -eo alldgs list | grep -Ev '^sd*'，請觀察資訊

# vxdisk -eo alldgs list | grep -Ev '^sd*'
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
cl8080b_0_11 auto:cdsdisk   -            -           online clone_disk    sdi              tprclm
cl8080b_0_12 auto:cdsdisk   -            -           online clone_disk    sdk              tprclm
cl8080b_0_13 auto:cdsdisk   -            -           online clone_disk    sdm              tprclm
cl8080b_0_14 auto:cdsdisk   testdbdg12    testdbdg     online clone_disk    sdg              tprclm
cl8080b_0_15 auto:cdsdisk   testdbdg10    testdbdg     online clone_disk    sdc              tprclm
cl8080b_0_16 auto:cdsdisk   testdbdg11    testdbdg     online clone_disk    sde              tprclm
cl8080b_0_17 auto:cdsdisk   -            -           online clone_disk    sdl              tprclm
cl8080b_0_18 auto:cdsdisk   testdbdg15    testdbdg     online clone_disk    sdf              tprclm
cl8080b_0_19 auto:cdsdisk   -            -           online clone_disk    sdj              tprclm
cl8080b_0_20 auto:cdsdisk   -            -           online clone_disk    sdh              tprclm

再執行 lsscsi，請觀察資訊

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb
[2:0:2:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdc
[2:0:3:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdd
[2:0:4:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sde
[2:0:5:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdf
[2:0:6:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdg
[2:0:8:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdh
[2:0:9:0]    disk    NETAPP   LUN C-Mode       9000  /dev/sdi
[2:0:10:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdj
[2:0:11:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdk
[2:0:12:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdl
[2:0:13:0]   disk    NETAPP   LUN C-Mode       9000  /dev/sdm
[2:0:14:0]   disk    VMware   Virtual disk     1.0   /dev/sdn
[2:0:15:0]   disk    VMware   Virtual disk     1.0   /dev/sdo
[3:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sdp
[3:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdq

可以透過觀察上面二個資訊後，可以得到 OS_NATIVE_NAME 會對應到 lsscsi 中的 /dev/sd* 這樣就可以來做徹底清空 Disk Path

執行 echo 1 > /sys/class/scsi_device/2:0:9:0/device/delete，下面三種可以選一個自已喜歡的方式

echo 1 >  /sys/class/scsi_device/2:0:9:0/device/delete

or

echo 1 > /sys/block/sdi/device/delete

or 

echo "scsi remove-single-device 2 0 9 0" > /proc/scsi/scsi

請再次執行 lsscsi | grep sdi，請觀察資訊

請再次執行 $ cat /proc/scsi/scsi，請觀察資訊

$ cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 02 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 03 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 04 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 05 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 06 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 08 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 10 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 11 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 12 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 13 Lun: 00
  Vendor: NETAPP   Model: LUN C-Mode       Rev: 9000
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi2 Channel: 00 Id: 14 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 15 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi3 Channel: 00 Id: 01 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 02

請再次執行 vxdisk -eo alldgs list | grep -Ev '^sd*'，請觀察資訊

$ vxdisk -eo alldgs list | grep -Ev '^sd*'
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
cl8080b_0_11 auto           -            -           error                sdi              tprclm
cl8080b_0_12 auto:cdsdisk   -            -           online clone_disk    sdk              tprclm
cl8080b_0_13 auto:cdsdisk   -            -           online clone_disk    sdm              tprclm
cl8080b_0_14 auto:cdsdisk   testdbdg12    testdbdg     online clone_disk    sdg              tprclm
cl8080b_0_15 auto:cdsdisk   testdbdg10    testdbdg     online clone_disk    sdc              tprclm
cl8080b_0_16 auto:cdsdisk   testdbdg11    testdbdg     online clone_disk    sde              tprclm
cl8080b_0_17 auto:cdsdisk   -            -           online clone_disk    sdl              tprclm
cl8080b_0_18 auto:cdsdisk   testdbdg15    testdbdg     online clone_disk    sdf              tprclm
cl8080b_0_19 auto:cdsdisk   -            -           online clone_disk    sdj              tprclm
cl8080b_0_20 auto:cdsdisk   -            -           online clone_disk    sdh              tprclm

依據上面的觀察得知，確認 /dev/sdi 已經被清除了，但執行 vxdisk -eo alldgs list | grep -Ev '^sd*' 還是會有 error，這個等等再來修正 先把要刪除的 Disk Path 都給執行完，我會用不同的刪除方法執行

# echo 1 > /sys/block/sdk/device/delete
# lsscsi | grep sdk

# echo "scsi remove-single-device 2:0:13:0" > /proc/scsi/scsi
# lsscsi | grep sdm

請再次執行 lsscsi | grep sdi，請觀察資訊

請再次執行 $ cat /proc/scsi/scsi，請觀察資訊

以上執行完後，就可以了，最後就是要來修正執行 vxdisk -eo alldgs list | grep -Ev '^sd*' 還是會有 error的問題 其實很簡單，只要執行 vxdisk scandisks 這樣就行了。

Referenece:

• Removing a volume
• Removing a disk from a disk group
• Moving volumes to disk partitions
• vxdg rmdisk errors with ERROR V-5-1-553 Disk DM_name is used by one or more subdisks
• Hot add, remove, rescan of SCSI devices on Linux
• Linux lsscsi - list SCSI devices (or hosts) and their attributes