Beetle is a GitHub repository bundled with custom linux repos and handy scripts, targeted to simplify initial configurations of linux hosts during POCs.
The main purpose of this tool is to streamline some of the unexpected scenarios we often encounter when we walk in to the remote POC datacenters with unique policies and perceptions. As is unexpected things will always be there, hope we could enhance this tool and address them as much as possible using this tool.
One of those scenarios is walking in to the lab or datacenter with no access to the internet or yum/zypper repositories. Beetle can be downloaded locally to your laptop or external media from GitHub and port it as needed. The custom linux repositories included in beetle covers all the required linux packages required during POC.
This package also includes the custom multipath configuration file and recommended performance optimization settings file. These are installed in the client hosts through install scripts.
In addition, we had an opportunity to develop some useful shortcuts (unix functions to sound boring) over period of time to help with some of the frequently performed tasks. These shortcuts do assume some non-minimal packages are installed on the Linux hosts. These are bundled with the custom repos and install script will take care of them.
Also a simple bash script is included, to perform the basic storage provisioning on a vexata array.
Last but not least some sample(handy) work profiles for FIO and VDBENCH tool are included in bettle.
Run on any Linux host with access to github:
git clone https://github.com/vxtools/beetle.git /opt/beetle
Note: As of now, please download this git repo on /opt in the Linux test machine. There is a dependency on custom repos for this. Will update this as and when we find an altenative.
- CentOS/Redhat/OEL/Fedora (version 7)
- SLES12 Note: Support to other OSes can be added on an as-needed basis.
Identify the Linux distro that you are working on and run respective install scripts (on all Linux hosts connected to the array):
- Configures the custom repos
- Installs all the required packages
- Install device-mapper and configures with vexata specific settings
- Installs udev rules optmized for vexata array
- Cleans up the custom repo
# ./install_linux.sh
# ./install_sles.sh
For inbox drivers, please run the following commands or use update_emulex_modprobe.sh script to distribute the interrupts across all cores in optimal way. Since irqbalance is a system level setting, disabling it is not included in the install_<distro>.sh script.
PS: Make sure there are no other devices like ethernet or inifiniband cards need irqbalance settings with in the system before disabling it.
# systemctl stop irqbalance
# systemctl disable irqbalance
# cp -pr /opt/beetle/opt/emulex/elx-lpfc-vector-map.conf /etc/modprobe.d/
# cp -pr /opt/beetle/opt/emulex/lpfc /etc/modprobe.d/
If you are able to install the latest driver from Broadcom Website, you may just run following command to increase the lpfc_fcp_imax value.
# sed -i "s/lpfc \$CMD/lpfc lpfc_fcp_imax=200000 \$CMD/g" /etc/modprobe.d/elx-lpfc-vector-map.conf
With inbox or latest drivers, you could run multipath -F; sleep 5 ; rmmod lpfc and modprobe lpfc or reboot the host for the above settings to get effective.
File location: vx_scripts/configure_vexata_array.sh
Please note that this script needs to be run from array.
- Copy the file to the array
# scp /opt/beetle/vx_scripts/configure_vexata_array.sh root@<array-name>:/root/
- Update the
HOSTSlist variable in the script to reflect all the linux host names.
HOSTS=("host1" "host2") # Replace this with linux hosts assosiated with this setup
- If possible enable ssh password less loging from array to linux host. If its not an option for some reason, just type the host password when prompted.
# /root/configure_vexata_array.sh <No. of Volumes> <Each Volume Size in GiB>
Now the fun part begins, let us make sure the volumes provisioned are visible to hosts:
- Login to individual Linux host and run rescan
# rescan-scsi-bus.sh
- Make sure multipath identifies vexata paths correctly.
# source /opt/beetle/.test_env
# vc
Vexata SDs : 224
Vexata DMs : 7
dm-10 : 32
dm-9 : 32
dm-8 : 32
dm-7 : 32
dm-6 : 32
dm-12 : 32
dm-11 : 32
- If you don't see the expected device count for
Vexata SDs. Make sure all fiber channel ports are online on linux host
# ws
host11 0x10000090fa9277ec Online 16 Gbit
host12 0x10000090fa9277ed Online 16 Gbit
- Make sure all FC ports are online on Vexata Array.
[root@bolt41-ioc0 ~]$ vxcli sa portlist
Num SA Ports = 16
PortNum Node/Ioc/Port Type WWN IniCnt VolCnt RefCnt State
--------------------------------------------------------------------------------------------
00 0/0/0 FC 10:00:3c:91:2b:00:75:00 20 24 2 Online
01 0/0/1 FC 10:00:3c:91:2b:00:75:01 20 24 2 Online
02 0/0/2 FC 10:00:3c:91:2b:00:75:02 20 24 2 Online
03 0/0/3 FC 10:00:3c:91:2b:00:75:03 20 24 2 Online
04 0/0/4 FC 10:00:3c:91:2b:00:75:04 20 24 2 Online
05 0/0/5 FC 10:00:3c:91:2b:00:75:05 20 24 2 Online
06 0/0/6 FC 10:00:3c:91:2b:00:75:06 20 24 2 Online
07 0/0/7 FC 10:00:3c:91:2b:00:75:07 20 24 2 Online
08 0/1/0 FC 10:00:3c:91:2b:00:75:08 20 24 2 Online
09 0/1/1 FC 10:00:3c:91:2b:00:75:09 20 24 2 Online
10 0/1/2 FC 10:00:3c:91:2b:00:75:0a 20 24 2 Online
11 0/1/3 FC 10:00:3c:91:2b:00:75:0b 20 24 2 Online
12 0/1/4 FC 10:00:3c:91:2b:00:75:0c 20 24 2 Online
13 0/1/5 FC 10:00:3c:91:2b:00:75:0d 20 24 2 Online
14 0/1/6 FC 10:00:3c:91:2b:00:75:0e 20 24 2 Online
15 0/1/7 FC 10:00:3c:91:2b:00:75:0f 20 24 2 Online
- Login to SAN FC switch or contact the SAN administrator to make sure all ports are logged in properly to the FC switch. If Zoning is enabled, make sure it is set properly.
switch:admin> switchshow # to check the port login status
switch:admin> zoneshow --validate # to verify all members of the zone are active or not.
Defined configuration:
cfg: cfg1 zone1; zone10; zone2
zone: zone1 10:00:00:90:fa:92:77:ec*; 10:00:3c:91:2b:00:75:00
Effective configuration:
cfg: cfg1
zone: zone1 10:00:00:90:fa:92:77:ec*
10:00:3c:91:2b:00:75:00
------------------------------------
~ - Invalid configuration
* - Member does not exist
# - Invalid usage of broadcast zone
# show fcns database vsan 1
# show zoneset active vsan 1
zoneset name b33 vsan 1
zone name bolt35 vsan 1
fcid 0x487500 [pwwn 10:00:00:90:fa:92:77:ed]
* fcid 0x487600 [pwwn 10:00:3c:91:2b:00:75:01]
<truncated>
* if link is up
- If no obvious issues are found from above steps, please refer to Linux best practices guide to try all other rescaning options.
Following are the list of handy shortcuts available to help with some routine tasks. These are not imported by default. It need to explicitly sourced on each linux host to use them.
# source /opt/beetle/.test_env
# vhelp
vdblist : Create sd list for vdbench profiles
fiolist : List device to be used with FIO command.
wwnlist : WWN List of all FC ports
ws : Status check of FC ports
dlist : list the dm-X devices in one line
sdlist : list the sdX devices in one line
lstune : Lists the values of all frequently tuned settings for devices.
tune_all : Set the recommended values to all frequently tuned settings for devices.
tune_random : Set the requested value to enable/disable entrophy on all vexata volumes(default
tune_affinity : Set the requested affinity value to all vexata volumes (default
tune_nr : Increase / decrease the nr_requests on all vexata volumes (default
tune_sched : set I/O scheduler on all vexata volumes (default
rescan_all : Rescan all scsi devices asoosiated with FC
rmdev : Remove stale device entries
mpconfig : List the effective multipath configuration.
vc : List the number of SD volumes and DM volumes.
dl : list all dm-X devices with new-line. Usefull to pass on to test tools.
devlist : list all dm-X devices with full path name in single line. Usefull to pass on to test tools.
ver : Display the current OS version.
hbaver : Display HBA version installed on the system,
check_interrupts : Check interrupt values on the system
chdev_to : Change timeout for SD devices
lsdev_to : List timeout for SD devices
get_max : List the max_sectors_kb value on all vexata volumes
You may view individual shortcut code by using type
# type vc
vc is a function
vc ()
{
echo "Vexata SDs : $(lsscsi |grep Vexata|wc -l)";
tf=$(multipath -ll);
echo "Vexata DMs : $(echo "$tf" | grep Vexata |wc -l)";
echo "$tf" | awk '/Vexata/ {print $3}' | while read mp; do
echo "$mp : $(echo "$tf" | sed -n '/'${mp}'/,/^mpath/p' | awk '/^ \|-|^ `/ {print $0}' | wc -l)";
done
}
Feel free to reach out to [email protected] for any questions. Appreciate any feedback or suggestions to make this tool more useful.