Creation of SPDK NVMe-OF TCP targets

Creation of SPDK NVMe-OF TCP targets

We can easily create NVMe targets using SPDK software. This document shall give us clear steps to configure and use SPDK targets and initiator.

These targets can be used and NVMe LUNs can be exposed through them.

Documentation 
https://spdk.io/
https://spdk.io/doc/getting_started.html

SPDK download :
git clone https://github.com/spdk/spdk
cd spdk
git submodule update –init  

INSTALL DEPENDENCIES
sudo su
scripts/pkgdep.sh

BUILDING
./configure
make

SETUP TO alloc HUGEPAGES
sudo su
scripts/setup.sh

STORAGE ARRAY PREPARATION:
https://spdk.io/doc/nvmf.html
sudo su   - always
build/bin/nvmf_tgt &

scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
scripts/rpc.py bdev_malloc_create -b Malloc0 512 512
scripts/rpc.py bdev_malloc_create -b Malloc1 512 512

scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc1

scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 100.XXX.XXX.XX -s 4420

scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 100.XXX.XXX.XX -s 4420

FROM HOST :

sudo nvme connect -t tcp '--hostnqn=myhostnqn' -n "nqn.2016-06.io.spdk:cnode1" -a 100.XXX.XXX.XX -s 4420
sudo nvme connect -t tcp '--hostnqn=myhostnqn' -n "nqn.2016-06.io.spdk:cnode1" -a 100.XXX.XXX.XX -s 4420

Kernel devices :

/dev/nvme0n1
/dev/nvme0n2
/dev/nvme1n1

using the SPDK IO submission : 

./build/examples/perf -q 16 -s 1024 -o 512 -w read -r 'trtype:rdma adrfam:IPv4 traddr:10.XXX.XXX.ZZZ trsvcid:4420' -t 5

ANA reporting modifications : 

For ANA feature we need to create SPDK subsystem with ANA reporting ON :
scripts/rpc.py nvmf_create_subsystem -h 
  -r, --ana-reporting   Enable ANA reporting feature

Steps : 
Modification while creating the subsystem :
scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1 -m 128 -r

Make IPs as optimised and non-optimised :
scripts/rpc.py nvmf_subsystem_listener_set_ana_state nqn.2016-06.io.spdk:cnode1 -t rdma -a 10.XX.XX.XX -s 4420 -n optimized
scripts/rpc.py nvmf_subsystem_listener_set_ana_state nqn.2016-06.io.spdk:cnode1 -t rdma -a 10.XX.XY.YY -s 4420 -n non_optimized