Name: tcmu-runner
Owner: Ceph
Description: A daemon that handles the userspace side of the LIO TCM-User backstore.
Forked from: open-iscsi/tcmu-runner
Created: 2017-02-13 17:02:49.0
Updated: 2018-05-06 00:26:02.0
Pushed: 2018-05-06 00:39:57.0
Homepage: null
Size: 1206
Language: C
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
A daemon that handles the userspace side of the LIO TCM-User backstore.
LIO is the SCSI target in the Linux kernel. It is entirely kernel code, and allows exported SCSI logical units (LUNs) to be backed by regular files or block devices. But, if we want to get fancier with the capabilities of the device we're emulating, the kernel is not necessarily the right place. While there are userspace libraries for compression, encryption, and clustered storage solutions like Ceph or Gluster, these are not accessible from the kernel.
The TCMU userspace-passthrough backstore allows a userspace process to handle requests to a LUN. But since the kernel-user interface that TCMU provides must be fast and flexible, it is complex enough that we'd like to avoid each userspace handler having to write boilerplate code.
tcmu-runner handles the messy details of the TCMU interface – UIO, netlink, pthreads, and DBus – and exports a more friendly C plugin module API. Modules using this API are called “TCMU handlers”. Handler authors can write code just to handle the SCSI commands as desired, and can also link with whatever userspace libraries they like.
One goal of TCMU is that configuring a userspace-backed LUN should be as easy as configuring a kernel-backed LUN. We're not quite there yet. This will require cooperation with the LIO configuration tool, targetcli
. targetcli
should list user-backed backstores along with the built-in kernel backstores, and ensure tcmu-runner is started if a user-backed backstore is created.
tcmu-runner is Apache 2.0 licensed.
Tarballs are available from https://fedorahosted.org/released/tcmu-runner/ .
We encourage pull requests and issues tracking via Github, and the target-devel mailing list (list info) may be used for discussion.
./extra/install_dep.sh
to install development packages for dependencies, or you can do it manually:cmake .
-Dwith-glfs=false
and -Dwith-qcow=false
cmake parameters respectively.-DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr
should be passed to cmake, so files are installed to the correct location.make
make install
tcmu-runner.conf
to /etc/dbus-1/system.d/
. This allows tcmu-runner to be on the system bus, which is privileged.org.kernel.TCMUService1.service
to /usr/share/dbus-1/system-services/
and tcmu-runner.service
to /lib/systemd/system
.If setting up tcmu-runner in a HA configuration, the ceph-iscsi-cli (https://github.com/ceph/ceph-iscsi-cli) tool is the preferred management tool.
Bug reports should be made to the tcmu-runner github: https://github.com/open-iscsi/tcmu-runner/issues, but can be made to ceph-users@ceph.com mailing list.
Gluster management must be done with the gluster-block tools (https://github.com/gluster/gluster-block).
Bug reports must be made to the gluster-block github: https://github.com/gluster/gluster-block/issues
Support for the user/tcmu backstore is supported in targetcli-fb/rtslib-fb:
https://github.com/open-iscsi/targetcli-fb https://github.com/open-iscsi/rtslib-fb
# targetcli
/> cd /backstores/
kstores> ls
ackstores .......................................................... [...]
user:glfs .......................................... [Storage Objects: 0]
user:qcow .......................................... [Storage Objects: 0]
user:rbd ........................................... [Storage Objects: 0]
user:file .......................................... [Storage Objects: 0]
user:zbc ........................................... [Storage Objects: 0]
/backstores> cd user:rbd
/backstores/user:rbd> create cfgstring=pool/rbd1;osd_op_timeout=30 name=rbd0 size=1G Created user-backed storage object rbd0 size 1073741824.
Note that the cfgstring is handler specific. The format is:
For the zbc handler, the available options are shown in the table below.
| Option | Description | Default value | | — | — | — | | model-type | Device model type, HA for host aware or HM for host managed | HM | lba-size (B) | LBA size in bytes (512 or 4096) | 512 | zsize-size (MiB) | Zone size in MiB | 256 MiB | conv-num | Number of conventional zones at LBA 0 (can be 0) | Number of zones corresponding to 1% of the device capacity | open-num | Optimal (for host aware) or maximum (for host managed) number of open zones | 128
Example:
tring=model-HM/zsize-128/conv-100@/var/local/zbc.raw
will create a host-managed disk with 128 MiB zones and 100 conventional zones, stored in the file /var/local/zbc.raw.
There are 5 logging levels supported:
And the default logging level is 3, if you want to change the default level, uncomment the following line in /etc/tcmu/tcmu.conf and set your level number:
# log_level = 3
The priority of the logdir setting can be managed via following options:
Cli argument
eg: –tcmu_log_dir/-l LOG_DIR_PATH
[Highest prio]
Environment variable eg: export TCMU_LOGDIR=“/var/log/mylogdir/”
Configuration file eg: uncommenting and adjusting value of 'log_dir_path' at /etc/tcmu/tcmu.conf
Default logdir as hard coded i.e. '/var/log/' [Least prio]
System configuration:
The default configuration file is installed into /etc/tcmu/tcmu.conf.
Tcmu-runner's configuration systems supports dynamic reloading without restarting the daemon. To change values open /etc/tcmu/tcmu.conf, update the value, and then close the file.
If your version of targetcli/rtslib does not support tcmu, setup can be done manually through configfs:
target_core_user
kernel module is loaded.mkdir -p /sys/kernel/config/target/core/user_1/test
cd /sys/kernel/config/target/core/user_1/test
echo -n dev_size=16777216 > control
echo -n dev_config=baz/addl_info_for_baz_handler > control
echo -n 1 > enable
/sys/class/uio
.To delete:
rmdir /sys/kernel/config/target/core/user_1/test
rmdir /sys/kernel/config/target/core/user_1
There are two different ways to write a TCMU handler. The primary difference is who is responsible for the event loop.
There are two different ways to write a tcmu-runner plugin handler:
With the option 1, tcmu-runner is in charge of the event loop
for your plugin, and your handler's handle_cmd
function is called
repeatedly to respond to each incoming SCSI command. While your
handler sees all SCSI commands, there are helper functions provided
that save each handler from writing boilerplate code for mandatory
SCSI commands, if desired.
The file_optical
handler is an examples of this type.
With the option 2, tcmu-runner will be partially or fully in charge of the event loop and SCSI command handling for your plugin, and your handler's registered functions are called repeatedly to handle storage requests as required by the upper SCSI layer, which will handle most of the SCSI commands for you.
The file_example
handler is an example of this type.
If you want to add handling of TCMU devices to an existing daemon or
other program that already is processing its own event loop, the best
option is to use tcmulib directly. This requires your code to keep
track of tcmulib's file descriptors. While tcmulib's 'master' file
descriptor must be handled with tcmulib_master_fd_ready()
single-threadedly, per-device fds can be handled on the main thread
(with tcmulib_get_next_command
and tcmulib_command_complete
) or
separate threads if desired. SCSI command-processing helper functions
are still available for use.
tcmu-runner
itself uses tcmulib in this manner and may be used as an
example of multi-threaded tcmulib use. The consumer.c
example
demonstrates single-threaded tcmulib processing.