HardenedBSD/sys/cam/ctl

/* $FreeBSD$ */

CTL - CAM Target Layer Description

Revision 1.4 (December 29th, 2011)
Ken Merry <ken@FreeBSD.org>

Table of Contents:
=================

Introduction
Features
Configuring and Running CTL
Revision 1.N Changes
To Do List
Code Roadmap
Userland Commands

Introduction:
============

CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003.  It has been shipping in
Copan (now SGI) products since 2005.

It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
available under a BSD-style license.  The intent behind the agreement was
that Spectra would work to get CTL into the FreeBSD tree.

Features:
========

 - Disk and processor device emulation.
 - Tagged queueing
 - SCSI task attribute support (ordered, head of queue, simple tags)
 - SCSI implicit command ordering support.  (e.g. if a read follows a mode
   select, the read will be blocked until the mode select completes.)
 - Full task management support (abort, LUN reset, target reset, etc.)
 - Support for multiple ports
 - Support for multiple simultaneous initiators
 - Support for multiple simultaneous backing stores
 - Persistent reservation support
 - Mode sense/select support
 - Error injection support
 - High Availability support (1)
 - All I/O handled in-kernel, no userland context switch overhead.

(1) HA Support is just an API stub, and needs much more to be fully
    functional.  See the to-do list below.

Configuring and Running CTL:
===========================

 - After applying the CTL patchset to your tree, build world and install it
   on your target system.

 - Add 'device ctl' to your kernel configuration file.

 - If you're running with a 8Gb or 4Gb Qlogic FC board, add
   'options ISP_TARGET_MODE' to your kernel config file.  Keep in mind that
   the isp(4) driver can run in target or initiator mode, but not both on
   the same machine.  'device ispfw' or loading the ispfw module is also
   recommended.

 - Rebuild and install a new kernel.

 - Reboot with the new kernel.

 - To add a LUN with the RAM disk backend:

	ctladm create -b ramdisk -s 10485760000000000000
	ctladm port -o on

 - You should now see the CTL disk LUN through camcontrol devlist:

scbus6 on ctl2cam0 bus 0:
<FREEBSD CTLDISK 0001>             at scbus6 target 1 lun 0 (da24,pass32)
<>                                 at scbus6 target -1 lun -1 ()

   This is visible through the CTL CAM SIM.  This allows using CTL without
   any physical hardware.  You should be able to issue any normal SCSI
   commands to the device via the pass(4)/da(4) devices.

   If any target-capable HBAs are in the system (e.g. isp(4)), and have
   target mode enabled, you should now also be able to see the CTL LUNs via
   that target interface.

   Note that all CTL LUNs are presented to all frontends.  There is no
   LUN masking, or separate, per-port configuration.

 - Note that the ramdisk backend is a "fake" ramdisk.  That is, it is
   backed by a small amount of RAM that is used for all I/O requests.  This
   is useful for performance testing, but not for any data integrity tests.

 - To add a LUN with the block/file backend:

	truncate -s +1T myfile
	ctladm create -b block -o file=myfile
	ctladm port -o on

 - You can also see a list of LUNs and their backends like this:

# ctladm devlist
LUN Backend       Size (Blocks)   BS Serial Number    Device ID       
  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0     
  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1     
  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2     
  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3     
  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4     
  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5     
  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6     
  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7     
  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8     
  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9     
 10 block            2147483648  512 MYSERIAL  10     MYDEVID  10     
 11 block            2147483648  512 MYSERIAL  11     MYDEVID  11    

 - You can see the LUN type and backing store for block/file backend LUNs
   like this:

# ctladm devlist -v
LUN Backend       Size (Blocks)   BS Serial Number    Device ID       
  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0     
      lun_type=0
      num_threads=14
      file=testdisk0
  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1     
      lun_type=0
      num_threads=14
      file=testdisk1
  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2     
      lun_type=0
      num_threads=14
      file=testdisk2
  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3     
      lun_type=0
      num_threads=14
      file=testdisk3
  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4     
      lun_type=0
      num_threads=14
      file=testdisk4
  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5     
      lun_type=0
      num_threads=14
      file=testdisk5
  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6     
      lun_type=0
      num_threads=14
      file=testdisk6
  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7     
      lun_type=0
      num_threads=14
      file=testdisk7
  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8     
      lun_type=0
      num_threads=14
      file=testdisk8
  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9     
      lun_type=0
      num_threads=14
      file=testdisk9
 10 ramdisk                   0    0 MYSERIAL   0     MYDEVID   0     
      lun_type=3
 11 ramdisk     204800000000000  512 MYSERIAL   1     MYDEVID   1     
      lun_type=0


Revision 1.4 Changes
====================
 - Added in the second HA mode (where CTL does the data transfers instead
   of having data transfers done below CTL), and abstracted out the Copan
   HA API.

 - Fixed the phantom device problem in the CTL CAM SIM and improved the
   CAM SIM to automatically trigger a rescan when the port is enabled and
   disabled.
 
 - Made the number of threads in the block backend configurable via sysctl,
   loader tunable and the ctladm command line.  (You can now specify
   -o num_threads=4 when creating a LUN with ctladm create.)

 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection
   of LUN numbers up to 1023.

 - General cleanup.

 - This version intended for public release.

Revision 1.3 Changes
====================
 - Added descriptor sense support to CTL.  It can be enabled through the
   control mode page (10), but is disabled by default.

 - Improved error injection support.  The number of errors that can be
   injected with 'ctladm inject' has been increased, and any arbitrary
   sense data may now be injected as well.

 - The port infrastructure has been revamped.  Individual ports and types
   of ports may now be enabled and disabled from the command line.  ctladm
   now has the ability to set the WWNN and WWPN for each port.

 - The block backend can now send multiple I/Os to backing files.  Multiple
   writes are only allowed for ZFS, but multiple readers are allowed for
   any filesystem.

 - The block and ramdisk backends now support setting the LUN blocksize.
   There are some restrictions when the backing device is a block device,
   but otherwise the blocksize may be set to anything.

Revision 1.2 Changes
====================

 - CTL initialization process has been revamped.  Instead of using an
   ad-hoc method, it is now sequenced through SYSINIT() calls.

 - A block/file backend has been added.  This allows using arbitrary files
   or block devices as a backing store.

 - The userland LUN configuration interface has been completely rewritten.
   Configuration is now done out of band.

 - The ctladm(8) command line interface has been revamped, and is now
   similar to camcontrol(8).

To Do List:
==========

 - Make CTL buildable as a module.  Work needs to be done on initialization,
   and on freeing resources and LUNs when it is built as a module.

 - Use devstat(9) for CTL's statistics collection.  CTL uses a home-grown
   statistics collection system that is similar to devstat(9).  ctlstat
   should be retired in favor of iostat, etc., once aggregation modes are
   available in iostat to match the behavior of ctlstat -t and dump modes
   are available to match the behavior of ctlstat -d/ctlstat -J.

 - ZFS ARC backend for CTL.  Since ZFS copies all I/O into the ARC
   (Adaptive Replacement Cache), running the block/file backend on top of a
   ZFS-backed zdev or file will involve an extra set of copies.  The
   optimal solution for backing targets served by CTL with ZFS would be to
   allocate buffers out of the ARC directly, and DMA to/from them directly.
   That would eliminate an extra data buffer allocation and copy.

 - Switch CTL over to using CAM CCBs instead of its own union ctl_io.  This
   will likely require a significant amount of work, but will eliminate
   another data structure in the stack, more memory allocations, etc.  This
   will also require changes to the CAM CCB structure to support CTL.

 - Full-featured High Availability support.  The HA API that is in ctl_ha.h
   is essentially a renamed version of Copan's HA API.  There is no
   substance to it, but it remains in CTL to show what needs to be done to
   implement active/active HA from a CTL standpoint.  The things that would
   need to be done include:
	- A kernel level software API for message passing as well as DMA
	  between at least two nodes.
	- Hardware support and drivers for inter-node communication.  This
	  could be as simples as ethernet hardware and drivers.
	- A "supervisor", or startup framework to control and coordinate
	  HA startup, failover (going from active/active to single mode),
	  and failback (going from single mode to active/active).
	- HA support in other components of the stack.  The goal behind HA
	  is that one node can fail and another node can seamlessly take
	  over handling I/O requests.  This requires support from pretty
	  much every component in the storage stack, from top to bottom.
	  CTL is one piece of it, but you also need support in the RAID
	  stack/filesystem/backing store.  You also need full configuration
	  mirroring, and all peer nodes need to be able to talk to the
	  underlying storage hardware.

Code Roadmap:
============

CTL has the concept of pluggable frontend ports and backends.  All
frontends and backends can be active at the same time.  You can have a
ramdisk-backed LUN present along side a file backed LUN.

ctl.c:
-----

This is the core of CTL, where all of the command handlers and a lot of
other things live.  Yes, it is large.  It started off small and grew to its
current size over time.  Perhaps it can be split into more files at some
point.

Here is a roadmap of some of the primary functions in ctl.c.  Starting here
and following the various leaf functions will show the command flow.

ctl_queue() 		This is where commands from the frontend ports come
			in.

ctl_queue_sense()	This is only used for non-packetized SCSI.  i.e.
			parallel SCSI prior to U320 and perhaps U160.

ctl_work_thread() 	This is the primary work thread, and everything gets
			executed from there.

ctl_scsiio_precheck() 	This where all of the initial checks are done, and I/O
			is either queued for execution or blocked.

ctl_scsiio() 		This is where the command handler is actually
			executed.  (See ctl_cmd_table.c for the mapping of
			SCSI opcode to command handler function.)

ctl_done()		This is the routine called (or ctl_done_lock()) to
			initiate the command completion process.

ctl_process_done()	This is where command completion actually happens.

ctl.h:
-----

Basic function declarations and data structures.

ctl_backend.c,
ctl_backend.h:
-------------

These files define the basic CTL backend API.  The comments in the header
explain the API.

ctl_backend_block.c
ctl_backend_block.h:
-------------------

The block and file backend.  This allows for using a disk or a file as the
backing store for a LUN.  Multiple threads are started to do I/O to the
backing device, primarily because the VFS API requires that to get any
concurrency.

ctl_backend_ramdisk.c:
---------------------

A "fake" ramdisk backend.  It only allocates a small amount of memory to
act as a source and sink for reads and writes from an initiator.  Therefore
it cannot be used for any real data, but it can be used to test for
throughput.  It can also be used to test initiators' support for extremely
large LUNs.

ctl_cmd_table.c:
---------------

This is a table with all 256 possible SCSI opcodes, and command handler
functions defined for supported opcodes.  It is included in ctl.c.

ctl_debug.h:
-----------

Simplistic debugging support.

ctl_error.c,
ctl_error.h:
-----------

CTL-specific wrappers around the CAM sense building functions.

ctl_frontend.c,
ctl_frontend.h:
--------------

These files define the basic CTL frontend port API.  The comments in the
header explain the API.

ctl_frontend_cam_sim.c:
----------------------

This is a CTL frontend port that is also a CAM SIM.  The idea is that this
frontend allows for using CTL without any target-capable hardware.  So any
LUNs you create in CTL are visible via this port.


ctl_frontend_internal.c
ctl_frontend_internal.h:
-----------------------

This is a frontend port written for Copan to do some system-specific tasks
that required sending commands into CTL from inside the kernel.  This isn't
entirely relevant to FreeBSD in general, but can perhaps be repurposed or
removed later.

ctl_ha.h:
--------

This is a stubbed-out High Availability API.  See the comments in the
header and the description of what is needed as far as HA support above.

ctl_io.h:
--------

This defines most of the core CTL I/O structures.  union ctl_io is
conceptually very similar to CAM's union ccb.  

ctl_ioctl.h:
-----------

This defines all ioctls available through the CTL character device, and
the data structures needed for those ioctls.

ctl_mem_pool.c
ctl_mem_pool.h:
--------------

Generic memory pool implementation.  This is currently only used by the
internal frontend.  The internal frontend can probably be rewritten to use
UMA zones and this can be removed.

ctl_private.h:
-------------

Private data structres (e.g. CTL softc) and function prototypes.  This also
includes the SCSI vendor and product names used by CTL.

ctl_scsi_all.c
ctl_scsi_all.h:
--------------

CTL wrappers around CAM sense printing functions.

ctl_ser_table.c:
---------------

Command serialization table.  This defines what happens when one type of
command is followed by another type of command.  e.g., what do you do when
you have a mode select followed by a write?  You block the write until the
mode select is complete.  That is defined in this table.

ctl_util.c
ctl_util.h:
----------

CTL utility functions, primarily designed to be used from userland.  See
ctladm for the primary consumer of these functions.  These include CDB
building functions.

scsi_ctl.c:
----------

CAM target peripheral driver and CTL frontend port.  This is the path into
CTL for commands from target-capable hardware/SIMs.

Userland Commands:
=================

ctladm(8) fills a role similar to camcontrol(8).  It allow configuring LUNs,
issuing commands, injecting errors and various other control functions.

ctlstat(8) fills a role similar to iostat(8).  It reports I/O statistics
for CTL.