From e270a6246833d5f4bdcd1b03f64ec498bdbbb602 Mon Sep 17 00:00:00 2001 From: Jeroen Janssen Date: Sun, 28 May 2023 10:49:27 +0200 Subject: [PATCH] Add upstream code --- .github/FUNDING.yml | 12 + LICENSE | 24 + Makefile | 50 + README.md | 339 ++++++ lib/vm-base | 61 ++ lib/vm-cmd | 237 ++++ lib/vm-config | 199 ++++ lib/vm-core | 1080 +++++++++++++++++++ lib/vm-datastore | 500 +++++++++ lib/vm-guest | 196 ++++ lib/vm-info | 503 +++++++++ lib/vm-migration | 278 +++++ lib/vm-rctl | 88 ++ lib/vm-run | 1028 ++++++++++++++++++ lib/vm-switch | 467 ++++++++ lib/vm-switch-manual | 142 +++ lib/vm-switch-netgraph | 119 +++ lib/vm-switch-standard | 405 +++++++ lib/vm-switch-vale | 128 +++ lib/vm-switch-vxlan | 204 ++++ lib/vm-util | 405 +++++++ lib/vm-zfs | 519 +++++++++ rc.d/vm | 30 + sample-templates/alpine.conf | 11 + sample-templates/arch.conf | 9 + sample-templates/centos6.conf | 11 + sample-templates/centos7.conf | 9 + sample-templates/config.sample | 503 +++++++++ sample-templates/coreos.conf | 14 + sample-templates/debian.conf | 9 + sample-templates/default.conf | 7 + sample-templates/dragonfly.conf | 14 + sample-templates/freebsd-zvol.conf | 8 + sample-templates/freepbx.conf | 12 + sample-templates/gentoo.conf | 13 + sample-templates/linux-zvol.conf | 8 + sample-templates/netbsd.conf | 9 + sample-templates/openbsd.conf | 10 + sample-templates/resflash.conf | 10 + sample-templates/ubuntu.conf | 7 + sample-templates/windows.conf | 21 + vm | 49 + vm.8 | 1605 ++++++++++++++++++++++++++++ 43 files changed, 9353 insertions(+) create mode 100644 .github/FUNDING.yml create mode 100644 LICENSE create mode 100644 Makefile create mode 100644 README.md create mode 100644 lib/vm-base create mode 100644 lib/vm-cmd create mode 100644 lib/vm-config create mode 100644 lib/vm-core create mode 100644 lib/vm-datastore create mode 100644 lib/vm-guest create mode 100644 lib/vm-info create mode 100644 lib/vm-migration create mode 100644 lib/vm-rctl create mode 100644 lib/vm-run create mode 100644 lib/vm-switch create mode 100644 lib/vm-switch-manual create mode 100644 lib/vm-switch-netgraph create mode 100644 lib/vm-switch-standard create mode 100644 lib/vm-switch-vale create mode 100644 lib/vm-switch-vxlan create mode 100644 lib/vm-util create mode 100644 lib/vm-zfs create mode 100644 rc.d/vm create mode 100644 sample-templates/alpine.conf create mode 100644 sample-templates/arch.conf create mode 100644 sample-templates/centos6.conf create mode 100644 sample-templates/centos7.conf create mode 100644 sample-templates/config.sample create mode 100644 sample-templates/coreos.conf create mode 100644 sample-templates/debian.conf create mode 100644 sample-templates/default.conf create mode 100644 sample-templates/dragonfly.conf create mode 100644 sample-templates/freebsd-zvol.conf create mode 100644 sample-templates/freepbx.conf create mode 100644 sample-templates/gentoo.conf create mode 100644 sample-templates/linux-zvol.conf create mode 100644 sample-templates/netbsd.conf create mode 100644 sample-templates/openbsd.conf create mode 100644 sample-templates/resflash.conf create mode 100644 sample-templates/ubuntu.conf create mode 100644 sample-templates/windows.conf create mode 100644 vm create mode 100644 vm.8 diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..c5f338e --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,12 @@ +# These are supported funding model platforms + +github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] +patreon: # Replace with a single Patreon username +open_collective: freebsd-vm-bhyve +ko_fi: # Replace with a single Ko-fi username +tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel +community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry +liberapay: # Replace with a single Liberapay username +issuehunt: # Replace with a single IssueHunt username +otechie: # Replace with a single Otechie username +custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..1c2529b --- /dev/null +++ b/LICENSE @@ -0,0 +1,24 @@ +Copyright (c) 2015-2016, churchers +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..eb13a2e --- /dev/null +++ b/Makefile @@ -0,0 +1,50 @@ +# +# vm-bhyve Makefile +# + +PREFIX?=/usr/local +BINDIR=$(DESTDIR)$(PREFIX)/sbin +EXAMPLESDIR=$(DESTDIR)${PREFIX}/share/examples/vm-bhyve +LIBDIR=$(DESTDIR)$(PREFIX)/lib/vm-bhyve +MANDIR=$(DESTDIR)$(PREFIX)/man/man8 +RCDIR=$(DESTDIR)$(PREFIX)/etc/rc.d + +CP=/bin/cp +INSTALL=/usr/bin/install +LN=/bin/ln +MKDIR=/bin/mkdir + +PROG=vm +MAN=$(PROG).8 + +install: + $(MKDIR) -p $(BINDIR) + $(INSTALL) -m 544 $(PROG) $(BINDIR)/ + + $(MKDIR) -p $(LIBDIR) + $(INSTALL) lib/* $(LIBDIR)/ + + $(MKDIR) -p $(EXAMPLESDIR) + $(INSTALL) sample-templates/* $(EXAMPLESDIR)/ + + $(MKDIR) -p $(RCDIR) + $(INSTALL) -m 555 rc.d/* $(RCDIR)/ + + $(MKDIR) -p $(MANDIR) + gzip -fk $(MAN) + $(INSTALL) $(MAN).gz $(MANDIR)/ + rm -f -- $(MAN).gz + $(LN) -sf $(MANDIR)/$(MAN).gz $(MANDIR)/vm-bhyve.8.gz + +vmdir: + @if [ -z "${PATH}" ]; then \ + echo "Usage: make vmdir PATH=/path"; \ + else \ + ${MKDIR} -p "${PATH}/.templates"; \ + ${MKDIR} -p "${PATH}/.iso"; \ + ${MKDIR} -p "${PATH}/.config"; \ + ${CP} sample-templates/* "${PATH}/.templates/"; \ + fi; + +.MAIN: clean +clean: ; diff --git a/README.md b/README.md new file mode 100644 index 0000000..af8e47d --- /dev/null +++ b/README.md @@ -0,0 +1,339 @@ +## vm-bhyve + +Management system for FreeBSD bhyve virtual machines + +Some of the main features include: + +* Windows/UEFI support +* Simple commands to create/start/stop bhyve instances +* Simple configuration file format +* Virtual switches supporting vlans & automatic device creation +* ZFS support +* FreeBSD/MidnightBSD/NetBSD/OpenBSD/Linux guest support +* Automatic assignment of console devices to access guest console +* Integration with rc.d startup/shutdown +* Guest reboot handling +* Designed with multiple compute nodes + shared storage in mind (NFS/iSCSI/etc) +* Multiple datastores +* VNC graphics & tmux support (1.1+ only. See wiki for instructions) +* Dependency free** + +** Some additional packages may be required in certain circumstances - + +* The port has a dependancy on ca_root_nss added by the ports maintainers to help avoid any SSL errors when downloading FreeBSD ISO files using the `vm iso` command. +* `sysutils/grub2-bhyve` is required to run Linux or any other guests that need a Grub bootloader. +* `sysutils/bhyve-firmware` is required to run UEFI guests +* `sysutils/tmux` is needed to use tmux console access instead of cu/nmdm + + +##### See the GitHub wiki for more information and examples. + +For most users, I recommend using the version in ports (1.1+). +Main development happens in the master branch on GitHub and it may contain broken or incomplete features. + +## Quick-Start + +A simple overview of the commands needed to install vm-bhyve and start a freebsd guest. +See the sections below for more in-depth details. + + 1. pkg install vm-bhyve + 2. zfs create pool/vm + 3. sysrc vm_enable="YES" + 4. sysrc vm_dir="zfs:pool/vm" + 5. vm init + 6. cp /usr/local/share/examples/vm-bhyve/* /mountpoint/for/pool/vm/.templates/ + 7. vm switch create public + 8. vm switch add public em0 + 9. vm iso https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-amd64-bootonly.iso + 10. vm create myguest + 11. vm install [-f] myguest FreeBSD-11.2-RELEASE-amd64-bootonly.iso + 12. vm console myguest + +- [ ] Line 1 +Install vm-bhvye + +- [ ] Line 2 +Create a dataset for your virtual machines. +If you're not using ZFS, just create a normal directory. + +- [ ] Lines 3-4 +Enable vm-bhyve in /etc/rc.conf and set the dataset to use. +If not using ZFS, just set `$vm_dir="/my/vm/folder"`. + +- [ ] Line 5 +Run the `vm init` command to create the required directories under $vm_dir and load kernel modules. + +- [ ] Line 6 +Install the sample templates that come with vm-bhyve. + +- [ ] Lines 7-8 +Create a virtual switch called 'public' and attach your network interface to it. +Replace `em0` with whatever interface connects your machine to the network. + +- [ ] Line 9 +Download a copy of FreeBSD from the ftp site. + +- [ ] Lines 10-12 +Create a new guest using the `default.conf` template, run the installer and +then connect to its console. At this point proceed through the installation +as normal. By specifying the `-f` option before the install command, the guest +will run directly on your terminal so the `console` command is not required. (Bear +in mind that you won't get back to your terminal until the guest is fully shutdown) + +## Install + +Download the latest release from GitHub, or install `sysutils/vm-bhyve` + +To install, just run the following command inside the vm-bhyve source directory + + # make install + +If you want to run guests other than FreeBSD, you will need the grub2-bhyve package; + + # pkg install grub2-bhyve + +## Initial configuration + +First of all, you will need a directory to store all your virtual machines and vm-bhyve configuration. +If you are not using ZFS, just create a normal directory: + + # mkdir /somefolder/vm + +If you are using ZFS, create a dataset to hold vm-bhyve data + + # zfs create pool/vm + +Now update /etc/rc.conf to enable vm-bhyve, and tell it where your directory is + + vm_enable="YES" + vm_dir="/somefolder/vm" + +Or with ZFS: + + vm_enable="YES" + vm_dir="zfs:pool/vm" + +This directory will be referred to as $vm_dir in the rest of this readme. + +Now run the following command to create the directories used to store vm-bhvye configuration and +load any necessary kernel modules. This needs to be run once after each host reboot, which is +normally handled by the rc.d script + + # vm init + +## Virtual machine templates + +When creating a virtual machine, you use a template which defines how much memory to give the guest, +how many cpu cores, and networking/disk configuration. The templates are all stored inside $vm_dir/.templates. +To install the sample templates, run the following command: + + # cp /usr/local/share/examples/vm-bhyve/* /my/vm/path/.templates/ + +If you look inside the template files with a text editor, you will see they are very simple. You +can create as many templates as you like. For example you could have web-server.conf, containing the setting +for your web servers, or freebsd-large.conf for large FreeBSD guests, and so on. This is the contents of +the default template: + + guest="freebsd" + loader="bhyveload" + cpu=1 + memory=256M + disk0_type="virtio-blk" + disk0_name="disk0.img" + network0_type="virtio-net" + network0_switch="public" + +You will notice that each template is set to create one network interface. You can easily add more network +interfaces by duplicating the two network configuration options and incrementing the number. In general you +will not want to change the type from 'virtio-net', but you will notice the first interface is set to connect +to a switch called 'public'. See the next section for details on how to configure virtual switches. + +I recommend reading the man page or `sample-templates/config.sample` for a full list of supported template +options and a description of their purpose. Almost all bhyve functionality is supported and a large variety +of network/storage configurations can be achieved. + +## Virtual Switches + +When a guest is started, each network interface is automatically connected to the virtual switch specified +in the configuration file. By default all the sample templates connect to a switch called 'public', although +you can use any name. The following section shows how to create a switch called 'public', and configure various +settings: + + # vm switch create public + +If you just want to bridge guests to your physical network, add the appropriate real interface to the switch. +Obviously you will need to replace em0 here with the correct interface name on your system: + + # vm switch add public em0 + +If you want guest traffic to be on a specific VLAN when leaving the host, specify a vlan number. To turn +off vlans, just set the vlan number to 0: + + # vm switch vlan public 10 + # vm switch vlan public 0 + +You can view current switch configuration using the list command: + + # vm switch list + +## Creating virtual machines + +Use one of the following command to create a new virtual machine: + + # vm create testvm + # vm create -t templatename -s 50G testvm + +The first example uses the default.conf template, and will create a 20GB disk image. The second +example specifies the templatename.conf template, and tells vm-bhyve to create a 50GB disk. + +You will need an ISO to install the guest with, so download one using the iso command: + + # vm iso https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-amd64-disc1.iso + +To start a guest install, run the following command. vm-bhyve will run the machine in the background, +so use the console command to connect to it and finish installation. + + # vm install testvm FreeBSD-11.2-RELEASE-amd64-disc1.iso + # vm console testvm + +You can also specify the foreground option to run the guest directly on your terminal: + + # vm install -f testvm FreeBSD-11.2-RELEASE-amd64-disc1.iso + +Once installation has finished, you can reboot the guest from inside the console and it will boot up into +the new OS (assuming installation was successful). Further reboots will work as expected and +the guest can be shutdown in the normal way. As the console uses the cu command, type ~+Ctrl-D to exit +back to your host. + +The following commands start and stop virtual machines: + + # vm start testvm + # vm stop testvm + +The basic configuration of each machine and state can be viewed using the list command: + + # vm list + NAME GUEST LOADER CPU MEMORY AUTOSTART STATE + alpine linux default 1 512M No Stopped + c7 linux default 1 512M Yes [2] Stopped + centos linux default 1 512M No Stopped + debian linux default 1 512M No Stopped + fbsd freebsd default 1 256M No Stopped + netbsd generic grub 1 256M No Stopped + openbsd generic grub 1 256M No Stopped + pf freebsd default 1 256M Yes [1] Stopped + ubuntu linux default 1 512M No Stopped + wintest windows default 2 2G No Running (2796) + +All running machines can be stopped using the stopall command + + # vm stopall + +On host boot, vm-bhyve will use the 'vm startall' command to start all machines. You can +control which guests start automatically using the following variables in /etc/rc.conf: + + vm_list="vm1 vm2" + vm_delay="5" + +The first defines the list of machines to start on boot, and the order to start them. The second +is the number of seconds to wait between starting each one. 5 seconds is the recommended setting, +although a longer delay is useful if you have disk intensive guests and don't want them all booting +at the same time. + +There's also a command which opens a guest's configuration file in your default text editor, allowing +you to easily make changes to the configuration. Please note that changes only take effect after +a full shutdown and restart of the guest + + # vm configure testvm + +See the man page for a full description of all available commands. + + # man vm + +## Using cloud images + +You can use cloud images to create virtual machines. The `vm img` command will download the image to datastore and +uncompress it if needed (.xz, .tar.gz, and .gz files are supported). The image should be in RAW or QCOW2 format. +To use this feature you'll need install qemu-tools package: + + # pkg install qemu-tools + +To launch FreeBSD using official cloud image: + + # vm img https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/FreeBSD-11.2-RELEASE-amd64.raw.xz + # vm create -t freebsd-zvol -i FreeBSD-11.2-RELEASE-amd64.raw freebsd-cloud + # vm start freebsd-cloud + +To list downloaded images: + + # vm img + DATASTORE FILENAME + default CentOS-7-x86_64-GenericCloud-20180930_02.raw + default debian-9-openstack-amd64.qcow2 + default Fedora-AtomicHost-28-1.1.x86_64.raw + default FreeBSD-11.2-RELEASE-amd64.raw + default xenial-server-cloudimg-amd64-uefi1.img + +## Using cloud init + +vm-bhyve has basic support for providing cloud-init configuration to the guest. You can enable it with `-C` option +to `vm create` command. You can also pass public SSH key to be injected into the guest with option `-k `. + +Example: + + # vm create -t linux -i xenial-server-cloudimg-amd64-uefi1.img -C -k ~/.ssh/id_rsa.pub cloud-init-ubuntu + # vm start cloud-init-ubuntu + Starting cloud-init-ubuntu + * found guest in /zroot/vm/cloud-init-ubuntu + * booting... + # ssh ubuntu@192.168.0.91 + The authenticity of host '192.168.0.91 (192.168.0.91)' can't be established. + ECDSA key fingerprint is SHA256:6s9uReyhsIXRv0dVRcBCKMHtY0kDYRV7zbM7ot6u604. + No matching host key fingerprint found in DNS. + Are you sure you want to continue connecting (yes/no)? yes + Warning: Permanently added '192.168.0.91' (ECDSA) to the list of known hosts. + Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-141-generic x86_64) + +## Adding custom disks + +Scenario: If you have a vm on one zpool and would like to add a new virtual disk to it that resides on a different zpool. + +Manually create a sparse-zvol (in this case 50G in size). + + # zfs create -sV 50G -o volmode=dev "zpool2/vm/yourvm/disk1" + +Add it to your vm config file. +Please note, for Windows guests the type will need to be `ahci-hd`, as it does not have virtio-blk drivers. + + # vm configure yourvm + + disk1_name="/dev/zvol/zpool2/vm/yourvm/disk1" + disk1_type="virtio-blk" + disk1_dev="custom" + +Restart your vm. + +## Windows Support + +Please see the Windows section in the [Wiki](https://github.com/churchers/vm-bhyve/wiki/Running-Windows) + +## Autocomplete + +If you are using the default csh/tcsh shell built into FreeBSD, running the following command should allow +autocomplete to work for all the currently supported functions. This is especially useful for viewing +and completing guest & ISO file names. Please note that there's three occurrences of '/path/to/vm' which +need to be changed to the directory containing your virtual machines. + +To make the autocomplete features available permanently, add the following to your `$HOME/.cshrc` file. Then either +logout/login, or run `source ~/.cshrc` to cause the `.cshrc` file to be reloaded. + + complete vm \ + 'p@1@(list create install start stop console configure reset poweroff destroy clone snapshot rollback add switch iso)@' \ + 'n@create@n@' \ + 'n@list@n@' \ + 'n@iso@n@' \ + 'n@switch@(list create add remove destroy vlan nat)@' \ + 'N@switch@`sysrc -inqf /path/to/vm/.config/switch switch_list`@' \ + 'N@install@`ls -1 /path/to/vm/.iso`@' \ + 'N@nat@(off on)@' \ + 'p@2@`ls -1 /path/to/vm | grep -v "^\." | grep -v "^images"`@' diff --git a/lib/vm-base b/lib/vm-base new file mode 100644 index 0000000..5333b60 --- /dev/null +++ b/lib/vm-base @@ -0,0 +1,61 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2018 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +VERSION=1.6-devel +VERSION_INT=106001 +VERSION_BSD=$(uname -K) +PATH=${PATH}:/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin + +. /etc/rc.subr +load_rc_config "vm" + +# check informational commands +cmd::parse_info "$@" + +# we should be enabled in rc.conf +# or call it using forcestart +[ -z "$rc_force" ] && ! checkyesno vm_enable && util::err "\$vm_enable is not enabled in /etc/rc.conf!" + +# check we can run bhyve +util::check_bhyve_support + +# init for zfs +zfs::init + +# create directories as needed +[ ! -d "${vm_dir}" ] && util::err "\$vm_dir has not been configured or is not a valid directory" +[ ! -d "${vm_dir}/.config" ] && mkdir "${vm_dir}/.config" +[ ! -e "${vm_dir}/.config/null.iso" ] && touch "${vm_dir}/.config/null.iso" +[ ! -d "${vm_dir}/.templates" ] && mkdir "${vm_dir}/.templates" +[ ! -d "${vm_dir}/.iso" ] && mkdir "${vm_dir}/.iso" +[ ! -d "${vm_dir}/.img" ] && mkdir "${vm_dir}/.img" + +# load core configuration +config::core::load +datastore::load + +# run the requested command +cmd::parse "$@" diff --git a/lib/vm-cmd b/lib/vm-cmd new file mode 100644 index 0000000..a9bffc8 --- /dev/null +++ b/lib/vm-cmd @@ -0,0 +1,237 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +CMD_VALID_LIST="init,switch,datastore,image,get,set,list,create,destroy,rename,install,start,stop,restart" +CMD_VALID_LIST="${CMD_VALID_LIST},add,reset,poweroff,startall,stopall,console,iso,img,configure,passthru,_run" +CMD_VALID_LIST="${CMD_VALID_LIST},info,clone,snapshot,rollback,migrate,version,usage" + +# cmd: vm ... +# +# handle simple information commands that don't need any +# priviledged access or bhyve support +# +# @param string _cmd the command right after 'vm ' +# +cmd::parse_info(){ + local _cmd + + cmd::find "_cmd" "$1" "${CMD_VALID_LIST}" + + case "${_cmd}" in + version) util::version && exit ;; + usage) util::usage ;; + esac +} + +# cmd: vm ... +# +# process the vm command line to see which function is requested +# +# @param string _cmd the command right after 'vm ' +# +cmd::parse(){ + local _cmd + + # try to find a matching command + cmd::find "_cmd" "$1" "${CMD_VALID_LIST}" || util::usage + shift + + case "${_cmd}" in + init) util::setup + switch::init ;; + switch) cmd::parse_switch "$@" ;; + datastore) cmd::parse_datastore "$@" ;; + image) cmd::parse_image "$@" ;; + get) core::get "$@" ;; + set) core::set "$@" ;; + list) core::list "$@" ;; + create) core::create "$@" ;; + destroy) core::destroy "$@" ;; + rename) core::rename "$@" ;; + install) core::install "$@" ;; + start) core::start "$@" ;; + stop) core::stop "$@" ;; + restart) core::restart "$@" ;; + add) core::add "$@" ;; + reset) core::reset "$@" ;; + poweroff) core::poweroff "$@" ;; + startall) core::startall ;; + stopall) core::stopall ;; + console) core::console "$@" ;; + iso) core::iso "$@" ;; + img) core::img "$@" ;; + configure) core::configure "$@" ;; + passthru) core::passthru ;; + _run) vm::run "$@" ;; + info) info::guest "$@" ;; + clone) zfs::clone "$@" ;; + snapshot) zfs::snapshot "$@" ;; + rollback) zfs::rollback "$@" ;; + migrate) migration::run "$@" ;; + *) util::err "unknown command '${_user_cmd}'. please run 'vm usage' or view the manpage for help" ;; + esac +} + +# cmd: vm switch ... +# +# parse switch command +# we've already shifted once, so $1 is the switch function +# +# @param string _cmd the command right after 'vm switch ' +# +cmd::parse_switch(){ + local _cmd + + # try to find a matching command + cmd::find "_cmd" "$1" "create,list,destroy,add,remove,vlan,nat,address,private,info" || util::usage + shift + + case "${_cmd}" in + create) switch::create "$@" ;; + list) switch::list ;; + destroy) switch::remove "$@" ;; + add) switch::add_member "$@" ;; + remove) switch::remove_member "$@" ;; + vlan) switch::vlan "$@" ;; + nat) switch::nat "$@" ;; + address) switch::address "$@" ;; + private) switch::private "$@" ;; + info) info::switch "$@" ;; + *) util::err "unknown command '${_user_cmd}'. please run 'vm usage' or view the manpage for help" ;; + esac +} + +# cmd: vm datastore ... +# +# parse a datastore command +# +# @param string _cmd the command after 'vm datastore ...' +# +cmd::parse_datastore(){ + local _cmd + + # try to find a matching command + cmd::find "_cmd" "$1" "list,add,remove,iso,img" || util::usage + shift + + case "${_cmd}" in + list) datastore::list ;; + add) datastore::add "$@" ;; + remove) datastore::remove "$@" ;; + iso) datastore::iso "$@" ;; + img) datastore::img "$@" ;; + *) util::err "unknown command '${_user_cmd}'. please run 'vm usage' or view the manpage for help" ;; + esac +} + +# cmd 'vm image ...' +# parse the image command set +# +# @param string _cmd the command after 'vm image ' +# +cmd::parse_image(){ + local _cmd + + [ -z "${VM_ZFS}" ] && util::err "\$vm_dir must be a ZFS datastore to use these functions" + + # try to find a matching command + cmd::find "_cmd" "$1" "list,create,provision,destroy" || util::usage + shift + + case "${_cmd}" in + list) zfs::image_list ;; + create) zfs::image_create "$@" ;; + provision) zfs::image_provision "$@" ;; + destroy) zfs::image_destroy "$@" ;; + *) util::err "unknown command '${_user_cmd}'. please run 'vm usage' or view the manpage for help" ;; + esac +} + +# many commands accept the same arguments (force being the obvious one) +# provide a function to parse these so we don't have to keep +# repeating the same getopt code. the return value here is the number +# of arguments the caller needs to shift. +# +# note that start/install/_run use -f for foreground mode +# +# @param _arglist[multiple] the callers $@ +# @return number of arguments caller should shift over +# +cmd::parse_args(){ + local _opt _count + + while getopts fitv _opt; do + case ${_opt} in + f) VM_OPT_FORCE="1" + VM_OPT_FOREGROUND="1" ;; + i) VM_OPT_INTERACTIVE="1" ;; + t) VM_OPT_TMUX="1" ;; + v) VM_OPT_VERBOSE="1" ;; + esac + done + + [ -n "${VM_OPT_FOREGROUND}" ] && [ -n "${VM_OPT_INTERACTIVE}" ] && \ + util::err "foreground and interactive mode are mutually exclusive" + + return $((OPTIND - 1)) +} + +# try to match part of a command name against a list of valid commands +# if we find more than one match we return an error +# if we only get one match, return the full command name +# +# @param string _var variable to put full command name into +# @param string _user_cmd the value provided by the user +# @param string _valid comma-separated list of valid choices +# @return success if we find one match +# +cmd::find(){ + local _var="$1" + local _user_cmd="$2" + local _valid="$3" + local _opt _choice _found="" + local IFS="," + + [ -n "${_user_cmd}" ] || util::err "no command specified" + + for _opt in ${_valid}; do + # exact match? + if [ "${_user_cmd}" = "${_opt}" ]; then + setvar "${_var}" "${_opt}" + return 0 + fi + + if echo "${_opt}" | grep -iqs "^${_user_cmd}"; then + [ -n "${_found}" ] && util::err "ambiguous command '${_user_cmd}'" + + _found=1 + _choice="${_opt}" + fi + done + + [ -z "${_found}" ] && return 1 + setvar "${_var}" "${_choice}" +} diff --git a/lib/vm-config b/lib/vm-config new file mode 100644 index 0000000..5cbc912 --- /dev/null +++ b/lib/vm-config @@ -0,0 +1,199 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# load a configuration file +# this reads the specfied file into the global VM_CONFIG variable. +# we have very basic parsing that uses # for comments and requires +# all variables to be at the beginning of the line in lowercase. +# Note also that a # within double quotes will still be treated +# as the start of a comment. +# +# @param string _file full path of the file to read +# @modifies VM_CONFIG +# +config::load(){ + local _file="$1" + + # read config file + # we kick out any lines that don't start with a letter, + # scrap anything after a # character, and remove double-quotes + VM_CONFIG=$(grep '^[a-z]' "${_file}" 2>/dev/null | awk -F# '{print $1}' | sed -e 's@ *$@@' | tr -d '"') +} + +# get a configuration value from the current config file +# +# @param string _var the variable to put value into +# @param string _key the name of the config key to retrieve +# @param optional string _def default value to return if setting not found +# @return true if setting found +# +config::get(){ + local _c_var="$1" + local _c_key="$2" + local _c_def="$3" + local _c_line + local IFS=$'\n' + + for _c_line in ${VM_CONFIG}; do + if [ "${_c_key}" = "${_c_line%%=*}" ]; then + setvar "${_c_var}" "${_c_line#*=}" + return 0 + fi + done + + # not found + setvar "${_c_var}" "${_c_def}" + return 1 +} + +# simple wrapper to check a config setting to see if it's +# set to yes/no true/false etc +# +# @param string _key the config key to check +# @param string _def default value if config key doesn't exist +# @return true(0) if set to anything other than no/false/off/0 +# +config::yesno(){ + local _key="$1" + local _def="$2" + local _value + + config::get "_value" "${_key}" "${_def}" + util::yesno "${_value}" +} + +# set a value in guest configuration file +# we check for newline at the end as sysrc won't add it +# and that will mess up the new key and the existing one +# from the end of the file +# +# @param string _name guest name +# @param string _key config key to set +# @param string _value value +# @param int _skip_newline_check skip the check for newline +# @return true if sysrc successful +# +config::set(){ + local _name="$1" + local _key="$2" + local _value="$3" + local _skip_newline_check="$4" + local _newline + + if [ -z "${_skip_newline_check}" ]; then + _newline=$(tail -1 "${VM_DS_PATH}/${_name}/${_name}.conf" | wc -l | tr -d " ") + [ "${_newline}" -eq "0" ] && echo "" >> "${VM_DS_PATH}/${_name}/${_name}.conf" + fi + + sysrc -inqf "${VM_DS_PATH}/${_name}/${_name}.conf" "${_key}=${_value}" >/dev/null 2>&1 +} + +# remove a value from guest config +config::remove(){ + local _name="$1" + local _key="$2" + + sysrc -inxqf "${VM_DS_PATH}/${_name}/${_name}.conf" "${_key}" >/dev/null 2>&1 +} + +# load core configuration file +# +# @modifies VM_CORE_CONFIG VM_CONFIG_USER +# +config::core::load(){ + VM_CONFIG_USER="console;compress;decompress;" + + # check config file exists + # this is mainly for upgrades to make sure switch/datastore config are migrated + # DEPRECATED 1.3, remove after + if [ ! -e "${vm_dir}/.config/system.conf" ]; then + cat "${vm_dir}/.config/switch" > "${vm_dir}/.config/system.conf" 2>/dev/null + cat "${vm_dir}/.config/datastore" >> "${vm_dir}/.config/system.conf" 2>/dev/null + fi + + VM_CORE_CONFIG=$(grep '^[a-z]' "${vm_dir}/.config/system.conf" 2>/dev/null | awk -F# '{print $1}' | sed -e 's@ *$@@' | tr -d '"') +} + +# get a value from core config +# +# @param string _c_var variable name to put value into +# @param string _c_key config key to look for +# @param string _c_def default value if not value +# @return 0 if found +# +config::core::get(){ + local _c_var="$1" + local _c_key="$2" + local _c_def="$3" + local _c_line + local IFS=$'\n' + + for _c_line in ${VM_CORE_CONFIG}; do + if [ "${_c_key}" = "${_c_line%%=*}" ]; then + setvar "${_c_var}" "${_c_line#*=}" + return 0 + fi + done + + # not found + setvar "${_c_var}" "${_c_def}" + return 1 +} + +# add a value to core configuration +# +# @param string _var variable to set +# @param string _value new value +# @param string _append non-empty to append to existing value +# +config::core::set(){ + local _var="$1" + local _value="$2" + local _append="$3" + + if [ -n "${_append}" ]; then + sysrc -inqf "${vm_dir}/.config/system.conf" "${_var}"+="${_value}" >/dev/null 2>&1 + else + sysrc -inqf "${vm_dir}/.config/system.conf" "${_var}"="${_value}" >/dev/null 2>&1 + fi +} + +# remove a value from core configuration +# +# @param string _var variable to remove +# @param string _value if non-empty we will try to remove just this value from setting +# @return non-zero on error +# +config::core::remove(){ + local _var="$1" + local _value="$2" + + if [ -n "${_value}" ]; then + sysrc -inqf "${vm_dir}/.config/system.conf" "${_var}"-="${_value}" >/dev/null 2>&1 + else + sysrc -inxqf "${vm_dir}/.config/system.conf" ${_var} >/dev/null 2>&1 + fi +} diff --git a/lib/vm-core b/lib/vm-core new file mode 100644 index 0000000..45fb6c2 --- /dev/null +++ b/lib/vm-core @@ -0,0 +1,1080 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# 'vm list' +# list virtual machines +# +core::list(){ + local _name _loader _cpu _our_host + local _memory _run _vm _auto _num _vnc _pid + local _state _pcpu _rss _uptime + local _format="%s^%s^%s^%s^%s^%s^%s^%s\n" + + cmd::parse_args "$@" + shift $? + + _our_host=$(hostname) + + vm::running_load + + [ -n "${VM_OPT_VERBOSE}" ] && _format="%s^%s^%s^%s^%s^%s^%s^%5s^%8s^%14s^%s\n"; + + # pass everything below here to column(1) + { + if [ -n "${VM_OPT_VERBOSE}" ]; then + printf "${_format}" "NAME" "DATASTORE" "LOADER" "CPU" "MEMORY" "VNC" "AUTO" "%CPU" "RSZ" "UPTIME" "STATE" + else + printf "${_format}" "NAME" "DATASTORE" "LOADER" "CPU" "MEMORY" "VNC" "AUTO" "STATE" + fi + + for _ds in ${VM_DATASTORE_LIST}; do + datastore::get "${_ds}" || continue + + ls -1 "${VM_DS_PATH}" 2>/dev/null | \ + while read _name; do + [ ! -e "${VM_DS_PATH}/${_name}/${_name}.conf" ] && continue + + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + config::get "_loader" "loader" "none" + config::get "_cpu" "cpu" + config::get "_memory" "memory" + + # defaults + _vnc="-" + _pid="" + _state="" + _pcpu="-" + _rss="-" + _uptime="-" + + # check if the guest is running + if vm::running_check "_run" "_pid" "${_name}" || \ + [ -e "${VM_DS_PATH}/${_name}/run.lock" -a "$(head -n1 ${VM_DS_PATH}/${_name}/run.lock 2>/dev/null)" = "${_our_host}" ]; then + + # if running and graphics, try to get vnc port + if config::yesno "graphics"; then + _vnc=$(grep vnc "${VM_DS_PATH}/${_name}/console" 2>/dev/null |cut -d= -f2) + [ -z "${_vnc}" ] && _vnc="-" + fi + fi + + if [ -n "${_pid}" -a -n "${VM_OPT_VERBOSE}" ]; then + _state=$(ps -o"%cpu"= -o"rss"= -o"etime"= -p "${_pid}") + + if [ -n "${_state}" ]; then + util::get_part "_pcpu" "${_state}" 1 + util::get_part "_rss" "${_state}" 2 + util::get_part "_uptime" "${_state}" 3 + + [ -n "${_rss}" ] && _rss=$(info::__bytes_human "${_rss}" 1 2) + _uptime=$(echo "${_uptime}" |sed 's/\-/d /') + fi + fi + + _num=1 + _auto="No" + + # find out if we auto-start this vm, and get sequence number + for _vm in ${vm_list}; do + [ "${_vm}" = "${_name}" ] && _auto="Yes [${_num}]" + _num=$((_num + 1)) + done + + # if stopped, see if it's locked by another host + if [ "${_run}" = "Stopped" -a -e "${VM_DS_PATH}/${_name}/run.lock" ]; then + _run=$(head -n1 "${VM_DS_PATH}/${_name}/run.lock") + _run="Locked (${_run})" + fi + + if [ -n "${VM_OPT_VERBOSE}" ]; then + printf "${_format}" "${_name}" "${_ds}" "${_loader}" "${_cpu}" "${_memory}" "${_vnc}" "${_auto}" "${_pcpu}" "${_rss}" "${_uptime}" "${_run}" + else + printf "${_format}" "${_name}" "${_ds}" "${_loader}" "${_cpu}" "${_memory}" "${_vnc}" "${_auto}" "${_run}" + fi + done + done + } | column -ts^ +} + +# 'vm create' +# create a new virtual machine +# +# @param optional string (-t) _template the template to use (default = default) +# @param optional string (-s) _size guest size (default = 20G) +# @param string _name the name of the guest to create +# +core::create(){ + local _name _opt _size _vmdir _disk _disk_dev _num=0 + local _zfs_opts _disk_size _template="default" _ds="default" _ds_path _img _cpu _memory _uuid + local _enable_cloud_init _cloud_init_dir _ssh_public_key _ssh_key_file _network_config _mac + + while getopts d:t:s:i:c:m:Ck:n: _opt ; do + case $_opt in + t) _template=${OPTARG} ;; + s) _size=${OPTARG} ;; + d) _ds=${OPTARG} ;; + c) _cpu=${OPTARG} ;; + m) _memory=${OPTARG} ;; + i) _img=${OPTARG} ;; + k) _ssh_key_file=${OPTARG} ;; + C) _enable_cloud_init='true' ;; + n) _network_config=${OPTARG} ;; + + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _name=$1 + + [ -z "${_name}" ] && util::usage + + # check guest name + util::check_name "${_name}" || util::err "invalid virtual machine name - '${_name}'" + datastore::get_guest "${_name}" && util::err "virtual machine already exists in ${VM_DS_PATH}/${_name}" + datastore::get "${_ds}" || util::err "unable to load datastore - '${_ds}'" + + [ ! -f "${vm_dir}/.templates/${_template}.conf" ] && \ + util::err "unable to find template ${vm_dir}/.templates/${_template}.conf" + + # we need to get disk0 name and device type from the template + config::load "${vm_dir}/.templates/${_template}.conf" + config::get "_disk" "disk0_name" + config::get "_disk_dev" "disk0_dev" + config::get "_disk_size" "disk0_size" "20G" + config::get "_zfs_opts" "zfs_dataset_opts" + + # make sure template has a disk before we start creating anything + [ -z "${_disk}" ] && util::err "template is missing disk0_name specification" + + if [ -n "${_enable_cloud_init}" ]; then + if ! which genisoimage > /dev/null; then + util::err "Error: genisoimage is required to work with cloud init! Run 'pkg install cdrkit-genisoimage'." + fi + fi + + # get ssh public key for cloud-init from file + if [ -n "${_ssh_key_file}" ]; then + + [ -z "${_enable_cloud_init}" ] && util::err "cloud-init is required for injecting public key. Use -C to enable it." + [ ! -r "${_ssh_key_file}" ] && util::err "can't read file with public key (${_ssh_key_file})" + _ssh_public_key="$(cat "${_ssh_key_file}")" + fi + + # if we're on zfs, make a new filesystem + zfs::make_dataset "${VM_DS_ZFS_DATASET}/${_name}" "${_zfs_opts}" + + [ ! -d "${VM_DS_PATH}/${_name}" ] && mkdir "${VM_DS_PATH}/${_name}" >/dev/null 2>&1 + [ ! -d "${VM_DS_PATH}/${_name}" ] && util::err "unable to create virtual machine directory ${VM_DS_PATH}/${_name}" + + cp "${vm_dir}/.templates/${_template}.conf" "${VM_DS_PATH}/${_name}/${_name}.conf" + [ $? -eq 0 ] || util::err "unable to copy template to virtual machine directory" + + # generate a uuid + _uuid=$(uuidgen) + config::set "${_name}" "uuid" ${_uuid} + + # get any zvol options + config::get "_zfs_opts" "zfs_zvol_opts" + + # generate mac address - it's saved in _mac variable + vm::generate_static_mac + + # Optional overrides + [ -n "${_cpu}" ] && config::set "${_name}" "cpu" "${_cpu}" + [ -n "${_memory}" ] && config::set "${_name}" "memory" "${_memory}" + # use cmd line size for disk 0 if specified + [ -n "${_size}" ] && _disk_size="${_size}" + + # create each disk + while [ -n "${_disk}" ]; do + case "${_disk_dev}" in + zvol) + zfs::make_zvol "${VM_DS_ZFS_DATASET}/${_name}/${_disk}" "${_disk_size}" "0" "${_zfs_opts}" + [ $_num -eq 0 ] && [ ! -z "$_img" ] && core::write_img "/dev/zvol/${VM_DS_ZFS_DATASET}/${_name}/${_disk}" "${_img}" + ;; + sparse-zvol) + zfs::make_zvol "${VM_DS_ZFS_DATASET}/${_name}/${_disk}" "${_disk_size}" "1" "${_zfs_opts}" + [ $_num -eq 0 ] && [ ! -z "$_img" ] && core::write_img "/dev/zvol/${VM_DS_ZFS_DATASET}/${_name}/${_disk}" "${_img}" + ;; + custom) + ;; + iscsi) + ;; + *) + truncate -s "${_disk_size}" "${VM_DS_PATH}/${_name}/${_disk}" + [ $? -eq 0 ] || util::err "failed to create sparse file for disk image" + + # make sure only owner can read the disk image + chmod 600 "${VM_DS_PATH}/${_name}/${_disk}" + [ $_num -eq 0 ] && [ ! -z "$_img" ] && core::write_img "${VM_DS_PATH}/${_name}/${_disk}" "${_img}" + ;; + esac + + # scrap size option from guest template + sysrc -inxqf "${VM_DS_PATH}/${_name}/${_name}.conf" "disk${_num}_size" + + # look for another disk + _num=$((_num + 1)) + config::get "_disk" "disk${_num}_name" + config::get "_disk_dev" "disk${_num}_dev" + config::get "_disk_size" "disk${_num}_size" "20G" + done + + if [ -n "${_enable_cloud_init}" ]; then + core::create_cloud_init + fi + + exit 0 +} + +core::create_cloud_init(){ + # create disk with metadata for cloud-init + _cloud_init_dir="${VM_DS_PATH}/${_name}/.cloud-init" + # Use VM's name as a hostname by default + _hostname="${_name}" + + mkdir -p "${_cloud_init_dir}" + + if [ -n "${_network_config}" ]; then + # Example netconfig param: "ip=10.0.0.2/24;gateway=10.0.0.1;nameservers=1.1.1.1,8.8.8.8" + _network_config_ipaddress="$(echo "${_network_config}" | pcregrep -o "ip=\K[^;]*")" + _network_config_gateway="$(echo "${_network_config}" | pcregrep -o "gateway=\K[^;]*")" + _network_config_nameservers="$(echo "${_network_config}" | pcregrep -o "nameservers=\K[^;]*")" + _network_config_hostname="$(echo "${_network_config}" | pcregrep -o "hostname=\K[^;]*")" + + # Override default hostname when network config is passed to cloud-init + if [ ! -z "${_network_config_hostname}" ]; then + _hostname="${_network_config_hostname}" + fi + + cat << EOF > "${_cloud_init_dir}/network-config" +version: 2 +ethernets: + id0: + set-name: eth0 + match: + macaddress: "${_mac}" + addresses: + - ${_network_config_ipaddress} + gateway4: ${_network_config_gateway} + nameservers: + search: [] + addresses: [${_network_config_nameservers}] +EOF + fi + + cat << EOF > "${_cloud_init_dir}/meta-data" +instance-id: ${_uuid} +local-hostname: ${_hostname} +EOF + + cat << EOF > "${_cloud_init_dir}/user-data" +#cloud-config +resize_rootfs: True +manage_etc_hosts: localhost +EOF + + if [ -n "${_ssh_public_key}" ]; then + cat << EOF >> "${_cloud_init_dir}/user-data" +ssh_authorized_keys: + - ${_ssh_public_key} +EOF + fi + + genisoimage -output "${VM_DS_PATH}/${_name}/seed.iso" -volid cidata -joliet -rock ${_cloud_init_dir}/* > /dev/null 2>&1 || util::err "Can't write seed.iso for cloud-init" + config::set "${_name}" "disk${_num}_type" "ahci-cd" + config::set "${_name}" "disk${_num}_name" "seed.iso" + config::set "${_name}" "disk${_num}_dev" "file" +} + +# write cloud image to disk image +# +# @private +# @param string _disk_dev device to write image to +# @param string _img the img file in $vm_dir/.img to use +# +core::write_img(){ + local _disk_dev _img _imgpath + + cmd::parse_args "$@" + shift $? + _disk_dev="${1}" + _img="$2" + timeout=30 + i=0 + + # wait a few seconds for newly created zvol device to appear + while [ $i -lt $timeout ]; do + if [ ! -r "${_disk_dev}" ]; then + sleep 1 + i=$(($i+1)) + else + break + fi + done + + # just run start with an iso + datastore::img_find "_imgpath" "${_img}" || util::err "unable to locate img file - '${_img}'" + qemu-img dd -O raw if="${_imgpath}" of="${_disk_dev}" bs=1M + if [ $? -ne 0 ]; then + util::err "failed to write img file with qemu-img" + fi +} + +# 'vm add' +# add a device to an existing guest +# +# @param string (-d) _device=network|disk the type of device to add +# @param string (-t) _type for disk, the type of disk - file|zvol|sparse-zvol +# @param string (-s) _sopt for disk the size, for network the virtual switch name +# @param string _name name of the guest +# +core::add(){ + local _name _device _type _sopt _opt + + while getopts d:t:s: _opt; do + case $_opt in + d) _device=${OPTARG} ;; + t) _type=${OPTARG} ;; + s) _sopt=${OPTARG} ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _name="$1" + + # check guest + [ -z "${_name}" ] && util::usage + datastore::get_guest "${_name}" || "${_name} does not appear to be a valid virtual machine" + + case "${_device}" in + disk) core::add_disk "${_name}" "${_type}" "${_sopt}" ;; + network) core::add_network "${_name}" "${_sopt}" ;; + *) util::err "device must be one of the following: disk network" ;; + esac +} + +# add a disk to guest +# this creates the disk image or zvol and updates configuration file +# we use the same emulation as the existing disk(s) +# +# @private +# @param string _name name of the guest +# @param string _device type of device file|zvol|sparse-zvol +# @param string _size size of the disk to create +# +core::add_disk(){ + local _name="$1" + local _device="$2" + local _size="$3" + local _num=0 _curr _diskname _emulation _zfs_opts + + : ${_device:=file} + + [ -z "${_size}" ] && util::usage + + # get the last existing disk + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + config::get "_zfs_opts" "zfs_zvol_opts" + + while true; do + config::get "_curr" "disk${_num}_name" + [ -z "${_curr}" ] && break + config::get "_emulation" "disk${_num}_type" + _num=$((_num + 1)) + done + + [ -z "${_emulation}" ] && util::err "failed to get emulation type of the existing guest disks" + + # create the disk first, then update config if no problems + case "${_device}" in + zvol) + zfs::make_zvol "${VM_DS_ZFS_DATASET}/${_name}/disk${_num}" "${_size}" "0" "${_zfs_opts}" + _diskname="disk${_num}" + ;; + sparse-zvol) + zfs::make_zvol "${VM_DS_ZFS_DATASET}/${_name}/disk${_num}" "${_size}" "1" "${_zfs_opts}" + _diskname="disk${_num}" + ;; + file) + truncate -s "${_size}" "${VM_DS_PATH}/${_name}/disk${_num}.img" + [ $? -eq 0 ] || util::err "failed to create sparse file for disk image" + _diskname="disk${_num}.img" + ;; + *) + util::err "device type must be one of the following: zvol sparse-zvol file" + ;; + esac + + # update configuration + config::set "${_name}" "disk${_num}_name" "${_diskname}" + config::set "${_name}" "disk${_num}_type" "${_emulation}" "1" + config::set "${_name}" "disk${_num}_dev" "${_device}" "1" + [ $? -eq 0 ] || util::err "disk image created but errors while updating guest configuration" +} + +# add network interface to guest +# +# @private +# @param string _name name of the guest +# @param string _switch the switch name for this interface +# +core::add_network(){ + local _name="$1" + local _switch="$2" + local _num=0 _curr _emulation + + [ -z "${_switch}" ] && util::usage + + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + + while true; do + _emulation="${_curr}" + config::get "_curr" "network${_num}_type" + [ -z "${_curr}" ] && break + _num=$((_num + 1)) + done + + # handle no existing network + : ${_emulation:=virtio-net} + + # update configuration + config::set "${_name}" "network${_num}_type" "${_emulation}" + config::set "${_name}" "network${_num}_switch" "${_switch}" "1" + [ $? -eq 0 ] || util::err "errors encountered while updating guest configuration" +} + +# 'vm install' +# install os to a virtual machine +# +# @param string _name the guest to install to +# @param string _iso the iso file in $vm_dir/.iso to use +# +core::install(){ + local _name _iso _fulliso + + cmd::parse_args "$@" + shift $? + _name="$1" + _iso="$2" + + [ -z "${_name}" -o -z "${_iso}" ] && util::usage + + # just run start with an iso + datastore::iso_find "_fulliso" "${_iso}" || util::err "unable to locate iso file - '${_iso}'" + core::__start "${_name}" "${_fulliso}" +} + +# 'vm startall' +# start all virtual machines listed in rc.conf:$vm_list +# +core::startall(){ + [ -z "${vm_list}" ] && exit + core::start ${vm_list} +} + +# 'vm stopall' +# stop all bhyve instances +# note this will also stop instances not started by vm-bhyve +# +core::stopall(){ + local _pids=$(pgrep -f 'bhyve:' | tr '\n' ' ') + local _stop_parallel _stop_list _running _list_rev _curr + local _stop_p_pids _pid + + cmd::parse_args "$@" + : ${vm_delay:=2} + + # get a list of running bhyve instances + _running=$(pgrep -lf 'bhyve:' | cut -d" " -f3 | tr '\n' ' ') + [ -z "${_running}" ] && return 0 + + # do we have any guests to stop in order + if [ -z "${VM_OPT_FORCE}" -a -n "${vm_list}" ]; then + # reverse the configured start order + for _curr in ${vm_list}; do + _list_rev="${_curr} ${_list_rev}" + done + + # go through each autostart guest + # if running add to stop list. + # we are searching in reverse start order, so should get an ordered shutdown + for _curr in ${_list_rev}; do + echo "${_running}" | grep -qs "${_curr} " + + if [ $? -eq 0 ]; then + _stop_list="${_stop_list}${_stop_list:+ }${_curr}" + fi + done + + # look for anything running that isn't in the ordered stop list + for _curr in ${_running}; do + echo "${_stop_list}" | grep -qs "${_curr}\b" + if [ $? -ne 0 ]; then + _stop_parallel="${_stop_parallel}${_stop_parallel:+ }${_curr}" + util::getpid "_pid" "bhyve: ${_curr}\$" + [ $? -eq 0 ] && _stop_p_pids="${_stop_p_pids}${_stop_p_pids:+ }${_pid}" + fi + done + else + # nothing ordered, or a force shutdown + # just do everything at once + _stop_parallel="${_running}" + _stop_p_pids="${_pids}" + fi + + echo "Beginning shutdown process" + + # have any guests to stop in parallel + if [ -n "${_stop_p_pids}" ]; then + echo " * parallel shutdown of non-ordered guests: ${_stop_parallel}" + kill ${_stop_p_pids} >/dev/null 2>&1 + sleep 1 + kill ${_stop_p_pids} >/dev/null 2>&1 + fi + + if [ -n "${_stop_list}" ]; then + _pid="" + for _curr in ${_stop_list}; do + [ -n "${_pid}" ] && echo " * waiting ${vm_delay} second(s)" && sleep ${vm_delay} + util::getpid "_pid" "bhyve: ${_curr}\$" + + if [ $? -eq 0 ]; then + echo " * stopping ${_curr}" + kill ${_pid} >/dev/null 2>&1 + sleep 1 + kill ${_pid} >/dev/null 2>&1 + fi + done + fi + + echo "" + wait_for_pids ${_pids} +} + +# 'vm start' +# start a virtual machine +# +# @param string[multiple] _name the name of the guest(s) to start +# +core::start(){ + local _name + local _done + + cmd::parse_args "$@" + shift $? + _name="$1" + + [ -z "${_name}" ] && util::usage + : ${vm_delay:=2} + + # disable foreground/interactive if we're starting more than one + if [ $# -ge 2 ]; then + VM_OPT_FOREGROUND="" + VM_OPT_INTERACTIVE="" + fi + + while [ -n "${_name}" ]; do + [ -n "${_done}" ] && echo "Waiting ${vm_delay} second(s)" && sleep ${vm_delay} + core::__start "${_name}" + shift + _name="$1" + _done="1" + done +} + +# actually start a virtual machine +# +# @param string _name the name of the guest to start +# @param optional string _iso iso file is this is an install (can only be provided through 'vm install' command) +# +core::__start(){ + local _name="$1" + local _iso="$2" + local _cpu _memory _disk _guest _loader _console + local _tmux_cmd _tmux_name _util _uefi + + [ -z "${_name}" ] && util::usage + + echo "Starting ${_name}" + + # try to find guest + if ! datastore::get_guest "${_name}"; then + echo " ! ${_name} does not seem to be a valid virtual machine" + return 1 + fi + + echo " * found guest in ${VM_DS_PATH}/${_name}" + + # confirm we aren't running + if ! vm::confirm_stopped "${_name}" "1" >/dev/null 2>&1; then + echo " ! guest appears to be running already" + return 1 + fi + + # check basic settings before going into background mode + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + config::get "_cpu" "cpu" + config::get "_memory" "memory" + config::get "_loader" "loader" + + # check minimum configuration + if [ -z "${_cpu}" -o -z "${_memory}" ]; then + echo " ! incomplete virtual machine configuration" + return 1 + fi + + # we can only load freebsd without unrestricted guest support + if [ -n "${VM_NO_UG}" -a "${_loader}" != "bhyveload" ]; then + echo " ! no unrestricted guest support in cpu. only single vcpu FreeBSD guests supported" + return 1 + fi + + # check loader + if [ "${_loader}" = "grub" ]; then + _util=$(which grub-bhyve) + + if [ -z "${_util}" ]; then + echo " ! grub requested but sysutils/grub2-bhyve not installed?" + return 1 + fi + fi + + # check for tmux + config::core::get "_console" "console" "nmdm" + _tmux_cmd=$(which tmux) + + if [ "${_console}" = "tmux" -a -z "${_tmux_cmd}" ]; then + echo " ! tmux support enabled but misc/tmux not found" + return 1 + fi + + echo " * booting..." + + # run background process to actually start bhyve + # this will run as long as vm is running, including restarting bhyve after guest reboot + if [ -n "${VM_OPT_FOREGROUND}" ]; then + $0 _run -f "${_name}" "${_iso}" + elif [ "${_console}" = "tmux" ]; then + # can't have dots in tmux session :( (looks like it may use . to separate window.pane) + # use ~ which we don't normally allow + _tmux_name=$(echo "${_name}" | tr "." "~") + + # start session and connect if in interactive mode + if [ -n "${VM_OPT_INTERACTIVE}" ]; then + ${_tmux_cmd} new -s "${_tmux_name}" $0 _run -tf "${_name}" "${_iso}" + else + ${_tmux_cmd} new -ds "${_tmux_name}" $0 _run -tf "${_name}" "${_iso}" + fi + else + $0 _run "${_name}" "${_iso}" >/dev/null 2>&1 & + fi +} + +# 'vm restart' +# restart a guest +# all we do is create a "restart" file which vm-run looks for +# +# @param string _name name of the guest to restart +# +core::restart(){ + local _name="$1" + + datastore::get_guest "${_name}" || util::err "unable to locate specified guest" + echo "Setting guest restart flag" + touch "${VM_DS_PATH}/${_name}/restart" >/dev/null 2>&1 + core::stop "${_name}" +} + +# 'vm stop' +# send a kill signal to the specified guest +# +# @param string[multiple] _name name of the guest to stop +# +core::stop(){ + local _name="$1" + local _pid _loadpid + + [ -z "${_name}" ] && util::usage + + while [ -n "${_name}" ]; do + if [ ! -e "/dev/vmm/${_name}" ]; then + util::warn "${_name} doesn't appear to be a running virtual machine" + else + _pid=$(pgrep -fx "bhyve: ${_name}") + _loadpid=$(pgrep -fl "grub-bhyve|bhyveload" | grep " ${_name}\$" |cut -d' ' -f1) + + if [ -n "${_pid}" ]; then + echo "Sending ACPI shutdown to ${_name}" + kill "${_pid}" >/dev/null 2>&1 + sleep 1 + kill "${_pid}" >/dev/null 2>&1 + elif [ -n "${_loadpid}" ]; then + if util::confirm "Guest ${_name} is in bootloader stage, do you wish to force exit"; then + echo "Killing ${_name}" + kill "${_loadpid}" + bhyvectl --destroy --vm=${_name} >/dev/null 2>&1 + fi + else + util::warn "unable to locate process id for ${_name}" + fi + fi + shift + _name="$1" + done +} + +# 'vm reset' +# force reset +# +# @param string _name name of the guest +# +core::reset(){ + local _name + + cmd::parse_args "$@" + shift $? + _name="$1" + + [ -z "${_name}" ] && util::usage + [ ! -e "/dev/vmm/${_name}" ] && util::err "${_name} doesn't appear to be a running virtual machine" + + if [ -z "${VM_OPT_FORCE}" ]; then + util::confirm "Are you sure you want to forcefully reset this virtual machine" || exit 0 + fi + + bhyvectl --force-reset --vm=${_name} +} + +# 'vm poweroff' +# force poweroff +# +# @param string _name name of the guest +# +core::poweroff(){ + local _name + + cmd::parse_args "$@" + shift $? + _name="$1" + + [ -z "${_name}" ] && util::usage + [ ! -e "/dev/vmm/${_name}" ] && util::err "${_name} doesn't appear to be a running virtual machine" + + if [ -z "${VM_OPT_FORCE}" ]; then + util::confirm "Are you sure you want to forcefully poweroff this virtual machine" || exit 0 + fi + + bhyvectl --force-poweroff --vm=${_name} +} + +# 'vm destroy' +# completely remove a guest +# +# @param string _name name of the guest +# +core::destroy(){ + local _name + + cmd::parse_args "$@" + shift $? + _name="$1" + + [ -z "${_name}" ] && util::usage + + # trying to remove a snapshot? + echo "${_name}" | grep -qs "@" + + if [ $? -eq 0 ]; then + zfs::remove_snapshot "${_name}" + exit $? + fi + + # make sure it's stopped! + datastore::get_guest "${_name}" || util::err "${_name} doesn't appear to be a valid virtual machine" + vm::confirm_stopped "${_name}" || exit 1 + + if [ -z "${VM_OPT_FORCE}" ]; then + util::confirm "Are you sure you want to completely remove this virtual machine" || exit 0 + fi + + [ -n "${VM_DS_ZFS_DATASET}" ] && zfs::destroy_dataset "${VM_DS_ZFS_DATASET:?}/${_name:?}" + [ -e "${VM_DS_PATH}/${_name}" ] && rm -R "${VM_DS_PATH:?}/${_name:?}" + + exit 0 +} + +# 'vm rename' +# rename an existing guest +# +# @param string _old the existing guest name +# @param string _new the new guest name +# +core::rename(){ + local _old="$1" + local _new="$2" + + [ -z "${_old}" -o -z "${_new}" ] && util::usage + util::check_name "${_new}" || util::err "invalid virtual machine name - '${_name}'" + + datastore::get_guest "${_new}" && util::err "directory ${VM_DS_PATH}/${_new} already exists" + datastore::get_guest "${_old}" || util::err "${_old} doesn't appear to be a valid virtual machine" + + # confirm guest stopped + vm::confirm_stopped "${_old}" || exit 1 + + # rename zfs dataset + zfs::rename_dataset "${_old}" "${_new}" + + # rename folder if it still exists (shouldn't if zfs mode and rename worked) + if [ -d "${VM_DS_PATH}/${_old}" ]; then + mv "${VM_DS_PATH}/${_old}" "${VM_DS_PATH}/${_new}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "failed to rename guest directory" + fi + + # rename config file + mv "${VM_DS_PATH}/${_new}/${_old}.conf" "${VM_DS_PATH}/${_new}/${_new}.conf" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "changed guest directory but failed to rename configuration file" +} + +# 'vm console' +# connect to the console (nmdm) of the specified guest +# we store the nmdm for com1 & com2 in $vm_dir/{guest}/console +# if no port is specified, we use the first one that is specified in the configuration file +# so if comports="com2 com1", it will connect to com2 +# the boot loader always using the nmdm device of the first com port listed +# +# @param string _name name of the guest +# @param string _port the port to connect to (default = first in configuration) +# +core::console(){ + local _name="$1" + local _port="$2" + local _console _tmux _tmux_cmd + + [ -z "${_name}" ] && util::usage + + datastore::get_guest "${_name}" || util::err "${_name} doesn't appear to be a valid virtual machine" + [ ! -e "/dev/vmm/${_name}" ] && util::err "${_name} doesn't appear to be a running virtual machine" + + if [ -e "${VM_DS_PATH}/${_name}/console" ]; then + + # did user specify a com port? + # if not, get first in the file (the first will also be the console used for loader) + if [ -n "${_port}" ]; then + _console=$(grep "${_port}=" "${VM_DS_PATH}/${_name}/console" | cut -d= -f2) + else + _console=$(head -n 1 "${VM_DS_PATH}/${_name}/console" | grep "^com" | cut -d= -f2) + fi + fi + + # is this a tmux console? + if [ "${_console%%/*}" = "tmux" ]; then + _tmux_cmd=$(which tmux) + + if [ -n "${_tmux_cmd}" ]; then + _tmux=$("${_tmux_cmd}" ls |grep "^${_name}:") + + if [ -n "${_tmux}" ]; then + ${_tmux_cmd} attach -t ${_console##*/} + exit + fi + fi + fi + + [ -z "${_console}" ] && util::err "unable to locate console device for this virtual machine" + cu -l "${_console}" +} + +# 'vm configure' +# configure a machine (edit the configuration file) +# +# @param string _name name of the guest +# +core::configure(){ + local _name="$1" + + [ -z "${_name}" ] && util::usage + [ -z "${EDITOR}" ] && EDITOR="vi" + + datastore::get_guest "${_name}" || \ + util::err "cannot locate configuration file for virtual machine: ${_name}" + + $EDITOR "${VM_DS_PATH}/${_name}/${_name}.conf" +} + +# 'vm iso' +# list iso images or get a new one +# +# @param string _url if specified, the url will be fetch'ed into $vm_dir/.iso +# +core::iso(){ + local _url _ds="default" + + while getopts d:u _opt ; do + case $_opt in + d) _ds=${OPTARG} ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _url=$1 + + if [ -n "${_url}" ]; then + datastore::get_iso "${_ds}" || util::err "unable to locate path for datastore '${_ds}'" + fetch -o "${VM_DS_PATH}" "${_url}" + else + datastore::iso_list + fi +} + +# uncompress cloud image +# +# @private +# @param string _filepath path to file to decompress +# +core::decompress(){ + local _filepath + + cmd::parse_args "$@" + shift $? + _filepath="${1}" + + if echo "${_filepath}" | grep "\.xz$" > /dev/null; then + xz -d "${_filepath}" + elif echo "${_filepath}" | grep "\.tar\.gz$" > /dev/null; then + tar Szxf "${_filepath}" -C "$(dirname "${_filepath}")" + rm -f "${_filepath}" + elif echo "${_filepath}" | grep "\.gz$" > /dev/null; then + gunzip "${_filepath}" + fi +} + +# 'vm img' +# list cloud images or get a new one +# +# @param string _url if specified, the url will be fetch'ed into $vm_dir/.img +# +core::img(){ + local _url _ds="default" _filename + if ! which qemu-img > /dev/null; then + util::err "Error: qemu-img is required to work with cloud images! Run 'pkg install qemu-tools'." + fi + + while getopts d:u _opt ; do + case $_opt in + d) _ds=${OPTARG} ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _url=$1 + + if [ -n "${_url}" ]; then + datastore::get_img "${_ds}" || util::err "unable to locate path for datastore '${_ds}'" + _filename=$(basename "${_url}") + fetch -o "${VM_DS_PATH}" "${_url}" + core::decompress "${VM_DS_PATH}/${_filename}" + else + datastore::img_list + fi +} + +# 'vm passthru' +# show a list of available passthrough devices +# and their device number +# +core::passthru(){ + local _dev _sbf _desc _ready + local _format="%-10s %-12s %-12s %s\n" + + printf "${_format}" "DEVICE" "BHYVE ID" "READY" "DESCRIPTION" + + pciconf -l | awk -F'[@:]' '{ print $1,$3 "/" $4 "/" $5}' | \ + while read _dev _sbf; do + + _ready=$(echo "${_dev}" | grep ^ppt) + [ -n "${_ready}" ] && _ready="Yes" + + _desc=$(pciconf -lv | grep -A2 "^${_dev}@" | tail -n1 | grep device | cut -d\' -f2) + printf "${_format}" "${_dev}" "${_sbf}" "${_ready:-No}" "${_desc:--}" + done +} + +# 'vm get' +# get a core configuration setting +# +core::get(){ + local _var="$1" + local _val _format="%-20s%s\n" + local IFS=";" + + printf "${_format}" "SETTING" "VALUE" + + if [ "${_var}" = "all" ]; then + + for _var in ${VM_CONFIG_USER}; do + config::core::get "_val" "${_var}" + printf "${_format}" "${_var}" "${_val:--}" + done + else + while [ -n "${_var}" ]; do + if util::valid_config_setting "${_var}"; then + config::core::get "_val" "${_var}" + printf "${_format}" "${_var}" "${_val:--}" + fi + + shift + _var="$1" + done + fi +} + +# 'vm set' +# set a core configuration setting +# +core::set(){ + local _pair="$1" + local _key _val + + while [ -n "${_pair}" ]; do + _key="${_pair%%=*}" + _val="${_pair#*=}" + + if util::valid_config_setting "${_key}"; then + config::core::set "${_key}" "${_val}" + else + util::err "invalid configuration setting - '${_key}'" + fi + + shift + _pair="$1" + done +} diff --git a/lib/vm-datastore b/lib/vm-datastore new file mode 100644 index 0000000..c10c9e2 --- /dev/null +++ b/lib/vm-datastore @@ -0,0 +1,500 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# 'vm datastore list' +# show configured datastores +# +datastore::list(){ + local _format="%-15s %-11s %-25s %s" + local _name _type _dataset _path _spec + + # headings + printf "${_format}\n" "NAME" "TYPE" "PATH" "ZFS DATASET" + + # add the default datastore + _name="default" + _path="${vm_dir}" + + # get the type and path + if [ "${VM_ZFS}" ]; then + _type="zfs" + _dataset="${VM_ZFS_DATASET}" + else + _type="directory" + _dataset="-" + fi + + printf "${_format}\n" "${_name}" "${_type}" "${_path}" "${_dataset}" + + for _name in ${VM_DATASTORE_LIST}; do + [ "${_name}" = "default" ] && continue + + config::core::get "_spec" "path_${_name}" + [ -z "${_spec}" ] && continue + + if [ "${_spec%%:*}" = "zfs" ]; then + _type="zfs" + _dataset="${_spec#*:}" + datastore::__resolve_path "_path" "${_spec}" + elif [ "${_spec%%:*}" = "iso" ]; then + _type="iso" + _path="${_spec#*:}" + _dataset="-" + elif [ "${_spec%%:*}" = "img" ]; then + _type="img" + _path="${_spec#*:}" + _dataset="-" + else + _type="directory" + _path="${_spec}" + _dataset="-" + fi + + printf "${_format}\n" "${_name}" "${_type}" "${_path}" "${_dataset}" + done +} + +# 'vm datastore add' +# create a new datastore +# +# we don't try and create directories or datasets. +# the user should do that first +# +# @param string _name datastore name +# @param string _spec specification (either /path or zfs:dataset) +# +datastore::add(){ + local _name="$1" + local _spec="$2" + local _mount _num=0 _curr + + [ -z "${_name}" -o -z "${_spec}" ] && util::usage + util::check_name "${_name}" || util::err "invalid datastore name - '${_name}'" + + # check name not in use + for _curr in ${VM_DATASTORE_LIST}; do + [ "${_curr}" = "${_name}" ] && util::err "datstore '${_name}' already exists!" + done + + # look for zfs + if [ "${_spec%%:*}" = "zfs" ]; then + # try to find mountpoint + _mount=$(mount | grep "^${_spec#*:} " |cut -d' ' -f3) + [ -z "${_mount}" ] && util::err "${_spec} doesn't seem to be a valid, mounted dataset" + else + # make sure it's a directory + [ ! -d "${_spec}" ] && util::err "${_spec} doesn't seem to be a valid directory" + + _mount="${_spec}" + fi + + # see if this is already our default datastore + [ "${_mount}" = "${vm_dir}" ] && util::err "specified path already exists as default datastore" + + # save + config::core::set "datastore_list" "${_name}" "1" + config::core::set "path_${_name}" "${_spec}" + [ $? -eq 0 ] || util::err "error saving settings to configuration file" +} + +# remove a datastore +# we don't actually delete anything, just remove from config +# +# @param string _name name of dataset +# +datastore::remove(){ + local _name="$1" + local _ds _found + + [ "${_name}" = "default" ] && util::err "cannot remove default datastore" + + for _ds in ${VM_DATASTORE_LIST}; do + [ "${_ds}" = "${_name}" ] && _found="1" + done + + # found the dataset? + [ -z "${_found}" ] && util::err "unable to locate the specified dataset" + + config::core::remove "datastore_list" "${_name}" + config::core::remove "path_${_name}" + [ $? -eq 0 ] || util::err "error removing settings from configuration file" +} + +# get the filesystem path for the specified dataset spec +# this can either be just a path, or ZFS dataset +# +# @param string _var variable to put path into +# @param string _spec the path spec (either directory or zfs:dataset) +# @return non-zero on error +# +datastore::__resolve_path(){ + local _var="$1" + local _spec="$2" + + if [ "${_spec%%:*}" = "zfs" ]; then + _mount=$(mount | grep "^${_spec#*:} " |cut -d' ' -f3) + + if [ -n "${_mount}" ]; then + setvar "${_var}" "${_mount}" + return 0 + fi + elif [ "${_spec%%:*}" = "iso" ] || [ "${_spec%%:*}" = "img" ]; then + setvar "${_var}" "${_spec#*:}" + return 0 + else + if [ -d "${_spec}" ]; then + setvar "${_var}" "${_spec}" + return 0 + fi + fi + + setvar "${_var}" "" + return 1 +} + +# load list of datastores into VM_DATASTORE_LIST +# +# @modifies VM_DATASTORE_LIST +# +datastore::load(){ + config::core::get "VM_DATASTORE_LIST" "datastore_list" + VM_DATASTORE_LIST="default${VM_DATASTORE_LIST:+ }${VM_DATASTORE_LIST}" +} + +# init global settings for a vm +# we take a guest name and try to find it in all +# datastores. we don't allow duplicate names, and +# if there is, we will just return the first found. +# +# @param string _guest guest name +# @return non-zero on error +# @modifies VM_DS_NAME VM_DS_PATH VM_DS_ZFS VM_DS_ZFS_DATASET +# +datastore::get_guest(){ + local _guest="$1" + local _ds _spec _path _found _zfs _dataset + + # look in default store + if [ -f "${vm_dir}/${_guest}/${_guest}.conf" ]; then + _found="1" + _ds="default" + _path="${vm_dir}" + _zfs="${VM_ZFS}" + _dataset="${VM_ZFS_DATASET}" + fi + + # look on other datastores + if [ -z "${_found}" ]; then + for _ds in ${VM_DATASTORE_LIST}; do + [ "${_ds}" = "default" ] && continue + + config::core::get "_spec" "path_${_ds}" + if [ "${_spec%%:*}" = "iso" ] || [ "${_spec%%:*}" = "img" ]; then + continue + fi + + datastore::__resolve_path "_path" "${_spec}" + + if [ -f "${_path}/${_guest}/${_guest}.conf" ]; then + [ "${_spec%%:*}" = "zfs" ] && _zfs="1" && _dataset="${_spec#*:}" + + _found="1" + break + fi + done + fi + + # make sure we have a path + [ -z "${_found}" ] && return 1 + + # set variables + VM_DS_NAME="${_ds}" + VM_DS_PATH="${_path}" + VM_DS_ZFS="${_zfs}" + VM_DS_ZFS_DATASET="${_dataset}" +} + +# get the path and details for a datastore +# put into same variables as datastore_get_guest +# +# @param string _ds datastore name +# @return non-zero on error +# @modifies VM_DS_PATH VM_DS_ZFS VM_DS_ZFS_DATASET +# +datastore::get(){ + local _ds="$1" + local _spec _path _zfs _dataset + + # check for default + if [ "${_ds}" = "default" ]; then + VM_DS_NAME="default" + VM_DS_PATH="${vm_dir}" + VM_DS_ZFS="${VM_ZFS}" + VM_DS_ZFS_DATASET="${VM_ZFS_DATASET}" + return 0 + fi + + config::core::get "_spec" "path_${_ds}" + [ -z "${_spec}" ] && return 1 + + # skip iso and img stores + if [ "${_spec%%:*}" = "iso" ] || [ "${_spec%%:*}" = "img" ]; then + return 1 + fi + + datastore::__resolve_path "_path" "${_spec}" || return 1 + [ "${_spec%%:*}" = "zfs" ] && _zfs="1" && _dataset="${_spec#*:}" + + # set variables + VM_DS_NAME="${_ds}" + VM_DS_PATH="${_path}" + VM_DS_ZFS="${_zfs}" + VM_DS_ZFS_DATASET="${_dataset}" +} + +# get the path for an iso datastore +# +# @param string _ds datastore name +# +datastore::get_iso(){ + local _ds="$1" + + # default? + # we use the .iso subdir in that case + if [ "${_ds}" = "default" ]; then + VM_DS_PATH="${vm_dir}/.iso" + return 0 + fi + + config::core::get "_spec" "path_${_ds}" + [ -z "${_spec}" ] && return 1 + + # should be an iso ds + [ "${_spec%%:*}" = "iso" ] || return 1 + + datastore::__resolve_path "_path" "${_spec}" || return 1 + VM_DS_PATH="${_path}" +} + +# get the path for an img datastore +# +# @param string _ds datastore name +# +datastore::get_img(){ + local _ds="$1" + + # default? + # we use the .img subdir in that case + if [ "${_ds}" = "default" ]; then + VM_DS_PATH="${vm_dir}/.img" + return 0 + fi + + config::core::get "_spec" "path_${_ds}" + [ -z "${_spec}" ] && return 1 + + # should be an img ds + [ "${_spec%%:*}" = "img" ] || return 1 + + datastore::__resolve_path "_path" "${_spec}" || return 1 + VM_DS_PATH="${_path}" +} + +# add a datastore for iso files +# +# @param string _name the name of the datastore +# @param string _path filesystem path +# +datastore::iso(){ + local _name="$1" + local _path="$2" + + [ -z "${_name}" -o -z "${_path}" ] && util::usage + util::check_name "${_name}" || util::err "invalid datastore name - '${_name}'" + + # check name not in use + for _curr in ${VM_DATASTORE_LIST}; do + [ "${_curr}" = "${_name}" ] && util::err "datstore '${_name}' already exists!" + done + + # make sure directory exists + [ ! -d "${_path}" ] && util::err "specified directory does not appear to be valid" + + # save + config::core::set "datastore_list" "${_name}" "1" + config::core::set "path_${_name}" "iso:${_path}" + [ $? -eq 0 ] || util::err "error saving settings to configuration file" +} + +# add a datastore for img files +# +# @param string _name the name of the datastore +# @param string _path filesystem path +# +datastore::img(){ + local _name="$1" + local _path="$2" + + [ -z "${_name}" -o -z "${_path}" ] && util::usage + util::check_name "${_name}" || util::err "invalid datastore name - '${_name}'" + + # check name not in use + for _curr in ${VM_DATASTORE_LIST}; do + [ "${_curr}" = "${_name}" ] && util::err "datstore '${_name}' already exists!" + done + + # make sure directory exists + [ ! -d "${_path}" ] && util::err "specified directory does not appear to be valid" + + # save + config::core::set "datastore_list" "${_name}" "1" + config::core::set "path_${_name}" "img:${_path}" + [ $? -eq 0 ] || util::err "error saving settings to configuration file" +} + +# find an iso file by looking in the default location +# and any "iso" datastores +# +# @param string _var variable name to put full iso path into +# @param string _file iso filename to look for +# @return int success if found +# +datastore::iso_find(){ + local _var="$1" + local _file="$2" + local _ds _spec + + # given a full path? + if [ -z "${_file%%/*}" ] && [ -r "${_file}" ]; then + setvar "${_var}" "${_file}" + return 0 + fi + + # file exists in current dir? + if [ -r "$(pwd)/${_file}" ]; then + setvar "${_var}" "$(pwd)/${_file}" + return 0 + fi + + # look in default store + if [ -r "${vm_dir}/.iso/${_file}" ]; then + setvar "${_var}" "${vm_dir}/.iso/${_file}" + return 0 + fi + + for _ds in ${VM_DATASTORE_LIST}; do + config::core::get "_spec" "path_${_ds}" + [ "${_spec%%:*}" != "iso" ] && continue + + if [ -r "${_spec#*:}/${_file}" ]; then + setvar "${_var}" "${_spec#*:}/${_file}" + return 0 + fi + done + + return 1 +} + +# find an img file by looking in the default location +# and any "img" datastores +# +# @param string _var variable name to put full img path into +# @param string _file img filename to look for +# @return int success if found +# +datastore::img_find(){ + local _var="$1" + local _file="$2" + local _ds _spec + + # given a full path? + if [ -z "${_file%%/*}" ] && [ -r "${_file}" ]; then + setvar "${_var}" "${_file}" + return 0 + fi + + # file exists in current dir? + if [ -r "$(pwd)/${_file}" ]; then + setvar "${_var}" "$(pwd)/${_file}" + return 0 + fi + + # look in default store + if [ -r "${vm_dir}/.img/${_file}" ]; then + setvar "${_var}" "${vm_dir}/.img/${_file}" + return 0 + fi + + for _ds in ${VM_DATASTORE_LIST}; do + config::core::get "_spec" "path_${_ds}" + [ "${_spec%%:*}" != "img" ] && continue + + if [ -r "${_spec#*:}/${_file}" ]; then + setvar "${_var}" "${_spec#*:}/${_file}" + return 0 + fi + done + + return 1 +} + +# list iso files +# +datastore::iso_list(){ + local _ds _spec _format="%-20s%s\n" + + printf "${_format}" "DATASTORE" "FILENAME" + + # look for default iso location + [ -d "${vm_dir}/.iso" ] && ls -1 "${vm_dir}/.iso" | awk '{printf "'${_format}'","default",$1}' + + # look for iso datastores + for _ds in ${VM_DATASTORE_LIST}; do + config::core::get "_spec" "path_${_ds}" + [ "${_spec%%:*}" != "iso" ] && continue + + [ -d "${_spec#*:}" ] && ls -1 ${_spec#*:} | awk '{printf "'${_format}'","'${_ds}'",$1}' + done +} + +# list img files +# +datastore::img_list(){ + local _ds _spec _format="%-20s%s\n" + + printf "${_format}" "DATASTORE" "FILENAME" + + # look for default img location + [ -d "${vm_dir}/.img" ] && ls -1 "${vm_dir}/.img" | awk '{printf "'${_format}'","default",$1}' + + # look for img datastores + for _ds in ${VM_DATASTORE_LIST}; do + config::core::get "_spec" "path_${_ds}" + [ "${_spec%%:*}" != "img" ] && continue + + [ -d "${_spec#*:}" ] && ls -1 ${_spec#*:} | awk '{printf "'${_format}'","'${_ds}'",$1}' + done +} diff --git a/lib/vm-guest b/lib/vm-guest new file mode 100644 index 0000000..3e69e0f --- /dev/null +++ b/lib/vm-guest @@ -0,0 +1,196 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# guest::load +# this function is responsible for doing any pre-load tasks for a guest. +# for non uefi guests this normally means running bhyveload or grub-bhyve. +# this function should return a non-zero value if there's a problem +# or 0 on success. +# As this is called from within the scope of vm::run, +# the following variables are already set (among others) +# +# _name: guest name +# _loader: boot loader to use (grub|bhyveload) +# _com: com port - /dev/nmdmXA +# _conf: full path to guest config file +# _cpu: cpu count +# _memory: RAM +# _guest: guest type +# _bootdisk: full path to primary disk +# +# I've written append wrong as it just needs to be something other than 'write', +# and is much more readable when all the util::log* calls line up +# +# @param optional string _iso set to the boot iso on install, or not given for normal run +# @return int 0=success, 15=vm-bhyve error (see log), other=bhyveload|grub-bhyve error code +# +guest::load(){ + local _iso="$1" + local _args _command _timeout _grub_opt _bsd_loader _custom_args + + # require a boot disk + if [ -z "${_bootdisk}" ]; then + util::log "guest" "${_name}" "fatal; non-uefi loaders require a boot disk device" + return 15 + fi + + # all loaders have same console and wired memory options + [ -z "${VM_OPT_FOREGROUND}" ] && _args="-c ${_com}" + [ "${_wiredmem}" = "1" ] && _args="${_args}${_args:+ }-S" + + # get timeout + config::get "_timeout" "loader_timeout" "3" + + case "${_loader}" in + bhyveload) + _command="bhyveload" + _args="${_args}${_args:+ }-m ${_memory} -e smbios.system.uuid=${_uuid} -e autoboot_delay=${_timeout} -e bhyve_vm_name=${_name}" + + # look for custom bhyveload arguments + config::get "_custom_args" "bhyveload_args" + [ -n "${_custom_args}" ] && _args="${_args} ${_custom_args}" + + # have a custom guest loader specified? + config::get "_bsd_loader" "bhyveload_loader" + [ -n "${_bsd_loader}" ] && _args="${_args} -l ${_bsd_loader}" + + if [ -n "${_iso}" ]; then + _args="${_args} -d ${_iso}" + else + _args="${_args} -d ${_bootdisk}" + fi + ;; + grub) + _command=$(which grub-bhyve) + + # check we have grub-bhyve + if [ $? -ne 0 ]; then + util::log "guest" "${_name}" "fatal; grub requested but sysutils/grub2-bhyve not installed?" + return 15 + fi + + # add device map path and memory + _args="${_args}${_args:+ }-m ${VM_DS_PATH}/${_name}/device.map -M ${_memory}" + + if [ -n "${_iso}" ]; then + _root="cd0" + util::log_and_write "write" "${_name}" "device.map" "(cd0) ${_iso}" + util::log_and_write "appnd" "${_name}" "device.map" "(hd0) ${_bootdisk}" + guest::__map_all_disks + + # if we have local grub config, we need to point grub-bhyve at the host. + # if not, just use defaults + if guest::__write_config "install"; then + _args="${_args} -r host -d ${VM_DS_PATH}/${_name}" + else + _args="${_args} -r ${_root}" + fi + else + _root="hd0,1" + util::log_and_write "write" "${_name}" "device.map" "(hd0) ${_bootdisk}" + guest::__map_all_disks + + config::get "_grub_opt" "grub_run_partition" + [ -n "${_grub_opt}" ] && _root="hd0,${_grub_opt}" + + # if we have local config, point grub-bhyve at it + # otherwise we use defaults, or directory and file specified by user + if guest::__write_config "run"; then + _args="${_args} -r host -d ${VM_DS_PATH}/${_name}" + else + _args="${_args} -r ${_root}" + + config::get "_grub_opt" "grub_run_dir" + [ -n "${_grub_opt}" ] && _args="${_args} -d ${_grub_opt}" + config::get "_grub_opt" "grub_run_file" + [ -n "${_grub_opt}" ] && _args="${_args} -g ${_grub_opt}" + fi + fi + ;; + *) + util::log "guest" "${_name}" "unsupported loader - '${_loader}'" + return 15 + ;; + esac + + # run the command + util::log "guest" "${_name}" "${_command} ${_args} ${_name}" + ${_command} ${_args} ${_name} +} + +# Add all extra/non-boot disks to the device.map file +# Some users may need to access additional disks from the loader +# +guest::__map_all_disks(){ + local _disk _dev _path _num=1 + + config::get "_disk" "disk${_num}_name" + + while [ -n "${_disk}" ]; do + config::get "_dev" "disk${_num}_dev" + vm::get_disk_path "_path" "${_name}" "${_disk}" "${_dev}" + util::log_and_write "appnd" "${_name}" "device.map" "(hd${_num}) ${_path}" + + _num=$((_num + 1)) + config::get "_disk" "disk${_num}_name" + done +} + +# See if the user has configured grub commands. +# If so we write them to a grub.cfg file and +# tell grub-bhyve to use it via (host) device +# +# @param string _type=install|run which commands to load +# @return int true (0) if commands were loaded +# +guest::__write_config(){ + local _type="$1" + local _command _num=0 + + # make sure original boot command file grub.cmd is gone + # we've switched to grub.cfg now as this is the + # default for grub-bhyve and makes one less option needed + rm "${VM_DS_PATH}/${_name}/grub.*" >/dev/null 2>&1 + + config::get "_command" "grub_${_type}${_num}" + [ -z "${_command}" ] && return 1 + + util::log_and_write "write" "${_name}" "grub.cfg" "timeout=${_timeout}" + util::log_and_write "appnd" "${_name}" "grub.cfg" "menuentry '${_name} (bhyve ${_type})' {" + util::log_and_write "appnd" "${_name}" "grub.cfg" " root=${_root}" + + while [ -n "${_command}" ]; do + # we don't need boot command anymore + [ "${_command}" != "boot" ] && util::log_and_write "appnd" "${_name}" "grub.cfg" " ${_command}" + + _num=$((_num + 1)) + config::get "_command" "grub_${_type}${_num}" + done + + util::log_and_write "appnd" "${_name}" "grub.cfg" "}" + + return 0 +} diff --git a/lib/vm-info b/lib/vm-info new file mode 100644 index 0000000..6310997 --- /dev/null +++ b/lib/vm-info @@ -0,0 +1,503 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# 'vm info' +# display a wealth of information about all guests, or one specified +# +# @param optional string[multiple] _name name of the guest to display +# +info::guest(){ + local _name="$1" + local _bridge_list=$(ifconfig -g vm-switch) + local _ds + + vm::running_load + + # see if guest name(s) provided + if [ -n "${_name}" ]; then + while [ -n "${_name}" ]; do + datastore::get_guest "${_name}" || util::err "unable to locate virtual machine '${_name}'" + info::guest_show "${_name}" + + shift + _name="$1" + done + + exit + fi + + # show all guests from all datastores + for _ds in ${VM_DATASTORE_LIST}; do + datastore::get "${_ds}" || continue + + ls -1 "${VM_DS_PATH}" 2>/dev/null | \ + while read _name; do + [ -e "${VM_DS_PATH}/${_name}/${_name}.conf" ] && info::guest_show "${_name}" + done + done +} + +# 'vm switch info' +# display config of each virtual switch as well as stats and connected guests +# +# @param optional string[multiple] _switch name of switch to display +# +info::switch(){ + local _switch="$1" + local _list + + # load config file manually using non-core function + # this means we can share the config_output function with guest + config::load "${vm_dir}/.config/system.conf" + config::get "_list" "switch_list" + + if [ -n "${_switch}" ]; then + while [ -n "${_switch}" ]; do + info::switch_show "${_switch}" + + shift + _switch="$1" + done + + exit + fi + + for _switch in ${_list}; do + info::switch_show "${_switch}" + done +} + +# display info for one virtual switch +# +# @param string _switch the name of the switch +# +info::switch_show(){ + local _switch="$1" + local _type _bridge _vale _netgraph _id + local _INDENT=" " + + [ -z "${_switch}" ] && return 1 + + config::get "_type" "type_${_switch}" "standard" + config::get "_bridge" "bridge_${_switch}" + config::get "_vale" "vale_${_switch}" + config::get "_netgraph" "netgraph_${_switch}" + + + echo "------------------------" + echo "Virtual Switch: ${_switch}" + echo "------------------------" + + echo "${_INDENT}type: ${_type}" + switch::id "_id" "${_switch}" + + # we don't have a bridge for vale and netgraph switches + [ "${_type}" != "vale" ] && [ "${_type}" != "netgraph" ] && _bridge="${_id}" + + echo "${_INDENT}ident: ${_id:--}" + + info::__output_config "vlan_${_switch}" "vlan" + info::__output_config "ports_${_switch}" "physical-ports" + + if [ -n "${_bridge}" ]; then + _stats=$(netstat -biI "${_bridge}" |grep '1; i--) printf("%s ",$i); print $1; }' | awk '{print $5,$2}') + + if [ -n "${_stats}" ]; then + _b_in=$(info::__bytes_human "${_stats%% *}") + _b_out=$(info::__bytes_human "${_stats##* }") + + echo "${_INDENT}bytes-in: ${_stats%% *} (${_b_in})" + echo "${_INDENT}bytes-out: ${_stats##* } (${_b_out})" + fi + + # show guest ports + info::switch_ports + fi + + echo "" +} + +# get all guest ports for current bridge +# +info::switch_ports(){ + local _port_list=$(ifconfig "${_bridge}" |grep 'member: tap' |awk '{print $2}') + local _port _guest + + for _port in ${_port_list}; do + _guest=$(ifconfig "${_port}" |grep 'description: vmnet' |awk '{print $2}' |cut -d'/' -f2) + + echo "" + echo "${_INDENT}virtual-port" + echo "${_INDENT}${_INDENT}device: ${_port}" + echo "${_INDENT}${_INDENT}vm: ${_guest:--}" + done +} + +# display the info for one guest +# +# @param string _name name of the guest to display +# +info::guest_show(){ + local _name="$1" + local _conf="${VM_DS_PATH}/${_name}/${_name}.conf" + local _INDENT=" " + local _RUN="0" + local _res_mem _b_res_mem _global_run _port _opt _pid + + [ -z "${_name}" ] && return 1 + [ ! -f "${_conf}" ] && return 1 + + config::load "${_conf}" + + # check local and global runstate + [ -e "/dev/vmm/${_name}" ] && _RUN="1" + vm::running_check "_global_run" "_pid" "${_name}" + _global_run=$(echo "${_global_run}" | tr '[:upper:]' '[:lower:]') + + echo "------------------------" + echo "Virtual Machine: ${_name}" + echo "------------------------" + + echo "${_INDENT}state: ${_global_run}" + echo "${_INDENT}datastore: ${VM_DS_NAME}" + + # basic guest configuration + info::__output_config "loader" "" "none" + info::__output_config "uuid" "" "auto" + info::__output_config "cpu" + + # check for a cpu topology + config::get "_opt" "cpu_sockets" + + if [ -n "${_opt}" ]; then + echo -n "${_INDENT}cpu-topology: sockets=${_opt}" + config::get "_opt" "cpu_cores" + [ -n "${_opt}" ] && echo -n ", cores=${_opt}" + config::get "_opt" "cpu_threads" + [ -n "${_opt}" ] && echo -n ", threads=${_opt}" + echo "" + fi + + info::__output_config "memory" + + # running system details + if [ "${_RUN}" = "1" ]; then + _res_mem=$(bhyvectl --get-stats --vm="${_name}" |grep 'Resident memory' |awk '{print $3}') + + if [ -n "${_res_mem}" ]; then + _b_res_mem=$(info::__bytes_human "${_res_mem}") + echo "${_INDENT}memory-resident: ${_res_mem} (${_b_res_mem})" + fi + + # show com ports + if [ -e "${VM_DS_PATH}/${_name}/console" ]; then + echo "" + echo "${_INDENT}console-ports" + + cat "${VM_DS_PATH}/${_name}/console" | \ + while read _port; do + echo "${_INDENT}${_INDENT}${_port%%=*}: ${_port##*=}" + done + + # virtio-console devices + info::guest_vtcon + fi + fi + + # network interfaces + info::guest_networking + + # disks + info::guest_disks + + # zfs data + info::guest_zfs + + echo "" +} + +# display any virtio consoles +# +info::guest_vtcon(){ + local _INDENT=" " + local _console _num=0 + + config::get "_console" "virt_console0" + [ -z "${_console}" ] && return 0 + + echo "" + + while [ -n "${_console}" -a ${_num} -lt 16 ]; do + # if set to "yes/on/1", just use the console number as port name + case "${_console}" in + [Yy][Ee][Ss]|[Oo][Nn]|1) _console="${_num}" ;; + esac + + echo "${_INDENT}vtcon${_num}: ${VM_DS_PATH}/${_name}/vtcon.${_console}" + + _num=$((_num + 1)) + config::get "_console" "virt_console${_num}" + done +} + +# display zfs snapshot/origin data +# +info::guest_zfs(){ + local _INDENT=" " + local _data + + [ -z "${VM_DS_ZFS}" ] && return 1 + + _data=$(zfs list -o name,used,creation -s creation -rHt snapshot "${VM_DS_ZFS_DATASET}/${_name}" | sed "s/^/${_INDENT}/") + + if [ -n "${_data}" ]; then + echo "" + echo " snapshots" + echo "${_data}" + fi + + _data=$(zfs get -Ho value origin "${VM_DS_ZFS_DATASET}/${_name}") + [ "${_data}" = "-" ] && return 0 + + echo "" + echo " clone-origin" + echo " ${_data}" +} + +# display disks +# +info::guest_disks(){ + local _num=0 + local _disk _type _dev _path _size _b_size _used _b_used + local _INDENT=" " + + while true; do + config::get "_disk" "disk${_num}_name" + config::get "_type" "disk${_num}_type" + config::get "_dev" "disk${_num}_dev" "file" + [ -z "${_disk}" -o -z "${_type}" ] && break + + vm::get_disk_path "_path" "${_name}" "${_disk}" "${_dev}" + + echo "" + echo " virtual-disk" + echo "${_INDENT}number: ${_num}" + + info::__output_config "disk${_num}_dev" "device-type" "file" + info::__output_config "disk${_num}_type" "emulation" + info::__output_config "disk${_num}_opts" "options" + + echo "${_INDENT}system-path: ${_path:--}" + + _size="" + _used="" + + if [ -n "${_path}" ]; then + case "${_dev}" in + file) + _size=$(stat -f%z "${_path}") + _used=$(du "${_path}" | awk '{print $1}') + _used=$((_used * 1024)) + ;; + zvol|sparse-zvol) + _size=$(zfs get -Hp volsize "${_path#/dev/zvol/}" |cut -f3) + _used=$(zfs get -Hp refer "${_path#/dev/zvol/}" |cut -f3) + ;; + iscsi) + _size=$(sysctl -b kern.geom.conftxt | awk "/ ${_path#/dev/} /{print \$4}") + _used=${_size} + esac + + if [ -n "${_size}" -a -n "${_used}" ]; then + _b_size=$(info::__bytes_human "${_size}") + _b_used=$(info::__bytes_human "${_used}") + echo "${_INDENT}bytes-size: ${_size} (${_b_size})" + echo "${_INDENT}bytes-used: ${_used} (${_b_used})" + fi + fi + + _num=$((_num + 1)) + done +} + +# display networking configuration +# +info::guest_networking(){ + local _num=0 + local _int _id _tag _switch _stats _b_in _b_out + local _INDENT=" " + + while true; do + config::get "_int" "network${_num}_type" + [ -z "${_int}" ] && break + + echo "" + echo " network-interface" + echo "${_INDENT}number: ${_num}" + + # basic interface config + info::__output_config "network${_num}_type" "emulation" + info::__output_config "network${_num}_switch" "virtual-switch" + info::__output_config "network${_num}_mac" "fixed-mac-address" + info::__output_config "network${_num}_device" "fixed-device" + + # if running, try to get some more interface details + if [ "${_RUN}" = "1" ]; then + config::get "_switch" "network${_num}_switch" + + _int=$(ifconfig | grep -B1 "vmnet/${_name}/${_num}/" | head -n1 | cut -d' ' -f1,6) + _id=${_int%%:*} + _tag=$(ifconfig | grep "vmnet/${_name}/${_num}/" | cut -d' ' -f2) + + info::__find_bridge "_bridge" "${_id}" + + echo "${_INDENT}active-device: ${_id:--}" + echo "${_INDENT}desc: ${_tag:--}" + echo "${_INDENT}mtu: ${_int##* }" + echo "${_INDENT}bridge: ${_bridge:--}" + + if [ -n "${_id}" ]; then + _stats=$(netstat -biI "${_id}" |grep '1; i--) printf("%s ",$i); print $1; }' |awk '{print $2,$5}') + _b_in=$(info::__bytes_human "${_stats%% *}") + _b_out=$(info::__bytes_human "${_stats##* }") + + echo "${_INDENT}bytes-in: ${_stats%% *} (${_b_in})" + echo "${_INDENT}bytes-out: ${_stats##* } (${_b_out})" + fi + fi + + _num=$((_num + 1)) + done +} + +# output a single configuration variable +# alwasy called once guest configuration has been loaded +# +# @param string _option config option to display +# @param optional string _title title to display instead of using option name +# @param optional string _default default value to display if not - +# +info::__output_config(){ + local _option="$1" + local _title="$2" + local _default="$3" + local _var + + config::get "_var" "${_option}" "${_default:--}" + [ -z "${_title}" ] && _title="${_option}" + + echo "${_INDENT}${_title}: ${_var}" +} + +# try and find the bridge an interface is a member of. +# we do this rather than just use switch::id as +# this should be able to locate the bridge even for devices +# that have been bridged manually and have no switch name configured +# +# @param string _var variable to put value into +# @param string _interface interface to look for +# +info::__find_bridge(){ + local _var="$1" + local _interface="$2" + local _br _found + + for _br in ${_bridge_list}; do + _found=$(ifconfig "${_br}" |grep member: |awk '{print $2}' |tr "\n" "," | grep "${_interface},") + + if [ -n "${_found}" ]; then + setvar "${_var}" "${_br}" + return 0 + fi + done + + setvar "${_var}" "" + return 1 +} + +# format bytes to human readable +# convert to k,m,g or t +# output rounded number +# +# @param int _val the value to convert +# +info::__bytes_human(){ + local _val="$1" _int _ext + local _dec="$2" + local _num="$3" + + : ${_dec:=3} + : ${_val:=0} + : ${_num:=1} + _int=${_val%%.*} + + while [ ${_int} -ge 1024 -a ${_num} -lt 5 ]; do + _val=$(echo "scale=3; ${_val}/1024" | bc) + _int=${_val%%.*} + _num=$((_num + 1)) + done + + case "${_num}" in + 1) _ext="B" ;; + 2) _ext="K" ;; + 3) _ext="M" ;; + 4) _ext="G" ;; + 5) _ext="T" ;; + esac + + export LC_ALL="C" + printf "%.${_dec}f%s" "${_val}" "${_ext}" +} + +info::__find_iscsi() { + local _var="$1" + # _target format: [iqn.*]unique_name[/N] where N is the lun# and defaults + # to 0. The address before the unique_name can be omitted so long as + # the unique_name is sufficient to select only one output of iscsictl -L. + local _target="$2" + local _lun _col _found + + # If no lun is specified, assume /0 + _lun=${_target##*/} + [ "${_lun}" = "${_target}" ] && _lun=0 + _target=${_target%/*} + + _found=$(iscsictl -L -w 10 | grep ${_target} | wc -l) + if [ "${_found}" -ne 1 ]; then + setvar "${_var}" "" + util::err "Unable to locate unique iSCSI device ${_target}" + fi + + # _col to be the column of iscsictl -L we want + _col=$((_lun + 4)) + _found=$(iscsictl -L | awk "/${_target}/{print \$${_col}}") + if echo "${_found}" | egrep -q '^da[0-9]+$'; then + setvar "${_var}" /dev/"${_found}" + return 0 + fi + + util::err "Unable to locate iSCSI device ${_target}/${_lun}" +} diff --git a/lib/vm-migration b/lib/vm-migration new file mode 100644 index 0000000..d42142a --- /dev/null +++ b/lib/vm-migration @@ -0,0 +1,278 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2021 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# vm migrate ... +# +# @param string name the guest to send +# @param string host host to send guest to +# +migration::run(){ + local _name + local _ds="default" + local _start="1" + local _renaming="0" + local _config _opt _stage _inc _triple _rdataset _pid _exists _rname _running + local _snap1 _snap2 _snap3 _destroy + local _count=0 + + while getopts cn12txr:d:i: _opt; do + case $_opt in + c) _config="1" ;; + r) _rname="${OPTARG}" ;; + n) _start="" ;; + i) _inc="${OPTARG}" ;; + 1) _stage="1" ;; + 2) _stage="2" ;; + t) _triple="1" ;; + x) _destroy="1" ;; + d) _ds="${OPTARG}" ;; + esac + done + + # get the name and host + shift $((OPTIND -1)) + _name="$1" + _host="$2" + + # do we want to output our config? + # sender uses the config option to pull config from the recieve end + if [ -n "${_config}" ]; then + migration::__check_config "${_ds}" + exit + fi + + # basic checks + [ -z "${_name}" -o -z "${_host}" ] && util::usage + datastore::get_guest "${_name}" || util:err "unable to locate guest - '${_name}'" + [ -z "${VM_DS_ZFS}" ] && util:err "the source datastore must be ZFS to support migration" + [ -n "${_stage}" -a -n "${_triple}" ] && util::err "single stage and triple stage are mutually exclusive" + [ "${_stage}" = "2" -a -z "${_inc}" ] && util::err "source snapshot must be given when running stage 2" + + if [ -n "${_rname}" ]; then + util::check_name "${_rname}" || util::err "invalid virtual machine name - '${_rname}'" + _renaming="1" + else + _rname="${_name}" + fi + + # check guest can be sent + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + migration::__check_compat + + # check running state + vm::confirm_stopped "${_name}" "1" >/dev/null 2>&1 + _state=$? + [ ${_state} -eq 2 ] && util::err "guest is powered up on another host" + [ ${_state} -eq 1 ] && _running="1" + + # try to get pid + if [ -n "${_running}" ]; then + _pid=$(pgrep -fx "bhyve: ${_name}") + [ -z "${_pid}" ] && util::err "guest seems to be running but can't find its pid" + fi + + # try to get remote config + _rdataset=$(ssh "${_host}" vm migrate -cd "${_ds}" 2>/dev/null) + [ $? = "1" -o -z "${_rdataset}" ] && util::err "unable to get config from ${_host}" + + echo "Attempting to send ${_name} to ${_host}" + echo " * remote dataset ${_rdataset}/${_rname}" + [ -n "${_running}" ] && echo " * source guest is powered on (#${_pid})" + + # STAGE 1 + # we send a full snapshot of the guest + if [ -z "${_stage}" -o "${_stage}" = "1" ]; then + _snap1="$(date +'%Y%m%d%H%M%S-s1')" + echo " * stage 1: taking snapshot ${_snap1}" + zfs snapshot -r "${VM_DS_ZFS_DATASET}/${_name}@${_snap1}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err_inline "error taking local snapshot" + + # send this snapshot + migrate::__send "1" "${_snap1}" "${_inc}" + fi + + # STAGE 1B + # do it again in triple mode + # for a big guest, hopefully not too much changed during full send + # this will therefore complete fairly quick, leaving very few changes for stage 2 + if [ -n "${_triple}" ]; then + _snap2="$(date +'%Y%m%d%H%M%S-s1b')" + echo " * stage 1b: taking snapshot ${_snap2}" + zfs snapshot -r "${VM_DS_ZFS_DATASET}/${_name}@${_snap2}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err_inline "error taking local snapshot" + + # send this snapshot + migrate::__send "1b" "${_snap2}" "${_snap1}" + fi + + # only running first stage + if [ "${_stage}" = "1" ]; then + echo " * done" + exit + fi + + # if it's running we now need to stop it + if [ -n "${_running}" ]; then + echo -n " * stage 2: attempting to stop guest" + + kill ${_pid} >/dev/null 2>&1 + + while [ ${_count} -lt 60 ]; do + sleep 2 + kill -0 ${_pid} >/dev/null 2>&1 || break + echo -n "." + _count=$((_count + 1)) + done + + echo "" + fi + + # has it stopped? + kill -0 ${_pid} >/dev/null 2>&1 && util:err_inline "failed to stop guest" + echo " * stage 2: guest powered off" + + # only needed if running or specifically doing a stage 2 + if [ -n "${_running}" -o "${_stage}" = "2" ]; then + _snap3="$(date +'%Y%m%d%H%M%S-s2')" + echo " * stage 2: taking snapshot ${_snap3}" + zfs snapshot -r "${VM_DS_ZFS_DATASET}/${_name}@${_snap3}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err_inline "error taking local snapshot" + + # send this snapshot + if [ "${_triple}" = "1" ]; then + migrate::__send "2" "${_snap3}" "${_snap2}" + elif [ "${_stage}" = "2" ]; then + migrate::__send "2" "${_snap3}" "${_inc}" + else + migrate::__send "2" "${_snap3}" "${_snap1}" + fi + fi + + # do we need to rename? + [ "${_renaming}" = "1" ] && migrate::__rename_config + + # start + if [ -n "${_start}" -a -n "${_running}" ]; then + echo " * attempting to start ${_rname} on ${_host}" + ssh ${_host} vm start ${_rname} + fi + + if [ -n "${_destroy}" ]; then + echo " * removing source guest" + zfs destroy -r "${VM_DS_ZFS_DATASET}/${_name}" + else + echo " * removing snapshots" + [ -n "${_snap1}" ] && zfs destroy "${VM_DS_ZFS_DATASET}/${_name}@${_snap1}" >/dev/null 2>&1 + [ -n "${_snap2}" ] && zfs destroy "${VM_DS_ZFS_DATASET}/${_name}@${_snap2}" >/dev/null 2>&1 + [ -n "${_snap3}" ] && zfs destroy "${VM_DS_ZFS_DATASET}/${_name}@${_snap3}" >/dev/null 2>&1 + fi + + echo " * done" +} + +# updates the config file for a renamed guest +# god knows why I didn't just use "guest.conf" +# +migrate::__rename_config(){ + local _path + + # we need the mount path first + _path=$(ssh "${_host}" mount | grep "^${_rdataset} " | cut -wf3) + + if [ $? -ne 0 -o -z "${_path}" ]; then + echo " ! failed to find remote datastore path. guest may not start" + return 1 + fi + + # make sure it's mounted on remote + ssh "${_host}" zfs mount "${_rdataset}/${_rname}" >/dev/null 2>&1 + + echo " * renaming configuration file to ${_rname}.conf" + ssh "${_host}" mv "${_path}/${_rname}/${_name}.conf" "${_path}/${_rname}/${_rname}.conf" >/dev/null 2>1 + + if [ $? -ne 0 ]; then + echo " ! failed to find rename remote configuration file. guest may not start" + return 1 + fi +} + +migrate::__send(){ + local _stage="$1" + local _snap="$2" + local _inc="$3" + + # are we sending incremental? + if [ -n "${_inc}" ]; then + echo " * stage ${_stage}: sending ${VM_DS_ZFS_DATASET}/${_name}@${_snap} (incremental source ${_inc})" + zfs send -Ri "${_inc}" "${VM_DS_ZFS_DATASET}/${_name}@${_snap}" | ssh ${_host} zfs recv "${_rdataset}/${_rname}" + else + echo " * stage ${_stage}: sending ${VM_DS_ZFS_DATASET}/${_name}@${_snap}" + zfs send -R "${VM_DS_ZFS_DATASET}/${_name}@${_snap}" | ssh ${_host} zfs recv "${_rdataset}/${_rname}" + fi + + [ $? -eq 0 ] || util::err_inline "error detected while sending snapshot" + echo " * stage ${_stage}: snapshot sent" +} + +# currently just outputs zfs path or error if datastore isn't zfs +# in future may also return some data we can use to verify compat, etc +# +# @param string _ds the datastore to get details of +# +migration::__check_config(){ + local _ds="$1" + + datastore::get "${_ds}" + [ -z "${VM_DS_ZFS}" ] && exit 1 + + # output the datastore dataset + # sender needs this to do a zfs recv + echo "${VM_DS_ZFS_DATASET}" +} + +# see if a guest can be migrated. +# there are a few guest settings that are likely to +# cause the guest to break if it's moved to another host +# +migration::__check_compat(){ + local _setting _err _num=0 + + # check pass through + config::get "_setting" "passthru0" + [ -n "${_setting}" ] && _err="pci pass-through enabled" + + # check for custom disks + # file/zvol are under guest dataset and should go across ok + # custom disks could be anywhere + while true; do + config::get "_setting" "disk${_num}_type" + [ -z "${_setting}" ] && break + [ "${_setting}" = "custom" ] && _err="custom disk(s) configured" && break + _num=$((_num + 1)) + done + + [ -n "${_err}" ] && util::err "migration is not supported for this guest (${_err})" +} diff --git a/lib/vm-rctl b/lib/vm-rctl new file mode 100644 index 0000000..e0245d6 --- /dev/null +++ b/lib/vm-rctl @@ -0,0 +1,88 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 ramdaron (https://github.com/ramdaron) +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# set limits to virtual machine +# this is the background process +# +rctl::set(){ + local _pcpu _rbps _wbps _riops _wiops + local _pid _pri + + # get limit settings + config::get "_pri" "priority" + config::get "_pcpu" "limit_pcpu" + config::get "_rbps" "limit_rbps" + config::get "_wbps" "limit_wbps" + config::get "_riops" "limit_riops" + config::get "_wiops" "limit_wiops" + + # wait till bhyve starts and get pid + sleep 1 + _pid=$(pgrep -fx "bhyve[: ].* ${_name}") + [ -z "${_pid}" ] && return 1 + + # check for a priority + [ -n "${_pri}" ] && renice ${_pri} ${_pid} >/dev/null 2>&1 + + # return if there are no limits + [ -z "${_pcpu}${_rbps}${_wbps}${_riops}${_wiops}" ] && return 1 + + # see if rctl works + /usr/bin/rctl >/dev/null 2>&1 + [ $? -ne 0 ] && \ + util::log "guest" "${_name}" "RCTL support requested but RCTL not available" && return 1 + + util::log "guest" "${_name}" "applying rctl limits" + + if [ -n "${_pcpu}" ]; then + /usr/bin/rctl -a process:${_pid}:pcpu:deny=${_pcpu} >/dev/null 2>&1 + [ $? -eq 0 ] && util::log "guest" "${_name}" " pcpu=${_pcpu}" + fi + + # at this point we can return if < FreeBSD 11 + [ ${VERSION_BSD} -lt 1100000 ] && return 0 + + if [ -n "${_rbps}" ]; then + /usr/bin/rctl -a process:${_pid}:readbps:throttle=${_rbps} >/dev/null 2>&1 + [ $? -eq 0 ] && util::log "guest" " readbps=${_rbps}" + fi + + if [ -n "${_wbps}" ]; then + /usr/bin/rctl -a process:${_pid}:writebps:throttle=${_wbps} >/dev/null 2>&1 + [ $? -eq 0 ] && util::log "guest" " writebps=${_wbps}" + fi + + if [ -n "${_riops}" ]; then + /usr/bin/rctl -a process:${_pid}:readiops:throttle=${_riops} >/dev/null 2>&1 + [ $? -eq 0 ] && util::log "guest" " readiops=${_riops}" + fi + + if [ -n "${_wiops}" ]; then + /usr/bin/rctl -a process:${_pid}:writeiops:throttle=${_wiops} >/dev/null 2>&1 + [ $? -eq 0 ] && util::log "guest" " writeiops=${_wiops}" + fi +} diff --git a/lib/vm-run b/lib/vm-run new file mode 100644 index 0000000..301b801 --- /dev/null +++ b/lib/vm-run @@ -0,0 +1,1028 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# 'vm _run' +# run a virtual machine +# this is the background process that does all the work +# in most cases this should not be run directly +# +# @param string _name the name of the guest to run +# @param optional string _iso the iso file for an install +# +vm::run(){ + local _name _iso _iso_dev + local _cpu _memory _bootdisk _bootdisk_dev _guest _wiredmem + local _guest_support _uefi _uuid _debug _hostbridge _loader + local _opts _devices _slot _install_slot _func=0 _taplist _exit _passdev + local _com _comports _comstring _logpath="/dev/null" _run=1 + local _bhyve_options _action + + cmd::parse_args "$@" + shift $? + _name="$1" + _iso="$2" + + # try to load datstore details + datastore::get_guest "${_name}" || exit 5 + + # bail out immediately if guest running + vm::confirm_stopped "${_name}" "1" || exit 10 + + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + config::get "_memory" "memory" + config::get "_loader" "loader" + config::get "_bootdisk" "disk0_name" + config::get "_bootdisk_dev" "disk0_dev" "file" + config::get "_hostbridge" "hostbridge" "standard" + config::get "_comports" "comports" "com1" + config::get "_uuid" "uuid" + config::get "_debug" "debug" "no" + config::get "_bhyve_options" "bhyve_options" + config::get "_slot" "start_slot" "4" + config::get "_install_slot" "install_slot" "3" + + # generate a uuid if we don't have one already + if [ -z "${_uuid}" ]; then + _uuid=$(uuidgen) + config::set "${_name}" "uuid" "${_uuid}" + fi + + # get cpu topology + vm::__cpu "_cpu" + + util::log_rotate "guest" "${_name}" + util::log "guest" "${_name}" \ + "initialising" \ + " [loader: ${_loader}]" \ + " [cpu: ${_cpu}]" \ + " [memory: ${_memory}]" \ + " [hostbridge: ${_hostbridge}]" \ + " [com ports: ${_comports}]" \ + " [uuid: ${_uuid}]" \ + " [debug mode: ${_debug}]" \ + " [primary disk: ${_bootdisk}]" \ + " [primary disk dev: ${_bootdisk_dev}]" + + # check basic settings + if [ -z "${_loader}" -o -z "${_cpu}" -o -z "${_memory}" ]; then + util::log "guest" "${_name}" "fatal; unable to start - missing required configuration" + exit 15 + fi + + # check ug + if [ -n "${VM_NO_UG}" ]; then + + # only FreeBSD guests. these start direct in 64bit mode and don't need UG + if [ "${_loader}" != "bhyveload" ]; then + util::log "guest" "${_name}" "fatal; unable to start - no unrestricted guest support" + exit 15 + fi + + # only 1 vcpu + if [ "${_cpu}" != "1" ]; then + _cpu=1 + util::log "guest" "${_name}" "warning; no unrestricted guest support. reducing vcpu count to 1" + fi + fi + + # default bhyve options + _opts="-AHP" + + # ignore access to unimplemented Model Specific Registers? + config::yesno "ignore_msr" && _opts="${_opts}w" + + # if uefi, make sure we have bootrom, then update options for uefi support + if [ "${_loader%-*}" = "uefi" ]; then + vm::uefi + fi + + # add any custom bhyve options + [ -n "${_bhyve_options}" ] && _opts="${_opts} ${_bhyve_options}" + + # if we have passthru, check vt-d or amdvi support now and exit + config::get "_passdev" "passthru0" + + if [ -n "${_passdev}" ] && ! util::check_bhyve_iommu; then + util::log "guest" "${_name}" "fatal; pci passthrough not supported on this system (no VT-d or amdvi)" + exit 15 + fi + + # wired memory? + if config::yesno "wired_memory"; then + _wiredmem="1" + _opts="${_opts} -S" + fi + + # set cpu/memory and uuid in opts + _opts="-c ${_cpu} -m ${_memory} ${_opts}" + [ -n "${_uuid}" ] && _opts="${_opts} -U ${_uuid}" + + # set utc time in opts if requested + if config::yesno "utctime" yes; then + if [ ${VERSION_BSD} -ge 1002000 ]; then + _opts="${_opts} -u" + else + util::log "guest" "${_name}" "warning; utc time requested but not available pre FreeBSD 10.2" + fi + fi + + # send bhyve output to bhyve.log if debug=yes + util::yesno "${_debug}" && _logpath="${VM_DS_PATH}/${_name}/bhyve.log" + + # complete the boot disk path + [ -n "${_bootdisk}" ] && vm::get_disk_path "_bootdisk" "${_name}" "${_bootdisk}" "${_bootdisk_dev}" + + # build bhyve device string + vm::bhyve_device_comports + vm::bhyve_device_basic + vm::bhyve_device_disks + vm::bhyve_device_networking + vm::bhyve_device_rand + vm::bhyve_device_passthru + vm::bhyve_device_fbuf + vm::bhyve_device_mouse + vm::bhyve_device_sound + vm::bhyve_device_console + + vm::lock + util::log "guest" "${_name}" "booting" + cd / + + while true; do + + # destroy existing vmm + # freebsd seems happy to run a bhyveload/bhyve loop + # grub-bhyve doesn't seem to like it for a lot of users + # Peter says don't destroy in Windows instructions, so don't if in UEFI mode + if [ -e "/dev/vmm/${_name}" -a -z "${_uefi}" ]; then + bhyvectl --vm="${_name}" --destroy >/dev/null 2>&1 + if [ $? -ne 0 ]; then + util::log "guest" "${_name}" "fatal; failed to destroy existing vmm device" + _exit=15 + break + fi + sleep 1 + fi + + # run any prestart script while guest is fully down + vm::prestart + + # add install iso or disk image + if [ -n "${_iso}" ]; then + if echo "${_iso}" | grep -iqs '.iso$'; then + _iso_dev="-s ${_install_slot}:0,ahci-cd,${_iso},ro" + else + _iso_dev="-s ${_install_slot}:0,ahci-hd,${_iso},ro" + fi + fi + + # use null.iso if not an install and uefi firmware + # old instructions but some windows versions apparently needed this present + [ -z "${_iso}" -a "${_loader}" = "uefi" ] && config::yesno "nulliso_fix" && \ + _iso_dev="-s ${_install_slot}:0,ahci-cd,${vm_dir}/.config/null.iso" + + # reasonably ugly hack to remove wait option after first run + [ "${_run}" -eq "2" ] && vm::bhyve_device_fbuf_clear_wait + + # load guest + if [ -z "${_uefi}" ]; then + + guest::load "${_iso}" + _exit=$? + + # check no errors + if [ ${_exit} -ne 0 ]; then + util::log "guest" "${_name}" "fatal; loader returned error ${_exit}" + break + fi + fi + + util::log "guest" "${_name}" " [bhyve options: ${_opts}]" \ + " [bhyve devices: ${_devices}]" \ + " [bhyve console: ${_comstring}]" + [ -n "${_iso_dev}" ] && util::log "guest" "${_name}" " [bhyve iso device: ${_iso_dev}]" + util::log "guest" "${_name}" "starting bhyve (run ${_run})" + + # call rctl now as next line will block until bhyve exits + rctl::set & + + # actually run bhyve! + # we're already in the background so we just wait for it to exit + bhyve ${_opts} \ + ${_devices} \ + ${_iso_dev} \ + ${_comstring} \ + ${_name} 2> "${_logpath}" + + # get bhyve exit code + _exit=$? + util::log "guest" "${_name}" "bhyve exited with status ${_exit}" + + # remove any console sockets + rm ${VM_DS_PATH}/${_name}/vtcon.* >/dev/null 2>&1 + + # decide what to do with exit code + vm::handle_exit "_action" ${_exit} + [ "${_action}" != "restart" ] && break + + util::log "guest" "${_name}" "restarting" + + # remove install iso so guest reboots from disk + # after install non-uefi guests will still get install cd until a full shutdown+restart + # as we don't reset _iso_dev + _iso="" + _run=$((_run + 1)) + done + + # destroy taps + for _devices in ${_taplist}; do + util::log "guest" "${_name}" "destroying network device ${_devices}" + ifconfig "${_devices}" destroy + done + + util::log "guest" "${_name}" "stopped" + [ -e "/dev/vmm/${_name}" ] && bhyvectl --destroy --vm=${_name} >/dev/null 2>&1 + + vm::unlock + exit ${_exit} +} + +# creates options to use a uefi bootrom +# +# @modifies _opts _uefi +# +vm::uefi(){ + local _bootrom + + if [ ${VERSION_BSD} -lt 1002509 ]; then + util::log "guest" "${_name}" "fatal; uefi guests can only be run on FreeBSD 10.3 or newer" + exit 15 + fi + + case "${_loader}" in + uefi-devel) + _bootrom="/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd" + ;; + uefi-csm) + _bootrom="/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd" + ;; + uefi-custom) + _bootrom="${VM_DS_PATH}/.config/BHYVE_UEFI.fd" + ;; + *) + _bootrom="/usr/local/share/uefi-firmware/BHYVE_UEFI.fd" + ;; + esac + + if [ ! -e "${_bootrom}" ]; then + util::log "guest" "${_name}" "fatal; unable to locate firmware ${_bootrom}" + exit 15 + fi + + # should we store uefi vars? + if config::yesno "uefi_vars"; then + + # do we already have a storage file for this guest? + if [ -e "${VM_DS_PATH}/${_name}/uefi-vars.fd" ]; then + : + elif [ -e "/usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd" ]; then + # create a copy and use + cp "/usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd" "${VM_DS_PATH}/${_name}/uefi-vars.fd" + else + util::log "guest" "${_name}" "fatal; unable to locate UEFI vars database or template" + exit 15 + fi + + _bootrom="${_bootrom},${VM_DS_PATH}/${_name}/uefi-vars.fd" + fi + + _opts="-Hwl bootrom,${_bootrom}" + _uefi="yes" +} + +# decide how to handle bhyve exit code +# +# @param string _var variable to put action into +# @param int _code bhyve exit code +# +vm::handle_exit(){ + local _var="$1" + local _code="$2" + + # check exit code + # get relevant action from config, defaulting to the behaviour + # we'd normally expect. we don't currently allow overriding shutdown + # as it makes it impossible to actually stop a guest cleanly + # + case "${_code}" in + 0) config::get "${_var}" "on_restart" "restart" ;; + 1) ;& + 2) if [ -e "${VM_DS_PATH}/${_name}/restart" ]; then + setvar "${_var}" "restart" + unlink "${VM_DS_PATH}/${_name}/restart" >/dev/null 2>&1 + else + setvar "${_var}" "shutdown" + fi + ;; + *) config::get "${_var}" "on_fault" "shutdown" ;; + esac +} + +# lock a vm +# stop another instance being started on this or another host +# we write hostname so vm-bhyve can inform user which host locked a vm +# +# @param string - the name of the guest to lock +# +vm::lock(){ + hostname > "${VM_DS_PATH}/${_name}/run.lock" +} + +# unlock a vm +# +# @param string - the name of the guest to unlock +# +vm::unlock(){ + unlink "${VM_DS_PATH}/${_name}/run.lock" >/dev/null 2>&1 + unlink "${VM_DS_PATH}/${_name}/console" >/dev/null 2>&1 +} + +# create string for guest com ports +# this builds the '-l comX' part of the bhyve command into _comstring +# _com is used by bhyveload|grub_bhyve so we set that to the first +# com port we come across. +# The nmdm devices are written to $vm_dir/{guest}/console so we can +# read them back later for the 'vm console' command +# +# @modifies _com _comstring +# +vm::bhyve_device_comports(){ + local _port _num=1 _tmux_name + local _port_name + + unlink "${VM_DS_PATH}/${_name}/console" >/dev/null 2>&1 + + for _port in ${_comports}; do + if [ ${_num} -eq 1 ]; then + # if tmux mode, log this to the console data + if [ -n "${VM_OPT_TMUX}" ]; then + _tmux_name=$(echo "${_name}" | tr "." "~") + echo "${_port}=tmux/${_tmux_name}" >> "${VM_DS_PATH}/${_name}/console" + fi + + # if foreground mode, we don't configure a serial port + if [ -n "${VM_OPT_FOREGROUND}" ]; then + _comstring="-l ${_port},stdio" + _num=$((_num + 1)) + continue + fi + fi + + # generate a port name unique to this vm and port number + _port_name="/dev/nmdm-${_name}.${_num}" + + # use first com port for the loader + [ ${_num} -eq 1 ] && _com="${_port_name}A" + + echo "${_port}=${_port_name}B" >> "${VM_DS_PATH}/${_name}/console" + _comstring="${_comstring}${_comstring:+ }-l ${_port},${_port_name}A" + _num=$((_num + 1)) + done +} + +# get bhyve device string for basic devices +# hostbridge & lpc on their own slots +# windows requires slot 0 & 31, nothing else cares fortunately +# +# @modifies _devices +# +vm::bhyve_device_basic(){ + + # add hostbridge + case "$_hostbridge" in + no*) ;; + amd) _devices="-s 0,amd_hostbridge" ;; + *) _devices="-s 0,hostbridge" ;; + esac + + # lpc + _devices="${_devices}${_devices:+ }-s 31,lpc" +} + +# get bhyve device string for disk devices +# read all disks starting at 0 and add to the _devices string +# this is done first so disks will start at slot 4. For uefi, +# we move through slots and stop at slot 6. For non-uefi we +# step through all functions and just keep going +# +# since r302459 the ahci controller supports +# up to 32 devices per controller. by default we set +# the device limit to 1 and use the original syntax, but +# this can be overridden by setting the ahci_device_limit +# guest option to an integer between 2 and 32. +# +# @modifies _devices _slot +# +vm::bhyve_device_disks(){ + local _disk _type _dev _path _opts _ahci _atype + local _ahci_num=0 _num=0 _add _ahci_multi + local _ahci_limit=1 + + # check if user has set a per-controller device limit + config::get "_ahci_multi" "ahci_device_limit" + + if [ ${VERSION_BSD} -ge 1200000 ] || \ + [ ${VERSION_BSD} -ge 1101000 ] || \ + [ ${VERSION_BSD} -ge 1004000 ]; then + + if [ -n "${_ahci_multi}" ]; then + + # see if it's numeric + echo "${_ahci_multi}" | egrep -iqs '^[0-9]+$' + + [ $? -eq 0 -a ${_ahci_multi} -gt 1 -a ${_ahci_multi} -le 32 ] && \ + _ahci_limit="${_ahci_multi}" + fi + fi + + # get all disks + while true; do + config::get "_disk" "disk${_num}_name" + config::get "_type" "disk${_num}_type" + [ -z "${_disk}" -o -z "${_type}" ] && break + + config::get "_dev" "disk${_num}_dev" + config::get "_opts" "disk${_num}_opts" + + if [ ${_func} -ge 8 ]; then + _func=0 + _slot=$((_slot + 1)) + fi + + vm::get_disk_path "_path" "${_name}" "${_disk}" "${_dev}" + + # ahci device and multi mode? + if [ ${_ahci_limit} -gt 1 -a "${_type%%-*}" = "ahci" ]; then + + # check device type + case "${_type}" in + ahci-cd) + _atype="cd" + ;& + ahci-hd) + [ -z "${_atype}" ] && _atype="hd" + _ahci="${_ahci},${_atype}:${_path}" + [ -n "${_opts}" ] && _ahci="${_ahci},${_opts}" + _ahci_num=$((_ahci_num + 1)) + + # we need to move to another controller if we get to the limit + if [ ${_ahci_num} -ge ${_ahci_limit} ]; then + _devices="${_devices} -s ${_slot}:${_func},ahci${_ahci}" + _ahci="" + _ahci_num=0 + _add=1 + fi + ;; + esac + + _atype="" + else + + _devices="${_devices} -s ${_slot}:${_func},${_type},${_path}" + [ -n "${_opts}" ] && _devices="${_devices},${_opts}" + _add=1 + fi + + # have we just added a device? + if [ -n "${_add}" ]; then + if [ -n "${_uefi}" ]; then + _slot=$((_slot + 1)) + + else + _func=$((_func + 1)) + fi + + _add="" + fi + + _num=$((_num + 1)) + done + + # have ahci devices left? + if [ -n "${_ahci}" ]; then + _devices="${_devices} -s ${_slot}:${_func},ahci${_ahci}" + [ -n "${_uefi}" ] && _slot=$((_slot + 1)) + fi + + # move to next slot if we have devices + # unless uefi as we already inc slot in uefi mode + if [ ${_num} -ge 1 -a -z "${_uefi}" ]; then + _slot=$((_slot + 1)) + _func=0 + fi +} + +# get bhyve device string for networking +# we dynamically create a new tap device for each interface +# if we can find the correct bridge, we then add the tap as a member +# we add each tap to __taplist from vm::run which it will +# use to desstroy them all on shutdown +# +# @modifies _devices _slot _taplist _func +# +vm::bhyve_device_networking(){ + local _emulation _num=0 + + while true; do + config::get "_emulation" "network${_num}_type" + [ -z "${_emulation}" ] && break + + # move slot if we've hit function 8 + if [ ${_func} -ge 8 ]; then + _func=0 + _slot=$((_slot + 1)) + fi + + switch::provision + _num=$((_num + 1)) + done + + if [ ${_num} -ge 1 ]; then + _slot=$((_slot + 1)) + _func=0 + fi +} + +# check if user wants a virtio-rand device +# +# @modifies _devices _slot +# +vm::bhyve_device_rand(){ + + if config::yesno "virt_random"; then + _devices="${_devices} -s ${_slot}:0,virtio-rnd" + _slot=$((_slot + 1)) + fi +} + +# add frame buffer output +# +vm::bhyve_device_fbuf(){ + local _port _listen _res _pass _vga _wait + local _fbuf_conf + + # only works in uefi mode + [ -z "${_uefi}" ] && return 0 + + # check graphics enabled + ! config::yesno "graphics" && return 0 + + # only available in 11+ + if [ ${VERSION_BSD} -lt 1100000 ]; then + util::log "guest" "${_name}" "warning; UEFI graphics is only available in FreeBSD 11 and newer" + return 1 + fi + + config::get "_port" "graphics_port" + config::get "_listen" "graphics_listen" + config::get "_res" "graphics_res" + config::get "_vga" "graphics_vga" + config::get "_pass" "vnc_password" + config::get "_wait" "graphics_wait" "auto" + + # check if graphics_wait is auto + # auto will count as yes so we need to unset it if we're not in an install. + [ "${_wait}" = "auto" -a -z "${_iso}" ] && _wait="no" + + # try to get port + # return if we can't + if [ -z "${_port}" ]; then + vm::find_available_net_port "_port" "5900" + + if [ -z "${_port}" ]; then + util::log "guest" "${_name}" "warning; unable to allocate a network port for graphics/vnc" + return 1 + fi + + util::log "guest" "${_name}" "dynamically allocated port ${_port} for vnc connections" + fi + + # add ip, port, resolution, wait + _fbuf_conf="tcp=${_listen:-0.0.0.0}:${_port}" + [ -n "${_res}" ] && _fbuf_conf="${_fbuf_conf},w=${_res%%x*},h=${_res##*x}" + [ -n "${_vga}" ] && _fbuf_conf="${_fbuf_conf},vga=${_vga}" + [ -n "${_pass}" ] && _fbuf_conf="${_fbuf_conf},password=${_pass}" + util::yesno "${_wait}" && _fbuf_conf="${_fbuf_conf},wait" + + # write vnc port to console data + echo "vnc=${_listen:-0.0.0.0}:${_port}" >> "${VM_DS_PATH}/${_name}/console" + + # add device + _devices="${_devices} -s ${_slot}:0,fbuf,${_fbuf_conf}" + _slot=$((_slot + 1)) +} + +# remove wait option if we're in auto mode +# fairly ugly code that uses sed to look for ",wait{word boundary}" +# and removes it. Hopefully this will never match anything else in +# the device string... +# +# @modifies _devices +# +vm::bhyve_device_fbuf_clear_wait(){ + local _wait + + [ -z "${_uefi}" ] && return + + config::get "_wait" "graphics_wait" "auto" + [ "${_wait}" = "auto" ] && _devices=$(echo "${_devices}" | sed 's@,wait[[:>:]]@@') +} + +# add a xhci mouse device +# +vm::bhyve_device_mouse(){ + + [ ${VERSION_BSD} -lt 1100000 ] && return 0 + + # add a tablet device if enabled + if config::yesno "xhci_mouse"; then + _devices="${_devices} -s ${_slot}:0,xhci,tablet" + _slot=$((_slot + 1)) + fi +} + +vm::bhyve_device_sound(){ + local _play _rec + config::get "_play" "sound_play" "/dev/dsp0" + config::get "_rec" "sound_rec" + + if config::yesno "sound"; then + _devices="${_devices} -s ${_slot}:0,hda,play=${_play}" + + if [ -n "${_rec}" ]; then + _devices="${_devices},rec=${_rec}" + fi + + _slot=$((_slot + 1)) + fi +} + +# add virtio_console devices to the guest +# +# @modifies _devices _slot +# +vm::bhyve_device_console(){ + local _console _curr=0 + local _dev_str + + [ ${VERSION_BSD} -lt 1102000 ] && return 0 + config::get "_console" "virt_console0" + [ -z "${_console}" ] && return 0 + + # add ports + while [ -n "${_console}" -a ${_curr} -lt 16 ]; do + # if set to "yes/on/1", just use the console number as port name + case "${_console}" in + [Yy][Ee][Ss]|[Oo][Nn]|1) _console="${_curr}" ;; + esac + + _dev_str="${_dev_str},${_console}=${VM_DS_PATH}/${_name}/vtcon.${_console}" + + _curr=$((_curr + 1)) + config::get "_console" "virt_console${_curr}" + done + + _devices="${_devices} -s ${_slot}:0,virtio-console${_dev_str}" + _slot=$((_slot + 1)) +} + +# get any pci passthrough devices +# FreeBSD 11 needs wired memory so update _opts in that case if +# we have any pass through devices +# +# @modifies _devices _slot _opts _wiredmem +# +vm::bhyve_device_passthru(){ + local _dev _orig_slot _func=0 + local _last_orig_slot + local _num=0 + + while true; do + config::get "_dev" "passthru${_num}" + [ -z "${_dev}" ] && break + + # see if there's an = sign + # we allow A/B/C=D:E to force D:E as the guest SLOT:FUNC + if echo "${_dev}" | grep -qs "="; then + _devices="${_devices} -s ${_dev##*=},passthru,${_dev%%=*}" + else + _orig_slot=${_dev%%/*} + + # only move to new slot if the original device is on a different slot to the last one. + # if user wants to passthru a device that has multiple functions which must stay together + # on one slot, they should be together in configuration file + if [ -n "${_last_orig_slot}" -a "${_last_orig_slot}" != "${_orig_slot}" ]; then + _slot=$((_slot + 1)) + _func=0 + fi + + _devices="${_devices} -s ${_slot}:${_func},passthru,${_dev}" + _last_orig_slot=${_orig_slot} + _func=$((_func + 1)) + fi + + _num=$((_num + 1)) + done + + if [ ${_num} -ge 1 ]; then + _slot=$((_slot + 1)) + + # add wired memory for 10.3+ + [ ${VERSION_BSD} -ge 1003000 ] && _opts="${_opts} -S" && _wiredmem="1" + fi +} + +# get the path to a disk image depending on type. +# the disk name in configuration is usually the name of the file +# or zvol, directly under the guest directory/dataset. +# if a user wants the disk image anywhere else, they can specify +# the following +# diskX_dev="custom" +# diskX_name="/full/path/to/disk/device/or/file" +# +# @param string _var variable to put disk path into +# @param string _name the name of the guest +# @param string _disk the name of the disk +# @param string _disk_dev=file|zvol|sparse-zvol|custom type of device +# +vm::get_disk_path(){ + local _var="$1" + local _name="$2" + local _disk="$3" + local _disk_dev="$4" + + case "${_disk_dev}" in + zvol) ;& + sparse-zvol) setvar "${_var}" "/dev/zvol/${VM_DS_ZFS_DATASET}/${_name}/${_disk}" ;; + custom) setvar "${_var}" "${_disk}" ;; + iscsi) info::__find_iscsi "${_var}" "${_disk}" ;; + *) setvar "${_var}" "${VM_DS_PATH}/${_name}/${_disk}" ;; + esac +} + +# looks for a prestart setting and runs it if possible +# this will execute before each start/reboot +# +# script is given the guest name and full path as arguments +# +vm::prestart(){ + local _script + + config::get "_script" "prestart" || return 0 + echo "${_script}" | grep -qs '^/' + [ $? -ne 0 ] && _script="${VM_DS_PATH}/${_name}/${_script}" + [ ! -x "${_script}" ] && return 1 + + util::log "guest" "${_name}" "running prestart script ${_script} ${_name} ${VM_DS_ZFS_DATASET}/${_name}" + + ( + cd "${VM_DS_PATH}/${_name}" + exec "${_script}" "${_name}" "${VM_DS_ZFS_DATASET}/${_name}" + ) +} + +# get a list of all running virtual machines +# this loads a list of all running guests into global variables +# both variables are in the following format with one guest per line +# "pid guest" +# +# VM_RUN_BHYVE = all running bhyve instances +# VM_RUN_LOAD = all loading guests (bhyveload|grub-bhyve) +# +# @modifies VM_RUN_BHYVE VM_RUN_LOAD +# +vm::running_load(){ + VM_RUN_BHYVE=$(pgrep -fl "bhyve:" | awk '{print $1,$NF}') + VM_RUN_LOAD=$(pgrep -fl "grub-bhyve|bhyveload" | awk '{print $1,$NF}') +} + +# see if a specific virtual machine is running +# we search the VM_RUN_BHYVE and VM_RUN_LOAD variables looking for the +# specified guest. If found we output "Running" or "Bootloader", along with +# the PID. If not running we output "Stopped" +# +# @param string _var variable to put result into +# @param string _name name of the guest to look for +# @return success if running +# +vm::running_check(){ + local _var="$1" + local _var2="$2" + local _name="$3" + local IFS=$'\n' + local _entry + + for _entry in ${VM_RUN_BHYVE}; do + if [ "${_entry##* }" = "${_name}" ]; then + setvar "${_var}" "Running (${_entry%% *})" + setvar "${_var2}" "${_entry%% *}" + return 0 + fi + done + + for _entry in ${VM_RUN_LOAD}; do + if [ "${_entry##* }" = "${_name}" ]; then + setvar "${_var}" "Bootloader (${_entry%% *})" + setvar "${_var2}" "${_entry%% *}" + return 0 + fi + done + + setvar "${_var}" "Stopped" + return 1 +} + +# make sure a virtual machine is not running +# display warning if it is. up to caller to decide whether to continue. +# on unclean shutdown, lock files may not be cleared up by vm-bhyve, +# we now provide option to skip the lock file if it references this host, +# and there is no /dev/vmm/{_name}. +# +# @param string _name the guest name +# @param int _skip_lock skip local lock file if no /dev/vmm found +# @return success if not running +# +vm::confirm_stopped(){ + local _name="$1" + local _skip_lock="$2" + local _host _our_host + + _our_host=$(hostname) + + # check vm-bhyve lock + # this will err even if guest is running on another node + if [ -e "${VM_DS_PATH}/${_name}/run.lock" ]; then + _host=$(head -n 1 "${VM_DS_PATH}/${_name}/run.lock") + + if [ "${_host}" != "${_our_host}" -o "${_skip_lock}" != "1" ]; then + util::warn "${_name} appears to be running on ${_host} (locked)" + return 2 + fi + fi + + # check local machine, just in case guest run manually + if [ -e "/dev/vmm/${_name}" ]; then + util::warn "${_name} appears to be running locally (vmm exists)" + return 1 + fi + + return 0 +} + +# generate a static mac address for a guest. +# we now make sure all interfaces have a static mac so +# there's no worry of guest problems due to it changing. +# bhyve is allocated 58:9c:fc:0x:xx:xx, so we try and +# randomise the last 20 bits (5 hex digits). +# using guest name, interface number and time as seed, +# with an incrementing integer just in case we happen to +# get 5 zeros. +# +vm::generate_static_mac(){ + local _time=$(date +%s) + local _key _part="0:00:00" + local _base _int=0 + + _base="${_name}:${_num}:${_time}" + + # generate the last 5 digits + # make sure we don't get 0:00:00 (see sys/net/ieee_oui.h) + while [ "${_part}" = "0:00:00" ]; do + _key="${_base}:${_int}" + _part=$(md5 -qs "${_key}" | awk 'BEGIN {FS="";OFS=":"}; {print $1,$2$3,$4$5}') + _int=$((_int + 1)) + done + + # add base bhyve OUI + _mac="58:9c:fc:0${_part}" + + util::log "guest" "${_name}" "generated static mac ${_mac} (based on '${_key}')" + config::set "${_name}" "network${_num}_mac" "${_mac}" +} + +# try to find an available network port to listen on +# this isn't clever enough to diffeential ports on unique ip's yet +# +# @param string _var variable to put port into +# @param int _curr the port to start searching from +# @return true if we found a port +# +vm::find_available_net_port(){ + local _var="$1" + local _curr="$2" + local _open _line _max _used + local IFS=$'\n' + + # stop searching after 200 ports + _max=$((_curr + 200)) + + # get list of open ports + _open=$(netstat -an | grep LISTEN | awk '{print $4}' | awk -F. '{print $NF}') + + while [ "${_curr}" -lt "${_max}" ]; do + + # reset used flag + _used="" + + # see if current port is used + for _line in ${_open}; do + [ "${_line}" = "${_curr}" ] && _used="1" + done + + # this port available? + if [ -z "${_used}" ]; then + setvar "${_var}" "${_curr}" + return 0 + fi + + _curr=$((_curr + 1)) + done + + # not found a port + setvar "${_var}" "" + return 1 +} + +# remove any values from configuration that should be unique +# such as uuid and mac addresses. these should be generated +# automatically at runtime +# +# used for cloning or imaging existing guests +# should this remove devices like passthru that cannot be shared? +# +# @param string _name name of guest to generalise +# +vm::generalise(){ + local _name="$1" + local _entry _num=0 + + config::load "${VM_DS_PATH}/${_name}/${_name}.conf" + config::remove "${_name}" "uuid" + + while true; do + config::get "_entry" "network${_num}_mac" + [ -z "${_entry}" ] && break + + config::remove "${_name}" "network${_num}_mac" + _num=$((_num + 1)) + done +} + +# get number of cpus from the configuration file, and also +# look for sockets/cores/threads options. +# returns the bhyve cpu config string +# +# @param string _var variable to put cpu configuration into +# +vm::__cpu(){ + local _var="$1" + local _c_cpu _c_socket _c_core _c_thread + local _config + + config::get "_c_cpu" "cpu" + [ -n "${_c_cpu}" ] || util::err "fatal; unable to determine cpu count" + _config="${_c_cpu}" + + config::get "_c_socket" "cpu_sockets" + config::get "_c_core" "cpu_cores" + config::get "_c_thread" "cpu_threads" + + [ -n "${_c_socket}" ] && _config="${_config},sockets=${_c_socket}" + [ -n "${_c_core}" ] && _config="${_config},cores=${_c_core}" + [ -n "${_c_thread}" ] && _config="${_config},threads=${_c_thread}" + + setvar "${_var}" "${_config}" +} diff --git a/lib/vm-switch b/lib/vm-switch new file mode 100644 index 0000000..5acaa4b --- /dev/null +++ b/lib/vm-switch @@ -0,0 +1,467 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# switch libraries +# +. "${LIB}/vm-switch-netgraph" +. "${LIB}/vm-switch-manual" +. "${LIB}/vm-switch-standard" +. "${LIB}/vm-switch-vale" +. "${LIB}/vm-switch-vxlan" + +# create switches from rc list on init +# this should run once per boot to make sure switches from the +# configuration file have bridge interfaces. If any new switches are +# created, the create function takes care of setting them up +# +switch::init(){ + local _switchlist _switch _type + + config::core::get "_switchlist" "switch_list" + + if [ -n "${_switchlist}" ]; then + for _switch in ${_switchlist}; do + # get the switch type + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + vxlan) switch::vxlan::init "${_switch}" ;; + manual) switch::manual::init "${_switch}" ;; + vale) ;; + netgraph) ;; + *) switch::standard::init "${_switch}" ;; + esac + done + fi +} + +# list switches configured +# +switch::list(){ + local _switchlist _switch _type + local _id _format="%s^%s^%s^%s^%s^%s^%s^%s\n" + + config::core::get "_switchlist" "switch_list" + + { + printf "${_format}" "NAME" "TYPE" "IFACE" "ADDRESS" "PRIVATE" "MTU" "VLAN" "PORTS" + + if [ -n "${_switchlist}" ]; then + for _switch in ${_switchlist}; do + # get the switch type + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + netgraph) switch::netgraph::show "${_switch}" "${_format}" ;; + vale) switch::vale::show "${_switch}" "${_format}" ;; + vxlan) switch::vxlan::show "${_switch}" "${_format}" ;; + manual) switch::manual::show "${_switch}" "${_format}" ;; + *) switch::standard::show "${_switch}" "${_format}" ;; + esac + done + fi + } | column -ts^ +} + +# create a new virtual switch +# +# @param string _switch name of the switch to create +# +switch::create(){ + local _switch + local _type="standard" + local _list _curr _vlan _if _bridge _addr _mtu _priv + + # process options + while getopts t:i:n:b:a:m:p _opt; do + case ${_opt} in + t) _type="${OPTARG}" ;; + i) _if="${OPTARG}" ;; + n) _vlan="${OPTARG}" ;; + b) _bridge="${OPTARG}" ;; + a) _addr="${OPTARG}" ;; + m) _mtu="${OPTARG}" ;; + p) _priv="yes" ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _switch="$1" + + # check for a valid switch name + util::check_name "${_switch}" || util::err "invalid switch name - '${_switch}'" + + # make sure it's not an existing name + config::core::get "_list" "switch_list" + + for _curr in ${_list}; do + [ "${_switch}" = "${_curr}" ] && util::err "switch ${_switch} already exists" + done + + # check vlan number + if [ -n "${_vlan}" ]; then + echo "${_vlan}" | egrep -qs '^[0-9]{1,4}$' + [ $? -eq 0 ] || util::err "invalid vlan number" + [ ${_vlan} -ge 4095 ] && util::err "invalid vlan number" + fi + + # check address + if [ -n "${_addr}" ]; then + echo "${_addr}" | egrep -qs '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/[0-9]{1,2}$' + [ $? -eq 0 ] || util::err "address must be supplied in CIDR notation (a.b.c.d/prefix-len)" + fi + + # check mtu + if [ -n "${_mtu}" ]; then + echo "${_mtu}" | egrep -qs '^[0-9]{3,4}$' + [ $? -eq 0 ] || util::err "invalid mtu" + [ ${_mtu} -gt 9000 ] && util::err "invalid mtu" + fi + + # check switch type + case "${_type}" in + standard) switch::standard::create ;; + manual) switch::manual::create ;; + netgraph) switch::netgraph::create ;; + vale) switch::vale::create ;; + vxlan) switch::vxlan::create ;; + *) util::err "invalid switch type - '${_type}'" ;; + esac +} + +# destroy a switch +# remove from configuration and unload any interfaces we created +# +# @param string _switch name of the switch to remove +# +switch::remove(){ + local _switch="$1" + local _type + + [ -z "${_switch}" ] && util::usage + + # get the type of switch + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + standard) switch::standard::remove "${_switch}" ;; + manual) switch::manual::remove "${_switch}" ;; + netgraph) switch::netgraph::remove "${_switch}" ;; + vale) switch::vale::remove "${_switch}" ;; + vxlan) switch::vxlan::remove "${_switch}" ;; + *) util::err "unable to remove switch of unknown type" ;; + esac + + # remove all configuration if there's no error + if [ $? -eq 0 ]; then + config::core::remove "switch_list" "${_switch}" + config::core::remove "ports_${_switch} vlan_${_switch} nat_${_switch} type_${_switch}" + config::core::remove "addr_${_switch} private_${_switch} mtu_${_switch}" + else + util::err "failed to remove virtual switch" + fi + + # make sure the exit status indicates success, + # even if config::core::remove did not + return 0 +} + +# add a new interface to a switch +# +# @param string _switch name of the switch +# @param string _if the interface to add +# +switch::add_member(){ + local _switch="$1" + local _if="$2" + local _type + + [ -z "${_switch}" -o -z "${_if}" ] && util::usage + + # get the type of switch + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + standard) switch::standard::add_member "${_switch}" "${_if}" ;; + manual) switch::manual::add_member "${_switch}" "${_if}" ;; + netgraph) switch::netgraph::add_member "${_switch}" "${_if}" ;; + vale) switch::vale::add_member "${_switch}" "${_if}" ;; + vxlan) switch::vxlan::add_member "${_switch}" "${_if}" ;; + *) util::err "unable to configure switch of unknown type" ;; + esac +} + +# remove a member interface from a virtual switch +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::remove_member(){ + local _switch="$1" + local _if="$2" + local _type + + [ -z "${_switch}" -o -z "${_if}" ] && util::usage + + # get the type of switch + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + standard) switch::standard::remove_member "${_switch}" "${_if}" ;; + manual) switch::manual::remove_member "${_switch}" "${_if}" ;; + netgraph) switch::netgraph::remove_member "${_switch}" "${_if}" ;; + vale) switch::vale::remove_member "${_switch}" "${_if}" ;; + vxlan) switch::vxlan::remove_member "${_switch}" "${_if}" ;; + *) util::err "unable to configure switch of unknown type" ;; + esac +} + +# change the vlan number on a virtual switch +# +# @param string _switch name of the switch +# @param int _vlan the vlan number (0 to turn vlan off) +# +switch::vlan(){ + local _switch="$1" + local _vlan="$2" + local _id _type + + [ -z "${_switch}" -o -z "${_vlan}" ] && util::usage + + switch::id "_id" "${_switch}" + switch::type "_type" "${_switch}" + [ -z "${_id}" ] && util::err "unable to locate specified virtual switch" + + echo "${_vlan}" | egrep -qs '^[0-9]{1,4}$' + [ $? -eq 0 ] || util::err "invalid vlan number" + [ ${_vlan} -ge 4095 ] && util::err "invalid vlan number" + + case "${_type}" in + standard) switch::standard::vlan "${_switch}" "${_vlan}" ;; + manual) switch::manual::vlan "${_switch}" "${_vlan}" ;; + netgraph) switch::netgraph::vlan "${_switch}" "${_vlan}" ;; + vale) switch::vale::vlan "${_switch}" "${_vlan}" ;; + vxlan) switch::vxlan::vlan "${_switch}" "${_vlan}" ;; + *) util::err "unable to configure switch of unknown type" ;; + esac +} + +# enable or diable private flag on a switch +# note that we don't update existing interfaces; this +# makes things easy for us and any guests booted after +# will get the new setting +# +# @param string _switch the switch to update +# @param string _priv on,yes|off,no +# +switch::private(){ + local _switch="$1" + local _priv="$2" + local _type + + # try to get switch type + [ -z "${_switch}" -o -z "${_priv}" ] && util::usage + switch::type "_type" "${_switch}" || util::err "specified switch does not appear to be valid" + + case "${_type}" in + standard|manual|vxlan) + if util::yesno "${_priv}"; then + config::core::set "private_${_switch}" "yes" + else + config::core::set "private_${_switch}" "no" + fi + ;; + netgraph) + util::err "unable to configure private mode on netgraph switches" + ;; + vale) + util::err "unable to configure private mode on vale switches" + ;; + *) + util::err "unable to configure switch of unknown type" + ;; + esac +} + +# enable or disable nat functionality on a virtual switch +# +# @param string _switch name of the switch +# @param string _nat on|off +# +switch::nat(){ + util::warn "internal nat support is currently disabled" + util::warn "please add an address to the virtual switch and configure your firewall for NAT manually" +} + +# set or remove ip address from a virtual switch +# +# @param string _switch name of the switch +# @param string _addr the ip address to add (or "none" to remove +# +switch::address(){ + local _switch="$1" + local _addr="$2" + local _id _type + + [ -z "${_switch}" ] && util::usage + + switch::id "_id" "${_switch}" + switch::type "_type" "${_switch}" + [ -z "${_id}" ] && util::err "unable to locate specified virtual switch" + + # check address + if [ -n "${_addr}" ] && [ "${_addr}" != "none" ]; then + echo "${_addr}" | egrep -qs '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/[0-9]{1,2}$' + [ $? -eq 0 ] || util::err "address must be supplied in CIDR notation (a.b.c.d/prefix-len)" + fi + + case "${_type}" in + standard) switch::standard::address "${_switch}" "${_addr}" ;; + manual) ;& + netgraph) ;& + vale) ;& + vxlan) util::err "feature not currently supported on switches of this type" ;; + *) util::err "unable to configure switch of unknown type" ;; + esac +} + +# return the type for a switch +# +# @param string _var variable to put type into +# @param string _switch the switch name +# +switch::type(){ + local _var="$1" + local _switch="$2" + + [ -z "${_switch}" ] && return 1 + config::core::get "${_var}" "type_${_switch}" "standard" +} + +# check if a switch is configured for private members +# +# @param string _switch switch name +# @return succes (0) if it's private +# +switch::is_private(){ + local _switch="$1" + local _priv + + config::core::get "_priv" "private_${_switch}" + util::yesno "${_priv}" +} + +# get the bridge id for a virtual switch +# +# @param string _var variable to put name into +# @param string _switch the name of the switch +# +switch::id(){ + local _var="$1" + local _switch="$2" + local _type + + [ -z "${_switch}" ] && return 1 + + # get switch type + config::core::get "_type" "type_${_switch}" + + case "${_type}" in + vale) switch::vale::id "${_var}" "${_switch}" ;; + netgraph) switch::netgraph::id "${_var}" "${_switch}" ;; + manual) switch::manual::id "${_var}" "${_switch}" ;; + *) switch::standard::id "${_var}" "${_switch}" ;; + esac +} + +# get a virt interface id for a port/switch +# +# @param string _var variable name to put result into +# @param string _switch switch name to get id for +# +switch::__viid(){ + local _hash=$(md5 -qs "${2}" | cut -c1-5) + setvar "$1" "viid-${_hash}@" +} + +# retrieve interface name, given a switch name +# we convert to viid then look for the matching group +# +# @param string _var variable to put interface name into +# @param string _switch the switch name +# +switch::find(){ + local _var="$1" + local _switch="$2" + local _viid _name + + switch::__viid "_viid" "${_switch}" + _name=$(ifconfig -g "${_viid}" 2>/dev/null) + [ -n "${_name}" ] && setvar "${_var}" "${_name}" +} + +# mark an interface with a unique viid +# i say unique, its 5 chars from an md5 hash which +# should be enough for half a dozen switches +# +# @parem string _switch switch name +# @param string _iface interface to mark +# +switch::set_viid(){ + local _switch="$1" + local _iface="$2" + local _viid + + switch::__viid "_viid" "${_switch}" + ifconfig "${_iface}" group "${_viid}" >/dev/null 2>&1 +} + +# create a network interface for a guest +# relies heavily on variables set in the main vm::run function +# +# @modifies _func _devices +# +switch::provision(){ + local _switch _mac _type + + config::get "_switch" "network${_num}_switch" + config::get "_mac" "network${_num}_mac" + + # set a static mac if we don't have one + [ -z "${_mac}" ] && vm::generate_static_mac + + switch::type "_type" "${_switch}" + + case "${_type}" in + vale) switch::vale::provision ;; + netgraph) switch::netgraph::provision ;; + standard) ;& + manual) ;& + vxlan) switch::standard::provision ;; + *) util::warn "unable to configure interface ${_num}" ;; + esac +} diff --git a/lib/vm-switch-manual b/lib/vm-switch-manual new file mode 100644 index 0000000..8ebafda --- /dev/null +++ b/lib/vm-switch-manual @@ -0,0 +1,142 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# for a manual switch the bridge interface should already exist +# we just assign our description so it's visible in ifconfig +# +# @param string _name name of the switch +# +switch::manual::init(){ + local _name="$1" + local _bridge="$2" + + # get bridge + if [ -z "${_bridge}" ]; then + config::core::get "_bridge" "bridge_${_name}" + [ -z "${_bridge}" ] && return 1 + fi + + # don't rename custom bridges nor set a description. + # manual interfaces are fully configured using rc.conf. + ifconfig "${_bridge}" group vm-switch up >/dev/null 2>&1 + switch::set_viid "${_name}" "${_bridge}" +} + +# show the configuration details for a manual switch +# +# @param string _name the switch name +# @param string _format output format +# +switch::manual::show(){ + local _name="$1" + local _format="$2" + local _bridge _priv + + config::core::get "_bridge" "bridge_${_name}" + config::core::get "_priv" "private_${_name}" "no" + printf "${_format}" "${_name}" "manual" "${_bridge}" "n/a" "${_priv}" "n/a" "n/a" "n/a" +} + +# create a manual switch +# we just assign our switch name to the existing bridge +# +switch::manual::create(){ + + # we need to have a bridge + [ -z "${_bridge}" ] && util::err "you must specify a bridge to import when creating a manual switch" + + # check we can find this bridge on the system + ifconfig "${_bridge}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "${_bridge} does not appear to be a valid existing bridge" + + # store configuration + config::core::set "switch_list" "${_switch}" "1" + config::core::set "type_${_switch}" "manual" + config::core::set "bridge_${_switch}" "${_bridge}" + + [ -n "${_priv}" ] && config::core::set "private_${_switch}" "${_priv}" + + # import + switch::manual::init "${_switch}" "${_bridge}" +} + +# remove a manual switch +# +# @param string _switch the name of the switch +# +switch::manual::remove(){ + local _switch="$1" + local _id + + switch::manual::id "_id" "${_switch}" + [ -z "${_id}" ] && return 0 + + # try to remove our description + # viid stays but it's not worth the extra hassle to remove that + ifconfig ${_id} -descr -group vm-switch >/dev/null 2>&1 +} + +# add a new interface to this switch +# manual switches should be managed by the user +# using rc.conf, hence "manual" +# +# @param string _switch name of the switch +# @param string _if the interface to add +# +switch::manual::add_member(){ + util::err "manual switches and member interfaces should be configured using /etc/rc.conf" +} + +# remove an interface +# manual switches should be managed by the user +# using rc.conf, hence "manual" +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::manual::remove_member(){ + util::err "manual switches and member interfaces should be configured using /etc/rc.conf" +} + +# set vlan id +# +# @param string _switch name of switch +# @param int _vlan vlan id to set +# +switch::manual::vlan(){ + util::err "manual switches and member interfaces should be configured using /etc/rc.conf" +} + +# get id for a manual switch +# in this case we need to return the bridge name +# +# @param string _var variable to put id into +# @param string _switch switch name +# @return 0 if switch id found +# +switch::manual::id(){ + config::core::get "$1" "bridge_$2" +} diff --git a/lib/vm-switch-netgraph b/lib/vm-switch-netgraph new file mode 100644 index 0000000..8b70c0d --- /dev/null +++ b/lib/vm-switch-netgraph @@ -0,0 +1,119 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2021 Benoit Chesneau (bchesneau@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + + +# show the configuration details for a netgraph switch +# +# @param string _name the switch name +# @param string _format output format +# +switch::netgraph::show(){ + local _name="$1" + local _format="$2" + local _id + + switch::netgraph::id "_id" "${_name}" + printf "${_format}" "${_name}" "netraph" "${_id}" "n/a" "n/a" "n/a" "n/a" "n/a" +} + +# create a netgraph switch +# +# @param string _switch the name of the switch +# +switch::netgraph::create(){ + config::core::set "switch_list" "${_switch}" "1" + config::core::set "type_${_switch}" "netgraph" +} + +# remove a netgraph switch +# +switch::netgraph::remove(){ } + +# add a new interface to this switch +# at the moment we require the user to manually +# set up any netgraph switches +# +# @param string _switch name of the switch +# @param string _if the interface to add +# +switch::netgraph::add_member(){ + util::err "physical interfaces must be added to the netgraph switch manually" +} + +# remove an interface +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::netgraph::remove_member(){ + util::err "physical interfaces must be removed from the netgraph switch manually" +} + +# set vlan id +# +# @param string _switch name of switch +# @param int _vlan vlan id to set +# +switch::netgraph::vlan(){ + util::err "vlan support is not currently implemented for netgraph switches" +} +# gets a unique linkname name for a ng_bridge interface +# we need to make sure the link is unique and the last one +# +# @param string _var name of variable to put port name into +# @param string _switch the name of the switch +# +switch::netgraph::id(){ + local _var="$1" + local _switch="$2" + + # Create a new interface to the bridge + num=2 + while ngctl msg "${_switch}:" getstats $num > /dev/null 2>&1 + do + num=$(( $num + 1 )) + done + setvar "${_var}" "netgraph,path=${_switch}:,peerhook=link$num" +} + +# create a netgraph interface for a guest +# relies heavily on variables set in the main vm::run function +# +# @modifies _func _devices +# @return 1 if we don't get a tap device +# +switch::netgraph::provision(){ + local _ngid + + # create a netgraph peer + switch::netgraph::id "_ngid" "${_switch}" + + util::log "guest" "${_name}" "adding netgraph interface ${_ngid} (${_switch})" + _devices="${_devices} -s ${_slot}:${_func},${_emulation},${_ngid}" + [ -n "${_mac}" ] && _devices="${_devices},mac=${_mac}" + + _func=$((_func + 1)) +} diff --git a/lib/vm-switch-standard b/lib/vm-switch-standard new file mode 100644 index 0000000..3e0f1a8 --- /dev/null +++ b/lib/vm-switch-standard @@ -0,0 +1,405 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# creaate bridge interface for a standard switch +# +# @param string _name name of the switch +# +switch::standard::init(){ + local _name="$1" + local _id _addr _mtu _len _ifconf + + # see if it already exists + switch::standard::id "_id" "${_name}" && return 0 + + # get the length of the switch name + # it's useful for other utilities to use switch name as interface name + # as it's static. can't do that if it's > 12 chars + _len=$(echo -n "${_name}" | wc -m) + + if [ ${_len} -le 12 ]; then + _ifconf="name vm-${_name}" + else + _ifconf="descr vm/${_name}" + fi + + # create a bridge for this switch + _id=$(ifconfig bridge create ${_ifconf} group vm-switch up 2>/dev/null) + [ $? -eq 0 ] || util::err "failed to create bridge interface for switch ${_name}" + + switch::set_viid "${_name}" "${_id}" + + # randomise mac if feature is available + [ ${VERSION_BSD} -ge 1102000 ] && ifconfig "${_id}" link random + + # try to set ip address + config::core::get "_addr" "addr_${_name}" + [ -n "${_addr}" ] && ifconfig "${_id}" inet ${_addr} 2>/dev/null + config::core::get "_addr" "addr6_${_name}" + [ -n "${_addr}" ] && ifconfig "${_id}" inet6 ${_addr} 2>/dev/null + + # custom mtu? + config::core::get "_mtu" "mtu_${_name}" + [ -n "${_mtu}" ] && ifconfig "${_id}" mtu ${_mtu} + + # add member interfaces + switch::standard::__add_members "${_name}" "${_id}" +} + +# show the configuration details for a switch +# +# @param string _name the switch name +# @param string _format output format +# +switch::standard::show(){ + local _name="$1" + local _format="$2" + local _id _vlan _ports _addr _mtu _priv + + switch::find "_id" "${_name}" + config::core::get "_ports" "ports_${_name}" + config::core::get "_vlan" "vlan_${_name}" + config::core::get "_addr" "addr_${_name}" + config::core::get "_mtu" "mtu_${_name}" + config::core::get "_priv" "private_${_name}" "no" + + printf "${_format}" "${_name}" "standard" "${_id:--}" "${_addr:--}" "${_priv}" "${_mtu:--}" \ + "${_vlan:--}" "${_ports:--}" +} + +# create a standard virtual switch +# +switch::standard::create(){ + + # store configuration + config::core::set "switch_list" "${_switch}" "1" + config::core::set "type_${_switch}" "standard" + + [ -n "${_if}" ] && config::core::set "ports_${_switch}" "${_if}" + [ -n "${_vlan}" ] && config::core::set "vlan_${_switch}" "${_vlan}" + [ -n "${_addr}" ] && config::core::set "addr_${_switch}" "${_addr}" + [ -n "${_priv}" ] && config::core::set "private_${_switch}" "${_priv}" + [ -n "${_mtu}" ] && config::core::set "mtu_${_switch}" "${_mtu}" + + config::core::load + switch::standard::init "${_switch}" +} + +# destroy a standard switch +# +# @param string _switch name of the switch to destroy +# +switch::standard::remove(){ + local _switch="$1" + local _id + + # get the bridge id + switch::standard::id "_id" "${_switch}" + [ $? -eq 0 ] || return 1 + + # remove all member interfaces + switch::standard::__remove_members "${_switch}" "${_id}" + + # destroy the bridge + ifconfig "${_id}" destroy >/dev/null 2>&1 +} + +# add a new interface to this switch +# +# @param string _switch name of the switch +# @param string _if the interface to add +# +switch::standard::add_member(){ + local _switch="$1" + local _if="$2" + local _id _vlan _mtu + + switch::standard::id "_id" "${_switch}" || util::err "unable to locate switch id" + config::core::get "_vlan" "vlan_${_switch}" + config::core::get "_mtu" "mtu_${_switch}" + switch::standard::__configure_port "${_switch}" "${_id}" "${_if}" "${_vlan}" "${_mtu}" + config::core::set "ports_${_switch}" "${_if}" "1" +} + +# remove a member interface from this switch +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::standard::remove_member(){ + local _switch="$1" + local _if="$2" + local _id _vlan + + switch::standard::id "_id" "${_switch}" || util::err "unable to locate switch id" + config::core::remove "ports_${_switch}" "${_if}" + config::core::get "_vlan" "vlan_${_switch}" + switch::standard::__unconfigure_port "${_switch}" "${_id}" "${_if}" "${_vlan}" +} + +# set vlan id +# +# @param string _switch name of switch +# @param int _vlan vlan id to set +# +switch::standard::vlan(){ + local _switch="$1" + local _vlan="$2" + local _id + + switch::standard::id "_id" "${_switch}" || util::err "unable to locate switch id" + switch::standard::__remove_members "${_switch}" "${_id}" + + # update configuration + if [ "${_vlan}" = "0" ]; then + config::core::remove "vlan_${_switch}" + else + config::core::set "vlan_${_switch}" "${_vlan}" + fi + + config::core::load + switch::standard::__add_members "${_switch}" "${_id}" +} + +# set or remove ip address +# +# @param string _swtich name of the switch +# @param string _addr address or "none" +# @scope _id switch if from parent switch::address +# +switch::standard::address(){ + local _switch="$1" + local _addr="$2" + local _curr + + if [ "${_addr}" = "none" ]; then + + config::core::get "_curr" "addr_${_switch}" + [ $? -eq 0 ] || util::err "unable to locate an existing address for this switch" + + config::core::remove "addr_${_switch}" + ifconfig "${_id}" "${_curr}" delete + else + + config::core::set "addr_${_switch}" "${_addr}" + ifconfig "${_id}" "${_addr}" + fi +} + +# add all member interfaces to a switch +# +# @param string _switch the name of the switch +# @param string _id interface id for the switch +# +switch::standard::__add_members(){ + local _switch="$1" + local _id="$2" + local _ports _vlan _port _mtu + + # get the id if not provided + if [ -z "${_id}" ]; then + switch::standard::id "_id" "${_switch}" || util:err "failed to get switch id while adding members" + fi + + config::core::get "_ports" "ports_${_switch}" + config::core::get "_vlan" "vlan_${_switch}" + config::core::get "_mtu" "mtu_${_switch}" + + if [ -n "${_ports}" ]; then + for _port in ${_ports}; do + switch::standard::__configure_port "${_switch}" "${_id}" "${_port}" "${_vlan}" "${_mtu}" + done + fi +} + +# remove member interfaces from a switch +# +# @param string _switch the name of the switch +# @param string _id bridge id if already known +# +switch::standard::__remove_members(){ + local _switch="$1" + local _id="$2" + local _ports _port _vlan + + # get id if not given to us + if [ -z "${_id}" ]; then + switch::standard::id "_id" "${_switch}" + [ $? -eq 0 ] || util::err "failed to get switch id while removing members" + fi + + # get full port list + config::core::get "_ports" "ports_${_switch}" + config::core::get "_vlan" "vlan_${_switch}" + + if [ -n "${_ports}" ]; then + for _port in ${_ports}; do + switch::standard::__unconfigure_port "${_switch}" "${_id}" "${_port}" "${_vlan}" + done + fi +} + +# configure a local port for our bridge +# +# @param string _switch the switch to add port to +# @param string _id the bridge id of the switch +# @param string _port the interface to add +# @param int _vlan vlan number if assigned to this switch +# @param int _mtu custom mtu to use for this port +# +switch::standard::__configure_port(){ + local _switch="$1" + local _id="$2" + local _port="$3" + local _vlan="$4" + local _mtu="$5" + local _viface _vname + + # try to set mtu of port? + [ -n "${_mtu}" ] && ifconfig "${_port}" mtu ${_mtu} >/dev/null 2>&1 + + # vlan enabled? + if [ -n "${_vlan}" ]; then + + # see if vlan interface already exists + _vname="${_port}.${_vlan}" + switch::standard::id "_viface" "${_vname}" + + # create if needed + if [ $? -ne 0 ]; then + # use our id as the interface name here. + # it should always be a valid name and interface.vlan-id is much easier to understand in ifconfig + # than a bunch of vlanX interfaces + _viface=$(ifconfig vlan create vlandev "${_port}" vlan "${_vlan}" descr "vm-vlan/${_switch}/${_vname}" name "${_vname}" group vm-vlan up 2>/dev/null) + [ $? -eq 0 ] || util::err "failed to create vlan interface for port ${_port} on switch ${_switch}" + fi + + switch::set_viid "${_vname}" "${_viface}" + ifconfig ${_id} addm ${_viface} >/dev/null 2>&1 + else + # add to bridge, nice and simple :) + ifconfig ${_id} addm ${_port} >/dev/null 2>&1 + fi + + [ $? -eq 0 ] || util::err "failed to add member ${_port} to the virtual switch ${_switch}" +} + +# unconfigure a local port +# +# @param string _switch the switch to remove port from +# @param string _id the bridge id of the switch +# @param string _port the interface to remove +# @param string _vlan vlan number if assigned to this switch +# +switch::standard::__unconfigure_port(){ + local _switch="$1" + local _id="$2" + local _port="$3" + local _vlan="$4" + local _vid + + if [ -n "${_vlan}" ]; then + # get vlan interface + switch::standard::id "_vid" "${_port}.${_vlan}" + + # remove the vlan interface, it will be removed from bridge automatically + [ $? -eq 0 ] && ifconfig ${_vid} destroy >/dev/null 2>&1 + else + ifconfig ${_id} deletem ${_port} >/dev/null 2>&1 + fi +} + +# get the id for a standard switch +# this returns the associated bridge name +# +# @param string _var variable to put id into +# @param string _switch the switch to look for +# @return 0 on success +# +switch::standard::id(){ + switch::find "$1" "$2" +} + +# creates a standard tap interface for a guest +# relies heavily on variables set in the main vm::run function +# +# @modifies _func _devices +# @return 1 if we don't get a tap device +# +switch::standard::provision(){ + local _tap _custom_tap _sid _mtu _member_type _iname + + config::get "_custom_tap" "network${_num}_device" + config::get "_iname" "network${_num}_name" + + # create interface + if [ -n "${_custom_tap}" ]; then + _tap="${_custom_tap}" + elif [ -n "${_iname}" ]; then + _tap=$(ifconfig tap create name "${_iname}") + else + _tap=$(ifconfig tap create) + fi + + [ -z "${_tap}" ] && return 1; + + util::log "guest" "${_name}" "initialising network device ${_tap}" + ifconfig "${_tap}" descr "vmnet/${_name}/${_num}/${_switch:-custom}" group vm-port >/dev/null 2>&1 + + if [ -n "${_switch}" ]; then + switch::id "_sid" "${_switch}" + + # should this be a span member? + _member_type="addm" + config::yesno "network${_num}_span" && _member_type="span" + + if [ -n "${_sid}" ]; then + _mtu=$(ifconfig "${_sid}" | head -n1 | awk '{print $NF}') + + if [ "${_mtu}" != "1500" ]; then + util::log "guest" "${_name}" "setting mtu of ${_tap} to ${_mtu}" + ifconfig "${_tap}" mtu "${_mtu}" >/dev/null 2>&1 + fi + + util::log "guest" "${_name}" "adding ${_tap} -> ${_sid} (${_switch} ${_member_type})" + ifconfig "${_sid}" "${_member_type}" "${_tap}" >/dev/null 2>&1 || util::log "guest" "${_name}" "failed to add ${_tap} to ${_sid}" + + util::log "guest" "${_name}" "bring up ${_tap} -> ${_sid} (${_switch} ${_member_type})" + ifconfig "${_tap}" up >/dev/null 2>&1 || util::log "guest" "${_name}" "failed to bring up ${_tap} in ${_sid}" + + # set private if configured + switch::is_private "${_switch}" && ifconfig "${_sid}" "private" "${_tap}" >/dev/null 2>&1 + else + util::log "guest" "${_name}" "failed to find virtual switch '${_switch}'" + fi + fi + + _devices="${_devices} -s ${_slot}:${_func},${_emulation},${_tap}" + [ -n "${_mac}" ] && _devices="${_devices},mac=${_mac}" + + _func=$((_func + 1)) + [ -z "${_custom_tap}" ] && _taplist="${_taplist}${_taplist:+ }${_tap}" +} diff --git a/lib/vm-switch-vale b/lib/vm-switch-vale new file mode 100644 index 0000000..3a6ebf6 --- /dev/null +++ b/lib/vm-switch-vale @@ -0,0 +1,128 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# show the configuration details for a vale switch +# +# @param string _name the switch name +# @param string _format output format +# +switch::vale::show(){ + local _name="$1" + local _format="$2" + local _id + + switch::vale::id "_id" "${_name}" + printf "${_format}" "${_name}" "vale" "${_id}" "n/a" "n/a" "n/a" "n/a" "n/a" +} + +# create a vale switch +# +# @param string _switch the name of the switch +# +switch::vale::create(){ + + config::core::set "switch_list" "${_switch}" "1" + config::core::set "type_${_switch}" "vale" +} + +# remove a vale switch +# +switch::vale::remove(){ } + +# add a new interface to this switch +# at the moment we require the user to manually +# set up any vale switches +# +# @param string _switch name of the switch +# @param string _if the interface to add +# +switch::vale::add_member(){ + util::err "physical interfaces must be added to the vale switch manually" +} + +# remove an interface +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::vale::remove_member(){ + util::err "physical interfaces must be removed from the vale switch manually" +} + +# set vlan id +# +# @param string _switch name of switch +# @param int _vlan vlan id to set +# +switch::vale::vlan(){ + util::err "vlan support is not currently implemented for vale switches" +} + +# gets a unique port name for a vale interface +# we need to make sure vale switch name is the same +# for all interfaces on the same switch, but port is +# different +# +# @param string _var name of variable to put port name into +# @param string _switch the name of the switch +# @param string _port unique port identifier (usually mac address) +# +switch::vale::id(){ + local _var="$1" + local _switch="$2" + local _port="$3" + local _id_s _id_p + + # get a switch id + _id_s=$(md5 -qs "${_switch}" | cut -c1-5) + + # given port? + if [ -n "${_port}" ]; then + _id_p=$(md5 -qs "${_port}" | cut -c1-5) + setvar "${_var}" "vale${_id_s}:${_id_p}" + else + setvar "${_var}" "vale${_id_s}" + fi +} + +# create a vale interface for a guest +# relies heavily on variables set in the main vm::run function +# +# @modifies _func _devices +# @return 1 if we don't get a tap device +# +switch::vale::provision(){ + local _vale_id + + # create a vale port id + switch::vale::id "_vale_id" "${_switch}" "${_mac}" + + util::log "guest" "${_name}" "adding vale interface ${_tap} (${_switch})" + _devices="${_devices} -s ${_slot}:${_func},${_emulation},${_vale_id}" + [ -n "${_mac}" ] && _devices="${_devices},mac=${_mac}" + + _func=$((_func + 1)) +} diff --git a/lib/vm-switch-vxlan b/lib/vm-switch-vxlan new file mode 100644 index 0000000..9ab733f --- /dev/null +++ b/lib/vm-switch-vxlan @@ -0,0 +1,204 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# create a vxlan switch +# we create a bridge and then add the vxlan interface to it +# +# @param string _name name of the switch +# +switch::vxlan::init(){ + local _name="$1" + local _id _vlan _if _maddr _addr _ifconf + + # see if the bridge already exists + switch::standard::id "_id" "${_name}" && return 0 + + # need a vlan id and interface + config::core::get "_vlan" "vlan_${_name}" + config::core::get "_if" "ports_${_name}" + [ -z "${_vlan}" -o -z "${_if}" ] && return 1 + + # get local address for this interface + _local=$(ifconfig ${_if} | grep "inet " | cut -d" " -f 2) + [ -z "${_local}" ] && return 1 + + # come up with an ip address for multicast data + switch::vxlan::__multicast "_maddr" "${_name}" + + # create the vxlan interface + ifconfig "vxlan${_vlan}" create vxlanid "${_vlan}" vxlanlocal "${_local}" vxlangroup "${_maddr}" \ + vxlandev "${_if}" descr "vm-vxlan/${_switch}" group vm-vlan up >/dev/null 2>&1 + [ $? -eq 0 ] || return 1 + + # get the length of the switch name + # it's useful for other utilities to use switch name as interface name + # as it's static. can't do that if it's > 12 chars + _len=$(echo -n "${_name}" | wc -m) + + if [ ${_len} -le 12 ]; then + _ifconf="name vm-${_name}" + else + _ifconf="descr vm/${_name}" + fi + + # create a bridge for this switch + _id=$(ifconfig bridge create ${_ifconf} group vm-switch up 2>/dev/null) + [ $? -eq 0 ] || util::err "failed to create bridge interface for switch ${_name}" + + switch::set_viid "${_name}" "${_id}" + + # randomise mac if feature is available + [ ${VERSION_BSD} -ge 1102000 ] && ifconfig "${_id}" link random + + # bridge vxlan to our guest switch + # static route traffic for this multicast address via our specified interface + ifconfig "${_id}" addm "vxlan${_vlan}" + route add -net ${_maddr}/32 -iface ${_if} >/dev/null 2>&1 + + # custom address for bridge? + config::core::get "_addr" "addr_${_name}" + [ -n "${_addr}" ] && ifconfig "${_id}" inet ${_addr} +} + +# show the configuration details for a vxlan switch +# +# @param string _name the switch name +# @param string _format output format +# +switch::vxlan::show(){ + local _name="$1" + local _format="$2" + local _id _vlan _port _addr _priv + + switch::standard::id "_id" "${_name}" + config::core::get "_vlan" "vlan_${_name}" + config::core::get "_port" "ports_${_name}" + config::core::get "_addr" "addr_${_name}" + config::core::get "_priv" "private_${_name}" "no" + + printf "${_format}" "${_name}" "vxlan" "${_id:--}" "${_addr:--}" "${_priv}" "n/a" "${_vlan}" "${_port}" +} + +# create a vxlan switch +# +switch::vxlan::create(){ + + # we must have an interface and vlan to use + [ -z "${_if}" -o -z "${_vlan}" ] && util::err "vxlan switches must be created with an interface and vlan id specified" + + # store configuration + config::core::set "switch_list" "${_switch}" "1" + config::core::set "type_${_switch}" "vxlan" + config::core::set "vlan_${_switch}" "${_vlan}" + config::core::set "ports_${_switch}" "${_if}" + + [ -n "${_addr}" ] && config::core::set "addr_${_switch}" "${_addr}" + [ -n "${_priv}" ] && config::core::set "private_${_switch}" "${_priv}" + + config::core::load + switch::vxlan::init "${_switch}" +} + +# destroy a vxlan switch +# +# @param string _switch the switch to remove +# +switch::vxlan::remove(){ + local _switch="$1" + local _id _vlan _maddr + + # try to get guest bridge and vxlan id + switch::standard::id "_id" "${_switch}" + [ $? -eq 0 ] || return 1 + + config::core::get "_vlan" "vlan_${_switch}" + [ -z "${_vlan}" ] && return 1 + + # get the multicast address we used for this switch + # and try to remove any route we may have added + switch::vxlan::__multicast "_maddr" "${_switch}" + route del -net "${_maddr}/32" >/dev/null 2>&1 + + # destroy the guest bridge + ifconfig ${_id} destroy >/dev/null 2>&1 + [ $? -eq 0 ] || return 1 + + # destroy the vxlan + ifconfig "vxlan${_vlan}" destroy >/dev/null 2>&1 +} + +# add a new interface to this switch +# we only allow a single physical interface for +# vxlan switches. this must be set at creation time +# so this just reports an error +# +# @param string _switch name of the vxlan switch +# @param string _if the interface to add +# +switch::vxlan::add_member(){ + util::err "vxlan interface must be configured at creation time" +} + +# remove an interface from a switch +# we don't support this here +# +# @param string _switch name of the switch +# @param string _if the interface to remove +# +switch::vxlan::remove_member(){ + util::err "vxlan interface must be configured at creation time" +} + +# set vlan id +# +# @param string _switch name of switch +# @param int _vlan vlan id to set +# +switch::vxlan::vlan(){ + util::err "vxlan id can only be set at creation time" +} + +# get the multicast address for a vxlan switch +# +# @param string _var variable to put address into +# @param string _switch the switch name +# +switch::vxlan::__multicast(){ + local _var="$1" + local _switch="$2" + local _hash _l_addr _octet _pos + + # come up with an ip address for multicast data + _hash=$(md5 -qs "${_switch}") + _l_addr="239" + + for _pos in 1-2 3-4 5-6; do + _octet=$(printf ".%d" "0x`echo "${_hash}"| cut -c ${_pos}`") + _l_addr="${_l_addr}${_octet}" + done + + setvar "${_var}" "${_l_addr}" +} diff --git a/lib/vm-util b/lib/vm-util new file mode 100644 index 0000000..50515ab --- /dev/null +++ b/lib/vm-util @@ -0,0 +1,405 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# make sure we have the right environment +# +util::setup(){ + util::load_module "nmdm" + util::load_module "if_bridge" + + # tap(4) & tun(4) were unified in r347241, this is closest ABI bump + if [ `uname -K` -ge 1300029 ]; then + util::load_module "if_tuntap" + else + util::load_module "if_tap" + fi + + sysctl net.link.tap.up_on_open=1 >/dev/null 2>&1 + + # do we have the default template example, but no default in our .templates? + # if so, get a copy, this at least allows a simple "vm create" to work out of the box + if [ -e "/usr/local/share/examples/vm-bhyve/default.conf" -a ! -e "${vm_dir}/.templates/default.conf" ]; then + cp "/usr/local/share/examples/vm-bhyve/default.conf" "${vm_dir}/.templates/" >/dev/null 2>&1 + fi +} + +# load a kernel module +# +# @param string _mod the module name +# +util::load_module(){ + local _mod="$1" + kldstat -qm ${_mod} >/dev/null 2>&1 + if [ $? -ne 0 ]; then + kldload ${_mod} >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "unable to load ${_mod}.ko!" + fi +} + +# check if system have bhyve support +# look for sysctls set by the vmm module. I can't get confirmation that this +# is a valid way to check vmm is working, even though it seems reasonable. +# the vm_disable_host_checks="yes" rc settings allows bypassing all this +# if your system should be supported but these checks break. +# +# @modifies VM_NO_UG +# +util::check_bhyve_support(){ + + # almost all our functionality requires access to things only root can do + [ `id -u` -ne 0 ] && util::err "virtual machines can only be managed by root" + + # try to load the vmm module + # we do this here to make sure the sysctls exist, and before disable_host_checks + # as we want this loaded anyway. FreeBSD version check removed as the + # module won't exist on systems too old for bhyve + util::load_module "vmm" + + # don't check if user wants to bypass host checks + util::yesno "$vm_disable_host_checks" && return 0 + + # check sysctls + # these only work for intel + # for AMD we give up trying to check for the time being + sysctl hw.model |grep Intel >/dev/null 2>&1 + + if [ $? -eq 0 ]; then + [ "`sysctl -n hw.vmm.vmx.initialized 2>/dev/null`" != "1" ] && util::err "kernel vmm not initialised (no VT-x / AMD SVM cpu support?)" + [ "`sysctl -n hw.vmm.vmx.cap.unrestricted_guest 2>/dev/null`" != "1" ] && VM_NO_UG="1" + fi +} + +# check for passthru support +# following neel@ wiki we search for DMAR acpi table for vt-d +# and we check sysctl if amdvi is present and enabled +# +# @return success if host has vt-d or amdvi +# +util::check_bhyve_iommu(){ + local _vtd + local _amdvi + + # don't check if user wants to bypass host checks + # think this check is fairly solid but there's probably someone somewhere + # with iommu support that our tests fail for. + util::yesno "$vm_disable_host_checks" && return 0 + + _vtd=$(acpidump -t |grep DMAR) + _amdvi=$(sysctl hw |grep 'vmm.amdvi.enable: 1') + [ -z "${_vtd}" -a -z "${_amdvi}" ] && return 1 + + return 0 +} + +# restart a local service +# checks if service is running and either starts or restarts +# +# @param string _serv the name of the service +# +util::restart_service(){ + local _serv="$1" + local _cmd="restart" + + # see if it's actually running + service ${_serv} status >/dev/null 2>&1 + [ $? -ne 0 ] && _cmd="start" + + service ${_serv} ${_cmd} >/dev/null 2>&1 + [ $? -ne 0 ] && util::warn "failed to ${_cmd} service ${_serv}" +} + +# show version +# +util::version(){ + echo "vm-bhyve: Bhyve virtual machine management v${VERSION} (rev. ${VERSION_INT})" +} + +# show version & usage information +# we exit after running this +# +util::usage(){ + util::version + cat << EOT +Usage: vm ... + version + init + set [setting=value] [...] + get [all|setting] [...] + switch list + switch info [name] [...] + switch create [-t type] [-i interface] [-n vlan-id] [-m mtu] [-a address/prefix-len] [-b bridge] [-p] + switch vlan + switch nat + switch private + switch add + switch remove + switch destroy + datastore list + datastore add + datastore remove + datastore add + list + info [name] [...] + create [-d datastore] [-t template] [-s size] [-m memory] [-c vCPUs] + install [-fi] + start [-fi] [...] + stop [...] + restart + console [com1|com2] + configure + rename + add [-d device] [-t type] [-s size|switch] + startall + stopall + reset [-f] + poweroff [-f] + destroy [-f] + passthru + clone + snapshot [-f] + rollback [-r] + iso [url] + img [url] + image list + image create [-d description] [-u] + image destroy + image provision [-d datastore] +EOT + exit 1 +} + +# err +# display an error message and exit immediately +# +# @param string - the message to display +# +util::err(){ + echo "${0}: ERROR: $1" >&2 + exit 1 +} + +# err_inline +# display an error inline with informational output +# +# @param string - message to display +# +util::err_inline(){ + echo " ! $1" + exit 1 +} + +# warn +# display warning, but do not exit +# +# @param string - the message to display +# +util::warn(){ + echo "${0}: WARNING: $1" >&2 +} + +# log_rotate +# simple rotation of log files +# if we hit 1MB, which should cover a fair amount of history, +# we move existing log and and create a new one. +# one keep 1 previous file, as that should be enough +# +# @param string _type whether to rotate guest or main log +# +util::log_rotate(){ + local _type="$1" + local _lf="vm-bhyve.log" + local _file _size _guest + + case "${_type}" in + guest) + _guest="$2" + _file="${VM_DS_PATH}/${_guest}/${_lf}" + ;; + system) + _file="${vm_dir}/${_lf}" + ;; + esac + + if [ -e "${_file}" ]; then + _size=$(stat -f %z "${_file}") + + if [ -n "${_size}" -a "${_size}" -ge 1048576 ]; then + unlink "${_file}.0.gz" >/dev/null 2>&1 + mv "${_file}" "${_file}.0" + gzip "${_file}.0" + fi + fi +} + +# log to file +# writes the date and a message to the specified log +# the global log is in $vm_dir/vm-bhyve.log +# guest logs are $vm_dir/{guest}/vm-bhyve.log +# +# @param string _type=guest|system log to global vm-bhyve log or guest +# @param optional string _guest if _type=guest, the guest name, otherwise do not provide at all +# @param string _message the message to log +# +util::log(){ + local _type="$1" + local _lf="vm-bhyve.log" + local _guest _message _file _date + + case "${_type}" in + guest) + _guest="$2" + _file="${VM_DS_PATH}/${_guest}/${_lf}" + shift 2 + ;; + system) + _file="${vm_dir}/${_lf}" + shift 1 + ;; + esac + + while [ -n "$1" ]; do + echo "$(date +'%b %d %T'): $1" >> "${_file}" + shift + done +} + +# write content to a file, and log what we +# did to the guest log file +# it's useful to be able to see what files vm-bhyve is creating +# and the contents so we write that to the log. +# The file is created in $vm_dir/{guest} +# +# @param string _type=write|appnd create file or append to it +# @param string _guest the guest name +# @param string _file the file name to write to +# @param string _message the data to write +# +util::log_and_write(){ + local _type="$1" + local _guest="$2" + local _file="${VM_DS_PATH}/${_guest}/$3" + local _message="$4" + + if [ "${_type}" = "write" ]; then + util::log "guest" "${_guest}" "create file ${_file}" + echo "${_message}" > "${_file}" + else + echo "${_message}" >> "${_file}" + fi + + util::log "guest" "${_guest}" " -> ${_message}" +} + +# confirm yes or no +# +# @param string _msh message to display +# @return int success if confirmed +# +util::confirm(){ + local _msg="$1" + local _resp + + while read -p "${_msg} (y/n)? " _resp; do + case "${_resp}" in + y*) return 0 ;; + n*) return 1 ;; + esac + done +} + +# our own checkyesno copy +# doesn't warn for unsupported values +# also returns as 'yes' unless value is specifically no/off/false/0 +# +# @param _value the value to test +# @return int 1 if set to "off/false/no/0", 0 otherwise +# +util::yesno(){ + local _value="$1" + + [ -z "${_value}" ] && return 1 + + case "$_value" in + [Nn][Oo]|[Ff][Aa][Ll][Ss][Ee]|[Oo][Ff][Ff]|0) + return 1 ;; + *) return 0 ;; + esac +} + +# 'vm check name' +# check name of virtual machine +# +# @param _name name to check +# @param _maxlen=30(229 on 13+) maximum name length (NOTE should be 2 less than desired) +# @return int 0 if name is valid +# +util::check_name(){ + local _name="$1" + local _maxlen="$2" + + if [ -z "${_maxlen}" ]; then + if [ ${VERSION_BSD} -ge 1300000 ]; then + : ${_maxlen:=229} + else + : ${_maxlen:=30} + fi + fi + + echo "${_name}" | egrep -iqs "^[a-z0-9][.a-z0-9_-]{0,${_maxlen}}[a-z0-9]\$" +} + +# check if the specified string is a valid core configuration +# setting that the user can change +# +# @param string the setting name to look for +# +util::valid_config_setting(){ + echo "${VM_CONFIG_USER}" | grep -iqs "${1};" +} + +# __getpid +# get a process id +# +# @param string _var variable to put pid into +# @param string _proc process to look for +# +util::getpid(){ + local _var="$1" + local _proc="$2" + local _ret + + _ret=$(pgrep -f "${_proc}") + [ $? -eq 0 ] || return 1 + setvar "${_var}" "${_ret}" +} + +util::get_part(){ + local _var="$1" + local _data="$2" + local _num="$3" + + setvar "${_var}" $(echo "${_data}" |cut -w -f${_num}) +} diff --git a/lib/vm-zfs b/lib/vm-zfs new file mode 100644 index 0000000..c210701 --- /dev/null +++ b/lib/vm-zfs @@ -0,0 +1,519 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# set us up for zfs use +# we need the zfs dataset name for zfs commands, and the file system path +# for bhyve. It's highly possible these will be different. +# We ask user to specify the dataset name as "zfs:pool/dataset" +# This can then be used for zfs commands, and we can retrieve the mountpoint +# to find out the file path for bhyve +# +# This then overwrites $vm_dir with the file system path, so that the +# rest of vm-bhyve can work normally, regardless of whether we are in zfs mode +# or not +# +# If zfs is enabled, the following global variables are set +# VM_ZFS=1 +# VM_ZFS_DATASET={pool/dataset} +# +# @modifies VM_ZFS VM_ZFS_DATASET vm_dir +# +zfs::init(){ + + # check for zfs storage location + # user should specify "zfs:pool/dataset" if they want ZFS support + if [ "${vm_dir%%:*}" = "zfs" ]; then + + # check zfs running + kldstat -qm zfs >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "ZFS support requested but ZFS not available" + + # global zfs details + VM_ZFS="1" + VM_ZFS_DATASET="${vm_dir#*:}" + + # update vm_dir + # this makes sure it exists, confirms it's mounted & gets correct path in one go + vm_dir=$(mount | grep "^${VM_ZFS_DATASET} " |cut -d' ' -f3) + [ -z "${vm_dir}" ] && util::err "unable to locate mountpoint for ZFS dataset ${VM_ZFS_DATASET}" + fi +} + +# make a new dataset +# this is always called when creating a new vm, but will do nothing +# +# @param string _name name of the dataset to create +# +zfs::make_dataset(){ + local _name="$1" + local _opts="$2" + + if [ -n "${_name}" -a "${VM_DS_ZFS}" = "1" ]; then + zfs::__format_options "_opts" "${_opts}" + zfs create ${_opts} "${_name}" + [ $? -eq 0 ] || util::err "failed to create new ZFS dataset ${_name}" + fi +} + +# destroy a dataset +# +# @param string _name name of the dataset to destroy +# +zfs::destroy_dataset(){ + local _name="$1" + + if [ -n "${_name}" -a "${VM_DS_ZFS}" = "1" ]; then + zfs destroy -rf "${_name}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "failed to destroy ZFS dataset ${_name}" + fi +} + +# rename a dataset +# as with other zfs functions, the arguments should just be the name +# of the dataset under $VM_ZFS_DATASET (in this case just guest name) +# +# @param string _old the name of the dataset to rename +# @param string _new the new name +# +zfs::rename_dataset(){ + local _old="$1" + local _new="$2" + + if [ -n "${_old}" -a -n "${_new}" -a "${VM_DS_ZFS}" = "1" ]; then + zfs rename "${VM_DS_ZFS_DATASET}/${_old}" "${VM_DS_ZFS_DATASET}/${_new}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "failed to rename ZFS dataset ${VM_DS_ZFS_DATASET}/${_old}" + fi +} + +# make a zvol for a guest disk image +# +# @param string _name name of the zvol to create +# @param string _size how big to create the dataset +# @param int _sparse=0 set to 1 for a sparse zvol +# +zfs::make_zvol(){ + local _name="$1" + local _size="$2" + local _sparse="$3" + local _user_opts="$4" + local _opt="-V" + + [ ! "${VM_DS_ZFS}" = "1" ] && util::err "cannot use ZVOL storage unless ZFS support is enabled" + [ "${_sparse}" = "1" ] && _opt="-sV" + + zfs::__format_options "_user_opts" "${_user_opts}" + zfs create ${_opt} ${_size} -o volmode=dev ${_user_opts} "${_name}" + [ $? -eq 0 ] || util::err "failed to create new ZVOL ${_name}" +} + +# format options for zfs commands +# options are stored in configuration separated by a space +# we need to replace that with -o +# +# @modifies $_val +# +zfs::__format_options(){ + local _val="$1" + local _c_opts="$2" + + if [ -n "${_c_opts}" ]; then + _c_opts=$(echo "${_c_opts}" |sed -e 's/\ / -o /') + _c_opts="-o ${_c_opts}" + setvar "${_val}" "${_c_opts}" + return 0 + fi + + setvar "${_val}" "" +} + +# 'vm snapshot' +# create a snapshot of a guest +# specify the snapshot name in zfs format guest@snap +# if no snapshot name is specified, Y-m-d-H:M:S will be used +# +# @param flag (-f) force snapshot if guest is running +# @param string _name the name of the guest to snapshot +# +zfs::snapshot(){ + local _name _snap + + cmd::parse_args "$@" + shift $? + _name="$1" + + # try to get snapshot name + # we support normal zfs syntax for this + echo "${_name}" | grep -qs "@" + + if [ $? -eq 0 ]; then + _snap=${_name##*@} + _name=${_name%%@*} + fi + + [ -z "${_name}" ] && util::usage + datastore::get_guest "${_name}" || util::err "${_name} does not appear to be an existing virtual machine" + [ ! "${VM_DS_ZFS}" = "1" ] && util::err "cannot snapshot guests on non-zfs datastores" + [ -z "${_snap}" ] && _snap=$(date +"%Y-%m-%d-%H:%M:%S") + + if ! vm::confirm_stopped "${_name}" >/dev/null; then + [ -z "${VM_OPT_FORCE}" ] && util::err "${_name} must be powered off first (use -f to override)" + fi + + zfs snapshot -r ${VM_DS_ZFS_DATASET}/${_name}@${_snap} + [ $? -eq 0 ] || util::err "failed to create recursive snapshot of virtual machine" +} + +# try to remove a snapshot +# +# @param string _name the guest name and snapshot (guest@snap) +# @return true if successful +# +zfs::remove_snapshot(){ + local _name="$1" + local _snap + + # split name and snapshot + _snap=${_name##*@} + _name=${_name%%@*} + + # try to load guest + datastore::get_guest "${_name}" || util::err "${_name} does not appear to be an existing virtual machine" + [ ! "${VM_DS_ZFS}" = "1" ] && util::err "cannot snapshot guests on non-zfs datastores" + + # remove + zfs destroy -r ${VM_DS_ZFS_DATASET}/${_name}@${_snap} + [ $? -eq 0 ] || util::err "failed to remove snapshot ${VM_DS_ZFS_DATASET}/${_name}@${_snap}" +} + +# 'vm rollback' +# roll a guest back to a previous snapshot +# we show zfs errors here as it will fail if the snapshot is not the most recent. +# zfs will output an error mentioning to use '-r', and listing the snapshots +# that will be deleted. makes sense to let user see this and just support +# that option directly +# +# @param flag (-r) force deletion of more recent snapshots +# @param string _name name of the guest +# +zfs::rollback(){ + local _name _snap _opt _force _fs _snap_exists + + while getopts r _opt; do + case $_opt in + r) _force="-r" ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + + _snap_exists=$(echo "${1}" | grep "@") + [ -z "${_snap_exists}" ] && util::err "a snapshot name must be provided in guest@snapshot format" + + _name="${1%%@*}" + _snap="${1##*@}" + + [ -z "${_name}" -o -z "${_snap}" ] && util::usage + + datastore::get_guest "${_name}" || util::err "${_name} does not appear to be an existing virtual machine" + [ ! "${VM_DS_ZFS}" = "1" ] && util::err "cannot rollback guests on non-zfs datastores" + + vm::confirm_stopped "${_name}" || exit 1 + + # list all datasets and zvols under guest + zfs list -o name -rHt filesystem,volume ${VM_DS_ZFS_DATASET}/${_name} | \ + while read _fs; do + zfs rollback ${_force} ${_fs}@${_snap} + [ $? -ne 0 ] && exit $? + done +} + +# 'vm clone' +# clone a vm +# this makes a true zfs clone of the specifies guest +# +# @param string _old the guest to clone +# @param string _new name of the new guest +# +zfs::clone(){ + local _old="$1" + local _name="$2" + local _fs _newfs _snap _snap_exists _fs_list _entry + local _num=0 _error=0 + local _uuid=$(uuidgen) + + # check args and make sure new guest doesn't already exist + [ -z "${_old}" -o -z "${_name}" ] && util::usage + datastore::get_guest "${_name}" && util::err "new guest already exists in ${VM_DS_PATH}/${_name}" + + # try to get snapshot name + # we support normal zfs syntax for this + _snap_exists=$(echo "${_old}" | grep "@") + + if [ -n "${_snap_exists}" ]; then + _snap=${_old##*@} + _old=${_old%%@*} + fi + + # make sure old guest exists + datastore::get_guest "${_old}" || util::err "${_old} does not appear to be an existing virtual machine" + [ ! "${VM_DS_ZFS}" = "1" ] && util::err "cannot clone guests on non-zfs datastores" + + # get list of datasets to copy + _fs_list=$(zfs list -rHo name -t filesystem,volume "${VM_DS_ZFS_DATASET}/${_old}") + [ $? -eq 0 ] || util::err "unable to list datasets for ${VM_DS_ZFS_DATASET}/${_old}" + + # generate a short uuid and create snapshot if no custom snap given + if [ -z "${_snap}" ]; then + vm::confirm_stopped "${_old}" || exit 1 + _snap=$(echo "${_uuid}" |awk -F- '{print $1}') + + zfs snapshot -r "${VM_DS_ZFS_DATASET}/${_old}@${_snap}" + [ $? -eq 0 ] || util::err "failed to create snapshot ${VM_DS_ZFS_DATASET}/${_old}@${_snap}" + else + for _fs in ${_fs_list}; do + zfs get creation "${_fs}@${_snap}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "snapshot ${_fs}@${_snap} doesn't seem to exist" + done + fi + + # clone + for _fs in ${_fs_list}; do + _newfs=$(echo "${_fs}" | sed "s@${VM_DS_ZFS_DATASET}/${_old}@${VM_DS_ZFS_DATASET}/${_name}@") + + zfs clone "${_fs}@${_snap}" "${_newfs}" + [ $? -eq 0 ] || util::err "error while cloning dataset ${_fs}@${_snap}" + done + + # update new guest files + unlink "${VM_DS_PATH}/${_name}/vm-bhyve.log" >/dev/null 2>&1 + mv "${VM_DS_PATH}/${_name}/${_old}.conf" "${VM_DS_PATH}/${_name}/${_name}.conf" >/dev/null 2>&1 + if [ $? -ne 0 ]; then + echo "Unable to rename configuration file to ${_name}.conf" + echo "This will need to be renamed manually" + echo "Please also remove any uuid or mac address settings, these will be regenerated automatically" + exit 1 + fi + + # generalise the clone + vm::generalise "${_name}" +} + +# 'vm image create' +# create an image of a vm +# this creates an archive of the specified guest, stored in $vm_dir/images +# we use a uuid just in case we want to provide the ability to share images at any point +# +# @param optional string (-d) description of the image +# @param string _name name of guest to take image of +# +zfs::image_create(){ + local _name _opt _desc + local _uuid _snap _date _no_compress _filename + local _compress _decompress + + while getopts d:u _opt ; do + case $_opt in + d) _desc=${OPTARG} ;; + u) _no_compress="1" ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _name=$1 + _uuid=$(uuidgen) + _snap=${_uuid%%-*} + _date=$(date) + + [ -z "${_desc}" ] && _desc="No description provided" + + datastore::get_guest "${_name}" || util::err "${_name} does not appear to be a valid virtual machine" + [ -z "${VM_ZFS}" -o -z "${VM_DS_ZFS}" ] && util::err "this command is only supported on zfs datastores" + + # create the image dataset if we don't have it + if [ ! -e "${vm_dir}/images" ]; then + zfs create "${VM_ZFS_DATASET}/images" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "failed to create image store ${VM_ZFS_DATASET}/images" + fi + + # try to snapshot + zfs snapshot -r "${VM_DS_ZFS_DATASET}/${_name}@${_snap}" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "failed to create snapshot of source dataset ${VM_DS_ZFS_DATASET}/${_name}@${_snap}" + + # copy source + if [ -n "${_no_compress}" ]; then + _filename="${_uuid}.zfs" + + echo "Creating guest image, this may take some time..." + zfs send -R "${VM_DS_ZFS_DATASET}/${_name}@${_snap}" > "${vm_dir}/images/${_filename}" + else + _filename="${_uuid}.zfs.z" + + config::core::get "_compress" "compress" + config::core::get "_decompress" "decompress" + + # use defaults if either of these settings are missing + # no point using user defined compress if we don't know how to decompress + if [ "${_compress}" = "" -o "${_decompress}" = "" ]; then + _compress="xz -T0" + _decompress="xz -d" + fi + + echo "Creating a compressed image, this may take some time..." + zfs send -R "${VM_DS_ZFS_DATASET}/${_name}@${_snap}" | ${_compress} > "${vm_dir}/images/${_filename}" + fi + + [ $? -ne 0 ] && exit 1 + + # done with the source snapshot + zfs destroy -r ${VM_DS_ZFS_DATASET}/${_name}@${_snap} + + # create a description file + sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" "description=${_desc}" >/dev/null 2>&1 + sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" "created=${_date}" >/dev/null 2>&1 + sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" "name=${_name}" >/dev/null 2>&1 + sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" "filename=${_filename}" >/dev/null 2>&1 + sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" "decompress=${_decompress}" >/dev/null 2>&1 + + echo "Image of ${_name} created with UUID ${_uuid}" +} + +# 'vm image provision' +# create a new vm from an image +# +# @param string _uuid the uuid of the image to use +# @param string _name name of the new guest +# +zfs::image_provision(){ + local _uuid _name _file _oldname _entry _num=0 _type + local _datastore="default" _decompress + + while getopts d: _opt ; do + case $_opt in + d) _datastore="${OPTARG}" ;; + *) util::usage ;; + esac + done + + shift $((OPTIND - 1)) + _uuid="$1" + _name="$2" + + [ -z "${_uuid}" -o -z "${_name}" ] && util::usage + [ ! -e "${vm_dir}/images/${_uuid}.manifest" ] && util::err "unable to locate image with uuid ${_uuid}" + datastore::get_guest "${_name}" && util::err "new guest already exists in ${VM_DS_PATH}/${_name}" + + # get the data filename + _file=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" filename) + _type=${_file##*.} + + _oldname=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" name) + [ -z "${_file}" -o -z "${_oldname}" ] && util::err "unable to locate required details from the specified image manifest" + [ ! -e "${vm_dir}/images/${_file}" ] && util::err "image data file does not exist: ${vm_dir}/images/${_file}" + + # get the datastore to create on + datastore::get "${_datastore}" || util::err "unable to locate datastore '${_datastore}'" + + # try to recieve + echo "Unpacking guest image, this may take some time..." + + # check format of image + case ${_type} in + zfs) cat "${vm_dir}/images/${_file}" | zfs recv "${VM_DS_ZFS_DATASET}/${_name}" ;; + xz) xz -dc "${vm_dir}/images/${_file}" 2>/dev/null | zfs recv "${VM_DS_ZFS_DATASET}/${_name}" ;; + z) _decompress=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" decompress) + [ -z "${_decompress}" ] && util::err "unable to locate decompression configuration" + ${_decompress} <"${vm_dir}/images/${_file}" 2>/dev/null | zfs recv "${VM_DS_ZFS_DATASET}/${_name}" ;; + *) util::err "unsupported guest image type - '${_type}'" ;; + esac + + # error unpacking? + [ $? -eq 0 ] || util::err "errors occured while trying to unpackage the image file" + + # remove the original snapshot + zfs destroy -r "${VM_DS_ZFS_DATASET}/${_name}@${_uuid%%-*}" >/dev/null 2>&1 + + # rename the guest configuration file + mv "${VM_DS_PATH}/${_name}/${_oldname}.conf" "${VM_DS_PATH}/${_name}/${_name}.conf" >/dev/null 2>&1 + [ $? -eq 0 ] || util::err "unpackaged image but unable to update guest configuration file" + + # update mac addresses and create a new uuid + _uuid=$(uuidgen) + + # remove unique settings from new image + vm::generalise "${_name}" + + # vm may be started when 'vm image create' is executed + rm -f "${VM_DS_PATH}/${_name}/run.lock" >/dev/null 2>&1 + rm -f "${VM_DS_PATH}/${_name}/vm-bhyve.log*" >/dev/null 2>&1 +} + +# 'vm image list' +# list available images +# +zfs::image_list(){ + local _file _uuid _ext + local _format="%s^%s^%s^%s\n" + + { + printf "${_format}" "UUID" "NAME" "CREATED" "DESCRIPTION" + + [ ! -e "${vm_dir}/images" ] && exit + + ls -1 ${vm_dir}/images/ | \ + while read _file; do + if [ "${_file##*.}" = "manifest" ]; then + _uuid=${_file%.*} + _desc=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" description) + _created=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" created) + _name=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" name) + + printf "${_format}" "${_uuid}" "${_name}" "${_created}" "${_desc}" + fi + done + } | column -ts^ +} + +# 'vm image destroy' +# destroy an image +# +# @param string _uuid the uuid of the image +# +zfs::image_destroy(){ + local _uuid="$1" + local _file + + [ -z "${_uuid}" ] && util::usage + [ ! -e "${vm_dir}/images/${_uuid}.manifest" ] && util::err "unable to locate image with uuid ${_uuid}" + + # get the image filename + _file=$(sysrc -inqf "${vm_dir}/images/${_uuid}.manifest" filename) + [ -z "${_file}" ] && util::err "unable to locate filename for the specified image" + + unlink "${vm_dir}/images/${_uuid}.manifest" + unlink "${vm_dir}/images/${_file}" +} diff --git a/rc.d/vm b/rc.d/vm new file mode 100644 index 0000000..caada6c --- /dev/null +++ b/rc.d/vm @@ -0,0 +1,30 @@ +#!/bin/sh +# +# $FreeBSD$ + +# PROVIDE: vm +# REQUIRE: NETWORKING SERVERS dmesg +# BEFORE: dnsmasq ipfw pf +# KEYWORD: shutdown nojail + +. /etc/rc.subr + +: ${vm_enable="NO"} + +name=vm +desc="Start and stop vm-bhyve guests on boot/shutdown" +rcvar=vm_enable + +load_rc_config $name + +command="/usr/local/sbin/${name}" +start_cmd="${name}_start" +stop_cmd="${command} stopall -f" + +vm_start() +{ + env rc_force="$rc_force" ${command} init + env rc_force="$rc_force" ${command} startall >/dev/null & +} + +run_rc_command "$1" diff --git a/sample-templates/alpine.conf b/sample-templates/alpine.conf new file mode 100644 index 0000000..8d58021 --- /dev/null +++ b/sample-templates/alpine.conf @@ -0,0 +1,11 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /boot/vmlinuz-vanilla initrd=/boot/initramfs-vanilla alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage,sr-mod" +grub_install1="initrd /boot/initramfs-vanilla" +grub_run0="linux /boot/vmlinuz-vanilla root=/dev/vda3 modules=ext4" +grub_run1="initrd /boot/initramfs-vanilla" diff --git a/sample-templates/arch.conf b/sample-templates/arch.conf new file mode 100644 index 0000000..bc51d53 --- /dev/null +++ b/sample-templates/arch.conf @@ -0,0 +1,9 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /arch/boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201611 ro" +grub_install1="initrd /arch/boot/x86_64/archiso.img" diff --git a/sample-templates/centos6.conf b/sample-templates/centos6.conf new file mode 100644 index 0000000..5d40c38 --- /dev/null +++ b/sample-templates/centos6.conf @@ -0,0 +1,11 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /isolinux/vmlinuz" +grub_install1="initrd /isolinux/initrd.img" +grub_run0="linux /vmlinuz-2.6.32-573.el6.x86_64 root=/dev/mapper/VolGroup-lv_root" +grub_run1="initrd /initramfs-2.6.32-573.el6.x86_64.img" diff --git a/sample-templates/centos7.conf b/sample-templates/centos7.conf new file mode 100644 index 0000000..1daa29b --- /dev/null +++ b/sample-templates/centos7.conf @@ -0,0 +1,9 @@ +loader="uefi" +graphics="yes" +xhci_mouse="yes" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" diff --git a/sample-templates/config.sample b/sample-templates/config.sample new file mode 100644 index 0000000..7e4183d --- /dev/null +++ b/sample-templates/config.sample @@ -0,0 +1,503 @@ +# This is a sample configuration file containing all supported options +# Please do not try and use this file itself for a guest +# For any option that contains a number in the name, such as "network0_type", +# you can add additional devices of that type by creating a new set of +# variables using the next number in sequence, e.g "network1_type" +# +# Please make sure all option names are specified in lowercase and +# at the beginning of the line. If there is any whitespace before +# the option name, the line will be ignored. +# The '#' character signifies the start of a comment, even within +# double-quotes, and so cannot be used inside any values. + +# loader +# Specify the loader to use for the guest. This can either be +# one of the original bhyve loaders (bhyveload/grub), or +# you can specify to use uefi firmware to load the guest +# +# Valid Options: bhyveload,grub,uefi,uefi-csm +# +loader="" + +# bhyveload_loader +# If using bhyveload, this option can be used to specify the path +# to the loader inside the guest to use +# +# the default is /boot/userboot.so +# +bhyveload_loader="" + +# bhyveload_args +# If using bhyveload, this option can be used to pass command line +# arguments to the loader +# +bhyveload_args="-e machdep.hyperthreading_allowed=0" + +# loader_timeout +# By default bhyveload & grub-bhyve will wait 3 seconds before booting the default +# option. This setting allows you to either reduce the timeout to +# make boot faster, or increase it so that it's easier to access +# the grub console before it starts booting +# +loader_timeout="3" + +# uefi_vars +# set to a true value to support persistent UEFI vars +# this relies on a version of uefi-firmware that comes with the BHYVE_UEFI_VARS.fd template, +# and support in bhyve +# +uefi_vars="no" + +# cpu (required) +# specify the number of cpu cores to give to the guest +# +cpu="1" + +# cpu_sockets +# manually configure the number of sockets that bhyve should +# expose to the guest. +# note that sockets*cores*threads should equal the above cpu count +# +cpu_sockets="1" + +# cpu_cores +# the number of cores to create per physical processor +# +cpu_cores="1" + +# cpu_threads +# the number of cpu threads per core +# +cpu_threads="1" + +# memory (required) +# specify the amount of ram to give to the guest. This can be +# followed by M or G. +# +memory="512M" + +# wired_memory +# All requested memory should be wired to the guest +# +wired_memory="no" + +# hostbridge +# Allows you to specify the type of hostbridge to use for the +# guest hardware. This can usually be left as default. The +# additional options are 'amd', for a hostbridge that advertises +# itself as AMD hardware and 'none' for no hostbridge. Note +# that there is no requirement to use the 'amd' hostbridge if +# you host has an AMD processor +# +# Default: standard +# Valid Options: standard,amd,none +# +hostbridge="" + +# ignore_bad_msr +# Instruct bhyve to ignore accesses to model specific registers +# that are not implemented in the current CPU. +# This appears to be required for AMD processors when using +# some guest operating systems. Note that this is enabled +# by default when running a UEFI guest +# +ignore_bad_msr="no" + +# bhyve_options +# any additional bhyve command line options +# +bhyve_options="-p 1:1" + +# comports +# This allows you to define the com ports which should be available. +# By default only com1 is connected, and can be accessed using the +# 'vm console' command. If more than one com port is specified, you +# can choose the port to connect to by running 'vm console guest com1|com2'. +# When using the 'vm console' command, if no com port is specified, +# you are connected to the first port listed in this string. +# +# Default: com1 +# Valid Options: com1,com2,com1 com2,com2 com1 +# +comports="" + +# utctime +# bhyve normally sets the guests RTC to the host's localtime. The utctime +# option causes bhyve to try and configure the guests RTC to UTC. +# +# As of vm-bhyve 1.2, this setting defaults to yes, giving the guest a +# UTC realtime clock. I consider this more consistent, and is actually +# expected by some guests. The guest should show correct time as long as +# its timezone is configured correctly. Note that the following command +# is useful to verify the time of a guest's "hardware" RTC: +# bhyvectl --vm={guestname} --get-rtc-time +# +# To revert to the default bhyve behaviour, explicitly set this to off/no/false/0 +# +# Additionally it is generally advised to run a time sync daemon, such as ntpd +# in the guest, as each OS will have its own clock that will inevitably drift. +# +# Default: yes +# +utctime="no" + +# debug +# Set to a value other than [empty]/no/off/false/0 to run vm-bhyve in debug mode. +# In this mode, all output from the bhyve process is written to +# $vm_dir/{guest}/bhyve.log. This is useful if the guest is crashing or +# exiting abnormally as the log will contain any output from bhyve. +# +# Default: no +# +debug="" + +# uuid +# This is set automatically by vm-bhyve when creating a new guest. Normally +# bhyve assigns a UUID at runtime based on host and guest name. This +# option allows you to specify a fixed UUID that will always be used. Remove +# this or leave blank to return to the normal bhyve behaviour. +# +uuid="" + +# ahci_device_limit +# By default all ahci devices (ahci-hd/ahci-cd) are configured on independent +# slots with their own controller. In FreeBSD 12 it's possible to put up +# to 32 devices on each controller. This setting allows you to configure +# the number of devices vm-bhyve will allocate on each controller. +# +# Valid Options: 2-32 +# Default: 1 +# +ahci_device_limit="8" + +# disk0_type (required) +# This specifies the emulation type for disk0. Please note that each disk requires +# at least a type and name. +# +# Valid Options: virtio-blk,ahci-hd,ahci-cd,nvme +# +disk0_type="virtio-blk" + +# disk0_dev +# The type of device used as the backing store for this disk. The default is 'file', +# which means a sparse file is used. This file is stored in the guest's directory. +# For the zvol options, the zvol must be directly under the guest dataset. +# There is also a 'custom' option, in which case the disk name should be the full path +# to the file or device you want to use. +# For 'iscsi', the disk name must be set to a unique target and lun combination +# when matched against iscsictl -L output. +# +# Default: file +# Valid Options: file,zvol,sparse-zvol,custom,iscsi +# +disk0_dev="" + +# disk0_name (required) +# The name of the file or zvol for this disk. If the device type is 'custom', it should +# be the full path to whichever device or file you want to use +# +# This value is translated to a path as follows, based on disk0_dev +# +# DEVICE TYPE DISK NAME BHYVE PATH USED +# file 'disk0.img' -> '$vm_dir/$name/disk0.img' +# zvol|sparse-zvol 'disk0' -> '/dev/zvol/pool/dataset/path/guest/disk0' +# custom '/dev/da10' -> '/dev/da10' +# iscsi 'tgt[/lun]' -> '/dev/daNN' (lun defaults to 0 if omitted) +# +disk0_name="disk0.img" + +# disk0_opts +# A comma separated list of additional options for the specified disk. +# The available options are listed below. See the bhyve(8) man page for +# more details +# +# Valid Options: direct,nocache,ro,sectorsize=logical[/physical] +# +disk0_opts="" + +# disk0_size +# When a new guest is created, vm will create a 20G disk image by +# default. This option can be used to specify a different size +# for this disk. Make sure to include a human readable suffix +# compatible with 'zfs create' (G for gigabytes, T terabytes, +# etc) +# +# The size of the first disk (disk0) can also be overridden +# using the -s option to 'vm create'. +# +# NOTE: This option is only supported in template files. This +# setting serves no purpose in real guests and would become +# misleading if a disk were resized manually. When provisioning +# a new guest, all 'diskX_size' options are stripped from +# its configuration file. +# +disk0_size="50G" + +# network0_type +# This specifies the emulation type to use for the first network interface. +# Networking is not required, although this field is mandatory if you do want +# to add an interface +# +# Valid Options: virtio-net,e1000 +# +network0_type="virtio-net" + +# network0_switch +# The name of the virtual switch to connect this interface to. When starting the +# guest, if this switch cannot be found, or no switch is specified, the interface +# is still created but will not be connected to anything. +# +# All default templates use a switch called 'public', although it's perfectly +# reasonable to use other switch names that make sense in your environment +# +network0_switch="public" + +# network0_device +# If you do not want vm-bhyve to create a new interface, but use an existing +# one, enter the interface name here. This allows you to preconfigure the network +# device in a custom configuration, then instruct vm-bhyve to use that rather +# than create all interfaces dynamically at run time. +# +network0_device="" + +# network0_name +# if specified, the interface will be given this name +# +network0_name="web1" + +# network0_mac +# This allows you to specify a fixed mac address for this interface inside the guest. +# When a guest is run, vm-bhyve will automatically assign a mac address for each +# interface if one is not specified. This mac address is then written to the +# configuration file using this option. If we didn't do this guests might get +# a different mac if the tap device changes (very possible in vm-bhyve as all +# tap devices are dynamic by default). Guests like Windows treat an interface +# with a different mac as a new interface, with a new set of default settings. +# +network0_mac="" + +# network0_span +# Set to any value other than [empty]/off/false/no/0 to create the specified +# port as a SPAN port rather than as an ordinary bridge member. +# +# NOTE: Does not work with VALE switches yet. +# +network0_span="no" + +# passthru0 +# Add a pass-through PCI device to the virtual machine. This allows the guest +# to access a hardware device no differently than if it was running on bare +# metal. The value of this option is the B/S/F of the appropriate device. +# e.g "3/0/0" +# +# The slot to use in bhyve can be specified as below. This example will +# force the host device 6/0/0 to use slot 2:0 in the guest +# +# passthru0="6/0/0=2:0" +# +# Please note that in order to stop the bhyve host from attaching to the device, +# there are some steps required to reserve the device in /boot/loader.conf. +# +# The 'vm passthru' command provides a convinient way of seeing the BSF of each +# device in your system, and whether any have been reserved ready for use +# in bhyve +# +# More details can be found in the FreeBSD bhyve wiki pages +# +passthru0="" + +# start_slot +# The slot to start creating devices at inside the guest. Note that +# we create disk devices first, and some UEFI guests require disks to +# be in slots 3-6. The default is 4, with 3 being left available for +# an installation ISO +# +start_slot="4" + +# install_slot +# The slot to use for an installation ISO. By default this is 3, +# which is the first available slot with the original UEFI firmware. +# Using this makes sure the ISO is the first device, and leaves +# 4-6 available for hd devices. Being able to change this may +# be useful for non-UEFI guests, especially if a passthru device +# requires this slot. +# +install_slot="3" + +# virt_random +# Set to any value other than [empty]/off/false/no/0 to create +# a virtio-rnd device for the guest +# +virt_random="" + +# graphics +# Set to a value other than [empty]/off/false/no/0 to enable +# the bhyve frame buffer device. This creates a graphics console +# in the guest, which is accessible using vnc +# +# By default this is set at 800x600, and we find an available vnc +# port starting at 5900. The port can be seen in vm list|info output. +# +graphics="yes" + +# graphics_port +# Use this option to specify a fixed network port that the vnc service +# should listen on. If specifying port numbers manually, please make +# sure all guests have a unique port. +# +graphics_port="5999" + +# graphics_listen +# By default, the vnc service will listen on 0.0.0.0, so you can connect by +# using any IP address assigned to the bhyve host. Use this option if you +# want to specify a specific IP address that the service should bind to +# +# Default: 0.0.0.0 +# +graphics_listen="10.0.0.1" + +# graphics_res +# This allows you to specify a resolution for the graphical console. +# Pleas note only the below options are supported +# +# Default: 800x600 +# Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480 +# +graphics_res="800x600" + +# graphics_wait +# Set to yes in order to make guest boot wait for the VNC console +# to be opened. This can help when installing operating systems +# that require immediate keyboard input (such as a timed 'enter setup' +# screen). The default setting of auto will add the wait option +# if the guest is run in install mode. Note that in auto mode +# the wait option will only be present on the first boot. If you +# need the guest to wait on every boot during install, the yes +# option should be used. +# +# Valid Options: no,yes,auto +# +graphics_wait="auto" + +# graphics_vga +# valid options for this are on/off/io. io is the default +# please see the bhyve man page for details on this option +# +graphics_vga="io" + +# xhci_mouse +# When graphics are enabled, a PS2 mouse is created by default. This +# doesn't track very well, and can be replaced with an XHCI mouse +# by setting this option to yes. Please note only some guests support +# this mouse +# +xhci_mouse="yes" + +# virt_console0 +# create up to 16 virtual console devices +# +# the value can be yes|on|1 to create a numbered port. FreeBSD < 12 +# only supports virtio consoles configured in this way +# +# For guests with named console support (FreeBSD 12+, Linux?), the +# value can be the name of the port to create. The name "org.freenas.byhve-agent" +# can be useful as it ties in with tools written to make use of the +# FreeNAS bhyve-agent interface. +# +virt_console0="org.freenas.byhve-agent" + +# grub_install0 +# use this to specify grub commands that should be run inside the +# guest when installing. +# +# If more than one command is needed, you can specify this option +# multiple times, incrementing the number on the end each time +# +grub_install0="..." +grub_install1="..." + +# grub_run0 +# use this to specify grub commands to run when starting the guest +# normally +# +grub_run0="linux ..." +grub_run1="initrd ..." + +# grub_run_partition +# by default 'hd0,1' is specified as the root when running grub-bhyve. +# to force 'hd0,X' instead, set this to 'X'. +# in most cases the default of partition 1 is correct, although +# this settings allows you to force grub-bhyve to look on a different +# partition if required. +# +grub_run_partition="msdos1" + +# grub_run_dir +# by default grub-bhyve will look in /boot/grub for the guests +# grub config file. use this to specify an alterate path +# +# Default: /boot/grub (set by grub-bhyve) +# +grub_run_dir="/grub" + +# grub_run_file +# by default grub-bhyve will look for a file called grub.cfg containing +# the guests grub configuration. use this to specify an alternate filename +# +# Default: grub.cfg (set by grub-bhyve) +# +grub_run_file="grub.conf" + +# zfs_dataset_opts +# A list of ZFS properties to set on any new dataset created for this +# guest. Multiple properties can be specified, separated by a space. +# As a dataset is created while provisioning a new guest, this option +# makes most sense when specified inside a template. +# +# Please note that spaces are currently not supported in the field values +# +zfs_dataset_opts="" + +# zfs_zvol_opts +# A list of ZFS properties to set on any new ZVOL created for this guest. +# As with dataset_opts, this should to be set inside the guest template +# if you need the properties to apply to the guest as it is created. +# Some options such as volblocksize cannot be changed once the guest +# disk has been created. +# +zfs_zvol_opts="" + +# prestart +# specify a script to run when the guest starts +# if just a name rather than full path is provided, we look in the guest directory +# the script must be executable and is run in the following way - +# +# {scriptname} [zfs-dataset?] +# +# we also change directory to before running the script +# note that if taking guest snapshots, the -f option must be used as although +# the guest is technically stopped when this script runs, vm-bhyve still has it +# locked +# +prestart="myscript.pl" + +# priority +# set a priority (nice value) for a guest +# valid range is -20 (highest) to 20 (only run when system idle), with +# 0 being the default system priority +# +priority="10" + +# limit_pcpu +# use rctl to limit guest to the specified cpu percentage +# +limit_pcpu="" + +# limit_rbps, limit_wbps, limit_riops, limit_wiops +# Configure additional rctl limits available on 11+ +# These limit read/write throughput and iops +# +limit_rbps="" +limit_wbps="" +limit_riops="" +limit_wiops="" diff --git a/sample-templates/coreos.conf b/sample-templates/coreos.conf new file mode 100644 index 0000000..e762bae --- /dev/null +++ b/sample-templates/coreos.conf @@ -0,0 +1,14 @@ +# Use GRUB when booting from an installation medium +#loader="grub" + +# Use UEFI when booting from a disk +loader="uefi" + +cpu=1 +memory=1024M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /coreos/vmlinuz coreos.autologin" +grub_install1="initrd /coreos/cpio.gz" diff --git a/sample-templates/debian.conf b/sample-templates/debian.conf new file mode 100644 index 0000000..3c7fffd --- /dev/null +++ b/sample-templates/debian.conf @@ -0,0 +1,9 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="ahci-hd" +disk0_name="disk0.img" +grub_run_partition="1" +grub_run_dir="/boot/grub" diff --git a/sample-templates/default.conf b/sample-templates/default.conf new file mode 100644 index 0000000..a95af04 --- /dev/null +++ b/sample-templates/default.conf @@ -0,0 +1,7 @@ +loader="bhyveload" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" diff --git a/sample-templates/dragonfly.conf b/sample-templates/dragonfly.conf new file mode 100644 index 0000000..aed3d06 --- /dev/null +++ b/sample-templates/dragonfly.conf @@ -0,0 +1,14 @@ +# DragonFly 4.6 and later can boot with UEFI. The Dragonfly installer works +# with UEFI beginning at 4.8 +loader="uefi" +# The live CD has a serial console, but the installer requires graphics +graphics="yes" +graphics_wait="no" +cpu=1 +# 4GB of RAM is the minimum when using HAMMER. +memory=4G +network0_type="virtio-net" +network0_switch="public" +# The installer requires ahci. It can't correctly partition a virtio-blk +disk0_type="ahci-hd" +disk0_name="disk0.img" diff --git a/sample-templates/freebsd-zvol.conf b/sample-templates/freebsd-zvol.conf new file mode 100644 index 0000000..9efa8e9 --- /dev/null +++ b/sample-templates/freebsd-zvol.conf @@ -0,0 +1,8 @@ +loader="bhyveload" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0" +disk0_dev="sparse-zvol" diff --git a/sample-templates/freepbx.conf b/sample-templates/freepbx.conf new file mode 100644 index 0000000..4a53b23 --- /dev/null +++ b/sample-templates/freepbx.conf @@ -0,0 +1,12 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /isolinux/vmlinuz vga=normal ramdisk_size=32768 ks=cdrom:/kickstart-simple-asterisk13.cfg asknetwork LANG=en_US.UTF-8 KEYTABLE=us SYSFONT=latarcyrheb-sun16 console=ttyS0" +grub_install1="initrd /isolinux/initrd.img" +grub_run0="linux /vmlinuz-2.6.32-642.6.2.el6.x86_64 root=/dev/mapper/VolGroup-lv_root LANG=en_US.UTF-8 KEYTABLE=us SYSFONT=latarcyrheb-sun16 rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 console=ttyS0 crashkernel=auto rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM" +grub_run1="initrd /initramfs-2.6.32-642.6.2.el6.x86_64.img" + diff --git a/sample-templates/gentoo.conf b/sample-templates/gentoo.conf new file mode 100644 index 0000000..60caf09 --- /dev/null +++ b/sample-templates/gentoo.conf @@ -0,0 +1,13 @@ +loader="grub" +cpu=1 +memory=512MB +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="linux /boot/gentoo root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs loop=/image.squashfs cdroot" +grub_install1="initrd /boot/gentoo.igz" +# Make sure to modify the "root" variable according to your partitioning scheme. +grub_run0="set root=(hd0,gpt2)" +grub_run1="set timeout=1" +grub_run2="configfile /grub/grub.cfg" diff --git a/sample-templates/linux-zvol.conf b/sample-templates/linux-zvol.conf new file mode 100644 index 0000000..3a17026 --- /dev/null +++ b/sample-templates/linux-zvol.conf @@ -0,0 +1,8 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_name="disk0" +disk0_dev="sparse-zvol" +disk0_type="virtio-blk" diff --git a/sample-templates/netbsd.conf b/sample-templates/netbsd.conf new file mode 100644 index 0000000..10c8a55 --- /dev/null +++ b/sample-templates/netbsd.conf @@ -0,0 +1,9 @@ +loader="grub" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="knetbsd -h -r cd0a /netbsd" +grub_run0="knetbsd -h -r dk0 /netbsd" diff --git a/sample-templates/openbsd.conf b/sample-templates/openbsd.conf new file mode 100644 index 0000000..5bcae3f --- /dev/null +++ b/sample-templates/openbsd.conf @@ -0,0 +1,10 @@ +loader="grub" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_install0="kopenbsd -h com0 /6.2/amd64/bsd.rd" +grub_run0="kopenbsd -h com0 -r sd0a /bsd" +bhyve_options="-w" diff --git a/sample-templates/resflash.conf b/sample-templates/resflash.conf new file mode 100644 index 0000000..1341559 --- /dev/null +++ b/sample-templates/resflash.conf @@ -0,0 +1,10 @@ +loader="grub" +cpu=1 +memory=256M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" +grub_run_partition="openbsd4" +grub_run0="kopenbsd -h com0 -r sd0d /bsd" +bhyve_options="-w" diff --git a/sample-templates/ubuntu.conf b/sample-templates/ubuntu.conf new file mode 100644 index 0000000..4024598 --- /dev/null +++ b/sample-templates/ubuntu.conf @@ -0,0 +1,7 @@ +loader="grub" +cpu=1 +memory=512M +network0_type="virtio-net" +network0_switch="public" +disk0_type="virtio-blk" +disk0_name="disk0.img" diff --git a/sample-templates/windows.conf b/sample-templates/windows.conf new file mode 100644 index 0000000..4566fce --- /dev/null +++ b/sample-templates/windows.conf @@ -0,0 +1,21 @@ +loader="uefi" +graphics="yes" +xhci_mouse="yes" +cpu=2 +memory=2G + +# put up to 8 disks on a single ahci controller. +# without this, adding a disk pushes the following network devices onto higher slot numbers, +# which causes windows to see them as a new interface +ahci_device_limit="8" + +# ideally this should be changed to virtio-net and drivers installed in the guest +# e1000 works out-of-the-box +network0_type="e1000" +network0_switch="public" + +disk0_type="ahci-hd" +disk0_name="disk0.img" + +# windows expects the host to expose localtime by default, not UTC +utctime="no" diff --git a/vm b/vm new file mode 100644 index 0000000..372727f --- /dev/null +++ b/vm @@ -0,0 +1,49 @@ +#!/bin/sh +#-------------------------------------------------------------------------+ +# Copyright (C) 2016 Matt Churchyard (churchers@gmail.com) +# All rights reserved +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted providing that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +# get libs +if [ -e "/usr/local/lib/vm-bhyve" ]; then + LIB="/usr/local/lib/vm-bhyve" +else + echo "unable to locate vm-bhyve libraries" + exit 1 +fi + +# load libs +. "${LIB}/vm-cmd" +. "${LIB}/vm-config" +. "${LIB}/vm-core" +. "${LIB}/vm-datastore" +. "${LIB}/vm-guest" +. "${LIB}/vm-info" +. "${LIB}/vm-migration" +. "${LIB}/vm-rctl" +. "${LIB}/vm-run" +. "${LIB}/vm-switch" +. "${LIB}/vm-util" +. "${LIB}/vm-zfs" + +. "${LIB}/vm-base" diff --git a/vm.8 b/vm.8 new file mode 100644 index 0000000..bb36f83 --- /dev/null +++ b/vm.8 @@ -0,0 +1,1605 @@ +.Dd November 16, 2016 +.Dt VM-BHYVE 8 +.Os +.Sh NAME +.Nm vm +.Nd "utility to manage bhyve virtual machines" +.Sh SYNOPSIS +.Nm +.Cm version +.Nm +.Cm init +.Pp +.Nm +.Cm get +.Ar all +.Op Ar setting +.Op Ar ... +.Nm +.Cm set +.Ar setting=value +.Op Ar ... +.Pp +.Nm +.Cm switch list +.Nm +.Cm switch info +.Op Ar name +.Op Ar ... +.Nm +.Cm switch create +.Op Fl t Ar type +.Op Fl i Ar interface +.Op Fl n Ar vlan-id +.Op Fl b Ar bridge +.Op Fl m Ar mtu +.Op Fl a Ar address +.Op Fl p +.Ar name +.Nm +.Cm switch vlan +.Ar name vlan-id +.Nm +.Cm switch nat +.Ar name on|off +.Nm +.Cm switch address +.Ar name a.b.c.d/xx|none +.Nm +.Cm switch private +.Ar name on|off +.Nm +.Cm switch add +.Ar name interface +.Nm +.Cm switch remove +.Ar name interface +.Nm +.Cm switch destroy +.Ar name +.Pp +.Nm +.Cm datastore list +.Nm +.Cm datastore add +.Ar name spec +.Nm +.Cm datastore remove +.Ar name +.Nm +.Cm datastore iso +.Ar name path +.Pp +.Nm +.Cm create +.Op Fl d Ar datastore +.Op Fl t Ar template +.Op Fl s Ar size +.Ar name +.Nm +.Cm +.Cm destroy +.Op Fl f +.Ar name +.Nm +.Cm list +.Nm +.Cm info +.Op Ar name +.Op Ar ... +.Nm +.Cm install +.Op Fl fi +.Ar name iso +.Nm +.Cm start +.Op Fl fi +.Ar name +.Ar ... +.Nm +.Cm stop +.Ar name +.Ar ... +.Nm +.Cm restart +.Ar name +.Nm +.Cm console +.Ar name +.Op Ar com1|com2 +.Nm +.Cm rename +.Ar name +.Ar new-name +.Nm +.Cm add +.Op Fl d Ar device +.Op Fl t Ar type +.Op Fl s Ar size|switch +.Ar name +.Nm +.Cm reset +.Op Fl f +.Ar name +.Nm +.Cm poweroff +.Op Fl f +.Ar name +.Nm +.Cm startall +.Nm +.Cm stopall +.Op Fl f +.Nm +.Cm configure +.Ar name +.Nm +.Cm passthru +.Nm +.Cm clone +.Ar name[@snapshot] +.Ar new-name +.Nm +.Cm snapshot +.Op Fl f +.Ar name|name@snapshot +.Nm +.Cm rollback +.Op Fl r +.Ar name@snapshot +.Nm +.Cm migrate +.Op Fl s12t +.Op Fl r Ar name +.Op Fl i Ar snapshot +.Op Fl d Ar datastore +.Ar guest host +.Pp +.Nm +.Cm iso +.Op Ar -d datastore +.Op Ar url +.Pp +.Nm +.Cm image list +.Nm +.Cm image create +.Op Fl d Ar description +.Op Fl u +.Ar name +.Nm +.Cm image provision +.Op Fl d Ar datastore +.Ar uuid +.Ar new-name +.Nm +.Cm image destroy +.Ar uuid +.\" ============ DESCRIPTION ============= +.Sh DESCRIPTION +The +.Nm +utility is used to provide simplified management of +.Xr bhyve 8 +virtual machines, +including networking and console access. +.Pp +Networking is handled by creating one or more virtual switches. +Each switch has a simple name which is referenced in the virtual machine +configuration file. +The +.Nm +utility automatically creates a +.Xr bridge 4 +device for each virtual switch and assigns virtual machine +.Xr tap 4 +interfaces dynamically. +.Pp +All configuration for virtual machines is stored in a simple rc style +configuration file. +When virtual machines are first created, the configuration file is copied from +a template which can be specified by the user. +Multiple templates can be created providing an easy way to provision guests +with specific configurations. +.Pp +.Nm +gracefully handles reboot and shutdown commands from inside the guests, whilst +providing full management of the virtual machine from the host system. +.\" ============ BASIC SETUP ============ +.Sh BASIC SETUP +Once +.Nm +is installed, create the directory which will store your virtual machine +configuration and data. +This directory will be referred to as +.Pa $vm_dir +throughput this man page. +.Pp +Add the following into +.Pa /etc/rc.conf +.Bd -literal -offset indent +vm_enable="YES" +vm_dir="/your/vm/path" +vm_list="" +vm_delay="5" +.Ed +.Pp +The first and second lines are required to enable the +.Nm +utility. +Please see the +.Cm startall +command description for more information on the third and fourth settings. +.Pp +Now run the +.Nm vm +.Cm init +command to finish initialisation. +This will create subdirectories inside +.Pa $vm_dir +to hold configuration and templates. +It will also load any required kernel modules. +This command needs to be run on each boot, which is normally handled by the +rc.d script. +.Pp +Sample templates are installed to +.Pa /usr/local/share/examples/vm-bhyve/ . +You can make use of these by copying them into your +.Pa $vm_dir/.templates/ +directory. +You can create and edit the templates as required. +It is recommended to keep a template called +.Pa default.conf , +as this will be used when no template is manually specified. +.\" ============ ZFS ============= +.Sh ZFS +If you are using a ZFS dataset to store your virtual machines, and want a new +child dataset created for each one, specify the dataset to use in +.Pa /etc/rc.conf +as follows: +.Bd -literal -offset indent +vm_dir="zfs:pool/dataset" +.Ed +.Pp +In contrast to earlier versions, if +.Pa $vm_dir +is a normal path, a standard subdirectory will be created for each virtual +machine, regardless +of the file system type. +However, +.Nm +is now able to handle situations where the dataset mountpoint does not match +the dataset name. +.\" ============ QUICKSTART ============= +.Sh QUICKSTART +Create a virtual switch called +.Sy public +(which is the switch name specified in the default templates) and attach it to +a real interface. +Use your own interface in place of +.Sy em0 +as required. +.Bd -literal -offset ident +# vm switch create public +# vm switch add public em0 +.Ed +.Pp +Download an ISO file to use for installation: +.Bd -literal -offset ident +# vm iso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso +.Ed +.Pp +Create a new guest using the default template and disk size, then start the +installation. +The +.Ar install +subcommand will immediately return you to your shell. +Once started, use the +.Ar console +command to connect to the guest and complete the installation. +.Bd -literal -offset ident +# vm create my-guest +# vm install my-guest FreeBSD-10.1-RELEASE-amd64-disc1.iso +# vm console my-guest +.Ed +.Pp +Please note that Linux guests currently require the +.Sy sysutils/grub2-bhyve +package to be installed. +This is used in place of +.Xr bhyveload 8 +to load the guest kernel into memory. +.\" ============== WINDOWS =============== +.Sh WINDOWS SUPPORT +Windows guests are supported on versions of +.Fx +that have UEFI +support in +.Xr bhyve 8 . +As of April 2016, UEFI support should be available in +.Fx 10.3-RELEASE +and +.Fx 11-CURRENT . +.Pp +You will also need a copy of the UEFI firmware. +This can either be installed using the +.Pa sysutils/uefi-edk2-bhyve +port, or you can manually download a copy (see URL below) to +.Pa $vm_dir/.config/BHYVE_UEFI.fd and configure a guest to use it by setting +.Sy loader="uefi-custom" . +.Pp +If you are running +.Fx 10 +, there is no VGA console in +.Xr bhyve 8 , +and so an unattended installation ISO is required which allows Windows to install and +boot without any user interaction. +Instructions for creating a suitable ISO can be found at the URL below. +.Pp +On +.Fx 11 , +VGA access can be enabled by setting the +.Sy graphics="yes" +option in the guest configuration file. +Once the guest has started, vnc IP & port details can be seen in +.Sy vm list +output. +See the configuration format documentation below for more detailed information +on configuring graphics. +If network drivers are required, I recommend re-running the +.Sy vm install +command once the guest has been installed, but providing an ISO of the +virtio-net drivers instead. +.Pp +Once the installation ISO is ready, has been placed in the +.Pa $vm_dir/.iso +directory, and you have the UEFI firmware, installation can be performed as +normal. +.Bd -literal -offset indent +# vm create -t windows -s 30G winguest +# vm install winguest win_repack.iso +.Ed +.Pp +Windows installation has been tested with 2012r2 and takes around 20-25 +minutes. +During install, the guest will reboot twice (three runs in total). +You can see the guest reboot by watching the log file +.Pa $vm_dir/guestname/vm-bhyve.log . +The third run should boot fully into Windows. +The +.Sy virtio +network adapter will request an IP address using DHCP. +Connect to the guest console and press +.Sy i +to see the IP address that has been assigned. +The default unattended installation files should make RDP available, using +Administrator and Test123 as the default login details. +.Pp +A pre-compiled copy of the UEFI firmware (BHYVE_UEFI_20160526.fd), as well as +instructions for creating an +unattended installation ISO can currently be obtained from +.Lk https://people.freebsd.org/~grehan/bhyve_uefi/ +.\" ============ SUBCOMMANDS ============= +.Sh SUBCOMMANDS +.Bl -tag -width indent +.It Cm version +Show the version number of vm-bhyve installed. +.It Cm init +.br +This should be run once after each host reboot before running any other +.Nm +commands. +The main function of the +.Cm init +command is as follows: +.Pp +o Load all necessary kernel modules if not already loaded +.br +o Set tap devices to come up automatically when opened +.br +o Create any configured virtual switches +.It Cm get Ar all|setting +Get a global configuration setting. +These are settings that affect the functionality of vm-bhyve, such as +configuring the type of serial console to use. +The keyword +.Sy all +can be used to retrieve all user configurable settings, or you can specify one or +more settings by name, separated by a space. +.It Cm set Ar setting=value +Sets the value of a global configuration setting. +Multiple settings can be changed at the same time by seperating the +.Sy setting=value +pairs with a space. +.Pp +These settings are stored in +.Pa $vm_dir/.config/system.conf +.It Cm switch list +List virtual switches. +This reads all configured virtual switches from the +.Pa $vm_dir/.config/switch +file and displays them. +If the virtual switches are loaded, it also tries to display the +.Xr bridge 4 +interface that has been assigned to each one. +.It Cm switch info Op Ar name Op Ar ... +This command shows detailed information about the specified virtual switch(es). +If no switch names are provided, information is output for all configured +switches. +Information displayed includes the following: +.Pp +o Basic switch settings +.br +o Overall bytes sent and received via this switch +.br +o Physical ports connected +.br +o Virtual ports, including the associated virtual machine +.It Xo +.Cm switch create +.Op Fl t Ar type +.Op Fl i Ar interface +.Op Fl n Ar vlan-id +.Op Fl b Ar bridge +.Op Fl m Ar mtu +.Op Fl a Ar address +.Op Fl p +.Ar name +.Xc +Create a new virtual switch. +The name must be supplied and may only contain +letters, numbers and dashes. +However, it may not contain a dash at the beginning or end. +Note that the maximum length of a switch name is also limited to +12 characters, due to the way we use this as the interface name. +.Pp +There are currently 4 types of virtual switch that can be created. +These are +.Sy standard , +.Sy manual , +.Sy netgraph , +.Sy vale +and +.Sy vxlan . +The default type is +.Sy standard , +which creates a basic +.Xr bridge 4 +interface and bridges clients to it. +.Sy manual +allows you to attach guests to a bridge that you have created and configured +manually. +.Sy netgraph +switches use the netgraph ng_bridg esystem to create a virtual switch connecting +guests. +.Sy vale +switches use the netmap VALE system to create a virtual switch connecting +guests. +.Sy vxlan +allows you to create virtual LANs (similar to a VLAN) which tunnel L2 guest +traffic over L3. +.Bl -tag -width 12n +.It Fl t Ar type +The type of virtual switch to create. +The available types are listed above. +This defaults to +.Sy standard +if not specified. +.It Fl i Ar interface +For +.Sy standard +and +.Sy vxlan +switches you can attach a physical interface at creation time. +This option is required for vxlan switches. +.It Fl n Ar vlan-id +Allows you to specify a VLAN ID for +.Sy standard +and +.Sy vxlan +switches. +This option is required for vxlan switches. +.It Fl b Ar bridge +If creating a manual switch using an existing bridge on your system, this +option allows +you to specify the name of the bridge interface you would like to use. +This option is required for manual switches. +.It Fl m Ar mtu +Specify an mtu to use for the bridge interface. +.It Fl a Ar address +This allows you to specify an IP address that is assigned to the bridge +interface. +This should be specified in +.Sy a.b.c.d/prefix-len +CIDR notation. +.It Fl p +Use this option to create a private switch. +If this is enabled, no traffic will be allowed between guests on the same +switch, however then will all be able to communicate with any physical +interfaces added to the switch. +.El +.It Cm switch vlan Ar name Ar vlan-id +Assign a VLAN number to a virtual switch. +The VLAN number must be between 0-4094. +.Pp +When adding an interface to a VLAN enabled virtual switch, a new +.Xr vlan 4 +interface is created. +This interface has the relevant parent interface and VLAN tag configured. +This vlan interface is then added to the virtual switch. +As such, all traffic between guests on the same switch is untagged and travels +freely. +However, all traffic exiting via physical interfaces is tagged. +.Pp +If the virtual switch already has physical interfaces assigned, they are all +removed from the bridge, reconfigured, then re-added. +.Pp +To remove the VLAN configuration from a virtual switch, specify a +.Ar vlan-id +of 0. +.It Cm switch address Ar name Ar a.b.c.d/xx|none +Configure an IP address for the specified virtual switch. The address should +be specified in CIDR notation. To remove an address, specify +.Pa none +in place of the address. +.Pp +If NAT funtionality is required, please configure an address on the switch to +become the gateway address for guests. Source NAT rules can then be created +using your choice of firewall or NAT daemon. If DHCP is desired, we recommend +using a manual switch and configuring this by hand. +.It Cm switch private Ar name Ar on|off +Enable of disable private mode for a virtual switch. +In private mode, guests will only be able to communicate with the physical +interface(s), not with each other. +.Pp +Please note that changing this setting does not affect guests that are already +running, but will be applied to any guests started from cold-boot thereafter. +.It Cm switch add Ar name Ar interface +Add the specified interface to the named virtual switch. +.Pp +The interface will immediately be added to the relevant bridge if possible, and +stored in the persistent switch configuration file. +If a +.Ar vlan-id +is specified on the virtual switch, this will cause a new +.Xr vlan 4 +interface to be created. +.It Cm switch remove Ar name Ar interface +Removes the specified interface from the named virtual switch and updates the +persistent configuration file. +.It Cm switch destroy Ar name +Completely remove the named virtual switch and all configuration. +The associated +.Xr bridge 4 +interface will be removed, as well as any +.Xr vlan 4 +interfaces if they are not in use by other virtual switches. +.It Cm datastore list +List the configured datastores. +Normally +.Sy vm-bhyve +will store all guests under the directory specified in +.Pa /etc/rc.conf . +This is the +.Sy default +datastore. +Additional datastores can be added, providing the ability to store guests in +multiple locations on your system. +.It Cm datastore add Ar name spec +Add a new datastore to the system. +The datastore name can only contain letters, numbers and _. characters. +The +.Pa spec +should use the same format as +.Sy $vm_dir . +A standard directory can be specified by just providing the path, whereas a ZFS +storage location should be specified in +.Sy zfs:pool/dataset +format. +.Pp +Please note that the directory or dataset should already exist. +We do not try to create it. +.It Cm datastore remove Ar name +Remove the specified datastore from the list. +This does not destroy the directory or dataset, leaving all files intact. +.It Cm datastore iso Ar name path +Adds a new datastore location for storing iso files. +Guests cannot be created in an iso store, but this provides an easy way to +configure vm-bhyve to look in any arbitrary location on your system (or mounted +network share) where you may want to store iso images. +.It Xo +.Cm create +.Op Fl d Ar datastore +.Op Fl t Ar template +.Op Fl s Ar size +.Ar name +.Xc +Create a new virtual machine. +.Pp +Unless specified, the +.Pa default.conf +template will be used and a 20GB virtual disk image is created. +This command will create the virtual machine directory +.Pa $vm_dir/$name , +and create the configuration file and empty disk image within. +.Bl -tag -width 12n +.It Fl d Ar datastore +Specify the datastore to create this virtual machine under. +If not specified, the +.Sy default +dataset will be used, which is the location specified in +.Pa /etc/rc.conf . +.It Fl t Ar template +Specifies the template to use from within the +.Pa $vm_dir/.templates +directory. +The +.Sy .conf +suffix should not be included. +.It Fl s Ar size +The size of disk image to create in bytes. +Unless specified, the guest image will be a sparse file 20GB in size. +.El +.It Cm destroy Ar name +Removes the specified virtual machine from the system, deleting all associated +disk images & configuration. +.It Cm list +.br +List all the virtual machines in the +.Pa $vm_dir +directory. +This will show the basic configuration for each virtual machine, and whether +they are currently running. +.It Cm info Op Ar name Op Ar ... +Shows detailed information about the specified virtual machine(s). +If no names are given, information for all virtual machines is displayed. +.Pp +This output includes detailed information about network and disk devices, +including the space usage for all virtual disks (excluding custom disk +devices). +If the guest is running, the output also shows the amount of host memory +currently in use, and additional network details including bytes sent/received +for each virtual interface. +.It Xo +.Cm install +.Op Fl fi +.Ar name Ar iso +.Xc +Start a guest installation for the named virtual machine, using the specified +ISO file or install disk image. +The +.Ar iso +argument should be the filename of an ISO or image file already downloaded into the +.Pa $vm_dir/.iso +directory (or any media datastore), a full path, or a file in the current +directory. +ISO files in the default .iso store can be downloaded using the +.Ar iso +subcommand described below. +.Pp +By default the installation is started in the background. +Use the +.Ar console +command to connect and begin the installation. +.Pp +After installation, the guest can be rebooted and will restart using its own +disk image to boot. +At this point the installation ISO file is still attached, allowing you to use +the CD/DVD image for any post installation tasks. +The ISO file will remain attached after each reboot until the guest is fully +stopped. +.Pp +If the +.Ar -f +option is specified, the guest will be started in the foreground on stdio. +The +.Ar -i +option starts the guest in interactive mode. +This requires tmux, and the global +.Sy console +setting must be set likewise. +In interactive mode the guest is started on a foreground tmux session, but this +can be detached using the standard tmux commands. +.It Xo +.Cm start +.Op Fl fi +.Ar name Ar ... +.Xc +Start the named virtual machine(s). +The guests will boot and run completely in the background. +Use the +.Ar console +subcommand to connect to it if required. +.Pp +For each network adapter specified in the guest configuration, a +.Xr tap 4 +interface will be created. +If possible, the tap interface will be attached the relevant +.Xr bridge 4 +interface, based on the virtual switch specified in the guest configuration. +.Pp +If the +.Ar -f +option is specified, the guest will be started in the foreground on stdio. +The +.Ar -i +option starts the guest in interactive mode. +This requires tmux, and the global +.Sy console +setting must be set likewise. +In interactive mode the guest is started on a foreground +tmux session, but this can be detached using the standard tmux commands. +.It Cm stop Ar name Ar ... +Stop a named virtual machine. +All +.Xr tap 4 +and +.Xr nmdm 4 +devices will be automatically cleaned up once the guest has exited. +.Pp +If a guest is stuck in the bootloader stage, you are given the option to forcibly stop it. +.Pp +Multiple guests can be specified to this command at the same time. +Each one will be sent a poweroff event. +.It Cm restart Ar name +Attempt to restart the specified guest. This causes a shutdown event to be sent to the +guest, however, vm-bhyve will restart the guest rather than stopping completely. +.Pp +A benfit of using this function is that vm-bhyve will not destroy and recreate network devices like +it would when using +.Sy stop/start . +Note that guest configuration is not re-loaded, so all guest settings will be as they were +when the guest was originally started. +.It Cm console Ar name Op Ar com1|com2 +Connect to the console of the named virtual machine. +Without network access, this is the primary way of connecting to the guest once +it is running. +.Pp +By default this will connect to the first com port specified in the client +configuration, which is usually com1. +Alternatively you can specify the com port to connect to. +.Pp +This looks for the +.Xr nmdm 4 +device associated with the virtual machine, and connects to it with +.Xr cu 1 . +Use ~+Ctrl-D to exit the console and return to the host. +.It Cm rename Ar name Ar new-name +Renames the specified virtual machine. +The guest must be stopped to use this function. +.It Xo +.Cm add +.Op Fl d Ar device +.Op Fl t Ar type +.Op Fl s Ar size|switch +.Ar name +.Xc +Add a new network or disk device to the named virtual machine. +The options depend on the type of device that is being added: +.Bl -tag -width 15n +.It Fl d Ar device +The type of device to add. +Currently this can either be +.Pa disk +or +.Pa network +.It Fl t Ar type +For disk devices, this specifies the type of disk device to create. +Valid options for this are +.Pa zvol , +.Pa sparse-zvol +and +.Pa file . +If not specified, this defaults to +.Pa file . +.It Fl s Ar size|switch +For disk devices, this is used to specify the size of the disk image to create. +For network devices, use this option to specify the virtual switch to connect +the network interface to. +.El +.Pp +For both types of device, the emulation type will be chosen automatically based +on the emulation used for the existing guest devices. +.It Xo +.Cm reset +.Op Fl f +.Ar name +.Xc +Forcefully reset the named virtual machine. +This can cause corruption to the guest file system just as with real hardware +and should only be used if necessary. +.It Xo +.Cm poweroff +.Op Fl f +.Ar name +.Xc +Forcefully power off the named virtual machine. +As with +.Ar reset +above, this does not inform the guest to shutdown gracefully and should only be +used if the guest +can not be shut down using normal methods. +.It Cm startall +Start all virtual machines configured for auto-start. +This is the command used by the rc.d scripts to start all machines on boot. +.Pp +The list of virtual machines should be specified using the +.Pa $vm_list +variable in +.Pa /etc/rc.conf . +This allows you to use shared storage for virtual machine data, whilst making +sure that the correct guests are started automatically on each host. +(Or to just make sure your required guests start on boot whilst leaving +test/un-needed guests alone) +.Pp +The delay between starting guests can be set using the +.Pa $vm_delay +variable, which defaults to 5 seconds. +Too small a delay can cause problems, as each guest doesn't have enough time to +claim a null modem device before the next guest starts. +Increasing this value can be useful if you have disk-intensive guests and want +to give each guest a chance to fully boot before the next starts. +.It Cm stopall +Stop all running virtual machines. +This sends a stop command to all +.Xr bhyve 8 +instances, regardless of whether they were starting using +.Nm +or not. +.It Cm configure Ar name +The +.Cm configure +command simply opens the virtual machine configuration file in your default +editor, allowing you to easily make changes. +Please note, changes do not take effect until the virtual machine is fully +shutdown and restarted. +.It Cm passthru +The +.Cm passthru +command lists all PCI devices in the system, the device ID required for bhyve, +and whether the device is currently ready to be used by a guest. +In order to make a device ready, it needs to be reserved on boot by adding the +device ID to the +.Sy pptdevs +variable in +.Pa /boot/loader.conf . +.Pp +Once a device is ready, it can be assigned to a guest by adding +.Sy passthruX="{ID}" +to the guest's configuration file. +.Sy X +should be an integer starting at 0 for the first passthrough device. +.Pp +More details can be found in the bhyve wiki. +.It Cm clone Ar name[@snapshot] Ar new-name +Create a clone of the virtual machine +.Pa name , +as long as it is currently powered off. +The new machine will be called +.Pa new-name , +and will be ready to boot with a newly assigned UUID and empty log file. +.Pp +If no snapshot name is given, a new snapshot will be taken of the guest and any +descendant datasets or ZVOLs. +If you wish to use an existing snapshot as the source for the clone, please +make sure the snapshot exists for the guest and any child ZVOLs, otherwise the +clone will fail. +.Pp +Please note that this function requires ZFS. +.It Xo +.Cm snapshot +.Op Fl f +.Ar name|name@snapshot +.Xc +Create a snapshot of the names virtual machine. +This command is only supported with ZFS and will take a snapshot of the guest +dataset and any descendant ZVOL devices. +.Pp +The guest and snapshot name can be specified in the normal +.Pa name@snapshot +way familiar to ZFS users. +If no snapshot name is given, the snapshot is based on the current timestamp in +.Pa Y-m-d-H:M:S +format. +.Pp +By default the guest must be stopped to use this command, although you can +force a snapshot of a running guest by using the +.Fl f +option. +.It Xo +.Cm rollback +.Op Fl r +.Ar name@snapshot +.Xc +Rollback the guest to the specified snapshot. +This will roll back the guest dataset and all descendant ZVOL devices. +.Pp +Normally, ZFS will only allow you to roll back to the most recent snapshot. +If the snapshot given is not the most recent, ZFS will produce a warning +detailing that you need to use the +.Fl r +option to remove the more recent snapshots. +It will also produce a list of the snapshots that will be destroyed if you use +this option. +The +.Fl r +option can be passed directly into +.Nm +.Cm rollback +.Pp +The guest must always be stopped to use this command. +.It Xo +.Cm migrate +.Op Fl s12t +.Op Fl r Ar name +.Op Fl i Ar snapshot +.Op Fl d Ar datastore +.Ar guest host +.Xc +The migrate command allows transferring a guest from one host to another. Note that +currently this involves shutting down the guest, and optionally restarting it once +migration is complete. +.Pp +The migration process uses ssh, and works best if key-based ssh is enabled between +your hosts without the requirement of a password. Transfer is still possible using a password, +but you will be prompted for this several times during the transfer process. +.Pp +Firstly a full snapshot of the guest is sent while the guest is still running. Optionally, an intermediate +incremental snapshot can then be sent to bring the remote guest up to date if it is expected that the full +send may take a long time, or that a large amount of data may change during this time. Once the remote end +is reasonably up to date, the guest is powered off so a final incremental snapshot can be sent. +.Bl -tag -width 12n +.It Fl r Ar name +Allows the remote guest to be given a different name to the source. +.It Fl d Ar datastore +Specify the datastore to store the guest on the destination host. +.It Fl s +Start the guest on the remote host once migration is complete. +.It Fl 1 +Run only the first stage of migration. This will take a full snapshot of the local guest and send it to +the destination host. +.It Fl 2 +Run only the second stage. This will second an incremental snapshot and then complete the migration. +This requires the +.Fl i +parameter to specify the source snapshot. +.It Fl t +Triple snapshot mode. This will send both a full snapshot, and one incremental, before shutting the guest down +and doing a final incremental send. This may be useful for large or busy guests where there could be a large number +of chages during the initial full send. The idea is that the first incremental send will bring the remote guest nearly +up to date, sending changes that have occured during the lengthy inital full send. This should reduce the size of the +final incremental send, minimising the amount of time the guest is powered off. +.It Fl x +Destroy the local guest once the migration is complete. +.It Fl i Ar snapshot +When running the second stage of migration, this parameter is used to specify the name of the snapshot +to base the incremental send on. This snapshot must exist on both hosts. +.El +.It Xo +.Cm iso +.Op Ar -d datastore +.Op Ar url +.Xc +List all the ISO files currently stored in the +.Pa $vm_dir/.iso +directory. +This is often useful during guest installation, allowing you to copy and paste +the ISO filename. +.Pp +If a +.Sy url +is specified, instead of listing ISO files, it attempts to download the given file using +.Xr fetch 1 +to the default datastore. The target datasource can be changed by specifying +.Sy -d datastore +with +.Sy url . +.It Cm image list +List available images. +Any virtual machine can be packaged into an image, which can then be used to +create additional machines. +All images have a globally unique ID (UUID) which is used to identify them. +The list command shows the UUID, the original machine name, the date it was +created and a short description of the image. +.Pp +Please note that these commands rely on using ZFS features to package/unpackage +the images, and as such are only available when using a ZFS dataset as the +storage location. +.It Xo +.Cm image create +.Op Fl d Ar description +.Op Fl u +.Ar name +.Xc +Create a new image from the named virtual machine. +This will create a compressed copy of the original guest dataset, which is +stored in the +.Pa $vm_dir/images +directory. +It also creates a +.Pa UUID.manifest +file which contains details about the image. +.Pp +Once complete, it will display the UUID which has been assigned to this image. +.Pp +If you do not want the image to be compressed, specify the +.Sy -u +option. +.It Xo +.Cm image provision +.Op Fl d Ar datastore +.Ar uuid Ar new-name +.Xc +Create a new virtual machine, named +.Pa new-name , +from the specified image UUID. +This will be created on the +.Sy default +datastore unless specified otherwise. +.It Cm image destroy Ar uuid +Destroy the specified image. +.El +.\" ============ GLOBAL CONFIGURATION =========== +.Sh GLOBAL CONFIGURATION +These configuration options are stored in +.Pa $vm_dir/.config/system.conf , +and affect the global functionality of vm-bhyve. +These settings can be changed by either editing the configuration file +manually, or using the +.Sy vm set +and +.Sy vm get +commands. +.Bl -tag -width 17n +.It console +Set the type of console to use, which defaults to +.Sy nmdm . +If you have the tmux port installed and would prefer to use that for guest +console access, you can set this option to +.Sy tmux . +.El +.\" ============ CONFIGURATION FORMAT =========== +.Sh GUEST CONFIGURATION FORMAT +Each virtual machine has a configuration file that specifies the hardware +configuration. +This uses a similar format to the +.Sy rc +files, making them easy to edit by hand. +The settings for each guest are stored in +.Pa $vm_dir/$vm_name/$vm_name.conf . +An overview of the available configuration options is listed below. +.Bl -tag -width 17n +.It loader +This option sets the loader to use for a guest and must be specified. +The valid options are +.Sy bhyveload , +.Sy grub , +.Sy uefi , +.Sy uefi-csm , +or +.Sy uefi-custom . +.It uefi_vars +Setting this option to a true value allows the persistent storage of UEFI variables. +This may be required for some guests that install boot firmware to a non-standard location +and rely on UEFI variables to locate it. The version of +.Sy uefi-firmware +installed must provide the template data file, and support also needs to be present +in +.Sy bhyve +.It bhyveload_loader +This option allows a custom path to be used for the loader inside the guest. +Passed to +.Sy bhyveload +using the +.Sy -l +argument. +.It bhyveload_args +This option allows extra arguments to be given for the loader inside the guest. +Appended verbatim to the +.Sy bhyveload +command line. +.It loader_timeout +By default the +.Sy bhyveload +and +.Sy grub +loaders will wait for 3 seconds before booting the default option. +If access to the grub console is needed, this can be increased to give more +time to connect to the console. +If access to the grub console is not required, it can also be reduced to speed +up overall boot. +.It cpu_sockets +Specify the number of CPU sockets that should be exposed to the guest. The product +of +.Sy sockets * cores * threads +should equal the number of cpus that has been configured. The ability to control +CPU topology on a per-guest basis requires +.Fx +12 or newer. On older systems, there are +.Sy vmm +sysctl variables available to configure these settings globally. +.It cpu_cores +The number of cores to create per CPU socket. +.It cpu_threads +The number of threads to create per CPU core. +.It memory +The amount of memory to assign to the guest. +This can be specified in megabytes or gigabytes using the +.Sy M +and +.Sy G +suffixes. +.It wired_memory +Set this to yes in order to have the requested amount of memory wired to the +guest. +.It hostbridge +This option allows you to specify the type of hostbridge used for the guest +hardware. +Normally you can leave this as default, which is to use a standard bhyve +hostbridge. +.Pp +There are two other options. +.Sy amd , +which is almost identical to the standard hostbridge, but advertises itself +with a vendor ID of AMD. +There are also some special cases where you may require no hostbridge at all, +which can be achieved using the +.Sy none +value. +.It comports +This option allows you to specify which com ports to create for the guest. +The default is to create a single +.Sy com1 +port. +Valid values for this are +.Sy com1 +and +.Sy com2 . +You can also connect two com ports by specifying both, separated by a space. +.It utctime +As of version 1.2, vm-bhyve defaults to +.Sy yes +for this option. +This causes bhyve to try and set the guests RTC clock to UTC rather than the +host's time. +I consider this more consistent, and should produce the correct time in the +guest as long as the timezone is correctly set. +Additionally, some guests actually expect a UTC realtime clock. +.Pp +If you require bhyve to use the host's time, as it would by default, explicitly +set this +to +.Sy no . +.It debug +If this is set to +.Sy yes , +all output from the +.Xr bhyve 8 +process will be written to +.Sy ${vm_dir}/guest/bhyve.log . +This is useful for debugging purposes as it allows you to see any error +messages that are being produced by +.Xr bhyve 8 +itself. +.It network0_type +The emulation to use for the first network adapter. +This option can be unspecified if no guest networking is required. +The recommended value for this is +.Sy virtio-net . +Additional network interfaces can be configured by adding additional +.Sy networkX_type +and +.Sy networkX_switch +values, replacing +.Sy X +with the next available integer. +.It network0_switch +The virtual switch to connect interface +.Sy 0 +to. +This should correspond to a virtual switch created using the +.Pa vm switch create +subcommand. +If the virtual switch is not found, an interface will still be assigned, but +not connected to any bridge. +.Pp +Note that this field is no longer strictly required. +If you are using a custom device for the networking that is already configured, +you may not need the interface connected to a virtual switch. +See the +.Sy network0_device +configuration option. +.It network0_device +Normally vm-bhyve will create a +.Xr tap 4 +device at run-time for each virtual network interface. +This may be an issue in more advanced configurations where you want to +pre-configure the networking manually in a way unsupported by vm-bhyve. +This option allows you to instruct vm-bhyve to use an existing network device +for this virtual interface, rather than creating one dynamically. +.It network0_mac +This option allows you to specify a mac address to use for this interface. +If not provided, +.Xr bhyve 8 +will generate a mac address. +.It network0_span +Set this option to +.Sy yes +to instruct vm-bhyve to add the virtual network interface to the switch as a +span port on the bridge. +The default is to add the port to the switch as an ordinary bridge member. +.It disk0_type +The emulation type for the first virtual disk. +At least one virtual disk is required. +Valid options for this are currently +.Sy virtio-blk , +.Sy ahci-hd , +.Sy ahci-cd , +and +.Sy nvme . +Additional disks can be added by adding additional +.Sy diskX_type +and +.Sy diskX_name +values, replacing +.Sy X +with the next available integer. +.It disk0_name +The filename for the first virtual disk. +The first disk is created automatically when provisioning a new virtual +machine. +If additional disks are added manually, the image will need to be created, +usually done using the +.Xr truncate 1 +or +.Xr zfs 8 +commands. +Alternatively, you can use the +.Pa vm add +command, which will create the disk image for you. +.Pp +Normally disk images or zvols are stored directly inside the guest. +To use a disk image that is stored anywhere else, you can specify the full path +in this option, and configure the device as +.Sy custom . +.Pp +To use an established iscsi device, specify a target 'session[/lun]' +(default /0) which matches a unique session from the +.Pf ' Xr iscsictl 8 +-L' command output, and configure the device as +.Sy iscsi . +.It disk0_dev +The type of device to use for the disk. +If not specified, this will default to +.Sy file , +and a sparse file, located in the guest directory, will be used as the disk +image. +Other options include: +.Sy zvol +or +.Sy sparse-zvol +(which will use a ZVOL as the disk image, created directly under the guest +dataset), +.Sy custom , +and +.Sy iscsi . +.Pp +When using +.Sy custom , +the +.Pa diskX_name +parameter must be set to the full path to the image file or device. +.Pp +Already attached iscsi devices can have their device nodes dynamically +detected and used by setting this option to +.Sy iscsi +and +.Pa diskX_name +as described above. +.It disk0_opts +Any additional options to use for this disk device. +Multiple options can be specified, separated by a comma. +Please see the +.Xr bhyve 8 +man page for more details on supported options. +.It disk0_size +This setting can be specified in templates to set the size of this disk. +When creating a guest, +.Nm +will default to creating a 20G image for each disk, unless an alternative size +is specified using this option. +The size of the first disk can be overridden using the +.Sy -s +command line option. +.Pp +NOTE: This setting is only supported in templates. +It has no function in real guest configuration, and is not copied over when a +new machine is provisioned. +.It ahci_device_limit +By default, all AHCI devices are added on their own controller in a unique +slot/function. +In +.Fx 12 +it is possible to put up to 32 devices on one controller. +This setting allows you to control the number of devices (ahci-hd/ahci-cd) that +vm-bhyve will put on a single controller. +The default is +.Sy 1 +and allowed values are +.Sy 2-32 . +.It uuid +This option allows you to specify a fixed UUID for the guests SMBIOS. +Normally, the UUID is generated by +.Xr bhyve 8 +based on the hostname and guest name. +Because this may change if guests are moved between systems, the +.Pa vm create +command automatically assigns a UUID to all newly created guests. +.It ignore_bad_msr +Set to +.Sy true|on|yes|1 +to configure +.Xr bhyve 8 +to ignore accesses to unimplemented model specific registers. +This is commonly required on AMD processors, although is enabled by default for +UEFI guests. +.It bhyve_options +Specify any additional command line arguments to pass to the bhyve command. +This allows the use of options such as cpu pinning or debug that are not +exposed by +.Sy vm-bhyve . +.It grub_installX +This option allows you to specify grub commands needed to boot the install +media for this guest. +.Sy X +should be an integer starting at 0, with additional grub commands using the +next numbers in sequence. +.Pp +If no install commands are specified, +.Sy grub-bhyve +will be run on the guests console, so you can use the standard +.Pa vm console +command to access the bootloader if needed. +.It grub_run_partition +Specify the partition that grub should look in for the grub configuration +files. +By default, vm-bhyve will specify partition 1, which is correct in most +standard cases. +.It grub_runX +The option allows you to specify the grub commands needed to boot the guest +from disk. +.Sy X +should be an integer starting at 0, with additional grub commands using the +next numbers in sequence. +.Pp +If no boot commands are specified, +.Sy grub-bhyve +will be run on the guests console, so you can use the standard +.Pa vm console +command to access the bootloader if needed. +.Pp +The sample templates contain examples of how the grub configuration variables +can be used. +.It grub_run_dir +By default +.Sy grub-bhyve +will look in the directory +.Sy /boot/grub +for the grub configuration file. +This option allows you to specify an alternate path to use when starting a +guest. +.It grub_run_file +Allows you to specify the grub configuration file that +.Sy grub-bhyve +will look for inside the guest, rather than the +default of +.Sy grub.cfg . +.It passthruX +Specify a device to pass through to the guest. +You will need to reserve the device first so that is it claimed by the ppt +driver on boot. +.Pp +Once the device is successfully reserved, you can add it to the guest by adding +.Sy passthruX="1/2/3" +to the guest configuration file, where +.Sy X +is an integer starting at 0, and +.Sy 1/2/3 +is the Base/Slot/Function of the device. +If you are passing through multiple functions on the same device, make sure +they are specified together in the configuration file in the +same sequence as the original device. +.Pp +Please see +.Lk https://wiki.freebsd.org/bhyve/pci_passthru +for more details on how this works. +.It virt_random +Set this option to +.Sy yes +if you want to create a +.Sy virtio-rnd +device for this guest. +.It graphics +If set to yes, a frame buffer is added to the guest. +This provides a graphical console that is accessible using VNC. +By default the console is 800x600, and will listen on +.Sy 0.0.0.0:5900 . +If port 5900 is not available, the next available port will be used. +The active address and port can be viewed in +.Sy vm list +and +.Sy vm info +output. +.It graphics_port +This option allows you to specific a fixed port that the VNC service should +listen on. +Please remember that all guests should ideally use a unique port to avoid any +problems. +.It graphics_listen +By default the graphical VNC console will listen on +.Sy 0.0.0.0 , +so is accessible by connecting to any IP address assigned to the bhyve host. +Use this option to specify a specific IP address that the VNC service should +bind to. +.It graphics_res +Specify the resolution of the graphical console in +.Pa WxH +format. +Please note that only a certain range of resolutions are currently supported. +Please set +.Pa config.sample +for a full up-to-date list. +.It graphics_wait +Set this to +.Sy yes +in order to make guest boot wait for the VNC console +to be opened. +This can help when installing operating systems that require immediate keyboard +input (such as a timed 'enter setup' screen). +Set to +.Sy no +in order to completely disable this function. +.Pp +The default is +.Sy auto , +in which case the console will wait if the guest is started in install mode. +Note that after the first boot, the system will boot immediately as normal. +To force the console to wait on each boot, the +.Sy yes +setting should be used. +.It graphics_vga +This configures how the graphics card is exposed to the guest. Valid options are +.Sy io +(default), +.Sy on +or +.Sy off . +Please see the +.Xr bhyve 8 +man page for more details on this option. +.It xhci_mouse +Set this option to +.Sy yes +in order to provide an XHCI mouse device to the guest. +This tracks much better than the default PS2 mouse in VNC settings, although +this mouse may not supported by older guests. +.It sound +Set this option to +.Sy yes +in order to provide HD Audio Emulation to the guest. Please see +.Sy bhyve(8) +for details. +.It sound_play +Set this to the desired audio output device of the host to the guest. Defaults to +.Sy '/dev/dsp0' +.It sound_rec +Set this to the desired audio input device of the host to the guest. If empty +no audio input device is configured. Defaults to +.Sy '' (empty) +.It virt_consoleX +Allows the creation of up to 16 virtio-console devices in the guest. The value +to this option can be +.Sy yes|on|1 +to create a numbered port. This is the only method supported by some guests. +.Pp +If any other value is provided, this will be used as the name of the port. The +name +.Sy org.freenas.bhyve-agent +can be useful, as it ties in with utilities written for the FreeNAS +bhyve-agent interface. +.It zfs_dataset_opts +This allows you to specify one or more ZFS properties to set on the dataset +when a guest is created. +Because properties are assigned as the dataset is created, this option is most +useful when specified inside a template. +As a guest is created, all properties listed in this option will be applied to +the guest dataset. +.Pp +Multiple properties can be specified, separated by a space. +Please note that spaces are not currently supported in the property values. +.It zfs_zvol_opts +Allows you to specify ZFS properties that should be assigned to any ZVOLs that +are created for a guest. +As with +.Pa zfs_dataset_opts , +this makes most sense when entered into a template, as the properties can be +assigned while a guest is being created. +Some ZVOL options, such as +.Pa volblocksize +can only be set at creation time. +.Pp +Multiple properties can be specified, separated by a space. +For example, the following will configure the ZVOL block size to 128k, and turn +compression off. +.Pp +zfs_zvol_opts="volblocksize=128k compress=off" +.It prestart +Allows you to specify a script or executable that will run before the guest starts, +including on reboot. This is provided the guest name, and ZFS dataset (if applicable) as arguments. +We also change directory to the guest path before running the script. +.Pp +This can be specified as a full path, or just a script filename. In the latter case we +look in the guest directory for the script. +.Pp +Note that although the guest is technically stopped when this process runs, calls to +.Nm +will still consider the guest locked. +.It priority +Allows a priority to be set for a guest by using the +.Xr nice 8 +facility. The default value is 0, and has a range from -20, which is the highest +priority, to 20. A priority of 20 will cause the guest to only run when the host +system is idle. +.It limit_pcpu +Limit the bhyve process to the specified cpu percentage. +.Pp +Please note this, as with all +.Sy limit +settings, requires +.Xr rctl 8 +to be enabled in your kernel. +.It limit_rbps +Limit guest disk read throughput to the specified bits per second. +.It limit_wbps +Limit guest disk write throughput to the specified bits per second. +.It limit_riops +Limit guest disk read iops to the specified number of operations per second. +.It limit_wiops +Limit guest disk write iops to the specified number of operations per second. +.El +.\" ============ SEE ALSO ============= +.Sh SEE ALSO +.Xr cu 1 , +.Xr fetch 1 , +.Xr tmux 1 , +.Xr truncate 1 , +.Xr bridge 4 , +.Xr nmdm 4 , +.Xr tap 4 , +.Xr vlan 4 , +.Xr bhyve 8 , +.Xr bhyveload 8 , +.Xr rctl 8 , +.Xr zfs 8 +.\" ============ BUGS ============= +.Sh KNOWN BUGS +If a guest is renamed, and then cloned using a snapshot taken before the +rename, vm-bhyve is unable to find the guest configuration file. +This is because the configuration file in the snapshot still refers to the old +guest name. +In this circumstance, vm-bhyve will output an error during cloning detailing +that the configuration file in the new guest will need to be renamed and +updated manually. +.Pp +On some systems it has been observed that bridging can cause interfaces to go +down for up to 10 seconds, which is enough to stall ssh sessions. +This is noticable when the first guest is started or when the last guest is +stopped. +Once there are at least 2 interfaces bridged (one real interface and a tap +interface), further guests can be started/stopped without issue. +.Pp +Please report all bugs/issues/feature requests to the GitHub project at +.Lk https://github.com/churchers/vm-bhyve +.\" ============ AUTHOR ============= +.Sh AUTHORS +.An Matt Churchyard Aq Mt churchers@gmail.com