HardenedBSD/sys/netipsec/ipsec_offload.h

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

202 lines
6.1 KiB
C
Raw Permalink Normal View History

ipsec_offload: kernel infrastructure Inline IPSEC offload moves almost whole IPSEC processing from the CPU/MCU and possibly crypto accelerator, to the network card. The transmitted packet content is not touched by CPU during TX operations, kernel only does the required policy and security association lookups to find out that given flow is offloaded, and then packet is transmitted as plain text to the card. For driver convenience, a metadata is attached to the packet identifying SA which must process the packet. Card does encryption of the payload, padding, calculates authentication, and does the reformat according to the policy. Similarly, on receive, card does the decapsulation, decryption, and authentification. Kernel receives the identifier of SA that was used to process the packet, together with the plain-text packet. Overall, payload octets are only read or written by card DMA engine, removing a lot of memory subsystem overhead, and saving CPU time because IPSEC algos calculations are avoided. If driver declares support for inline IPSEC offload (with the IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. Driver decides which policies and SAs can be offloaded based on hardware capacity, and acks/nacks each SA for given interface to kernel. Kernel needs to keep this information to make a decision to skip software processing on TX, and to assume processing already done on RX. This shadow SPD/SAD database of offloads is rooted from policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs (struct secasvar accel_ipfs, struct ifp_handle_sav). Some extensions to the PF_KEY socket allow to limit interfaces for which given SP/SA could be offloaded (proposed for offload). Also, additional statistics extensions allow to observe allocation/octet/use counters for specific SA. Since SPs and SAs are typically instantiated in non-sleepable context, while offloading them into card is expected to require costly async manipulations of the card state, calls to the driver for offload and termination are executed in the threaded taskqueue. It also solves the issue of allocating resources needed for the offload database. Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the owning SP/SA, the offload must be terminated before last reference is dropped. ipsec_accel only adds transient references to ensure safe pointer ownership by taskqueue. Maintaining the SA counters for hardware-accelerated packets is the duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() is provided to hide accel infrastructure from drivers which would use expected callout to query hardware periodically for updates. Reviewed by: rscheff (transport, stack integration), np Sponsored by: NVIDIA networking Differential revision: https://reviews.freebsd.org/D44219
2021-08-22 21:38:04 +02:00
/*-
* Copyright (c) 2021,2022 NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS `AS IS' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _NETIPSEC_IPSEC_OFFLOAD_H_
#define _NETIPSEC_IPSEC_OFFLOAD_H_
#ifdef _KERNEL
#include <sys/errno.h>
#include <net/if.h>
#include <net/if_var.h>
struct secpolicy;
struct secasvar;
struct inpcb;
struct ipsec_accel_out_tag {
struct m_tag tag;
uint16_t drv_spi;
};
struct ipsec_accel_in_tag {
struct m_tag tag;
uint16_t drv_spi;
};
#define IPSEC_ACCEL_DRV_SPI_BYPASS 2
#define IPSEC_ACCEL_DRV_SPI_MIN 3
#define IPSEC_ACCEL_DRV_SPI_MAX 0xffff
extern void (*ipsec_accel_sa_newkey_p)(struct secasvar *sav);
extern void (*ipsec_accel_sa_install_input_p)(struct secasvar *sav,
const union sockaddr_union *dst_address, int sproto, uint32_t spi);
extern void (*ipsec_accel_forget_sav_p)(struct secasvar *sav);
extern void (*ipsec_accel_spdadd_p)(struct secpolicy *sp, struct inpcb *inp);
extern void (*ipsec_accel_spddel_p)(struct secpolicy *sp);
extern int (*ipsec_accel_sa_lifetime_op_p)(struct secasvar *sav,
struct seclifetime *lft_c, if_t ifp, enum IF_SA_CNT_WHICH op,
struct rm_priotracker *sahtree_trackerp);
extern void (*ipsec_accel_sync_p)(void);
extern bool (*ipsec_accel_is_accel_sav_p)(struct secasvar *sav);
extern struct mbuf *(*ipsec_accel_key_setaccelif_p)(struct secasvar *sav);
extern void (*ipsec_accel_on_ifdown_p)(struct ifnet *ifp);
extern void (*ipsec_accel_drv_sa_lifetime_update_p)(struct secasvar *sav,
if_t ifp, u_int drv_spi, uint64_t octets, uint64_t allocs);
extern int (*ipsec_accel_drv_sa_lifetime_fetch_p)(struct secasvar *sav,
if_t ifp, u_int drv_spi, uint64_t *octets, uint64_t *allocs);
ipsec_offload: kernel infrastructure Inline IPSEC offload moves almost whole IPSEC processing from the CPU/MCU and possibly crypto accelerator, to the network card. The transmitted packet content is not touched by CPU during TX operations, kernel only does the required policy and security association lookups to find out that given flow is offloaded, and then packet is transmitted as plain text to the card. For driver convenience, a metadata is attached to the packet identifying SA which must process the packet. Card does encryption of the payload, padding, calculates authentication, and does the reformat according to the policy. Similarly, on receive, card does the decapsulation, decryption, and authentification. Kernel receives the identifier of SA that was used to process the packet, together with the plain-text packet. Overall, payload octets are only read or written by card DMA engine, removing a lot of memory subsystem overhead, and saving CPU time because IPSEC algos calculations are avoided. If driver declares support for inline IPSEC offload (with the IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. Driver decides which policies and SAs can be offloaded based on hardware capacity, and acks/nacks each SA for given interface to kernel. Kernel needs to keep this information to make a decision to skip software processing on TX, and to assume processing already done on RX. This shadow SPD/SAD database of offloads is rooted from policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs (struct secasvar accel_ipfs, struct ifp_handle_sav). Some extensions to the PF_KEY socket allow to limit interfaces for which given SP/SA could be offloaded (proposed for offload). Also, additional statistics extensions allow to observe allocation/octet/use counters for specific SA. Since SPs and SAs are typically instantiated in non-sleepable context, while offloading them into card is expected to require costly async manipulations of the card state, calls to the driver for offload and termination are executed in the threaded taskqueue. It also solves the issue of allocating resources needed for the offload database. Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the owning SP/SA, the offload must be terminated before last reference is dropped. ipsec_accel only adds transient references to ensure safe pointer ownership by taskqueue. Maintaining the SA counters for hardware-accelerated packets is the duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() is provided to hide accel infrastructure from drivers which would use expected callout to query hardware periodically for updates. Reviewed by: rscheff (transport, stack integration), np Sponsored by: NVIDIA networking Differential revision: https://reviews.freebsd.org/D44219
2021-08-22 21:38:04 +02:00
#ifdef IPSEC_OFFLOAD
/*
* Have to use ipsec_accel_sa_install_input_p indirection because
* key.c is unconditionally included into the static kernel.
*/
static inline void
ipsec_accel_sa_newkey(struct secasvar *sav)
{
void (*p)(struct secasvar *sav);
p = atomic_load_ptr(&ipsec_accel_sa_newkey_p);
if (p != NULL)
p(sav);
}
static inline void
ipsec_accel_forget_sav(struct secasvar *sav)
{
void (*p)(struct secasvar *sav);
p = atomic_load_ptr(&ipsec_accel_forget_sav_p);
if (p != NULL)
p(sav);
}
static inline void
ipsec_accel_spdadd(struct secpolicy *sp, struct inpcb *inp)
{
void (*p)(struct secpolicy *sp, struct inpcb *inp);
p = atomic_load_ptr(&ipsec_accel_spdadd_p);
if (p != NULL)
p(sp, inp);
}
static inline void
ipsec_accel_spddel(struct secpolicy *sp)
{
void (*p)(struct secpolicy *sp);
p = atomic_load_ptr(&ipsec_accel_spddel_p);
if (p != NULL)
p(sp);
}
static inline int
ipsec_accel_sa_lifetime_op(struct secasvar *sav,
struct seclifetime *lft_c, if_t ifp, enum IF_SA_CNT_WHICH op,
struct rm_priotracker *sahtree_trackerp)
{
int (*p)(struct secasvar *sav, struct seclifetime *lft_c, if_t ifp,
enum IF_SA_CNT_WHICH op, struct rm_priotracker *sahtree_trackerp);
p = atomic_load_ptr(&ipsec_accel_sa_lifetime_op_p);
if (p != NULL)
return (p(sav, lft_c, ifp, op, sahtree_trackerp));
return (ENOTSUP);
}
static inline void
ipsec_accel_sync(void)
{
void (*p)(void);
p = atomic_load_ptr(&ipsec_accel_sync_p);
if (p != NULL)
p();
}
static inline bool
ipsec_accel_is_accel_sav(struct secasvar *sav)
{
bool (*p)(struct secasvar *sav);
p = atomic_load_ptr(&ipsec_accel_is_accel_sav_p);
if (p != NULL)
return (p(sav));
return (false);
}
static inline struct mbuf *
ipsec_accel_key_setaccelif(struct secasvar *sav)
{
struct mbuf *(*p)(struct secasvar *sav);
p = atomic_load_ptr(&ipsec_accel_key_setaccelif_p);
if (p != NULL)
return (p(sav));
return (NULL);
}
#else
#define ipsec_accel_sa_newkey(a)
#define ipsec_accel_forget_sav(a)
#define ipsec_accel_spdadd(a, b)
#define ipsec_accel_spddel(a)
#define ipsec_accel_sa_lifetime_op(a, b, c, d, e)
#define ipsec_accel_sync()
#define ipsec_accel_is_accel_sav(a)
#define ipsec_accel_key_setaccelif(a)
#endif
void ipsec_accel_forget_sav_impl(struct secasvar *sav);
void ipsec_accel_spdadd_impl(struct secpolicy *sp, struct inpcb *inp);
void ipsec_accel_spddel_impl(struct secpolicy *sp);
#ifdef IPSEC_OFFLOAD
int ipsec_accel_input(struct mbuf *m, int offset, int proto);
bool ipsec_accel_output(struct ifnet *ifp, struct mbuf *m,
struct inpcb *inp, struct secpolicy *sp, struct secasvar *sav, int af,
int mtu, int *hwassist);
ipsec_offload: kernel infrastructure Inline IPSEC offload moves almost whole IPSEC processing from the CPU/MCU and possibly crypto accelerator, to the network card. The transmitted packet content is not touched by CPU during TX operations, kernel only does the required policy and security association lookups to find out that given flow is offloaded, and then packet is transmitted as plain text to the card. For driver convenience, a metadata is attached to the packet identifying SA which must process the packet. Card does encryption of the payload, padding, calculates authentication, and does the reformat according to the policy. Similarly, on receive, card does the decapsulation, decryption, and authentification. Kernel receives the identifier of SA that was used to process the packet, together with the plain-text packet. Overall, payload octets are only read or written by card DMA engine, removing a lot of memory subsystem overhead, and saving CPU time because IPSEC algos calculations are avoided. If driver declares support for inline IPSEC offload (with the IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. Driver decides which policies and SAs can be offloaded based on hardware capacity, and acks/nacks each SA for given interface to kernel. Kernel needs to keep this information to make a decision to skip software processing on TX, and to assume processing already done on RX. This shadow SPD/SAD database of offloads is rooted from policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs (struct secasvar accel_ipfs, struct ifp_handle_sav). Some extensions to the PF_KEY socket allow to limit interfaces for which given SP/SA could be offloaded (proposed for offload). Also, additional statistics extensions allow to observe allocation/octet/use counters for specific SA. Since SPs and SAs are typically instantiated in non-sleepable context, while offloading them into card is expected to require costly async manipulations of the card state, calls to the driver for offload and termination are executed in the threaded taskqueue. It also solves the issue of allocating resources needed for the offload database. Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the owning SP/SA, the offload must be terminated before last reference is dropped. ipsec_accel only adds transient references to ensure safe pointer ownership by taskqueue. Maintaining the SA counters for hardware-accelerated packets is the duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() is provided to hide accel infrastructure from drivers which would use expected callout to query hardware periodically for updates. Reviewed by: rscheff (transport, stack integration), np Sponsored by: NVIDIA networking Differential revision: https://reviews.freebsd.org/D44219
2021-08-22 21:38:04 +02:00
void ipsec_accel_forget_sav(struct secasvar *sav);
#else
#define ipsec_accel_input(a, b, c) (ENXIO)
#define ipsec_accel_output(a, b, c, d, e, f, g, h) ({ \
*h = 0; \
false; \
})
ipsec_offload: kernel infrastructure Inline IPSEC offload moves almost whole IPSEC processing from the CPU/MCU and possibly crypto accelerator, to the network card. The transmitted packet content is not touched by CPU during TX operations, kernel only does the required policy and security association lookups to find out that given flow is offloaded, and then packet is transmitted as plain text to the card. For driver convenience, a metadata is attached to the packet identifying SA which must process the packet. Card does encryption of the payload, padding, calculates authentication, and does the reformat according to the policy. Similarly, on receive, card does the decapsulation, decryption, and authentification. Kernel receives the identifier of SA that was used to process the packet, together with the plain-text packet. Overall, payload octets are only read or written by card DMA engine, removing a lot of memory subsystem overhead, and saving CPU time because IPSEC algos calculations are avoided. If driver declares support for inline IPSEC offload (with the IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. Driver decides which policies and SAs can be offloaded based on hardware capacity, and acks/nacks each SA for given interface to kernel. Kernel needs to keep this information to make a decision to skip software processing on TX, and to assume processing already done on RX. This shadow SPD/SAD database of offloads is rooted from policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs (struct secasvar accel_ipfs, struct ifp_handle_sav). Some extensions to the PF_KEY socket allow to limit interfaces for which given SP/SA could be offloaded (proposed for offload). Also, additional statistics extensions allow to observe allocation/octet/use counters for specific SA. Since SPs and SAs are typically instantiated in non-sleepable context, while offloading them into card is expected to require costly async manipulations of the card state, calls to the driver for offload and termination are executed in the threaded taskqueue. It also solves the issue of allocating resources needed for the offload database. Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the owning SP/SA, the offload must be terminated before last reference is dropped. ipsec_accel only adds transient references to ensure safe pointer ownership by taskqueue. Maintaining the SA counters for hardware-accelerated packets is the duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() is provided to hide accel infrastructure from drivers which would use expected callout to query hardware periodically for updates. Reviewed by: rscheff (transport, stack integration), np Sponsored by: NVIDIA networking Differential revision: https://reviews.freebsd.org/D44219
2021-08-22 21:38:04 +02:00
#define ipsec_accel_forget_sav(a)
#endif
struct ipsec_accel_in_tag *ipsec_accel_input_tag_lookup(const struct mbuf *);
void ipsec_accel_on_ifdown(struct ifnet *ifp);
void ipsec_accel_drv_sa_lifetime_update(struct secasvar *sav, if_t ifp,
u_int drv_spi, uint64_t octets, uint64_t allocs);
int ipsec_accel_drv_sa_lifetime_fetch(struct secasvar *sav,
if_t ifp, u_int drv_spi, uint64_t *octets, uint64_t *allocs);
ipsec_offload: kernel infrastructure Inline IPSEC offload moves almost whole IPSEC processing from the CPU/MCU and possibly crypto accelerator, to the network card. The transmitted packet content is not touched by CPU during TX operations, kernel only does the required policy and security association lookups to find out that given flow is offloaded, and then packet is transmitted as plain text to the card. For driver convenience, a metadata is attached to the packet identifying SA which must process the packet. Card does encryption of the payload, padding, calculates authentication, and does the reformat according to the policy. Similarly, on receive, card does the decapsulation, decryption, and authentification. Kernel receives the identifier of SA that was used to process the packet, together with the plain-text packet. Overall, payload octets are only read or written by card DMA engine, removing a lot of memory subsystem overhead, and saving CPU time because IPSEC algos calculations are avoided. If driver declares support for inline IPSEC offload (with the IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. Driver decides which policies and SAs can be offloaded based on hardware capacity, and acks/nacks each SA for given interface to kernel. Kernel needs to keep this information to make a decision to skip software processing on TX, and to assume processing already done on RX. This shadow SPD/SAD database of offloads is rooted from policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs (struct secasvar accel_ipfs, struct ifp_handle_sav). Some extensions to the PF_KEY socket allow to limit interfaces for which given SP/SA could be offloaded (proposed for offload). Also, additional statistics extensions allow to observe allocation/octet/use counters for specific SA. Since SPs and SAs are typically instantiated in non-sleepable context, while offloading them into card is expected to require costly async manipulations of the card state, calls to the driver for offload and termination are executed in the threaded taskqueue. It also solves the issue of allocating resources needed for the offload database. Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the owning SP/SA, the offload must be terminated before last reference is dropped. ipsec_accel only adds transient references to ensure safe pointer ownership by taskqueue. Maintaining the SA counters for hardware-accelerated packets is the duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() is provided to hide accel infrastructure from drivers which would use expected callout to query hardware periodically for updates. Reviewed by: rscheff (transport, stack integration), np Sponsored by: NVIDIA networking Differential revision: https://reviews.freebsd.org/D44219
2021-08-22 21:38:04 +02:00
#endif /* _KERNEL */
#endif /* _NETIPSEC_IPSEC_OFFLOAD_H_ */