These were reported by `mandoc -T lint ...` as warnings:
- unusual Xr order
- unusual Xr punctuation
Fixes made by script in https://github.com/Tarsnap/freebsd-doc-scripts
Signed-off-by: Graham Percival <gperciva@tarsnap.com>
Reviewed by: mhorne, Alexander Ziaee <concussious.bugzilla@runbox.com>
Sponsored by: Tarsnap Backup Inc.
Pull Request: https://github.com/freebsd/freebsd-src/pull/1464
ifmedia_add() allocates an ifmedia_entry during ena_attach.
Current code doesn't release this memory during ena_detach()
This commit calls ifmedia_removeall() to properly free the
allocated memory during ena_detach().
Also, in case ena_attach fails, we need to detach ifmedia
which was allocated within ena_setup_ifnet().
This bug was first described in:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278100
Reviewed by: zlei
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Large LLQ depth size is currently calculated by dividing the maximum
possible size of LLQ by 2.
In newer paltforms, starting from r8g the size of BAR2,
which contains LLQ, will be increased, and the maximum depth of
wide LLQ will be set according to a value set by the device, instead of
hardcoded division by 2.
The new value will be stored by the device in max_wide_llq_depth field
for drivers that expose ENA_ADMIN_LLQ_FEATURE_VERSION_1 or higher to
the device.
There is an assumption that max_llq_depth >= max_wide_llq_depth, since
they both use the same bar, and if it is possible to have a wide LLQ
of size max_wide_llq_depth, it is possible to have a normal LLQ of the
same size, since it will occupy half of the space.
Also moved the large LLQ case calculation of max_tx_queue_size
before its rounddown.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit adds support for receiving LLQ entry size recommendation
from the device. The driver will use the recommended entry size, unless
the user specifically chooses to use regular or large LLQ entry.
Also added enum ena_llq_header_size_policy_t and llq_plociy field in
order to support the new feature.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit adds a handler for the new aenq message
ENA_ADMIN_DEVICE_REQUEST_RESET,
which in turn causes the driver to trigger reset of a new type:
ENA_REGS_RESET_DEVICE_REQUEST. Also adds counting of such occurrences in
a new statistic for it.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
When attaching ENA driver, ena_netmap_attach() is invoked which, in turn
calls netmap_attach which, initializes a struct netmap_adapter,
allocating the struct's netmap_ring and the struct selinfo.
When we change the interface number of queues we need to reinit the
netmap adapter struct as well, so we need to detach it in order to free
the memory allocated by netmap_attach and allocate new memory based on
the new parameters like number of rings, ring size etc...
Without detaching and attaching the netmap interface, if we're to change
the number of queues from 8 to 2 for example and try to enable netmap,
the kernel will panic since the original netmap struct within the
kernel's possession still thinks that the driver has 8 queues which will
eventually cause a non-allocated virtual address access fault.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
When processing packets within the rx-flow
ena_netmap_rx_load_desc doesn't know the number of descriptors, so it
sets NS_MOREFRAG to all the slots to indicate that there are more
fragments for this packet.
The code calls ena_netmap_rx_load_desc() for every descriptor in
this packet to map the relevant buffer into the netmap shared memory.
After ena_netmap_rx_load_desc() calls, we need to unset the NS_MOREFRAG
for the last fragment to indicate that this is the last fragment,
so we explicitly turn off NS_MOREFRAG flag.
Current code overrides all other flags and sets NS_BUF_CHANGED.
This patch unsets the relevant flag only.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Netmap index wraps around based on the number of netmap kernel ring
slots.
Currently the driver prefetches the next slot using nm_i + 1 which may
be wrong since it does not handle wrap around.
This patch fixes that by using the kernel API for fetching the next
netmap index.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
In case ena_com_prepare_tx() fails within the netmap tx flow,
the driver will unmap the last socket chain.
Currently, the driver unmaps the wrong socket within
ena_netmap_unmap_last_socket_chain().
Illustration of the flow:
1- ena_netmap_tx_frames()
2- ena_netmap_tx_frame()
3- ena_netmap_tx_map_slots()
3.1- Map slot
3.2- Advance to the next socket
4- ena_com_prepare_tx()
4.1- ena_com_prepare_tx() fails
5- ena_netmap_unmap_last_socket_chain()
In step 5, where the driver unmaps the socket, the netmap
index already points at the next entry, meaning we're unmapping the
wrong socket in case ena_com_prepare_tx() fails.
In order to fix that, the driver should first update the netmap index to
point at the previous entry and only then update the socket parameters.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit changes the code so all global counters will have the
same line break.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
The mbuf is NULL issue happens when the device sends the driver
a completion with a wrong request id.
Trigger a reset whenever this happens.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit adds differentiation for a reset caused by missing tx
completions, by verifying if the driver didn't receive tx
completions caused by missing interrupts.
The cleanup_running field was added to ena_ring because
cleanup_task.ta_pending is zeroed before ena_cleanup() runs.
Also ena_increment_reset_counter() API was added in order to support
only incrementing the reset counter.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit sets the default value for ena_min_poll_delay_us to 100.
This commit does not change the behavior of the driver, the delay is
calculated as MAX(ENA_MIN_ADMIN_POLL_US, delay_us), where the first
field is already defined as 100.
The second parameter, delay_us is taken from ena_min_poll_delay_us
which is currently unset - 0.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
There can be cases when we trigger reset if an admin interrupt
is missing.
In order to identify this use-case specifically,
this commit adds a new reset reason.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
RX completion descriptors may sometimes contain errors due
to corruption. Upon identifying such a case, the driver will
trigger a reset with an explicit reset reason
ENA_REGS_RESET_RX_DESCRIPTOR_MALFORMED.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
TX completion descriptors may sometimes contain errors due
to corruption. Upon identifying such a case, the driver will
trigger a reset with an explicit reset reason
ENA_REGS_RESET_TX_DESCRIPTOR_MALFORMED.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
The driver uses different reset reasons.
Some of them are counted and presented in the driver statistics.
There are cases where statistics are counted on a ring level,
but these are zeroed after a reset procedure takes place.
This commit makes the following changes:
1. Add statistics for the unrepresented reset reasons.
2. Add reset reasons which are counted on a ring level,
to be also global for better tracking.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit is part of the effort of notifying the user of non-optimal
or performance impacting practices.
A new interface is serving as a communication channel
between the device and the driver. One of the goals of this channel is
to create a new mechanism of notifying the driver and user in case of
sub-optimal configuration using a bitmap.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Currently we count all of the newly added and already existing
missing tx completions in each iteration of
check_missing_comp_in_tx_queue() causing duplicate counts
to missing_tx_comp stat.
This commit adds a new counter new_missed_tx within the relevant
function which only counts the newly added missing tx completions
in each iteration of check_missing_comp_in_tx_queue().
This will allow us to update missing_tx_comp stat accurately without
counting duplicates.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Upstream commit [1] made if_alloc_domain() never fail, then also do the
wrappers if_alloc(), if_alloc_dev(), and if_gethandle().
Upstream commit [2] removed the NULL check conducted by the driver.
This commit also removes err_customer_metrics_alloc goto label.
Commit [2] leaves behind a floating free() statement that
deallocates customer_metrics_array. This commit places the
deallocation statement where it belongs.
[1] commit 4787572d05 ("ifnet: make if_alloc_domain() never fail")
[2] commit aa3860851b ("net: Remove unneeded NULL check for the allocated ifnet")
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
When an exception class is unhandled by the kernel we handle it in
userspace by exiting the process. Rather than exiting raise an unknown
reason exception in the guest. The guest can then handle the exception
as it wishes.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46511
All handlers set x0 before returning, there is no need to also do so
in the data abort exception handler.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46814
When we get a data abort in an EFI runtime service the userspace pmap
will be locked by the current thread. Skip trying to lock it again as
it will be in a critical section and the lock may sleep.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46813
The Arm True Random Number Generator Firmware Interface provides a way
to query the SMCCC firmware for up to 192 bits of entropy. Use it to
provide another source of randomness to the kernel.
Reviewed by: cem, markm
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46989
This will be used to create an integer with a given version. It can
then be used to check if the SMCCC version is late enough for a driver.
Sponsored by: Arm Ltd
This will be used by other drivers that manage SMCCC firmware services
to use as an attachment point.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46988
Simplify the calls into the SMCCC firmware with the new
arm_smccc_invoke* macros.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46987
When calling into SMCCC functions we often only need a few arguments.
As the current function needs all 8 possible arguments to be set the
unused values will be zero.
Create a macro to pass in the used values, followed by enough zeros,
then the result pointer.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46986
- Remove -mllvm flags that don't exist in gcc
- Set the shadow offset with -fasan-shadow-offset
- Remove this when kasan is disabled as it's an error to use this
when not building for kasan under gcc
This allows the arm64 GENERIC-KASAN kernel to build with gcc13.
Reviewed by: brooks
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45300
It's intended to be used by both ipfw and pf.
Reviewed by: kp, markj
Approved by: kp (mentor)
Differential Revision: https://reviews.freebsd.org/D47110
In order to support reset request message from the device to the driver,
over AENQ, which in turn should cause the driver to trigger reset, the
following was added:
1. New AENQ group - ENA_ADMIN_DEVICE_REQUEST_RESET, to support the
new message from the device sent via the AENQ to the driver.
2. New reset reason - ENA_REGS_RESET_DEVICE_REQUEST, which is used in
order to indicate that the reset was triggered by the driver
following the device request.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Currently admin_queue->stats.aborted_cmd counter is incremented if an
admin command status is ENA_CMD_ABORTED and only if the admin queue is
in polling mode.
This commit fixes handling the case of incrementing
admin_queue->stats.aborted_cmd if the admin queue is in interrupt
mode as well.
Also added a verification that the command status is a valid
completion status which is currently verified only if the admin queue
is in polling mode.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
RX ring submission queue descriptors are always located in
instance memory.
This optimization replaces the generic update tail method with a
tailored method for host memory type descriptors to avoid unnecessary if
statement.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
The dma_rmb() memory barrier guarantees that the device set the
phase bit before continuing to read the rest of the descriptor.
Because the phase bit and the rest of the descriptor are in the same
cache line this ensures coherency of the data from the descriptor.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
This commit adds an API to query the aenq on whether there's a pending
keep alive notification.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
In order to support the change in the way drivers get the
maximal depth for wide LLQ,
max_wide_llq_depth was added to struct ena_admin_feature_llq_desc and
will hold the device's exact maximal size as reported by the device,
this way the driver doesn't have to calculate it in a hardcoded manner.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
There can be cases when we trigger reset if an admin interrupt
is missing.
In order to identify this use-case specifically,
this commit adds a new reset reason.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
The unlikely mechanism is used to reduce pipe flush,
caused by a wrong branch prediction.
Moreover, it increases readability by wrapping unexpected errors.
This commit adds unlikely to error checks that are unlikely to happen.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Bug description:
When invoking an admin command, in interrupt mode, if the interrupt
is received after timeout and also after the calling function finished
running, the response will be written into a memory that is no longer
valid.
We fix this by applying the following:
1. Verifying that comp_ctx is occupied during the interrupt handler,
otherwise it doesn't make sense to update its fields.
2. Set user_cqe to NULL when the admin command times out. When the
interrupt handler is later invoked, it will check if it is NULL and
will not use it.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Following this change, PHC can be activated on supported devices.
Driver and device will each hold its own max supported PHC version.
Initial supported PHC version including all changes to this check point
is ENA_ADMIN_PHC_FEATURE_VERSION_0,
all future PHC versions should always be backward compatible.
PHC mutual support is decided in the following steps:
1. Driver publishes PHC support
2. Device publishes PHC support
3. Driver publishes supported PHC version
4. Device publishes supported PHC version and settings
5. Driver enables PHC
DRIVER DEVICE
| (1) |
| Driver sends ENA_ADMIN_SET_FEATURE |
| - Set feature ID ENA_ADMIN_HOST_ATTR_CONFIG |
| - Enable bit ENA_ADMIN_HOST_INFO_PHC_MASK |
| ------------------------------------------------> |
| (2) |
| Driver sends ENA_ADMIN_GET_FEATURE |
| - Set feature ID ENA_ADMIN_DEVICE_ATTRIBUTES |
| ------------------------------------------------> |
| Device responds to ENA_ADMIN_GET_FEATURE |
| - Enable bit ENA_ADMIN_PHC_CONFIG |
| <------------------------------------------------ |
| (3) |
| Driver sends ENA_ADMIN_GET_FEATURE |
| - Set feature ID ENA_ADMIN_PHC_CONFIG |
| - Set PHC version ENA_ADMIN_PHC_FEATURE_VERSION_0 |
| ------------------------------------------------> |
| (4) |
| Device responds to ENA_ADMIN_GET_FEATURE |
| - Set PHC version |
| - Set doorbel offset |
| - Set timeouts |
| <------------------------------------------------ |
| (5) |
| Driver sends ENA_ADMIN_SET_FEATURE |
| - Set feature ID ENA_ADMIN_PHC_CONFIG |
| - Set output address/length |
| ------------------------------------------------> |
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Static variables are tied to the driver code and not to each driver
instance, thus, all driver instances, regardless of the devices they
manage, share the same static variables.
This modification switches the static variable "start_time" to be used
within phc struct.
In addition, rearranged several PHC variables for cache friendly
struct layout and refined several variable names and types for
enhanced clarity and accuracy.
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Under theoretical race condition, PHC may be destroyed while PHC
timestamp is retrieved. This will cause the PHC device memory to
be freed while being used in another thread.
To prevent such a race, PHC destroy flow will disable the PHC state
under PHC lock, (these lock and state variables are used on every
PHC timestamp retrieval).
Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
