A low-level Linux kernel improvement being worked on by engineers at Amazon Web Services (AWS) is for introducing a PCI Configuration Space Cache “PCSC” to help with dense SR-IOV deployments that can end up hitting significant overhead with the current Linux kernel.
Earlier this month Amazon engineer Evangelos Petrongonas sent out the initial patches for proposing the PCI Configuration Space Cache for the Linux kernel. Evangelos explained of the problem statement and approach for PCSC:
“Modern virtualization environments, particularly those leveraging SR-IOV with hundreds or thousands of Virtual Functions, expose a significant performance bottleneck in PCI configuration space accesses. During VM initialization, the repeated enumeration and configuration of assigned VFs creates substantial delays that scale linearly with deployment density. Each configuration space read triggers a hardware transaction, leading to bus contention and measurable impact on system startup times.
The fundamental issue stems from the PCI subsystem’s approach of treating every configuration space access as a direct hardware operation, even when reading registers that contain static, rarely-changing values. Capability registers, device/vendor IDs, and many control registers remain constant throughout a device’s lifetime, yet are repeatedly accessed during driver initialization, device enumeration, and system management operations.
This patch series introduces the PCI Configuration Space Cache (PCSC), a transparent caching layer that intercepts configuration space operations and maintains cached copies of register values. The implementation philosophy centers on complete transparency; existing drivers, applications, and system management tools continue functioning unchanged while benefiting from reduced hardware access overhead. Currently, only endpoint devices are cached. Bridges and Root Complexes utilize a passthrough approach.”
Linux PCSC has shown the potential to be of significant impact especially for hyperscalers / cloud providers:
“Performance characteristics demonstrate significant improvements, with cache hit rates reaching 49% in typical virtualization scenarios without persistence, and up to 81% when combined with kexec persistence. These metrics translate to substantial reductions in configuration space access latency, particularly during bulk VM operations where thousands of configuration space accesses would otherwise create significant bus traffic spikes.
Testing has covered diverse hardware configurations including high-density SR-IOV deployments. The implementation demonstrates particular effectiveness in cloud environments where rapid VM deployment and high device density create substantial configuration space access pressure.”
Beyond the performance benefits, the other important aspect of PCSC is not requiring any driver modifications. The PCSC caching layer can also be easily disabled if desired.
More details for those interested in this work can be found via this RFC patch series.
