Home

Awesome

Custom Firmware Development Guide for Full Device Emulation

I will update this soon, just been very busy

Table of Contents

Part 1: Foundational Concepts

  1. Introduction
  2. Key Definitions
  3. Device Compatibility
  4. Requirements
  5. Gathering Donor Device Information
  6. Initial Firmware Customization
  7. Vivado Project Setup and Customization

Part 2: Intermediate Concepts and Implementation

  1. Advanced Firmware Customization
  2. Emulating Device-Specific Capabilities
  3. Transaction Layer Packet (TLP) Emulation

Part 3: Advanced Techniques and Optimization

  1. Building, Flashing, and Testing
  2. Advanced Debugging Techniques
  3. Troubleshooting
  4. Emulation Accuracy and Optimizations
  5. Best Practices for Firmware Development
  6. Additional Resources
  7. Contact Information
  8. Support and Contributions

Part 1: Foundational Concepts

1. Introduction

1.1 Purpose of the Guide

The primary purpose of this guide is to provide a step-by-step approach to developing custom Direct Memory Access (DMA) firmware for FPGA-based devices to emulate PCIe hardware accurately. This enables applications such as hardware testing, system debugging, security research, and hardware emulation.

By following this guide, you will learn how to:

1.2 Target Audience

This guide is intended for:

1.3 How to Use This Guide

The guide is divided into three parts:

It is recommended to follow the guide sequentially to build a solid understanding before tackling advanced topics.

2. Key Definitions

Understanding the terminology is crucial for effectively following this guide. Below are key definitions related to PCIe, DMA, and device emulation:

3. Device Compatibility

3.1 Supported FPGA-Based Hardware

While this guide focuses on the Squirrel DMA (35T) card due to its accessibility, the methodologies are adaptable to other FPGA-based DMA hardware:

3.2 PCIe Hardware Considerations

To ensure smooth emulation, several PCIe-specific features must be addressed:

3.3 System Requirements

4. Requirements

4.1 Hardware

4.2 Software

4.3 Environment Setup

4.3.1 Install Xilinx Vivado Design Suite

4.3.2 Install Visual Studio Code

4.3.3 Clone the PCILeech-FPGA Repository

4.3.4 Set Up a Clean Development Environment

5. Gathering Donor Device Information

Accurate device emulation hinges on meticulously extracting and replicating critical information from the donor device. This comprehensive data collection enables your FPGA to faithfully mimic the target hardware's PCIe configuration and behavior, ensuring compatibility and functionality when interfacing with the host system.

5.1 Using Arbor for PCIe Device Scanning

Arbor is a robust and user-friendly tool designed for in-depth scanning of PCIe devices. It provides detailed insights into the configuration space of connected hardware, making it an invaluable resource for extracting the necessary information for device emulation.

5.1.1 Install Arbor

To begin utilizing Arbor for device scanning, you must first install the software on your system.

Steps:

  1. Visit the Arbor Download Page:

    • Navigate to the official Arbor Download Page using your preferred web browser.
    • Ensure you are accessing the site directly to avoid any malicious redirects.

  2. Create an Account (if required):

    • Arbor may require you to create a user account to access the download links.
    • Provide the necessary information, such as your name, email address, and organization.
    • Verify your email if prompted, to activate your account.

  3. Download Arbor:

    • Once logged in, locate the download section for Arbor.
    • Select the version compatible with your operating system (e.g., Windows 10/11 64-bit).
    • Click the Download button and save the installer to a known location on your computer.

  4. Install Arbor:

    • Locate the downloaded installer file (e.g., ArborSetup.exe).
    • Right-click the installer and select Run as administrator to ensure it has the necessary permissions.
    • Follow the on-screen instructions to complete the installation process.
      • Accept the license agreement.
      • Choose the installation directory.
      • Opt to create desktop shortcuts if desired.

  5. Verify Installation:

    • Upon completion, ensure that Arbor is listed in your Start Menu or on your desktop.
    • Launch Arbor to confirm it opens without errors.

5.1.2 Scan PCIe Devices

With Arbor installed, you can proceed to scan your system for connected PCIe devices.

Steps:

  1. Launch Arbor:

    • Double-click the Arbor icon on your desktop or find it via the Start Menu.
    • If prompted by User Account Control (UAC), allow the application to make changes to your device.

  2. Navigate to the Local System Tab:

    • In the Arbor interface, locate the navigation pane or tabs.
    • Click on Local System to access tools for scanning the local machine.

  3. Scan for PCIe Devices:

    • Look for a Scan or Rescan button, typically located at the top or bottom of the interface.
    • Click Scan/Rescan to initiate the detection process.
    • Wait for the scanning process to complete; this may take a few moments depending on the number of devices connected.

  4. Review Detected Devices:

    • Once the scan is complete, Arbor will display a list of all detected PCIe devices.
    • The devices are usually listed with their names, device IDs, and other identifying information.

5.1.3 Identify the Donor Device

Identifying the correct donor device is crucial for accurate emulation.

Steps:

  1. Locate Your Donor Device in the List:

    • Scroll through the list of devices detected by Arbor.
    • Look for the device matching the make and model of your donor hardware.
    • Devices may be listed by their vendor names, device types, or function.

  2. Verify Device Details:

    • Click on the device to select it.
    • Confirm that the Device ID and Vendor ID match those of your donor device.
      • Tip: These IDs are typically found in the device's documentation or on the manufacturer's website.

  3. View Detailed Configuration:

    • With the device selected, find and click on an option like View Details or Properties.
    • This will open a detailed view showing the device's configuration space and capabilities.

  4. Cross-Reference with Physical Hardware:

    • If multiple similar devices are listed, cross-reference the Slot Number or Bus Address with the physical slot where the donor device is installed.

5.1.4 Capture Device Data

Extracting detailed information from the donor device is essential for accurate emulation.

Information to Extract:

Steps:

  1. Navigate to the PCI Config Tab:

    • Within the device's detailed view, find and select the PCI Config or Configuration Space tab.

  2. Record Relevant Details:

    • Carefully document each of the required fields.
    • Use screenshots or copy the values into a text file or spreadsheet for accuracy.
    • Ensure hexadecimal values are noted correctly, including the 0x prefix if used.

  3. Expand Capability Lists:

    • Look for sections labeled Capabilities or Advanced Features.
    • Document each capability and its parameters (e.g., MSI count, power states supported).

  4. Examine BARs in Detail:

    • For each BAR, note:
      • BAR Number (e.g., BAR0):
      • Type (Memory or I/O):
      • Bit Width (32-bit or 64-bit):
      • Size (e.g., 256 MB):
      • Prefetchable Status (Yes/No):

  5. Save the Data for Reference:

    • Compile all the information into a well-organized document.
    • Label each section clearly for easy reference during firmware customization.

  6. Double-Check Entries:

    • Review all recorded data to ensure accuracy.
    • Correct any discrepancies by revisiting the Arbor interface.

5.2 Extracting and Recording Device Attributes

After capturing the data, it's crucial to understand the significance of each attribute and ensure they've been accurately documented.

Ensure You Have Accurately Recorded the Following:

  1. Device ID:

    • Purpose: Uniquely identifies the device model.
    • Usage: Essential for the host OS to load the correct driver.

  2. Vendor ID:

    • Purpose: Identifies the manufacturer.
    • Usage: Used in conjunction with Device ID to match device drivers.

  3. Subsystem ID and Subsystem Vendor ID:

    • Purpose: Specifies the subsystem's device and vendor IDs, allowing for differentiation between variants.
    • Usage: Important for devices with multiple configurations or OEM-specific versions.

  4. Revision ID:

    • Purpose: Indicates the hardware revision.
    • Usage: Helps in identifying specific hardware versions that may require different drivers or firmware.

  5. Class Code:

    • Purpose: Categorizes the device type (e.g., mass storage, network controller).
    • Usage: Allows the OS to understand the device's general function.

  6. Base Address Registers (BARs):

    • Purpose: Define the memory or I/O address regions that the device will use.
    • Usage: Critical for mapping device memory into the system address space.

  7. Capabilities:

    • Purpose: Lists the advanced features the device supports.
    • Examples:
      • MSI/MSI-X: Message Signaled Interrupts for efficient interrupt handling.
      • Power Management: States like D0, D1, D2, D3hot, D3cold.
      • PCIe Link Speed/Width: Determines the data transfer capabilities.

  8. Device Serial Number (DSN):

    • Purpose: A unique 64-bit identifier for the device.
    • Usage: Used for advanced identification and may be required by some drivers.

Best Practices:

6. Initial Firmware Customization

With the donor device's information meticulously documented, the next phase involves customizing your FPGA's firmware to emulate the donor device accurately. This involves modifying the PCIe configuration space and ensuring that memory mappings align correctly.

6.1 Modifying Configuration Space

The PCIe configuration space is a critical component that defines how the device is recognized and interacts with the host system. Customizing this space to match the donor device is essential for successful emulation.

6.1.1 Navigate to the Configuration File

The configuration space is defined within a specific SystemVerilog (.sv) file in your project.

Path:

6.1.2 Open the File in Visual Studio Code

Editing the configuration file requires a suitable code editor that supports syntax highlighting for SystemVerilog.

Steps:

  1. Launch Visual Studio Code:

    • Click on the VS Code icon or find it via the Start Menu.

  2. Open the File:

    • Use File > Open File or press Ctrl + O.
    • Navigate to the configuration file path mentioned above.
    • Select pcileech_pcie_cfg_a7.sv and click Open.

  3. Verify Syntax Highlighting:

    • Ensure that the editor recognizes the .sv file extension.
    • If necessary, install extensions for SystemVerilog support.

  4. Familiarize Yourself with the File Structure:

    • Scroll through the file to understand the existing assignments and comments.
    • Look for sections where configuration registers are defined.

6.1.3 Modify Device ID and Vendor ID

Updating these identifiers is crucial for the host system to recognize the emulated device as the donor.

Steps:

  1. Search for cfg_deviceid:

    • Use the search functionality (Ctrl + F).
    • Locate the line defining cfg_deviceid.

  2. Update Device ID:

    cfg_deviceid <= 16'hXXXX;  // Replace XXXX with the donor's Device ID
    • Example:
      • If the donor's Device ID is 0x1234, update as: 
        cfg_deviceid <= 16'h1234;

  3. Search for cfg_vendorid:

    • Locate the line defining cfg_vendorid.

  4. Update Vendor ID:

    cfg_vendorid <= 16'hYYYY;  // Replace YYYY with the donor's Vendor ID
    • Example:
      • If the donor's Vendor ID is 0xABCD, update as: 
        cfg_vendorid <= 16'hABCD;

  5. Ensure Correct Formatting:

    • Verify that hexadecimal values are prefixed with 16'h.
    • Maintain consistent indentation and commenting style.

6.1.4 Modify Subsystem ID and Revision ID

These identifiers provide additional details about the device variant and hardware revision.

Steps:

  1. Search for cfg_subsysid:

    • Locate the line defining cfg_subsysid.

  2. Update Subsystem ID:

    cfg_subsysid <= 16'hZZZZ;  // Replace ZZZZ with the donor's Subsystem ID
    • Example:
      • If the donor's Subsystem ID is 0x5678, update as: 
        cfg_subsysid <= 16'h5678;

  3. Search for cfg_subsysvendorid:

    • Locate the line defining cfg_subsysvendorid.

  4. Update Subsystem Vendor ID (if applicable):

    cfg_subsysvendorid <= 16'hWWWW;  // Replace WWWW with the donor's Subsystem Vendor ID
    • Example:
      • If the donor's Subsystem Vendor ID is 0x9ABC, update as: 
        cfg_subsysvendorid <= 16'h9ABC;

  5. Search for cfg_revisionid:

    • Locate the line defining cfg_revisionid.

  6. Update Revision ID:

    cfg_revisionid <= 8'hRR;   // Replace RR with the donor's Revision ID
    • Example:
      • If the donor's Revision ID is 0x01, update as: 
        cfg_revisionid <= 8'h01;

6.1.5 Update Class Code

The Class Code informs the host of the device type and function.

Steps:

  1. Search for cfg_classcode:

    • Locate the line defining cfg_classcode.

  2. Update Class Code:

    cfg_classcode <= 24'hCCCCCC;  // Replace CCCCCC with the donor's Class Code
    • Example:
      • If the donor's Class Code is 0x020000 (Ethernet Controller), update as: 
        cfg_classcode <= 24'h020000;

  3. Verify Correct Bit Width:

    • Ensure that the Class Code is a 24-bit value.
    • Hexadecimal values should be prefixed with 24'h.

6.1.6 Save Changes

After making all modifications, it's important to save and review the changes.

Steps:

  1. Save the File:

    • Click File > Save or press Ctrl + S.

  2. Review Changes:

    • Re-read the modified lines to confirm accuracy.
    • Check for any syntax errors or typos.

  3. Optional - Use Version Control:

    • If using Git or another version control system, commit your changes with a meaningful message.
      • Example: 
        git add pcileech_pcie_cfg_a7.sv
git commit -m "Updated PCIe configuration with donor device identifiers"

6.2 Inserting the Device Serial Number (DSN)

The Device Serial Number (DSN) is a unique identifier that some devices utilize for advanced features. Including it enhances the authenticity of the emulation.

6.2.1 Locate the DSN Field

The DSN is typically defined in the same configuration file.

Steps:

  1. Search for cfg_dsn:

    • In pcileech_pcie_cfg_a7.sv, use the search function (Ctrl + F) to find cfg_dsn.

  2. Understand the Existing Assignment:

    • The DSN may be set to a default value or zeroed out. 
      cfg_dsn <= 64'h0000000000000000;  // Default DSN

6.2.2 Insert the DSN

Updating the DSN involves setting it to the exact value from the donor device.

Steps:

  1. Update cfg_dsn:

    cfg_dsn <= 64'hXXXXXXXX_YYYYYYYY;  // Replace with donor DSN
    • Example:
      • If the donor's DSN is 0x0011223344556677, update as: 
        cfg_dsn <= 64'h0011223344556677;

  2. Handle DSN Unavailability:

    • If the donor device does not have a DSN or it is not required, set it to zero: 
      cfg_dsn <= 64'h0000000000000000;  // No DSN

  3. Ensure Correct Formatting:

    • The DSN is a 64-bit value; ensure it's properly formatted.
    • Use the 64'h prefix for hexadecimal values.

  4. Add Comments for Clarity:

    • Include a comment indicating the DSN source. 
      cfg_dsn <= 64'h0011223344556677;  // Donor DSN

6.2.3 Save Changes

Finalize the modifications by saving and reviewing.

Steps:

  1. Save the File:

    • Click File > Save or press Ctrl + S.

  2. Verify the Syntax:

    • Look for any red underlines or error indications in the editor.
    • Correct any issues before proceeding.

  3. Document the Changes:

    • If using version control, commit the updates with an appropriate message.
      • Example: 
        git commit -am "Inserted donor Device Serial Number (DSN) into configuration"

7. Vivado Project Setup and Customization

With the firmware files updated to reflect the donor device's configuration, the next step is to integrate these changes into the Vivado project. This involves generating the project files, customizing IP cores, and preparing the design for synthesis and implementation.

7.1 Generating Vivado Project Files

Vivado uses Tcl scripts to automate project creation and configuration. By running these scripts, you ensure that all settings are correctly applied based on your FPGA device.

7.1.1 Open Vivado

Starting with a fresh session of Vivado ensures that previous settings or projects do not interfere with your current work.

Steps:

  1. Launch Vivado:

    • Find the Vivado application in your Start Menu or desktop.
    • Click to open it.

  2. Select the Correct Version:

    • If multiple versions are installed, ensure you are using the one compatible with your FPGA (e.g., Vivado 2020.1).

  3. Wait for the Startup Screen:

    • Allow Vivado to fully initialize before proceeding.

7.1.2 Access the Tcl Console

The Tcl Console allows you to execute scripts and commands directly.

Steps:

  1. Open the Tcl Console:

    • In the Vivado interface, go to the menu bar.
    • Click on Window > Tcl Console.
    • The Tcl Console will appear at the bottom of the window.

  2. Adjust Console Size (Optional):

    • Drag the console's top border to resize it for better visibility.

  3. Clear Previous Commands:

    • If any commands are present, you can clear them for a clean start.

7.1.3 Navigate to the Project Directory

Ensure that the Tcl Console is pointing to the correct directory where your project scripts are located.

For Squirrel DMA (35T):

Path:

Steps:

  1. Set the Working Directory:

    • In the Tcl Console, enter: 
      cd C:/Users/YourUsername/Documents/pcileech-fpga/pcileech-wifi-main/
      • Replace the path with the actual location on your system.

  2. Verify the Directory Change:

    • Enter pwd in the Tcl Console.
    • The console should display the current directory, confirming the change.

7.1.4 Generate the Vivado Project

Running the appropriate Tcl script will set up the project with all necessary configurations.

Steps:

  1. Run the Tcl Script:

    • For Squirrel (35T): 
      source vivado_generate_project_squirrel.tcl -notrace
    • For Enigma-X1 (75T): 
      source vivado_generate_project_enigma_x1.tcl -notrace
    • For ZDMA (100T): 
      source vivado_generate_project_100t.tcl -notrace

  2. Wait for Script Completion:

    • The script will execute several commands:
      • Create the project.
      • Add source files.
      • Configure project settings.
    • Monitor the Tcl Console for progress messages.
    • Address any errors that may occur, such as missing files or incorrect paths.

  3. Confirm Project Generation:

    • Upon completion, the console will indicate that the project has been created.
    • The project files (.xpr and associated directories) will be present in the project directory.

7.1.5 Open the Generated Project

Now that the project is generated, you can open it within Vivado for further customization.

Steps:

  1. Open the Project:

    • In Vivado, click File > Open Project.
    • Navigate to your project directory.

  2. Select the Project File:

    • For Squirrel: 
      pcileech_squirrel_top.xpr
    • Click on the .xpr file to select it.

  3. Click Open:

    • Vivado will load the project, displaying the design hierarchy and sources.

  4. Verify Project Contents:

    • In the Project Manager window, ensure that all source files are listed.
    • Check for any warnings or errors upon opening.

7.2 Modifying IP Blocks

The PCIe IP core is a critical component that must be configured to match the donor device's specifications. Customizing the IP core ensures that the FPGA behaves identically to the donor hardware at the PCIe protocol level.

7.2.1 Access the PCIe IP Core

The PCIe IP core is an instantiated IP block within your Vivado project.

Steps:

  1. Locate the PCIe IP Core:

    • In the Sources pane, ensure the Hierarchy tab is selected.
    • Expand the design hierarchy to find the PCIe IP core.
      • It is typically named pcie_7x_0.xci or similar.

  2. Open the IP Customization Window:

    • Right-click on pcie_7x_0.xci.
    • Select Customize IP from the context menu.
    • The IP Configuration window will open.

  3. Wait for IP Settings to Load:

    • The IP customization interface may take a few moments to initialize.
    • Ensure that all options and tabs are fully loaded before proceeding.

7.2.2 Customize Device IDs and BARs

Configuring the device identifiers within the IP core is crucial for correct enumeration by the host system.

Steps:

  1. Navigate to Device and Vendor Identifiers:

    • In the IP customization window, select the Device and Vendor Identifiers tab or section.

  2. Enter the Device ID:

    • Find the field labeled Device ID.
    • Enter the donor's Device ID (e.g., 0x1234).

  3. Enter the Vendor ID:

    • Locate the Vendor ID field.
    • Input the donor's Vendor ID (e.g., 0xABCD).

  4. Enter the Subsystem ID and Subsystem Vendor ID:

    • Input the Subsystem ID (e.g., 0x5678).
    • Input the Subsystem Vendor ID (e.g., 0x9ABC).

  5. Set the Revision ID:

    • Enter the Revision ID (e.g., 0x01).

  6. Set the Class Code:

    • Enter the Class Code (e.g., 0x020000 for Ethernet Controller).

  7. Configure Other Identifiers (if available):

    • Some IP cores allow setting Programming Interface, Device Capabilities, etc.
    • Match these to the donor device as needed.

7.2.3 Configure BAR Sizes

The BARs define how the device maps its internal memory and registers to the host system.

Steps:

  1. Navigate to Base Address Registers (BARs):

    • Select the BARs tab or section in the IP customization window.

  2. Configure Each BAR:

    • For BAR0 to BAR5, set the following parameters based on the donor device:
      • Enable BAR: Check or uncheck to match donor device.
      • BAR Size: Select the size from the dropdown (e.g., 256 MB, 64 KB).
      • BAR Type:
        • Memory (32-bit Addressing)
        • Memory (64-bit Addressing)
        • I/O
      • Prefetchable: Check if the donor's BAR is prefetchable.

  3. Example Configuration:

    • BAR0:
      • Enabled
      • Size: 256 MB
      • Type: Memory (64-bit)
      • Prefetchable: Yes
    • BAR1:
      • Disabled (if the donor device does not use BAR1)

  4. Ensure Alignment and Non-Overlapping Spaces:

    • Verify that the total memory mapped does not exceed the FPGA's capabilities.
    • Ensure that BAR sizes align with PCIe specification requirements.

  5. Advanced Settings (if applicable):

    • Some devices may have special requirements, such as expansion ROM BAR.
    • Configure these settings if necessary.

7.2.4 Finalize IP Customization

After configuring all necessary settings, you need to apply the changes.

Steps:

  1. Review All Settings:

    • Go through each tab in the IP customization window.
    • Confirm that all entries match the donor device's specifications.

  2. Apply Changes:

    • Click OK or Generate to apply the settings.
    • If prompted, confirm that you wish to proceed with the changes.

  3. Regenerate IP Core:

    • Vivado will regenerate the IP core to reflect the new configurations.
    • Monitor the Messages pane for any errors or warnings.

  4. Update IP in Project:

    • Ensure that the updated IP core is correctly integrated into your project.
    • Vivado may prompt to update IP dependencies; allow it to do so.

7.2.5 Lock the IP Core

Locking the IP core prevents unintended changes during synthesis and implementation.

Purpose:

Steps:

  1. Open the Tcl Console:

    • In Vivado, if not already open, go to Window > Tcl Console.

  2. Execute the Lock Command:

    • Enter the following command: 
      set_property -name {IP_LOCKED} -value true -objects [get_ips pcie_7x_0]
    • Press Enter to execute.

  3. Verify the Lock:

    • Check the Messages pane for confirmation.
    • The IP core should now be marked as locked.

  4. Unlocking (if necessary):

    • To make further changes in the future, you can unlock the IP core: 
      set_property -name {IP_LOCKED} -value false -objects [get_ips pcie_7x_0]
    • Remember to re-lock it after making changes.

  5. Document the Action:

    • Note in your project documentation that the IP core has been locked.
    • This helps team members understand the project's configuration state.

Part 2: Intermediate Concepts and Implementation

8. Advanced Firmware Customization

To achieve a precise emulation of the donor device, further in-depth customization of the firmware is necessary. This involves aligning the PCIe parameters, adjusting Base Address Registers (BARs), and emulating power management and interrupt mechanisms to match the donor device's specifications. These steps ensure that the emulated device interacts seamlessly with the host system and behaves identically to the original hardware.

8.1 Configuring PCIe Parameters for Emulation

Accurate emulation requires that the PCIe parameters of your FPGA device are meticulously configured to match those of the donor device. This includes settings such as the PCIe link speed, link width, capability pointers, and maximum payload sizes. Proper configuration ensures compatibility with the host system and the correct operation of drivers and applications that interact with the device.

8.1.1 Matching PCIe Link Speed and Width

The PCIe link speed and width are critical parameters that determine the data throughput and performance of the device. Matching these settings with the donor device is essential for accurate emulation.

Steps:

  1. Access PCIe IP Core Settings:

    • Open Your Vivado Project:

      • Launch Vivado and open the project you previously created or modified.
      • Ensure that all source files are correctly added to the project.

    • Locate the PCIe IP Core:

      • In the Sources pane, expand the hierarchy to find the PCIe IP core instance, typically named pcie_7x_0.
      • The file associated with the IP core is usually pcie_7x_0.xci.

    • Customize the IP Core:

      • Right-click on pcie_7x_0.xci and select Customize IP.
      • The IP customization window will open, displaying various configuration options.

  2. Set Maximum Link Speed:

    • Navigate to Link Parameters:

      • In the IP customization window, click on the Link Parameters tab or section.
      • This section contains settings related to the PCIe link's characteristics.

    • Configure Maximum Link Speed:

      • Find the Maximum Link Speed option.
      • Set it to match the donor device's link speed.
        • Example:
          • If the donor device operates at Gen2 (5.0 GT/s), select 5.0 GT/s.
          • If it operates at Gen1 (2.5 GT/s) or Gen3 (8.0 GT/s), select the corresponding option.
      • Note: Ensure that your FPGA and the physical hardware support the selected link speed.

  3. Set Link Width:

    • Configure Link Width:
      • In the same Link Parameters section, locate the Link Width setting.
      • Set it to match the donor device's link width.
        • Example:
          • If the donor device uses a x4 link, set the Link Width to 4.
          • Options typically include 1, 2, 4, 8, 16 lanes.
      • Note: The physical connectors and the FPGA must support the selected link width.

  4. Save and Regenerate:

    • Apply Changes:

      • After configuring the link speed and width, click OK to apply the changes.
      • Vivado may prompt you to regenerate the IP core due to the changes made.
      • Confirm and allow the regeneration process to complete.

    • Verify Settings:

      • Once regeneration is complete, revisit the IP core settings to ensure the configurations are correctly applied.
      • Check for any warnings or errors in the Messages window.

8.1.2 Setting Capability Pointers

Capability pointers in the PCIe configuration space point to various capability structures, such as MSI, power management, and others. Correctly setting these pointers ensures that the host system can locate and utilize the device's capabilities.

Steps:

  1. Locate Capability Pointer in Firmware:

    • Open Configuration File:

      • In Visual Studio Code, open the pcileech_pcie_cfg_a7.sv file located at: 
        pcileech-fpga/pcileech-wifi-main/src/pcileech_pcie_cfg_a7.sv

    • Understand the Capability Pointer:

      • The capability pointer is an 8-bit register that points to the first capability structure in the PCIe configuration space, usually starting after the standard configuration header.

  2. Set Capability Pointer Value:

    • Find the Assignment for cfg_cap_pointer:

      • Search for the line in the code where cfg_cap_pointer is assigned. 
        cfg_cap_pointer <= 8'hXX; // Current value

    • Update the Capability Pointer:

      • Replace XX with the donor device's capability pointer value.
        • Example:
          • If the donor device's capability pointer is 0x60, update the line to: 
            cfg_cap_pointer <= 8'h60; // Updated to match donor device

    • Ensure Correct Alignment:

      • Capability structures must be aligned on a 4-byte boundary.
      • The capability pointer should point to a valid offset within the configuration space.

  3. Save Changes:

    • Save the Configuration File:

      • After making the changes, save the file by clicking File > Save or pressing Ctrl + S.

    • Verify Syntax:

      • Ensure there are no syntax errors introduced by the changes.

    • Comment for Clarity:

      • Add a comment explaining the change for future reference. 
        cfg_cap_pointer <= 8'h60; // Set to donor's capability pointer at offset 0x60

8.1.3 Adjusting Maximum Payload and Read Request Sizes

These parameters define the maximum amount of data that can be transferred in a single PCIe transaction. Matching these settings with the donor device ensures compatibility and optimal performance.

Steps:

  1. Set Maximum Payload Size:

    • Access Device Capabilities:

      • In the PCIe IP core customization window, navigate to the Device Capabilities or Capabilities tab.

    • Configure Max Payload Size Supported:

      • Find the Max Payload Size Supported setting.
      • Set it to the value supported by the donor device.
        • Options:
          • 128 bytes, 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes.
        • Example:
          • If the donor device supports a maximum payload size of 256 bytes, select 256 bytes.

  2. Set Maximum Read Request Size:

    • Configure Max Read Request Size Supported:
      • In the same tab, find the Max Read Request Size Supported setting.
      • Set it to match the donor device's capability.
        • Example:
          • If the donor supports a maximum read request size of 512 bytes, select 512 bytes.

  3. Adjust Firmware Parameters:

    • Open pcileech_pcie_cfg_a7.sv:

      • Ensure that the configuration file is open in Visual Studio Code.

    • Update Firmware Constants:

      • Locate the lines where max_payload_size_supported and max_read_request_size_supported are defined. 
        max_payload_size_supported <= 3'bZZZ; // Current value
max_read_request_size_supported <= 3'bWWW; // Current value

    • Set the Appropriate Values:

      • Replace ZZZ and WWW with the binary representations of the sizes.
        • Mapping:
          • 128 bytes: 3'b000
          • 256 bytes: 3'b001
          • 512 bytes: 3'b010
          • 1024 bytes: 3'b011
          • 2048 bytes: 3'b100
          • 4096 bytes: 3'b101
        • Example:
          • For 256 bytes payload size: 
            max_payload_size_supported <= 3'b001; // Supports up to 256 bytes
          • For 512 bytes read request size: 
            max_read_request_size_supported <= 3'b010; // Supports up to 512 bytes

  4. Save Changes:

    • Save the File:

      • After updating the values, save the file.

    • Verify Consistency:

      • Ensure that the values in the firmware match those configured in the PCIe IP core.

    • Add Comments:

      • Document the changes for future reference. 
        max_payload_size_supported <= 3'b001; // 256 bytes as per donor device
max_read_request_size_supported <= 3'b010; // 512 bytes as per donor device

8.2 Adjusting BARs and Memory Mapping

Base Address Registers (BARs) define the memory regions that the device exposes to the host. Correctly configuring the BARs and memory mapping is crucial for accurate emulation and proper operation of device drivers.

8.2.1 Setting BAR Sizes

Configuring the BAR sizes ensures that the device requests the correct amount of address space during enumeration and that the host maps these regions appropriately.

Steps:

  1. Access BAR Configuration:

    • Customize PCIe IP Core:

      • In Vivado, right-click on pcie_7x_0.xci and select Customize IP.

    • Navigate to BARs Tab:

      • In the IP customization window, click on the Base Address Registers (BARs) tab.

  2. Configure BAR Sizes and Types:

    • Match Donor Device's BARs:

      • For each BAR (BAR0 to BAR5), set the size and type to match the donor device.

    • Set BAR Sizes:

      • Select the appropriate size from the dropdown for each BAR.
        • Example:
          • If BAR0 is 64 KB, set BAR0 Size to 64 KB.
          • If BAR1 is 128 MB, set BAR1 Size to 128 MB.

    • Set BAR Types:

      • Choose between 32-bit or 64-bit addressing for each BAR.
      • Specify if the BAR is of type Memory or I/O.
      • Set Prefetchable status based on the donor device.

    • Enable or Disable BARs:

      • Ensure that only the BARs used by the donor device are enabled.

  3. Update BRAM Configurations:

    • Adjust BRAM IP Cores:

      • In the ip directory, locate the BRAM configurations corresponding to the BARs.
        • Files: 
          pcileech-fpga/pcileech-wifi-main/ip/bram_bar_zero4k.xci
pcileech-fpga/pcileech-wifi-main/ip/bram_pcie_cfgspace.xci

    • Modify BRAM Sizes:

      • Open each BRAM IP core and adjust the memory size to match the corresponding BAR size.
      • Ensure that the total memory does not exceed the FPGA's capacity.

  4. Save and Regenerate:

    • Apply Changes:

      • After configuring the BARs and updating BRAM sizes, click OK in the IP customization window.

    • Regenerate IP Cores:

      • Vivado may prompt you to regenerate the IP cores due to the changes.
      • Allow the regeneration to complete.

    • Check for Errors:

      • Review the Messages window for any warnings or errors related to BAR configurations.

8.2.2 Defining BAR Address Spaces in Firmware

With the BAR sizes and types set, you need to define how the firmware handles accesses to these BARs.

Steps:

  1. Open the BAR Controller File:

    • Locate the Source File:
      • In Visual Studio Code, open: 
        pcileech-fpga/pcileech-wifi-main/src/pcileech_tlps128_bar_controller.sv

  2. Map Address Ranges:

    • Define Address Decoding Logic:

      • Implement logic to detect when a BAR is accessed based on the address. 
        always_comb begin
  if (bar_hit[0]) begin
    // Handle accesses to BAR0
  end else if (bar_hit[1]) begin
    // Handle accesses to BAR1
  end
  // Continue for additional BARs
end

    • Implement BAR Access Handling:

      • For each BAR, define how reads and writes are managed.
        • Example: 
          if (bar_hit[0]) begin
  case (addr_offset)
    16'h0000: data_out <= reg0;
    16'h0004: data_out <= reg1;
    // Additional registers
    default: data_out <= 32'h0;
  endcase
end

  3. Implement Address Decoding Logic:

    • Calculate Address Offsets:

      • Use the incoming address to calculate offsets within the BAR. 
        addr_offset = incoming_address - bar_base_address[0];

    • Handle Data Transfers:

      • Implement logic for read and write operations. 
        if (cfg_write) begin
  // Write data to the appropriate register
end else if (cfg_read) begin
  // Read data from the appropriate register
end

  4. Save Changes:

    • Save the File:

      • After implementing the logic, save the pcileech_tlps128_bar_controller.sv file.

    • Verify Functionality:

      • Ensure that the logic correctly handles all possible accesses.

8.2.3 Handling Multiple BARs

Properly managing multiple BARs is essential for devices that expose multiple memory or I/O regions.

Steps:

  1. Implement Logic for Each BAR:

    • Separate Logic Blocks:

      • For clarity, create separate code blocks for each BAR within the controller. 
        // BAR0 Handling
if (bar_hit[0]) begin
  // BAR0 specific logic
end
// BAR1 Handling
if (bar_hit[1]) begin
  // BAR1 specific logic
end

    • Define Registers and Memories:

      • Allocate registers or memory blocks for each BAR as needed.

  2. Ensure Non-Overlapping Address Spaces:

    • Validate Address Ranges:

      • Confirm that the address spaces for each BAR do not overlap.
      • Align BAR sizes to power-of-two boundaries as per PCIe specifications.

    • Update Address Decoding:

      • Adjust the address decoding logic to account for the sizes and bases of each BAR.

  3. Test BAR Accesses:

    • Simulation Testing:

      • Use simulation tools to test read and write operations to each BAR.
      • Verify that the correct data is read or written.

    • Hardware Testing:

      • After programming the FPGA, use software tools on the host to access each BAR.
      • Example:
        • Use lspci on Linux to inspect BAR mappings.
        • Write test programs that perform memory-mapped I/O to the BARs.

8.3 Emulating Device Power Management and Interrupts

Emulating power management features and implementing interrupts are critical for devices that need to interact closely with the host operating system's power and interrupt handling mechanisms.

8.3.1 Power Management Configuration

Implementing power management allows the device to support various power states, contributing to system-wide power efficiency and compliance with operating system expectations.

Steps:

  1. Enable Power Management in PCIe IP Core:

    • Access Capabilities:

      • In the PCIe IP core customization window, select the Capabilities tab.

    • Enable Power Management:

      • Check the option for Power Management to include the capability in the device's configuration space.

  2. Set Power States Supported:

    • Configure Supported States:
      • Specify which power states the device supports, such as:
        • D0 (Fully On)
        • D1, D2 (Intermediate States)
        • D3hot, D3cold (Low Power States)
      • Match these settings to the donor device's capabilities.

  3. Implement Power State Logic in Firmware:

    • Open pcileech_pcie_cfg_a7.sv:

      • Modify the firmware to handle power state transitions.

    • Handle Power Management Registers:

      • Implement read and write access to the Power Management Control and Status Register (PMCSR). 
        // PMCSR Address
localparam PMCSR_ADDRESS = 12'h44; // Example address

// PMCSR Register
reg [15:0] pmcsr_reg;

// Handle PMCSR Writes
always @(posedge clk) begin
  if (cfg_write && cfg_address == PMCSR_ADDRESS) begin
    pmcsr_reg <= cfg_writedata[15:0];
    // Update power state based on pmcsr_reg[1:0]
  end
end

    • Manage Power State Effects:

      • Implement the logic to alter device behavior based on the current power state.

  4. Save Changes:

    • Save the Firmware File:

      • Ensure all modifications are saved.

    • Verify Functionality:

      • Test the power management features through simulation or hardware testing.

8.3.2 MSI/MSI-X Configuration

Implementing MSI/MSI-X allows the device to use message-based interrupts, which are more efficient and scalable than traditional pin-based interrupts.

Steps:

  1. Enable MSI/MSI-X in PCIe IP Core:

    • Access Interrupts Configuration:

      • In the PCIe IP core customization window, navigate to the Interrupts or MSI/MSI-X tab.

    • Select Interrupt Type:

      • Choose MSI or MSI-X based on the donor device.

    • Configure Number of Supported Vectors:

      • Set the number of interrupt vectors to match the donor device.
        • MSI supports up to 32 vectors.
        • MSI-X supports up to 2048 vectors.

    • Enable Capabilities:

      • Ensure that the MSI or MSI-X capabilities are included in the device's configuration space.

  2. Implement Interrupt Logic in Firmware:

    • Open pcileech_pcie_tlp_a7.sv:

      • Modify the firmware to handle interrupt generation.

    • Define Interrupt Signals:

      • Declare signals for MSI/MSI-X requests. 
        reg msi_req;

    • Implement Interrupt Generation Logic:

      • Define conditions under which an interrupt is triggered. 
        // Example Interrupt Condition
wire interrupt_condition = /* condition logic */;

// Generate MSI Interrupt
always @(posedge clk) begin
  if (interrupt_condition) begin
    msi_req <= 1'b1;
  end else begin
    msi_req <= 1'b0;
  end
end

    • Connect to PCIe Core:

      • Ensure that the msi_req signal is properly connected to the PCIe IP core's interrupt interface.

  3. Save Changes:

    • Save the Firmware File:

      • After implementing the interrupt logic, save the file.

    • Check for Timing Constraints:

      • Verify that the new logic does not introduce timing violations.

8.3.3 Implementing Interrupt Handling Logic

Defining when and how interrupts are generated is essential for the device's interaction with the host's interrupt handling mechanisms.

Steps:

  1. Define Interrupt Conditions:

    • Identify Trigger Events:

      • Determine specific events that should cause an interrupt.
        • Examples:
          • Data ready for processing.
          • Error conditions.
          • Completion of a task.

    • Implement Condition Logic:

      • Use combinational or sequential logic to detect these events.

  2. Create Interrupt Generation Module:

    • Modular Design:

      • Implement the interrupt logic as a separate module for clarity and reuse. 
        module interrupt_controller(
  input wire clk,
  input wire reset,
  input wire event_trigger,
  output reg msi_req
);
  always @(posedge clk or posedge reset) begin
    if (reset) begin
      msi_req <= 1'b0;
    end else if (event_trigger) begin
      msi_req <= 1'b1;
    end else begin
      msi_req <= 1'b0;
    end
  end
endmodule

    • Integrate with Main Firmware:

      • Instantiate the module and connect it to the main firmware logic.

  3. Ensure Proper Timing and Sequencing:

    • Adhere to PCIe Specifications:

      • Ensure interrupts are generated and cleared according to the protocol.

    • Manage Interrupt Latency:

      • Optimize logic to minimize delay between event occurrence and interrupt generation.

  4. Test Interrupt Delivery:

    • Simulation:

      • Use simulation tools to verify that interrupts are generated correctly.

    • Hardware Testing:

      • Program the FPGA and use host-side software to confirm that interrupts are received and handled.

    • Debugging Tools:

      • Utilize Integrated Logic Analyzer (ILA) cores to monitor signals in real time.

  5. Save Changes:

    • Finalize Code:

      • Ensure all changes are saved and documented.

    • Review and Refine:

      • Iterate on the design as needed based on testing results.

10. Transaction Layer Packet (TLP) Emulation

Transaction Layer Packets (TLPs) are the fundamental units of communication in PCIe. Accurate TLP emulation is crucial for the device to interact properly with the host system.

10.1 Understanding and Capturing TLPs

10.1.1 Learning the TLP Structure

10.1.2 Capturing TLPs from the Donor Device

10.1.3 Documenting Key TLP Transactions

10.2 Crafting Custom TLPs for Specific Operations

10.2.1 Implementing TLP Handling in Firmware

10.2.2 Handling Different TLP Types

10.2.3 Validating TLP Timing and Sequence

Part 3: Advanced Techniques and Optimization

11. Building, Flashing, and Testing

After all customizations, it's time to build the firmware, program it onto the FPGA, and thoroughly test it to ensure proper functionality.

11.1 Synthesis and Implementation

11.1.1 Running Synthesis

Synthesis converts your high-level code into a gate-level representation.

11.1.2 Running Implementation

Implementation maps the synthesized design onto the FPGA's resources.

11.1.3 Generating Bitstream

The bitstream is the binary file used to program the FPGA.

11.2 Flashing the Bitstream

11.2.1 Connecting the FPGA Device

11.2.2 Programming the FPGA

11.2.3 Verifying Programming

11.3 Testing and Validation

11.3.1 Verifying Device Enumeration

11.3.2 Testing Device Functionality

11.3.3 Monitoring for Errors

12. Advanced Debugging Techniques

When issues arise, advanced debugging tools and techniques can help identify and resolve problems efficiently.

12.1 Using Vivado's Integrated Logic Analyzer

The Integrated Logic Analyzer (ILA) allows real-time monitoring of internal FPGA signals.

12.1.1 Inserting ILA Cores

12.1.2 Configuring Trigger Conditions

12.1.3 Capturing and Analyzing Data

12.2 PCIe Traffic Analysis Tools

Using external tools can provide deeper insights into PCIe communications.

12.2.1 PCIe Protocol Analyzers

12.2.2 Software-Based Tools

13. Troubleshooting

This section provides solutions to common problems you may encounter during firmware development and testing.

13.1 Device Detection Issues

Problem: The FPGA device is not recognized by the host system.

Possible Causes and Solutions:

  1. Incorrect Device IDs:

    • Cause: Mismatch between the IDs in the firmware and what the host expects.
    • Solution: Verify and correct the Device ID, Vendor ID, and Subsystem ID in your firmware.

  2. PCIe Link Training Failure:

    • Cause: The PCIe link is not established.
    • Solution:
      • Check physical connections.
      • Ensure the Link Width and Link Speed are correctly configured.

  3. Power Issues:

    • Cause: Insufficient power to the FPGA device.
    • Solution: Verify power supply connections and voltage levels.

  4. Firmware Errors:

    • Cause: Errors in the firmware prevent proper operation.
    • Solution: Review code for syntax errors or misconfigurations.

13.2 Memory Mapping and BAR Configuration Errors

Problem: The device's memory regions are not accessible, or accessing them causes system errors.

Possible Causes and Solutions:

  1. Incorrect BAR Sizes or Types:

    • Cause: BAR configurations do not match the donor device.
    • Solution: Adjust BAR sizes and types in both the PCIe IP Core and firmware.

  2. Address Decoding Errors:

    • Cause: Firmware fails to correctly interpret addresses.
    • Solution: Debug the address decoding logic in your firmware.

  3. Overlapping Address Spaces:

    • Cause: BARs overlap or conflict with other devices.
    • Solution: Ensure BAR addresses are correctly aligned and do not overlap.

13.3 DMA Performance and TLP Errors

Problem: Slow data transfer rates or errors occur during DMA operations, or TLPs are malformed.

Possible Causes and Solutions:

  1. Inefficient DMA Logic:

    • Cause: DMA engine not optimized.
    • Solution: Implement buffering and pipelining to enhance throughput.

  2. Malformed TLPs:

    • Cause: Incorrect TLP formatting.
    • Solution: Review TLP assembly code to ensure compliance with the PCIe specification.

  3. Flow Control Issues:

    • Cause: Improper handling of flow control credits.
    • Solution: Implement proper flow control mechanisms in the firmware.

14. Emulation Accuracy and Optimizations

Enhancing emulation accuracy ensures compatibility and performance, making the emulated device indistinguishable from the donor.

14.1 Techniques for Accurate Timing Emulation

14.2 Dynamic Response to System Calls

15. Best Practices for Firmware Development

Following best practices helps maintain code quality, facilitates collaboration, and ensures the longevity of your project.

15.1 Continuous Testing and Documentation

15.2 Managing Firmware Versioning

15.3 Security Considerations

16. Additional Resources

Enhance your understanding and stay updated with the following resources:

17. Contact Information

If you need assistance, have questions, or wish to collaborate, feel free to reach out. I'm available to provide guidance, troubleshoot complex problems, or discuss ideas in detail.

Discord: VCPU | Server Invite Link

18. Support and Contributions

Your support helps maintain and improve this guide and related projects.

Donations

If you found this guide helpful and want to support ongoing work, consider contributing. Every donation helps in continuing to create, share, and support the community.

Special Bonus: If you donate, reach out on Discord (VCPU) to receive a personal thank you and possibly additional resources or assistance.

Note: If you need me to review your implementation or troubleshoot issues, please mark the relevant sections with //VCPU-REVIEW// and provide detailed explanations of the problems you're encountering.

End of Guide