ODA fiber interface link detection issue

Configuring fiber interfaces during an ODA installation can sometime been challenging. The configuration can depend on the kind of gigabit interface converter (GBIC) and switch been used. In this blog, I would like to show a real case faced during one of our customer deployment, and come with a solution.

Problem

We were facing link detection issue when using the GBIC converter delivered by ORACLE, although using a CISCO GBIC converter could make the link available.
em2 interface was used with a CISCO GBIC.
em3 interface was used with an ORACLE GBIC.

[root@TEST1 tmp]# ethtool em2
Settings for em2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes
        
[root@TEST1 tmp]# ethtool em3
Settings for em3:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: no

Solution

Forcing the interface on 10 Gb and configuring auto-negotiation to false could solve the link detection issue with the ORACLE GBIC converter.
This has been performed by updating both ifcfg-em2 and ifcfg-em3 (stored in /etc/sysconfig/network-scripts) with :

ETHTOOL_OPTS="speed 10000 duplex full autoneg off"

Both links could then be detected successfully.

[root@TEST1 ~]# ethtool em2
Settings for em2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes
        
[root@TEST1 ~]# ethtool em3
Settings for em3:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes

SFP28 Ethernet Controller firmware upgrade

The firmware version we were running is the 20.06.04.06.

[root@TEST1 ~]# ethtool -i em3
driver: bnxt_en
version: 1.8.0
firmware-version: 20.6.141/1.8.1 pkg 20.06.04.06
bus-info: 0000:18:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no

In case of further trouble, a firmware upgrade to 20.08.01.14 version can help, as described in following MOS note.
Using the onboard SFP28 ports on an ODA X7-2 server node (Doc ID 2373070.1)