Configuring fiber interfaces during an ODA installation can sometime been challenging. The configuration can depend on the kind of gigabit interface converter (GBIC) and switch been used. In this blog, I would like to show a real case faced during one of our customer deployment, and come with a solution.

Problem

We were facing link detection issue when using the GBIC converter delivered by ORACLE, although using a CISCO GBIC converter could make the link available.
em2 interface was used with a CISCO GBIC.
em3 interface was used with an ORACLE GBIC.

[root@TEST1 tmp]# ethtool em2
Settings for em2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes
        
[root@TEST1 tmp]# ethtool em3
Settings for em3:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: no

 

Solution

Forcing the interface on 10 Gb and configuring auto-negotiation to false could solve the link detection issue with the ORACLE GBIC converter.
This has been performed by updating both ifcfg-em2 and ifcfg-em3 (stored in /etc/sysconfig/network-scripts) with :

ETHTOOL_OPTS="speed 10000 duplex full autoneg off"

 

Both links could then be detected successfully.

[root@TEST1 ~]# ethtool em2
Settings for em2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes
        
[root@TEST1 ~]# ethtool em3
Settings for em3:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes

 

SFP28 Ethernet Controller firmware upgrade

The firmware version we were running is the 20.06.04.06.

[root@TEST1 ~]# ethtool -i em3
driver: bnxt_en
version: 1.8.0
firmware-version: 20.6.141/1.8.1 pkg 20.06.04.06
bus-info: 0000:18:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no

In case of further trouble, a firmware upgrade to 20.08.01.14 version can help, as described in following MOS note.
Using the onboard SFP28 ports on an ODA X7-2 server node (Doc ID 2373070.1)