The same MAC address for two interfaces on the same host?

Yes, sounds strange, but exactly this is what we faced today. In the end it is clear and not strange anymore but for a moment we really thought we are hitting a bug in the operating system, which was RedHat 7.4. For the little demo for this post I’ll use CentOS 7.4 but this should not really matter. Lets go.

The situation we faced looked like this:

root@:/etc/sysconfig/network-scripts/ [] ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d6:95:ab brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 86295sec preferred_lft 86295sec
    inet6 fe80::a00:27ff:fed6:95ab/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s8:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:5c:b0:e5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.99/24 brd 192.168.22.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe5c:b0e5/64 scope link 
       valid_lft forever preferred_lft forever
4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe85:2d8/64 scope link tentative 
       valid_lft forever preferred_lft forever

As you can see the interfaces enp0s8 and enp0s10 have exactly the same MAC address. In addition to that the bonding interface bond0 has the same mac address as well:

4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe85:2d8/64 scope link tentative 
       valid_lft forever preferred_lft forever

This was not so scary as we knew that the bonding interface will get one of the mac addresses of the slave interfaces. But both slaves had the same mac address, how that? It all depends on the bonding options. In our case:

root@:/etc/sysconfig/network-scripts/ [] cat ifcfg-bond0 
DEVICE=bond0
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=static
IPADDR=192.168.22.223
NETMASK=255.255.255.0
GATEWAY=192.168.22.1
ONBOOT=yes
BONDING_OPTS="mode=active-backup miimon=100"

What you need to know is that there is another parameter/option called fail_over_mac. The default value for this option (as we did not explicitly set it we are using the default) is 0/none which means all slave interfaces will use the same MAC address. The original MAC addresses can be seen here:

root@:/etc/sysconfig/network-scripts/ [] cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: enp0s9
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: enp0s9
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:85:02:d8
Slave queue ID: 0

Slave Interface: enp0s10
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e4:1c:63
Slave queue ID: 0

When we change that to 1/active and restart the network:

root@:/etc/sysconfig/network-scripts/ [] sed -i 's/BONDING_OPTS="mode=active-backup miimon=100"/BONDING_OPTS="mode=active-backup miimon=100 fail_over_mac=active"/g' ifcfg-bond0 
root@:/etc/sysconfig/network-scripts/ [] systemctl restart network

… we will have different MAC addresses:

4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:e4:1c:63 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:e4:1c:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fee4:1c63/64 scope link tentative 
       valid_lft forever preferred_lft forever

The downside of this approach is that the MAC address will change when the active interface goes down. The currently active interface is enp0s9 with the following MAC address:

root@:/etc/sysconfig/network-scripts/ [] cat /proc/net/bonding/bond0 | grep -i active
Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Currently Active Slave: enp0s9
root@:/etc/sysconfig/network-scripts/ [] ip a | grep -A 1 enp0s9
4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff

The bonding interface has the same MAC address:

root@:/etc/sysconfig/network-scripts/ [] ip a | grep -A 1 bond0
4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:e4:1c:63 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever

When we bring the currently active interface (enp0s9) down the MAC address of the bonding interface will switch to the MAC address of enp0s10 (which then becomes the active interface):

root@:/etc/sysconfig/network-scripts/ [] ifdown enp0s9
Device 'enp0s9' successfully disconnected.
root@:/etc/sysconfig/network-scripts/ [] ip a | grep -A 1 bond0
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:e4:1c:63 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:e4:1c:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever

The third an last option for fail_over_mac is 3/follow which will change the MAC address of the slave to the bonding’s MAC address when it becomes active. Documentation here.