Optical Fibre Cable & SFP Lights Good but Port Doesn't Come Up?

A good lesson learned.

I was scratching my head because we'd moved one side of a NetApp MetroCluster Cluster to a new site, all the cabling was done, everything looked good, but pretty much all the optical ports would not come up.

What I usually do is check to see light is coming out the optical cable, and light is coming out the SFP, and that the polarity is correct (light goes one way from the switch, and another way from the server/client/storage). Everything was perfect.

It turned out to be SFPs. The NetApp had (pretty much) all 25G SFP+ which don't down rev, and the switch had only 10G SFP+. Once the SFPs were corrected it all started working 🍻.

One interesting thing. The ports on the NetApp were set to autonegotiate. When I tried to manual set the port speed to 10000 (or 1000) I got a weird error which was something like "the port only supports 100Mbps". Must admit, I thought the network guys had configured their switch ports all for 100Mbps, but no, that  was a red-herring, the problem was the SFPs.