Intel(R) Advanced Network Services (ANS) for Linux* Operating Systems
=====================================================================
December 26, 2002
Contents
========
- Overview
- Installing ANS
- Configuring ANS
- Network Parameters Boot Configuration
- Structure of the ANS Build Tree
- Example Setup
- Troubleshooting
- Support
Overview
========
This file describes the Intel(R) Advanced Network Services (ANS) package,
version 2.3.x, for Linux* systems. Intel ANS provides both teaming
(including Multi-Vendor Teaming), and IEEE VLAN capabilities.
For ANS support, this package version must be used with the Linux base
driver version e100-2.2.x for PRO/100 adapters and/or driver version
e1000-5.0.x for PRO/1000 adapters. Base drivers provided in some
distributions do not contain the MII interface that is necessary for ANS.
This package is intended for 2.4.7 and higher kernels; it was tested
primarily on the Red Hat* 8.0 Linux 2.4.18 and Red Hat Advanced Server 2.1
Linux 2.4.9 kernels on SMP Intel processor-based systems. This package
includes support for Itanium(TM)-based systems.
New features for this release include:
- Switch Fault Tolerance ("SFT").
- IPv4 Receive Load Balancing ("RLB") in ALB teaming mode.
- Transmit Load Balancing of IPv6 traffic.
- Dynamic VLAN configuration with IEEE GVRP standard.
- Support of Zero Copy.
- Removed the proprietary VLAN and Teaming interfaces with e100 and e1000
base drivers.
This version of ANS may be configured with the PROCfg command-line utility.
This reporting and configuration tool reduces the command-line input required
by the manual configuration method described in this document. At this time,
PROCfg is only available for 32-bit architectures. PROCfg is available on the
Intel Customer Support website, http://support.intel.com.
NOTE: ANS does not work properly with drivers that are statically linked
into the kernel; it only works with loadable modules.
Teaming
-------
This package supports the following teaming modes:
- Adapter Fault Tolerance (AFT). Default mode when a team is created and
the mode is not specified. A secondary adapter in the team automatically
takes over on any failure of the primary connection (cable, adapter or
port). Only one network adapter is active at a time. After a failover,
if the connection to the primary adapter is restored, control passes
automatically back to that primary adapter. May mix speed and duplex
capabilities and settings. May use a hub or switch. Spanning Tree
Protocol must be turned off.
- Switch Fault Tolerance (SFT). Provides a fail-over between switches when
each adapter in the team is connected to a separate switch. Spanning
Tree Protocol must be turned on in the switches. SFT teaming mode has
been tested with the following configuration only: exactly two members
in a team connected to two switches, with one member set as preferred
primary and the other as a preferred secondary.
- Adaptive Load Balancing (ALB). Includes Receive Load Balancing (RLB). A
team of 2-8 adapters that performs receive and transmit load balancing
with no required switch configuration. Supports mixed-speed/duplex
settings among team members. Includes fault tolerance. Spanning Tree
Protocol must be turned off. An ALB team may be created with or without
RLB enabled.
If RLB is enabled (default), all active adapters with the highest speeds
load balance the IPv4 received traffic, and the primary adapter receives
traffic using any protocol. For ALB teams with RLB disabled, only the
primary adapter receives.
- Intel Link Aggregation, Cisco*'s Fast EtherChannel* Technology or static
802.3ad (FEC or FEC/LA/802.3ad: static). A team of 2-8 10/100
adapters which simultaneously receive and transmit data. Includes fault
tolerance and load balancing. Requires a switch that supports Intel Link
Aggregation, Cisco*'s FEC or static 802.3ad. Must match speed/duplex
settings on all team members. Spanning Tree Protocol must be turned off.
Must match switch aggregation requirements.
- Gigabit equivalent of FEC (GEC or GEC/LA/802.3ad: static). Requires
PRO/1000 or equivalent adapters and a switch that supports Intel Link
Aggregation, Cisco*'s GEC or static 802.3ad. Other requirements are
similar to FEC.
- IEEE 802.3ad: dynamic (802.3ad). This mode is the IEEE standard for the
technology incorporated into Cisco's FEC method. ANS support for dynamic
802.3ad teaming is similar to that for FEC and GEC teaming. In addition
to the benefits of Link Aggregation, IEEE 802.3ad Dynamic Link
Aggregation generally offers the following advantages: Automatic
Configuration, Rapid Configuration and Reconfiguration, and Deterministic
Behavior. It consists of a team of 2-8 adapters that simultaneously
receive and transmit data. Aggregation groups include only members with
full duplex and the same speed. Includes fault tolerance and load
balancing. Requires a switch that supports the IEEE 802.3ad dynamic
standard, or at least Intel Link Aggregation or Cisco's FEC. In case one
of the latter is used, the team will perform in 802.3ad static mode.
Spanning Tree Protocol must be turned off. Must match switch aggregation
requirements.
- NONE. A team of one Intel adapter that adds VLAN support on top of one
existing interface. Does not require an Intel server adapter. VLAN-
enabled must be set.
NOTES: You must have at least one Intel PRO/100 or PRO/1000 server adapter
to form a team.
For AFT, SFT, and ALB, you may specify which adapters in a team you
want to use as the primary and secondary adapters. The primary one
is the main adapter used in a team. For AFT/SFT, the primary one is
the only adapter used until a failure occurs. For ALB, the primary
one is the adapter that both transmits and receives. For ALB+RLB,
the primary adapter is the only adapter that receives non-IPv4
traffic. In a team of more than two adapters, the secondary adapter
fills the primary's role if the primary adapter fails. Also, fail
back will not occur unless the primary has been specified. If the
priority is not specified before the team is activated, the adapter
with the highest supported speed is chosen. If all adapters support
the same speed, the last one added is the primary adapter.
Probes
------
ANS supplies a periodic probes mechanism for checking the health of the
system and discovering connectivity problems. Probes can be used in all
non-Link Aggregation teaming modes, including AFT, SFT, and ALB(+RLB).
Fine tuning of the probe parameters can scale failover time and enhance
the system's reliability. Probes are on by default in teams with three or
more adapters, and off by default otherwise. See the ianscfg man page for
information on how to change probes parameters.
Multi-Vendor Teaming
--------------------
Multi-Vendor Teaming (MVT) adds the capability to include adapters from
selected vendors in a team (AFT, SFT, ALB(+RLB), FEC, GEC, 802.3ad).
MVT cannot be used with VLANs.
In order to activate MVT, you must have at least one Intel server adapter
in the team that is designated as the primary adapter. Adapters from other
vendors must be properly loaded. Use the same command line parameters for
building the team as with all Intel adapters. All members in an MVT operate
on a common feature set.
NOTES: For RLB teams, some adapters produced by non-Intel vendors do not
support setting the MAC address of the device while it is open.
Therefore, swapping MAC addresses during failover is impossible.
This creates a problem when RLB is enabled, as each team member
must use a unique MAC address. As a workaround, ANS sets the non-
Intel adapter to work in promiscuous mode when it becomes the
primary after a failover occurs. The adapter will stay promiscuous
as long as the MAC address that it is using differs from that of
the adapter's hardware.
Using ALB and RLB with non-Intel adapters requires the use of a
switch instead of a hub.
VLANs
-----
IEEE 802.3ac Virtual LANs (VLANs) are based on the addition of a tag to the
packet header. All equipment on the network that passes this traffic must
be capable of handling the extra length. Settings on the adapter must match
the VLAN settings on the switch. VLANs also add to load time and require
additional memory resources. There is a limit of 64 VLANs per server;
however, VLAN IDs may be numbered from 1 to 4094.
NOTES: When using ANS VLANs, the native VLAN module provided with the
kernel (8021q.o) must NOT be loaded.
In this release version, the proprietary VLAN and teaming interfaces
with e100 and e1000 base drivers have been removed. As a result,
VLAN hardware off-loading is supported only for kernels that support
VLAN hwaccel. These include stock Red Hat 7.3 and 8.0 distribution
kernels, or any 2.4.19 or later kernel. For older kernels, ANS
implements software VLAN tagging and stripping. For VLAN hardware
off-loading (enhanced performance) on older kernels you must use ANS
driver version 2.0.x or lower, plus the appropriate base driver
version(s) for that ANS driver.
Due to a third-party limitation, adding VLANs when the IPv6 module
is loaded might cause a network connectivity failure.
Setting VLANs statically
------------------------
When working with VLAN tagging, all the ports on the switch connected to
the server should be configured to be tagged. In some Cisco switches it is
possible to configure ports to be both tagged and untagged at the same time;
although, this configuration will result in communication loss.
When adding new VLAN IDs to an active team the switch ports attached to the
team's members must be configured with the new VLAN IDs prior to adding
them to the ANS team. Otherwise, ANS might disable members that are not
sensing probes on the new VLAN IDs because the switch is not configured
accordingly.
Setting VLANs dynamically - GVRP
--------------------------------
VLANs may be set dynamically with the GARP VLAN Registration Protocol (GVRP).
GVRP is an industry-standard protocol designed to dynamically create and
manage VLAN information from device to device. With GVRP, a single switch is
manually configured with all the desired VLANs for the network, and all
other switches on the network obtain the VLAN information dynamically.
If GVRP is enabled in the team and in the switch, the switch ports connected
to the server should not be marked as tagged, but will be dynamically
configured by ANS whenever VLAN IDs are added and removed from the team.
Installing ANS
==============
Installation assumes all necessary build tools, kernel sources, and headers
are in place and properly configured.
To compile and install the base driver package(s), see the readme included
with those drivers. Ensure no other base drivers are loaded or compiled
into the kernel for any of the Intel adapters (e.g., eepro, eepro100,
eexpress, e100, e1000). Ensure the configuration files have been edited to
alias the new driver(s). See the Example Setup section for an example file.
NOTE: Not for use with eepro100 (module removal may be necessary).
Verify that all interfaces are working and that you have the base driver
file(s), e100.o and/or e1000.o, located in:
/lib/modules/<kernel_version>/kernel/drivers/net/
The locations listed above may vary for certain Linux distributions. For
exceptions, refer to the ldistrib.txt file included with this package.
NOTE: If you already have an older version of ANS installed in your system
and you have installed this version using the INSTALL script that came
with the package, you must first uninstall it, using the UNINSTALL
script that was provided with that same ANS version. Failing to do so
may cause unexpected results when restoring ANS configuration during
boot.
1. Login as root.
2. Copy the iANS-x.x.x.tar.gz package file to a directory of your choice.
"x.x.x" specifies the iANS version number.
3. Open the package file by running 'tar xzf iANS-x.x.x.tar.gz'. This
creates a subdirectory for iANS-x.x.x under the current directory.
4. Enter the iANS-x.x.x/src directory. Run 'make' to compile the module.
NOTE: SMP/UP and MODVERSIONS are determined by 'make'. If there is a
mismatch between the configuration for the kernel source tree and
the running kernel, a warning is displayed. Run 'make options' to
list the options available for overriding the source tree settings.
Options include SMP, MODVERSIONS, and DEBUG (not listed). DEBUG
values are 0 (debugging is off - default) and 1 (debugging is on).
5. Run 'make install'.
NOTE: The default file locations are determined by 'make'. If there is a
mismatch between the configuration for the kernel source tree and
the running kernel, a warning is displayed. Either reboot to a
kernel that matches the kernel source tree or set the symbolic
link, usr/src/linux, to point to the kernel source tree that
matches the running kernel.
For a list of files and their default locations, see the Structure of
the ANS Build Tree section in this file.
Removing ANS
------------
Follow these instructions to remove ANS:
1. Login as root.
2. Run 'ifconfig' to see every open virtual adapter configured in the
system.
3. For each open virtual adapter, run 'ifconfig <vadapter_name> down'.
4. Remove the ANS module by running 'rmmod ians'.
5. Remove or disable each virtual adapter's network boot file, as
described in the Network Parameters Boot Configuration, Configuring
ANS section.
6. From the iANS-x.x.x/src directory, run 'make uninstall' to remove all
the files.
Configuring ANS
===============
You can choose one of these three methods for configuring ANS:
1. PROCfg Utility. Provides a simplified configuration tool that reduces
the amount of command-line input.
2. Scripted. Suggested for new users.
3. Manual, using the ianscfg utility.
Modifying ANS
-------------
ANS configuration and topology may be changed, with a few limitations.
After a team is activated, you may change member priorities, add or remove
team members, add or remove virtual adapters, or remove a team entirely.
You may not:
- Change the team mode.
- Enable or disable RLB teaming mode.
- Remove or add VLAN support.
- Enable or disable GVRP.
- Remove a team's last virtual adapter.
- Remove the last member of a team with open virtual adapters.
- Remove the last server adapter of a team with open virtual adapters.
If a team is left with no server adapter or with no members at all, it goes
into an idle state until members are added. During such an idle period,
resources used by the team are not freed.
For instructions on modifying ANS, see the instructions in either the
scripted or manual configuration methods.
Scripted Configuration
----------------------
For ease of configuration, Intel supplies an interactive script to help
configure an ANS topology. The script offers only the basic and most
common configuration capabilities. To be able to use all available
configuration options, refer to the Manual Configuration section, and to
the ianscfg man page.
Before using the interactive script to create and modify an ANS topology,
ensure the necessary base drivers (e.g. e100, e1000) are loaded.
To use the script, enter:
ianstool
You will be prompted with an interactive menu, offering several possible
actions to be performed on the topology. These actions include:
1. Adding a team
2. Deleting an existing team
3. Viewing an existing team's status
4. Configuring an existing team's topology (i.e. adding/removing members
and virtual adapters)
5. Saving the current topology
The script will prompt you for necessary information, such as the teaming
mode, as you go along. As noted above, the script does not offer all
possible configuration options that are available in ANS. For information
on configuring options that are not available with the script, refer to the
Manual Configuration section.
Manual Configuration
--------------------
1. After the required base drivers are loaded, bring down any Ethernet
adapters that are to be added to a team.
ifconfig -a - lists all current network devices
ifconfig ethx down - brings down adapters so ANS may use them
NOTE: Do not change the configuration of Ethernet adapters that are
not added to teams. This ensures that the network device name
assignment remains consistent after rebooting.
2. Load the ANS module:
insmod ians
3. Use the ianscfg utility to create and modify teams and VLANs. Refer to
the ianscfg man page for specific usage details.
4. Activate your teams using ianscfg, if you have not done so already.
Activate your virtual adapter(s), using:
ifconfig <vadapter_name> <IPaddr> [netmask <NETMASK>]
[broadcast <BROADCAST>]
At this point you should be able to send and receive packets via the
virtual adapter(s).
5. You can save the ANS topology or restore a previously saved topology
using ianscfg. See the man page for details.
6. If you want the new configurations to take affect after each reboot,
follow the instructions in Saving ANS Configuration Across Reboots.
Unattended Save Across Reboots
---------------------------------------
You can also save an ANS configuration across reboots from a previously
saved file or manually modified file, using the command:
ianscfg –b [-f <file name>]
Using this command will set the system to load the ANS configuration in the
specified file after the next boot. This is done without communication with
the ANS module at all – in fact, the ANS module does not have to be loaded
when this command is issued. Because of this, if the file specifies an illegal
ANS configuration there is no way to detect it until the ANS module will try
(unsuccessfully) to load the configuration at the next boot. Since some
distributions suppress messages during the boot sequence, you might need to
check your system’s log to find if any errors occurred during boot.
NOTES: Saving the configuration across reboots either creates or modifies
the network configuration files for the participating adapters. For
SuSE* 8.0 (and up) distributions, these files' locations and
formats are assumed to be as specified in the Network Parameters
Boot Configuration section, under _SuSE*_. For other distributions,
the location and format are assumed to be as specified in the same
section, under _RedHat*_. If your distribution uses a different
location/format for network configuration files (e.g. SuSE* 7.x),
ianscfg' most likely will not be able to save the configuration
across reboots.
If you modify the configuration such that some of the virtual
adapters that previously existed no longer exist, the system will
still make an (unsuccessful) attempt to load those (now non-
existent) virtual adapters when you next boot the machine. To avoid
this behavior, either use your distribution's network configuration
tools (e.g. netconf, setup) to disable (or remove) those virtual
adapters, or simply remove the relevant network configuration files
as specified in the Network Parameters Boot Configuration section.
Network Parameters Boot Configuration
====================================
If you want your virtual adapters' network parameters (i.e. IP address,
netmask, etc.) to load after the next reboot, you must first ensure you have
saved your topology to load on the next reboot. For details, see the
Scripted Configuration and Manual Configuration sections.
You can save your virtual adapters' network parameters for boot in one of
two ways:
1. Using the network and system configuration tools that are provided by
your distribution (e.g. netconf, setup, etc.)
2. Manually edit or create the virtual adapters' boot configuration files
(usually named ifcfg-<vadapter name>). This can be slightly different
for different distributions. Following are specific examples for common
distributions. If you have a different Linux distribution, the
configuration files' location and syntax should still be similar (if
nor identical) to one of the examples shown below. You should find the
directory that includes the adapters' configuration files in your
distribution, and follow the syntax of the other configuration files
there.
Red Hat:
- The files reside in /etc/sysconfig/network-scripts/.
- Sample file content (for file ifcfg-<vadapter_name>):
DEVICE="<vadapter_name>"
BOOTPROTO="none"
ONBOOT="yes"
IPADDR="<IP_address>"
NETMASK="<netmask>"
SuSE:
- The files reside in /etc/sysconfig/network/.
- Sample file content (for file ifcfg-<vadapter_name>):
STARTMODE="onboot"
IPADDR="<IP_address>"
NETMASK="<netmask>"
Structure of the ANS Build Tree
===============================
- Kernel dependant source files (src/*.[ch], src/lib/*.[ch])
- Kernel independent precompiled object file (bin/[ARCH]/ians_core.o)
- Makefiles
- ANS configuration utility (bin/[ARCH]/ianscfg)
- Configuration scripts (bin/iansonboot, bin/ianstool)
NOTE: [ARCH] is either "ia32" or "ia64".
Distribution of files
---------------------
By default, the components are distributed as follows:
- /lib/modules/<kernel_version>/kernel/drivers/net/ians.o
- /usr/sbin/ianscfg
- /usr/sbin/ianstool
- /usr/sbin/iansonboot
- /usr/share/man/man1/ianscfg.1.gz
Examples
========
modules.conf
------------
#alias eth0 eepro100
#alias eth1 eepro100
#alias eth2 eepro100
alias eth0 e100
alias eth1 e100
alias eth2 e100
alias parport_lowlevel parport_pc
alias scsi_hostadapter aic7xxx
Script for Setting up AFT Mode with VLANs: (two PRO/100 adapters)
-----------------------------------------------------------------------
insmod e100
insmod ians
ianscfg -a -t team1 -M AFT -V
ianscfg -at team1 -m eth0 -p primary
ianscfg -at team1 -m eth1 -p secondary
ianscfg -at team1 -v vadapt1 -i 10
ianscfg -at team1 -v vadapt2 -i 15
ianscfg -c team1
ianscfg -s
ifconfig vadapt1 192.168.1.1 netmask 255.255.255.0
ifconfig vadapt2 192.168.2.1 netmask 255.255.255.0
Troubleshooting
===============
NOTE: For distribution-specific information, see the ldistrib.txt file
included in the driver tar.
1. Always bind the protocol to the virtual adapter and NOT to the physical
adapter for adapters that are in a team. Also, when changing settings
such as the MTU or MAC address, always apply them to the virtual adapter.
Intel ANS cannot prevent such operations, and failing to do as
recommended might lead to unpredictable results.
2. If you receive the error message, "module not loaded," during
configuration of the team, this means that the target module does not
match the running kernel. Enter the iANS-x.x.x/src directory and run:
make veryclean
make install
3. If you have unsuccessfully attempted to add an adapter to a team with
Jumbo Frames, check the adapter's ability to operate with Jumbo Frames.
4. If you receive the error message, "Intel Advanced Network Services (iANS)
has determined you have selected network adapters that are not eligible
to participate as members of an iANS team. Please install Intel Server
Adapters in your system and re-install the advanced server features.",
this indicates that either:
a. the team needs at least one Intel server adapter, or
b. at least one adapter is from another vendor and is not capable of
participating in the team.
5. When trying to compile the driver by running make install, the following
error may occur:
"Linux kernel source not configured - missing version.h"
To solve this issue, create the version.h file by going to the Linux
source tree and doing a 'make include/linux/version.h'.
6. PCI Hot Plug Add/Remove.
- When executing a PCI Hot Plug Remove operation on an adapter that is
part of an ANS team, the adapter is automatically removed from the team.
- When reactivating an adapter with a PCI Hot Plug Add operation, the new
interface should be manually added back into the team by an ANS Hot Add
operation.
7. If you are running an AFT/SFT/ALB team and receive the following messages
numerous times, "Adapter ethx does not sense any probes. Possible reason:
Adapter deactivated, Adapter isolated, Partitioned team." and then,
"Adapter ethx rejoined.", probe packets are being dropped (by the kernel
queues, switch, etc.) resulting in the ANS team losing probes. This could
happen in a healthy system that is under heavy stress.
As a workaround, disable the probes by issuing the following command:
ianscfg -P -t <team_name> -d
8. Different switches react differently when the switch manager disables a
switch port. Some switches react with a link loss of the disabled port.
Others do not change the link status at all; they simply stop traffic.
If the link status is not affected when a port is disabled then ANS
cannot always detect that a switch port that is connected to an ANS
member has been disabled by the switch manager. ANS detects that the port
has been disabled even though no link status change occurs only if one of
two conditions exist:
- Probes are enabled and there are more than two enabled and active
members in the team (relevant to AFT, SFT, and ALB teaming modes only).
ANS detects that the member attached to the disabled port is missing
probes while others are receiving and transmitting probes successfully.
- Teaming mode is 802.3ad - ANS deactivates the member attached to the
disabled port since no LACPDUs will be received by ANS on the disabled
port.
To avoid problems of fault tolerance in FEC, GEC, and ALB teaming modes
the switch manager should physically disconnect the link on the disabled
port or remove the member connected to the disabled port from the team.
In AFT and SFT modes it is enough to set an active member that is attached
to an enabled port to be the preferred primary and make sure that the
priority of the member attached to the disabled port is "none".
9. When setting a new MAC address for an AFT/ALB team (for the virtual
adapter), if the new MAC address is already in use by a member of the
team and that MAC address was not chosen as the team's MAC address upon
committing the team, then traffic might become unstable. Intel ANS does
not automatically check for duplicate MAC addresses. Therefore, before
setting a new MAC address in this type of scenario, ensure that the
intended MAC address is not the same as those used for members of the
team.
10. If you load the ipv6 module when there are already active ANS teams, then
the ipv6 module will automatically add the default IPv6 link address to all
network interfaces, including ANS’s member interfaces, and ANS’s virtual
interfaces. This might cause duplicate entries in the IPv6 routing table,
which might affect the way the kernel chooses the correct interface to send
IPv6 traffic through. To fix this, run:
ifconfig <ethX>
where <ethX> is an ANS team member interface. For every IPv6 address that is
listed there, in the form:
inet6 addr: <addr>/<prefix len> Scope:<scope>
remove it by running:
ifconfig <ethX> del <addr>/<prefix len>
If the ipv6 module was already loaded in your system and you’ve manually
closed and then opened member interfaces of an ANS team, the same problem
will occur. Fix it as listed above for each member you’ve manually changed.
Support
=======
For general information, go to the Intel support website at:
http://support.intel.com
If an issue is identified with the released source code on the supported
kernel with a supported adapter, email the specific information related to
the issue to linux.nics@intel.com.
License
=======
This software program is released under the terms of a license agreement
between you ('Licensee') and Intel. Do not use or load this software or any
associated materials (collectively, the 'Software') until you have carefully
read the full terms and conditions of the licenses located in this software
package. By loading or using the Software, you agree to the terms of this
Agreement. If you do not agree with the terms of this Agreement, do not
install or use the Software.
* Other names and brands may be claimed as the property of others.
Download Driver Pack
After your driver has been downloaded, follow these simple steps to install it.
Expand the archive file (if the download file is in zip or rar format).
If the expanded file has an .exe extension, double click it and follow the installation instructions.
Otherwise, open Device Manager by right-clicking the Start menu and selecting Device Manager.
Find the device and model you want to update in the device list.
Double-click on it to open the Properties dialog box.
From the Properties dialog box, select the Driver tab.
Click the Update Driver button, then follow the instructions.
Very important: You must reboot your system to ensure that any driver updates have taken effect.
For more help, visit our Driver Support section for step-by-step videos on how to install drivers for every file type.