[Playground] Azure CNI initialization log/source code analysis

9 min readJan 11, 2021

Azure CNI in one of the CNI plugins AKS provides. Below example is based on an AKS cluster. The k8s version is 1.18.10 and CNI version is v1.2.0_hotfix.

What?

network

The info of virtual network CNI creates

endpoint

The info of pod’s networking CNI creates

netPlugin

The parent plugin created when Azure-CNI is invoked

CNI Plugin

The plugin used to accept/parse configuration

NetworkManager

The plugin used to create the network/endpoint

How?

Entry Point

Link
Detail

Initialize the report manager for telemetry

Link
Detail

Create new pointer of netPlugin object including the CNI plugin and NetworkManager plugin. The ‘name’ is set to ‘azure-vnet’ and ‘config’ is the ‘PluginConfig’ in the package ‘common’ like below.

Link
Detail

// Plugin common configuration.type PluginConfig struct {    Version  'v1.2.0_hotfix'    NetApi   NetApi    IpamApi  IpamApi    Listener *Listener    ErrChan  chan error    Store    store.KeyValueStore
}

It returns the pointer of below struct when the creation completes.

type netPlugin struct {    Plugin *cni.Plugin     nm network.NetworkManager}

The struct ‘cni.Plugin’ is below

// Plugin is the parent class for CNI plugins.type Plugin struct {    *common.Plugin    version 'v1.2.0_hotfix'}// common.Plugin is created like below// Plugin is the parent class that implements behavior common to all plugins.
type Plugin struct {
    Name    'azure-vnet'
    Version 'v1.2.0_hotfix'
    Options map[string]interface{}
}

The struct ‘network.NetworkManager’ is below

type networkManager struct {
    ExternalInterfaces map[string]*externalInterface
}

The struct ‘externalInterface’ is ‘a host network interface that bridges containers to external networks’

CNI plugin initializes the persistent storage

Link
Detail

The function creates json file ‘/var/run/azure-vnet.json’ and also set the struct ‘config’

type PluginConfig struct {    Version  'v1.2.0_hotfix'    NetApi   NetApi    IpamApi  IpamApi    Listener *Listener    ErrChan  chan error    Store    jsonFileStore{
                 fileName: '/var/run/azure-vnet.json',
                 data:     make(map[string]*json.RawMessage),
             }
}

netPlugin starts with config built previously

Link
Detail

During this step, firstly it invoke the method of anoymous field ‘Plugin’ to initialize the CNI at first. Following the code it invokes the ‘initialize’ in the pkg ‘common’ eventually

type Plugin struct {
    Name    'azure-vnet'
    Version 'v1.2.0_hotfix'
    Options map[string]interface{}
    ErrChan chan error
    Store   jsonFileStore{
                 fileName: '/var/run/azure-vnet.json',
                 data:     make(map[string]*json.RawMessage),
             }
}

If the initialization finishes successfully, it will print log like below.

After CNI plugin initialization, the ‘networkMangaer’ created previously will be initialized as well.

It eventually invokes the function below

type networkManager struct {    Version            'v1.2.0_hotfix'    ExternalInterfaces map[string]*externalInterface    Store    jsonFileStore{
                 fileName: '/var/run/azure-vnet.json',
                 data:     make(map[string]*json.RawMessage),
             }    sync.Mutex}

After the two fields of the networkManger are set, the networkManager will check if needs to restore the persisted state from persistent storage created previously. In linux OS, the variable ‘isRehydrationRequired’ is set to false.

Since this is still during initialization, the function will print out below log and return nil.

After the networkManager initialization finishes successfully, the netPlugin prints the log to indicate the plugin starts succesfully.

Once the netPlugin starts, the program gets the CNI command from the OS environment variable. For example, during the initialization, it would be ‘ADD’

Link
Detail

Since the command is not ‘UPDATE’, the netPlugin invokes ‘execute’ function.

Link
Detail

The input from kubelet is obtained via the function

Eventually it builds a struct ‘CmdArgs’ below.

CmdArgs{
    ContainerID: '9999',
    Netns:       '/run/netns/configureazcni',
    IfName:      'eth9999',
    Args:       'K8S_POD_NAMESPACE=default;K8S_POD_NAME=configureAzureCNI',
    Path:        '/opt/cni/bin',
    StdinData:   '{
                      "type": "azure-vnet",
                      "mode": "transparent",
                      "ipsToRouteViaHost": [
                           "169.254.20.10"
                      ],
                      "ipam": {
                           "type": "azure-vnet-ipam"
                      },
                      "name": "azure",
                      "cniVersion": "0.3.0"
                  }',
 }

If the command is ‘ADD’, the ‘Add’ method of the ‘netPlugin’ created previously will be invoked.

Link
Detail

Print out the input configuration from ‘CmdArgs’ built previously

Link
Detail

Parse and print out the network configuration from Stdin.

Link
Detail

The stdin data is parsed according to the struct . Meanwhile the configuration is set as the value of variable “nwCfg”

{
    CNIVersion:0.3.0 
    Name:azure 
    Type:azure-vnet 
    Mode:transparent 
    Master: 
    Bridge: 
    LogLevel: 
    LogTarget: 
    InfraVnetAddressSpace: 
    IPV6Mode: 
    ServiceCidrs: 
    VnetCidrs: 
    PodNamespaceForDualNetwork:[] 
    IPsToRouteViaHost:[169.254.20.10] 
    MultiTenancy:false 
    EnableSnatOnHost:false 
    EnableExactMatchForPodName:false 
    DisableHairpinOnHostInterface:false 
    DisableIPTableLock:false 
    CNSUrl: 
    Ipam:{
      Type:azure-vnet-ipam 
      Environment: 
      AddrSpace: 
      Subnet: 
      Address: 
      QueryInterval:
   } 
   DNS:{
     Nameservers:[] 
     Domain: 
     Search:[] 
     Options:[]
   } 
   RuntimeConfig:{
     PortMappings:[] 
     DNS:{
       Servers:[] 
       Searches:[] 
       Options:[]
     }
   } 
   AdditionalArgs:[]
}

Several variables are set according to the input from kubelet

Link
Detail

k8sPodName = "configureAzureCNI"
k8sNamespace = "default"
k8sContainerID = "9999"
k8sIfName = "eth9999"
networkId = "azure"
endpointId = k8sContainerID + "-" + k8sIfName = "9999-eth9999"
result = nil
cnsNetworkConfig = nil
subnetPrefix = net.IPNet{}
azIpamResult = nil
err = nil

It checks if the network exists via checking the target network detail

Link
Detail

Eventually it invokes the function. Since this is during initialization, it will return the error ‘errNetworkNotFound’ which is the value of ‘nwInfoErr’ as well

The ‘IpamInvoker ‘is created

Link
Detail

It returns below struct eventually

AzureIPAMInvoker{    plugin: netPlugin,    nwInfo: nil,}

As the ‘nwInfoErr’ is not empty, it starts to create the network.

Link
Detail

During this, firstly it calls the IPAM in the ‘nwCfg’ to allocate address pool at first

Some variables are set after the IPAM returns the result

Link
Detail

gateway = '10.240.0.1'
subnetPrefix.IP = '10.240.0.11'
masterIfName = 'eth0'

The map of master interface and subnet cidr are added

Link
Detail

During this, the struct ‘externalInterface’ is added to the ‘networkManager’ and it’s saved with key set to ‘Network’ to the json file created previously

externalInterface{
    Name:        'eth0',
    Networks:    make(map[string]*network),
    MacAddress:  hostIf.HardwareAddr,
    IPv4Gateway: net.IPv4zero,
    IPv6Gateway: net.IPv6unspecified, 
    Subnets:     ['10.240.0/16']
 }

Add DNS info from IPAM result since in the ‘nwCfg’ there is no DNS configuration

Link
Detail

Skip the ‘updateSubnetPrefix’ since it’s single tenant

Link
Detail

The struct “nwInfo” is configured and some info in it is used by ‘networkManager’ to create the network “azure”

Link
Detail

network.NetworkInfo{
   Id:           'azure',
   Mode:         'transparent',
   MasterIfName: 'eth0',
   Subnets: []network.SubnetInfo{
                  {
                      Family:  '10',
                      Prefix:  '10.240.0.0',
                      Gateway: '10.240.0.1',
                  },
            },
   BridgeName:                    '',
   EnableSnatOnHost:              'false',
   DNS:                           network.DNSInfo{
                                    Servers: '168.63.129.16',
                                    Suffix:  '',
                                    Options: []
                                  },
   Policies:                      [],
   NetNs:                         '/run/netns/configureazcni',
   DisableHairpinOnHostInterface: 'false',
   IPV6Mode:                      '',
   ServiceCidrs:                  [],
   IPAMTYPE:                      'azure-vnet-ipam',
   PodSubnet:                     network.SubnetInfo{
                                      Family:  '10',
                                      Prefix:  net.IPNet{
                                          IP: '10.240.0.0',
                                          Mask: 'ffff00000'
                                      },
                                      Gateway: '10.240.0.1'
                                  },
    Options:                      map['DeleteOnErr':'Error']}

When ‘networkManager’ creates the network, firstly a pointer of the struct ‘externalInterface’ is created and be assigned the same value as the field in the ‘networkManager’ itself

Link
Detail

&externalInterface{
   Name:        'eth0',
   Networks:    map[string]*network,
   MacAddress:  <eg. AA06xjUE>,
   IPv4Gateway: net.IPv4zero,
   IPv6Gateway: net.IPv6unspecified, 
   Subnets:     ['10.240.0/16']
 }

Then it calls the OS-specfic network implementation and update the network field in ‘extIf’

Link
Detail

During this, the pointer of struct ‘nw’ is created.

network{
  Id:               'azure',
  Mode:             'transparent',
  Endpoints:        map[string]*endpoint,
  extIf:            &externalInterface{
                       Name:        'eth0',
                       Networks:    make(map[string]*network),
                       MacAddress:  hostIf.HardwareAddr,
                       IPv4Gateway: net.IPv4zero,
                       IPv6Gateway: net.IPv6unspecified, 
                       Subnets:     ['10.240.0/16']
                     },
  VlanId:           '',
  DNS:              network.DNSInfo{
                        Servers: '168.63.129.16',
                        Suffix:  '',
                    },
  EnableSnatOnHost: 'false',
  Subnets: []network.SubnetInfo{
                  {
                      Family:  '10',
                      Prefix:  '10.240.0.0',
                      Gateway: '10.240.0.1',
                  },
            },
}

network.EndpointInfo{
  Id:                 endpointId,
  ContainerID:        '9999',
  NetNsPath:          '/run/netns/configureazcni',
  IfName:             'eth9999',
  Data:               map['vethname':'default.configureAzureCNI'],
  DNS:                network.DNSInfo{
                        Servers: '168.63.129.16',
                        Suffix:  , 
                        Options:[]
                      },
  Policies:           ,
  IPsToRouteViaHost:  [169.254.20.10],
  EnableSnatOnHost:   'false',
  EnableMultiTenancy: 'false',
  EnableInfraVnet:    'false',
  EnableSnatForDns:   'false',
  PODName:            'configureAzureCNI',
  PODNameSpace:       'default',
  SkipHotAttachEp:    'false', // Hot attach at the time of endpoint creation
  IPV6Mode:           ,
  VnetCidrs:          ,
  ServiceCidrs:       , 
  Routes:             []network.RouteInfo{
                        Dst: net.IPNet{
                               IP: '0.0.0.0',
                               Mask: '00000000'
                             },
                        Gw: '10.240.0.1', 
                        Src:nil
                       } 
  IPAddresses:         []net.IPNet{
                         IP: '10.240.0.11',
                         Mask: 'ffff0000'
                       } }

During the invoke, it get the network object created previously and then the object invokes the method to create the endpoint

Set the host network interface and container network interface name

link
Detail

hostIfName = ‘azv’ + generateVethName(key) eg. ‘azvd4c6c6c9b19’

contIfName = ‘azv’ + generateVethName(key) eg. ‘azvd4c6c6c9b192’

Create a point of ‘transparentEndpointClient’

link
Detail

&TransparentEndpointClient{
   bridgeName:        ,
   hostPrimaryIfName: 'eth0',
   hostVethName:      <eg. azvd4c6c6c9b19>,
   containerVethName: <eg. azvd4c6c6c9b192>,
   hostPrimaryMac:    <eg. AA06xjUE>,
   mode:              'transparent',
 }

The method of transparent client is invoked to create the endpoint

Link
Detail

Invoke method in the package ‘epcommon’ to create the endpoint which creates the veth pair of hostNetworkInterface and containerNetworkInterface like ‘ip link add’