[Playground] Azure CNI initialization log/source code analysis

Azure CNI in one of the CNI plugins AKS provides. Below example is based on an AKS cluster. The k8s version is 1.18.10 and CNI version is v1.2.0_hotfix.

What?

  • network

The info of virtual network CNI creates

  • endpoint

The info of pod’s networking CNI creates

  • netPlugin

The parent plugin created when Azure-CNI is invoked

  • CNI Plugin

The plugin used to accept/parse configuration

  • NetworkManager

The plugin used to create the network/endpoint

How?

  • Entry Point
  1. Link
  2. Detail
  • Initialize the report manager for telemetry
  1. Link
  2. Detail
  • Create new pointer of netPlugin object including the CNI plugin and NetworkManager plugin. The ‘name’ is set to ‘azure-vnet’ and ‘config’ is the ‘PluginConfig’ in the package ‘common’ like below.
  1. Link
  2. Detail
// Plugin common configuration.type PluginConfig struct {    Version  'v1.2.0_hotfix'    NetApi   NetApi    IpamApi  IpamApi    Listener *Listener    ErrChan  chan error    Store    store.KeyValueStore
}

It returns the pointer of below struct when the creation completes.

type netPlugin struct {    Plugin *cni.Plugin     nm network.NetworkManager}

The struct cni.Plugin’ is below

// Plugin is the parent class for CNI plugins.type Plugin struct {    *common.Plugin    version 'v1.2.0_hotfix'}// common.Plugin is created like below// Plugin is the parent class that implements behavior common to all plugins.
type Plugin struct {
Name 'azure-vnet'
Version 'v1.2.0_hotfix'
Options map[string]interface{}
}

The struct network.NetworkManager’ is below

type networkManager struct {
ExternalInterfaces map[string]*externalInterface
}

The struct ‘externalInterface’ is ‘a host network interface that bridges containers to external networks’

  • CNI plugin initializes the persistent storage
  1. Link
  2. Detail

The function creates json file ‘/var/run/azure-vnet.json’ and also set the struct config

type PluginConfig struct {    Version  'v1.2.0_hotfix'    NetApi   NetApi    IpamApi  IpamApi    Listener *Listener    ErrChan  chan error    Store    jsonFileStore{
fileName: '/var/run/azure-vnet.json',
data: make(map[string]*json.RawMessage),
}
}
  • netPlugin starts with config built previously
  1. Link
  2. Detail

During this step, firstly it invoke the method of anoymous field ‘Plugin to initialize the CNI at first. Following the code it invokes the ‘initialize’ in the pkg common’ eventually

type Plugin struct {
Name 'azure-vnet'
Version 'v1.2.0_hotfix'
Options map[string]interface{}
ErrChan chan error
Store jsonFileStore{
fileName: '/var/run/azure-vnet.json',
data: make(map[string]*json.RawMessage),
}
}

If the initialization finishes successfully, it will print log like below.

After CNI plugin initialization, the ‘networkMangaer’ created previously will be initialized as well.

It eventually invokes the function below

type networkManager struct {    Version            'v1.2.0_hotfix'    ExternalInterfaces map[string]*externalInterface    Store    jsonFileStore{
fileName: '/var/run/azure-vnet.json',
data: make(map[string]*json.RawMessage),
}
sync.Mutex}

After the two fields of the networkManger are set, the networkManager will check if needs to restore the persisted state from persistent storage created previously. In linux OS, the variable isRehydrationRequired’ is set to false.

Since this is still during initialization, the function will print out below log and return nil.

After the networkManager initialization finishes successfully, the netPlugin prints the log to indicate the plugin starts succesfully.

  • Once the netPlugin starts, the program gets the CNI command from the OS environment variable. For example, during the initialization, it would be ‘ADD’
  1. Link
  2. Detail
  • Since the command is not ‘UPDATE’, the netPlugin invokes ‘execute’ function.
  1. Link
  2. Detail
  • The input from kubelet is obtained via the function

Eventually it builds a struct CmdArgsbelow.

CmdArgs{
ContainerID: '9999',
Netns: '/run/netns/configureazcni',
IfName: 'eth9999',
Args: 'K8S_POD_NAMESPACE=default;K8S_POD_NAME=configureAzureCNI',
Path: '/opt/cni/bin',
StdinData: '{
"type": "azure-vnet",
"mode": "transparent",
"ipsToRouteViaHost": [
"169.254.20.10"
],
"ipam": {
"type": "azure-vnet-ipam"
},
"name": "azure",
"cniVersion": "0.3.0"
}',
}
  • If the command is ‘ADD’, the ‘Add’ method of the ‘netPlugin’ created previously will be invoked.
  1. Link
  2. Detail
  • Print out the input configuration from ‘CmdArgs’ built previously
  1. Link
  2. Detail
  • Parse and print out the network configuration from Stdin.
  1. Link
  2. Detail

The stdin data is parsed according to the struct . Meanwhile the configuration is set as the value of variablenwCfg

{
CNIVersion:0.3.0
Name:azure
Type:azure-vnet
Mode:transparent
Master:
Bridge:
LogLevel:
LogTarget:
InfraVnetAddressSpace:
IPV6Mode:
ServiceCidrs:
VnetCidrs:
PodNamespaceForDualNetwork:[]
IPsToRouteViaHost:[169.254.20.10]
MultiTenancy:false
EnableSnatOnHost:false
EnableExactMatchForPodName:false
DisableHairpinOnHostInterface:false
DisableIPTableLock:false
CNSUrl:
Ipam:{
Type:azure-vnet-ipam
Environment:
AddrSpace:
Subnet:
Address:
QueryInterval:
}
DNS:{
Nameservers:[]
Domain:
Search:[]
Options:[]
}
RuntimeConfig:{
PortMappings:[]
DNS:{
Servers:[]
Searches:[]
Options:[]
}
}
AdditionalArgs:[]
}
  • Several variables are set according to the input from kubelet
  1. Link
  2. Detail
k8sPodName = "configureAzureCNI"
k8sNamespace = "default"
k8sContainerID = "9999"
k8sIfName = "eth9999"
networkId = "azure"
endpointId = k8sContainerID + "-" + k8sIfName = "9999-eth9999"
result = nil
cnsNetworkConfig = nil
subnetPrefix = net.IPNet{}
azIpamResult = nil
err = nil
  • It checks if the network exists via checking the target network detail
  1. Link
  2. Detail

Eventually it invokes the function. Since this is during initialization, it will return the error ‘errNetworkNotFound’ which is the value of ‘nwInfoErr’ as well

  • The ‘IpamInvoker ‘is created
  1. Link
  2. Detail

It returns below struct eventually

AzureIPAMInvoker{    plugin: netPlugin,    nwInfo: nil,}
  • As the ‘nwInfoErr’ is not empty, it starts to create the network.
  1. Link
  2. Detail

During this, firstly it calls the IPAM in the ‘nwCfg’ to allocate address pool at first

  • Some variables are set after the IPAM returns the result
  1. Link
  2. Detail
gateway = '10.240.0.1'
subnetPrefix.IP = '10.240.0.11'
masterIfName = 'eth0'
  1. Link
  2. Detail

During this, the struct externalInterface’ is added to the ‘networkManager’ and it’s saved with key set to ‘Network’ to the json file created previously

externalInterface{
Name: 'eth0',
Networks: make(map[string]*network),
MacAddress: hostIf.HardwareAddr,
IPv4Gateway: net.IPv4zero,
IPv6Gateway: net.IPv6unspecified,
Subnets: ['10.240.0/16']
}
  • Add DNS info from IPAM result since in the ‘nwCfg’ there is no DNS configuration
  1. Link
  2. Detail
  • Skip the ‘updateSubnetPrefix’ since it’s single tenant
  1. Link
  2. Detail
  • The struct “nwInfo” is configured and some info in it is used by ‘networkManager’ to create the network “azure”
  1. Link
  2. Detail
network.NetworkInfo{
Id: 'azure',
Mode: 'transparent',
MasterIfName: 'eth0',
Subnets: []network.SubnetInfo{
{
Family: '10',
Prefix: '10.240.0.0',
Gateway: '10.240.0.1',
},
},
BridgeName: '',
EnableSnatOnHost: 'false',
DNS: network.DNSInfo{
Servers: '168.63.129.16',
Suffix: '',
Options: []
},
Policies: [],
NetNs: '/run/netns/configureazcni',
DisableHairpinOnHostInterface: 'false',
IPV6Mode: '',
ServiceCidrs: [],
IPAMTYPE: 'azure-vnet-ipam',
PodSubnet: network.SubnetInfo{
Family: '10',
Prefix: net.IPNet{
IP: '10.240.0.0',
Mask: 'ffff00000'
},
Gateway: '10.240.0.1'
},
Options: map['DeleteOnErr':'Error']
}
  • When ‘networkManager’ creates the network, firstly a pointer of the struct ‘externalInterface’ is created and be assigned the same value as the field in the ‘networkManager’ itself
  1. Link
  2. Detail
&externalInterface{
Name: 'eth0',
Networks: map[string]*network,
MacAddress: <eg. AA06xjUE>,
IPv4Gateway: net.IPv4zero,
IPv6Gateway: net.IPv6unspecified,
Subnets: ['10.240.0/16']
}
  • Then it calls the OS-specfic network implementation and update the network field in ‘extIf’
  1. Link
  2. Detail

During this, the pointer of struct ‘nw’ is created.

network{
Id: 'azure',
Mode: 'transparent',
Endpoints: map[string]*endpoint,
extIf: &externalInterface{
Name: 'eth0',
Networks: make(map[string]*network),
MacAddress: hostIf.HardwareAddr,
IPv4Gateway: net.IPv4zero,
IPv6Gateway: net.IPv6unspecified,
Subnets: ['10.240.0/16']
},
VlanId: '',
DNS: network.DNSInfo{
Servers: '168.63.129.16',
Suffix: '',
},
EnableSnatOnHost: 'false',
Subnets: []network.SubnetInfo{
{
Family: '10',
Prefix: '10.240.0.0',
Gateway: '10.240.0.1',
},
},
}
network.EndpointInfo{
Id: endpointId,
ContainerID: '9999',
NetNsPath: '/run/netns/configureazcni',
IfName: 'eth9999',
Data: map['vethname':'default.configureAzureCNI'],
DNS: network.DNSInfo{
Servers: '168.63.129.16',
Suffix: ,
Options:[]
},
Policies: ,
IPsToRouteViaHost: [169.254.20.10],
EnableSnatOnHost: 'false',
EnableMultiTenancy: 'false',
EnableInfraVnet: 'false',
EnableSnatForDns: 'false',
PODName: 'configureAzureCNI',
PODNameSpace: 'default',
SkipHotAttachEp: 'false', // Hot attach at the time of endpoint creation
IPV6Mode: ,
VnetCidrs: ,
ServiceCidrs: ,
Routes: []network.RouteInfo{
Dst: net.IPNet{
IP: '0.0.0.0',
Mask: '00000000'
},
Gw: '10.240.0.1',
Src:nil
}
IPAddresses: []net.IPNet{
IP: '10.240.0.11',
Mask: 'ffff0000'
}
}
  • Set the host network interface and container network interface name
  1. link
  2. Detail

hostIfName = ‘azv’ + generateVethName(key) eg. ‘azvd4c6c6c9b19’

contIfName = ‘azv’ + generateVethName(key) eg. ‘azvd4c6c6c9b192’

  • Create a point of ‘transparentEndpointClient
  1. link
  2. Detail
&TransparentEndpointClient{
bridgeName: ,
hostPrimaryIfName: 'eth0',
hostVethName: <eg. azvd4c6c6c9b19>,
containerVethName: <eg. azvd4c6c6c9b192>,
hostPrimaryMac: <eg. AA06xjUE>,
mode: 'transparent',
}
  • The method of transparent client is invoked to create the endpoint
  1. Link
  2. Detail
  • Invoke method in the package ‘epcommon’ to create the endpoint which creates the veth pair of hostNetworkInterface and containerNetworkInterface like ‘ip link add’
  1. link
  2. Detail
  • The MAC address of the host and container veth network interface are updated in the endpoint
  1. link
  2. Detail
  • Create routing rules like ‘ip route add <pod ip> veth <veth>’ to enable the request can be routed to pod via host veth
  1. link
  2. Detail
  • Enter the pod network namespace and set the link of the container veth and the network namespace
  1. link
  2. Detail
  • Since the interface name in the ‘epInfo’ is added, set the link of container veth and the network interface in the ‘epInfo
  1. link
  2. Detail
  • Configure the container veth like IP address assign and routes/ARP configure
  1. Link
  2. Detail
  • Update the network info in the ‘nw’ pointer
  1. Link
  2. Detail
  • ‘networkManager’ save the current configuration to persistent storage
  1. Link
  2. Detail
  • If no error, send CNI report and execute defer function to print out the result
  1. Link
  2. Detail

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store