Closed
Description
BUG REPORT
Versions
kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} - Cloud provider or hardware configuration:
Brightbox - OS (e.g. from /etc/os-release):
Ubuntu 18.04 LTS - Kernel (e.g.
uname -a):
Linux srv-d35vu 4.15.0-29-generic The product_uuid and the hostname should be unique across nodes #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux - Others:
What happened?
kube-proxy is scheduled on new worker nodes before the cloud-controller has initialised the node addresses. This causes kube-proxy to fail to pick up the node's IP address properly - with knock on effects to the proxy function managing load balancers.
ubuntu@srv-d35vu:~$ kubectl -n kube-system logs kube-proxy-xvd8m
I0727 14:10:07.854300 1 server_others.go:183] Using ipvs Proxier.
W0727 14:10:07.878777 1 server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: []
W0727 14:10:07.879942 1 proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I0727 14:10:07.880198 1 server_others.go:210] Tearing down inactive rules.
I0727 14:10:07.917875 1 server.go:448] Version: v1.11.1
I0727 14:10:07.931360 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0727 14:10:07.931503 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0727 14:10:07.934345 1 conntrack.go:83] Setting conntrack hashsize to 32768
I0727 14:10:07.945352 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0727 14:10:07.946073 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0727 14:10:07.946358 1 config.go:202] Starting service config controller
I0727 14:10:07.946479 1 controller_utils.go:1025] Waiting for caches to sync for service config controller
I0727 14:10:07.947422 1 config.go:102] Starting endpoints config controller
I0727 14:10:07.947586 1 controller_utils.go:1025] Waiting for caches to sync for endpoints config controller
I0727 14:10:08.046848 1 controller_utils.go:1032] Caches are synced for service config controller
I0727 14:10:08.047791 1 controller_utils.go:1032] Caches are synced for endpoints config controller
What you expected to happen?
kube-proxy should probably block on the uninitialised taint via the config (or respond to the event of the address update if that is possible).
Not sure if kubeadm uses its own ds specification or just picks up an upstream one.
How to reproduce it (as minimally and precisely as possible)?
Run a kubeadm init with 'cloud-provider: external' set and join a worker to the cluster. Kube-Proxy will schedule and run on all nodes even with the uninitialised taints in place.
Anything else we need to know?
Deleting the pod and causing a reload picks up the node ip on the worker.
ubuntu@srv-d35vu:~$ kubectl -n kube-system delete pods kube-proxy-xvd8m
pod "kube-proxy-xvd8m" deleted
ubuntu@srv-d35vu:~$ kubectl -n kube-system logs kube-proxy-
kube-proxy-b2k9z kube-proxy-l2r6m
ubuntu@srv-d35vu:~$ kubectl -n kube-system logs kube-proxy-l2r6m
I0727 14:11:34.441117 1 server_others.go:183] Using ipvs Proxier.
I0727 14:11:34.456983 1 server_others.go:210] Tearing down inactive rules.
E0727 14:11:34.495629 1 proxier.go:423] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I0727 14:11:34.500124 1 server.go:448] Version: v1.11.1
I0727 14:11:34.512962 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0727 14:11:34.513311 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0727 14:11:34.513647 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0727 14:11:34.514939 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0727 14:11:34.515210 1 config.go:102] Starting endpoints config controller
I0727 14:11:34.515325 1 controller_utils.go:1025] Waiting for caches to sync for endpoints config controller
I0727 14:11:34.515507 1 config.go:202] Starting service config controller
I0727 14:11:34.515605 1 controller_utils.go:1025] Waiting for caches to sync for service config controller
I0727 14:11:34.615648 1 controller_utils.go:1032] Caches are synced for endpoints config controller
I0727 14:11:34.615840 1 controller_utils.go:1032] Caches are synced for service config controller
kubeadm.conf is
apiVersion: kubeadm.k8s.io/v1alpha2
apiServerExtraArgs:
cloud-provider: external
controllerManagerExtraArgs:
cloud-provider: external
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
kind: MasterConfiguration
clusterName: kubernetes
api:
advertiseAddress: 0.0.0.0
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 172.30.0.0/16
bootstrapTokens:
- token: n14rmi.zutbuixp6uzbmcop
kubeProxy:
config:
mode: ipvs
ipvs:
scheduler: lc
Metadata
Metadata
Labels
Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.Categorizes issue or PR as related to documentation.Categorizes issue or PR as related to a new feature.Denotes an issue or PR that has aged beyond stale and will be auto-closed.Important over the long term, but may not be staffed and/or may need multiple releases to complete.Categorizes an issue or PR as relevant to SIG Scheduling.