v1.6.7 seems to have broken Kubernetes API connection

[hiiamelliott]

I have a daemonset of fluent-bit v1.6 pods running in three different (EKS) Kubernetes clusters.
For historical reasons, I have a cronjob to restart them every 6 hours, and after they restarted this morning, they could not contact https://kubernetes.default.svc:443 in any cluster.
After doing a bit of digging, I spotted that my daemonset was configured to

      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.6
        imagePullPolicy: Always

and that v1.6.7 was released a few hours ago.
Pinning the version to v1.6.6 resolved the issue, so there seems to be a bug in the new version.

The specific errors I was seeing in fluent-bit's logs were:

[2020/12/03 10:39:42] [error] [io] connection #42 failed to: kubernetes.default.svc:443
[2020/12/03 10:39:42] [error] [filter:kubernetes:kubernetes.0] upstream connection error

[shinebayar-g]

Fixed in 1.6.7 fluent/fluent-bit#2829

Unfortunately 1.6.8 also bugged fluent/fluent-bit#2830