Popeye — A Kubernetes Cluster Sanitizer

5 min readNov 28, 2021

Kubernetes and cloud native technologies have continued to gain momentum. As per the latest CNCF survey report, Kubernetes use in production has increased to 83%, up from 78% last year. Use of containers in production is the norm. Kubernetes simplifies the work of developers and operators, increasing agility and accelerating software delivery. While Kubernetes has been popular with developers for a number of years, it’s now moving steadily into production environments and well on its way to entering the IT mainstream.

The Rising Pain for Enterprise Businesses

As enterprises accelerate digital transformation and embrace the Kubernetes ecosystem, some of the enterprise businesses are experiencing growing pains due to a lack of expertise, complex deployments and challenges in integrating new and existing systems and deployment. In the latest State of Kubernetes 2021 survey conducted by VMware, almost 96% survey respondents reported difficulties selecting a Kubernetes distribution. Lack of internal experience and expertise remains the biggest challenge when making the choice (55%), but it has dropped 14% since last year, suggesting rapid improvement. Other notable challenges included: hard to hire needed expertise (37%), Kubernetes/cloud native speed of change (32%), and too many solutions to choose from (30%). Most of these challenges are likely to take care of themselves as more people gain familiarity and the ecosystem continues to mature.

In this blog, we will target the major pain of choosing the right tool for Kubernetes. We picked up the most popular and effective tools based on reviews, votes, social media comments and listed them below:

Introducing Popeye

Popeye is a utility that scans live Kubernetes clusters and reports potential issues with deployed resources and configurations. It sanitizes your cluster based on what’s deployed and not what’s sitting on disk. By scanning your cluster, it detects misconfigurations and helps you to ensure that best practices are in place, thus preventing future headaches. It aims at reducing the cognitive overload one faces when operating a Kubernetes cluster in the wild. Furthermore, if your cluster employs a metric-server, it reports potential resources over/under allocations and attempts to warn you should your cluster run out of capacity.

How is Popeye different from other existing tools?

Popeye is a readonly tool, it does not alter any of your Kubernetes resources in any way! Popeye scans your cluster for best practices and potential issues. Currently, Popeye only looks at nodes, namespaces, pods and services. More will come soon! We are hoping Kubernetes friends will pitch in to make Popeye even better. The aim of the sanitizers is to pick up on misconfigurations, i.e. things like port mismatches, dead or unused resources, metrics utilization, probes, container images, RBAC rules, naked resources, etc…

Installation

Popeye works best with Kubernetes 1.13+. You can containerize Popeye and run directly in your Kubernetes clusters as a one-off or CronJob. It is available on Linux, OSX and Windows platforms. Binaries for Linux, Windows and Mac are available as tarballs in the release page.

OSX

$ brew install derailed/popeye/popeye

Linux or Windows

Building from source Popeye was built using go 1.12+. In order to build Popeye from source you must:

Clone the repository
Add the following command in your go.mod file

replace (
  github.com/derailed/popeye => MY_POPEYE_CLONED_GIT_REPO
)

Build and run the executable

go run main.go

Cloning outside of GOPATH

git clone https://github.com/derailed/popeye
cd popeye
# Build and install
go install
# Run
popeye

Checking the version:

$ popeye version
 ___     ___ _____   _____                       K          .-'-.     
| _ \___| _ \ __\ \ / / __|                       8     __|      `\  
|  _/ _ \  _/ _| \ V /| _|                         s   `-,-`--._   `\
|_| \___/_| |___| |_| |___|                       []  .->'  a     `|-'
  Biffs`em and Buffs`em!                            `=/ (__/_       /  
                                                      \_,    `    _)  
                                                         `----;  |     
Version:   0.9.7
Commit:    4f12a172495e2acb7a621b29cffa924f1cd72580
Date:      2021-07-20T14:57:08Z
Logs:      /var/folders/7k/2jz4csrs4ss65_x0slwbl1540000gn/T/popeye.log

Popeye a cluster

$ popeyeGENERAL [DOCKER-DESKTOP]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Connectivity...................................................✅
  · MetricServer...................................................💥
CLUSTER (1 SCANNED)                            💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Version.........................................................✅
    ✅ [POP-406] K8s version OK.
CLUSTERROLES (60 SCANNE.                     💥 0 😱 0 🔊 15 ✅ 45 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · admin...........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · cluster-admin............................................... ....✅
  · edit.............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · kubeadm:getnodes.................................................✅
  · system:aggregate-to-admin.......................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:aggregate-to-edit........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:aggregate-to-view...............................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:auth-delegator..................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:basic-user.......................................✅

If you have just enabled Kubernetes under Docker Desktop with no Pods in operation, then you might end with the below score:

In order for Popeye to do his work, the signed-in user must have enough RBAC oomph to get/list the resources mentioned above.

Sample Popeye RBAC Rules:

---
# Popeye ServiceAccount.
apiVersion: v1
kind:       ServiceAccount
metadata:
  name:      popeye
  namespace: popeye---
# Popeye needs get/list access on the following Kubernetes resources.
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRole
metadata:
  name: popeye
rules:
- apiGroups: [""]
  resources:
   - configmaps
   - deployments
   - endpoints
   - horizontalpodautoscalers
   - namespaces
   - nodes
   - persistentvolumes
   - persistentvolumeclaims
   - pods
   - secrets
   - serviceaccounts
   - services
   - statefulsets
  verbs:     ["get", "list"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources:
  - clusterroles
  - clusterrolebindings
  - roles
  - rolebindings
  verbs:     ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources:
  - pods
  - nodes
  verbs:     ["get", "list"]---
# Binds Popeye to this ClusterRole.
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRoleBinding
metadata:
  name: popeye
subjects:
- kind:     ServiceAccount
  name:     popeye
  namespace: popeye
roleRef:
  kind:     ClusterRole
  name:     popeye
  apiGroup: rbac.authorization.k8s.io

Learn more about Popeye through this GITHUB link