How to troubleshoot KubeArmor policies in GKE

In this blog, we are going to see how to troubleshoot KubeArmor policies in GKE.

This blog assumes that you have read the concepts which explain all the components and fundamentals of KubeArmor. The blog will cover the following key steps:

• Creating policies
• Applying policies
• Testing policies
• Troubleshooting policies

Creating policies

1. To deploy KubeArmor in GKE apply this script. Github KubeArmor <<Deployments yaml>>
2. Another method is to deploy kubearmor in GKE follow this guide to deploy kubearmor in a very simple way.
3. Sample PostgreSQL policy is given below:

   
   Sample PostgreSQL policy

4. The above policy has been created for the PostgreSQL workload. Psql is the binary for PostgreSQL. If anyone tries to access the psql the policy will deny the access. Those binaries will be found easily in the bin folder inside the Linux directory.
5. Let’s apply the policy and see it in action!

Applying policies

1. When applying policies make sure to verify with YAML lint applications to verify the structure and indentation of the YAML files. If
   not verified, the policy will fail to apply.
   Note: The policy must follow the required policy structure of KubeArmor. View here
   

   KubeArmor structure

2. The above error shows that you may have missed the YAML indentation. As shown below.
   
3. The above error can simply be fixed by changing the indentation. Make sure to use appropriate YAML lint extensions in your code editor to avoid these kinds of errors.
   
4. Now we have fixed the policy. Let’s try to apply that policy and see it in action.

   [root@karmor]$ kubectl apply -f postgresql-policy.yaml
   kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql-console
   created

5. Now our policy has been created and applied in the GKE.
6. Check here for a complete guide and specification to write a kubearmor policy.

Testing policies

1. 1. When you are testing policies you have to check with KubeArmor Security policy specifications.
   2.  Make sure the wordings are spelled correctly. Otherwise, it throws an error like this shown below. These types of errors are likely to happen when we misspell words. So make sure to check in with the contribution guide mentioned above.
      

      Testing policy

1. The above error shows we misspelled the words “matchPath” instead of “matchPaths” and followed by “matchLabel” instead of “matchLabels” also “tag” instead of “tags”. Let’s fix those errors and test the policy again
2. This is the fixed policy with correct spellings.
   

   Kubearmor policy

   

   Code block

3. Now our policy is applied and ready.

   [root@kubearmor]$ kubectl apply -f postgres-policy.yaml
   kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql created

4. Let’s deploy PostgreSQL deployment in our cluster to test the policy.
5. To deploy a Postgres service in your cluster we have to apply this script
   

   Script-1

   

   Script-2

   

   Script-3

6. Now we have our working Postgres pod running in our k8s cluster.
   

   Postgres pod

7. Now let’s try to access the psql binary to test if it’s working or not. The goal is to block access to the psql binary.
   

   psql binary

8. As you can see in this above problem, psql binary is still accessible even after applying the policy. That policy is not working as we expected. Let’s troubleshoot where it went wrong and fix the policy.

Troubleshooting Policies

1. Even after using the correct format and parameters, the policy is not working and psql binary is still accessible. Why?
2. Let’s check the binary location with commands we know such as “whereis” and “which”. So we can figure out whether we added the correct path or not.
3. Here we checked psql binary location “whereis” command It shows that we added the correct path in our policy but still the policy is not working as we expected.
   
4. Let’s dive in deep and find if we have any other connections to the psql binary. If there is, we can add that to our policy.
   

   Kubearmor pod

5. Now when we check the psql with “ls -la” command. That psql binary is linked with “../share/postgresql-common/pg_wrapper”
6. So now we can see that it’s linked with another binary. So that’s why it’s not blocking the psql binary. We can add that in our policy and test that again.
7. Here’s the updated policy with that path included
   

   Kubearmor policy

8. Let’s apply this policy and check whether it is blocking the psql binary or not.
9. Now our policy is created without any errors.
   
   

   Policy template

10. Now we have our working policy that can block the psql binary in postgres
    pod. So we can authorize the use of the postgres server.

KubeArmor Logs

1. 1. To check kubearmor generated logs. We have to type this command to see that.
   2. kubectl -n kube-system get pods -A | grep kubearmor
   3. This command will list all kubearmor pods as shown below.
      

      Kubearmor pods

1. You have to select the first three pods that are named “kubearmor-1234” similarly. This name will differ in your environment. You have to enter the command shown below.
2. kubectl -n kube-system exec -it <kube-armor-pod> — tail /tmp/kubearmor.log
3. You will see the logs like this in the terminal as shown in the image below

   Kubearmor pod

4. We can format that as “pretty JSON” using a website like https://jsonformatter.org/.
5. Here is the generated command modified with JSON formatter.
   

   Policy template

6. The above logs will confirm that our policy is working as expected.

   psql binary

7. As you can see in this above problem, psql binary is still accessible even
   after applying the policy. That policy is not working as we expected. Let’s
   troubleshoot where it went wrong and fix the policy.