Harness: The Key to Efficiently Running Chaos Experiments

I recently happened to attend the Chaos Carnival where I learned about some amazing tools and it was my second year attending it. Chaos Engineering has gained so much value today that is not just a checklist item for the stability of your systems but also a mechanism by which you are ensuring that the steady state hypothesis of your systems is well known to you and how to better them.

Harness as a tool has a lot to offer, but in this article, I'll be trying out only the chaos engineering features that it provides.

To get started, the first step would be to sign up on Harness. You will be getting enterprise trial version for 15 days.
Create a project
Setup your environment
Enable Chaos
I'll be setting up a Kubernetes cluster, thanks to KodeKloud, I get to easily spin up a k3s cluster.
- Configure chaos infrastructure
  - Deploy chaos infrastructure on your k8s cluster using the reference yaml after you click download.
  - After a successful connection, you should see the heartbeat and status 'CONNECTED'
    - Now, time to create some chaos experiments.
      - You can either pick from the available template (LitmusChaos) or make your yaml template. In this case, I'll be using the Kubernetes node CPU hog template.
Run the attack
Check the execution status and logs
During the attack, try checking the CPU spike,
At the end of the experiment, you can check the resiliency score.
Re-run the experiment with additional steps. In the below example, I've added timechaos and kubelet service kill.

since the experiment did not succeed fully, the resilience score was updated accordingly to 33.33% because only 1/3 stages passed. The last 2 stages failed because I didn't configure them to run on the required pods/nodes.
If you re-run the same experiment after reconfiguration and the attack execution completes successfully, you'll see an improved resilience score.

I look to explore further more on the probes and combinations of different attacks.