Postmortem #1 - kinsing aka kdevtmpfsi

Summary

Date: 15.05.2020

Author: Giorgi Kabanashvili

Status: Complete

Short: Malware called kinsing/kdevtmpfsi infected one of our Digital Ocean droplets. DO administration disabled networking for that node after a while for suspicious traffic.

Effect: Services hosted on that node became unavailable.

Reason: kinsing malware started mining Bitcoin and scanning networks for similar penetrable machines.

Solution: Delete old droplets. Create new and redeploy apps. For now, it was the fastest solution for us. We closed access to docker for external connections and thus malware can't infect our node again.

Discovery: Administrators received a DO email explaining why our droplet has been disconnected from the internet.

Lessons Learned: Always hide all your network infrastructure behind private networks and firewalls if no external access from users is needed. And always double-check you really did it.

Investigation

Since networking interfaces were disabled, I had to log in from Digital Ocean Access. The web console is pretty clumsy, but it was the only way to find out more.

First of all, I listed outgoing connections:

netstat -tputw

There were no strange connections detected. In the case of finding one, it would be easy to see process PID and then get more details about executable

So next I listed running processes. grep was used to navigate output in the web terminal.

lsof -i | grep less

And here comes our first candidate: kinsing.

pstree -ps 2961

pstree command-line utility tells us when a program was evoked and by who.

Now we need to find an executable file:

find / -name kinsing
find / -name kdevtmpfsi

In my case, it was located in the docker overlay2 directory. So penetration was made through docker daemon. Some of the docker ports were open for public access for that node. I accidentally forgot to include a newly created droplet in our common firewall.

I attached volume to copy executable and after that unmounted it. From freshly created droplet booted with a volume and this way, I was able to download it locally for further investigation.

Virus Total tells us the rest. You can see the full details following the link.

Conclusion

And it's enough at the moment. The next steps would be to download the whole system image, run in a Virtual Machine with blocked outgoing access, and investigate how the virus got into the system. How it behaves and tries to spread more. But unfortunately, at the moment I don't have such local machine resources and the client wanted to delete droplet due to resource optimization.