Did John run rm –rf*? Why enterprises use session recordingAnirban Banerjee
For those of us who have been in Enterprise IT teams and DevOps groups it is a fact of life to be called in to investigate an incident. The incident in question can range from – we think we have been hacked to we are being sued we need to get data to prove our case. I am going to talk about the centre of this wide spectrum – employee activity validation.
What is the problem?
Your business runs on certain assumptions and verifiable actions. These assumptions include that your employees are performing to the best of their ability, they are honest, they are dedicated to the singular success of the company and more. Now, there are a few employees who often go off the deep end. It doesn’t even take an extreme breakdown for an employee to do something that is ethically challenging. Consider the case of shadow IT hired in countries where the ethical standards differ from where your headquarters are based. Offshore employees have often been found to try and “siphon” out information critical to the working of a business. An employee can do this on their own volition or be coerced. The point being that this happens more often than you think. We asked some of the largest IT companies who have development centres around the world and employ more than 3000 people. Every quarter there are at least 4-10 cases where investigation is needed. The number is higher as the mix of on site versus offshore employees grows.
What is employee activity validation?
Employee activity validation straddles two main groups of controls. The first one being putting checks and balances in place to make employee accounts can and cannot be used for some actions, this falls under the purview of privilege management. We are not going to discuss this now.
The second being activity monitoring. This is important to validate if the employee has actually done something distasteful or not. If it ever comes down to the wire your IT and DevOps team can product a clean report for a specific date, action taken, employee effected and so on. This is the area we are going to focus on.
Benefits of activity monitoring
Making sure your team has (1) validated (2) context specific information to help you in the case of understanding if something unethical has happened or not is critical. When it comes down to this stage, the team is already under a lot of pressure to provide proof quickly. Going through logs, combing through what the employee has accessed is not helpful.
If an employee is smart and they know what they are doing in many cases they can try to remove the tell tale signs of any activity they might have performed on the company’s account. Case in point – an employee decides to get access to a codebase they should not be looking at. They access a server hosting the codebase, do not use svn or git directly but instead go ahead and write commands manually on a terminal to copy code out and then remove all history, modify time stamps etc. This is quite doable for someone who is motivated.
Activity monitoring steps in here and allows for you to find out exactly what happened. It does not matter what the employee tries to do to cover up their tracks. Activity monitoring is usually implemented as an inline, real-time solution which tracks any and every aspect for what the employee is up to. In many cases it can be near invisible.
You can have conclusive reporting on what happened, be able to prove your case effectively and save everyone a lot of time and anguish by laying out the facts as they occurred. Each investigation takes anywhere from 5-40 hours. Consider how much time is spent on even 2 cases every quarter.
What can someone get to prove an action occurred?
Session recording comes in many forms, shapes and sizes. We are going to discuss SSH session recording. Why ? This is because SSH is one of the most popular ways for individuals and automated software to access servers, run commands, perform patching. You name it – SSH is involved somewhere in the pipeline of activities.
SSH is one of the most common ways for an employee to access a server. These servers run all critical applications and processes inside businesses. One of the most important pieces of information you can get access to is log information. Log information will help to show who accessed the server via ssh, when and from where. Inside Ubuntu Linux servers you will find logs placed in /var/log/auth containing information like the following:
May 1 16:17:02 owl CRON: pam_unix(cron:session): session closed for user root May 1 16:17:43 owl sshd: Accepted publickey for root from 192.168.0.101 port 37384 ssh2 May 1 16:17:43 owl sshd: pam_unix(sshd:session): session opened for user root by (uid=0)
On redhat based systems the same information can be accessed at /var/log/secure.
These logs can then be pumped into a SumoLogic/Splunk endpoint and analysed for detailed investigations. More information can be found in the discussion here – http://unix.stackexchange.com/questions/127432/logging-ssh-access-attempts
However, there is something better – full video SSH session recordings. You can use typescript to record SSH sessions and replay them back with contextual information like what commands were typed. An example video is present here: https://youtu.be/UXUKM-rTXS8 . Having a video recording of the entire session which the employee used to interact with your servers, annotated at various time points with the commands that were run and the output that was provided is a good way to investigate incidents. Having this type of information helps your IT and DevOps team answer questions faster and present forensic evidence without spending the hours usually required for something like this.
How can you get started?
There are two main steps here. The first one involves installing an SSH proxy of sorts. The good news is that there are many different choices available in the opensource community. You can pick whichever flavour you like. Once you have sintalled your choice of SSH proxy you will need to make sure to transfer DNS for each server you want to protect to the proxy. This makes sure that when someone types ssh firstname.lastname@example.org server.abcd.com points to your proxy. Subsequently the SSH connection gets routed to your SSH proxy.
The second step will be to use typescript to record and create a data files that stores the session information about input and output. You will need to accomplish session busting. Session busting means that you will need to terminate the SSH connection at your proxy and then connect to the server using a different SSH tunnel and then shovel data from one tunnel into the other. Fortunately most *nix systems come with the concept of pipes and it is not a very hard job to accomplish this.
Once these two parts are completed you can have a system that can provide you with video recording of detailed information about what has been done by an employee when they have accessed servers.
If you would like to build out your own SSH video session recording solution it will certainly help you pass PCI Level 3.1 and Level A certifications respectively. These kinds of solutions are also helpful for SOX and SOC 1, SOC 2 compliance. From experience with deploying systems like these with customers we can share that these types of solutions save IT and DevOps teams hours of time which would otherwise be spent on combing through logs and generating reports. We wish you and your team the best in your security endeavours and as always please connect with us, we love hearing about your opinion.
Talk to us about this
I would love to talk to you. Please feel free to schedule a time to talk with me using my calendar link here