Putting all of Your Eggs in One Basket - or How NOT to do Layoffs

Putting all of Your Eggs in One Basket - or How NOT to do Layoffs
The recent story about Jason Cornish, a disgruntled employee of pharmaceutical company Shionogi is getting a lot of attention this week. In a nutshell, he resigned after a dispute with management, and was kept on as a consultant for a few months after. The story then goes that he logged into the network remotely (ie - VPN'd in using his legitimate credentials), then logged into a "secret vSphere console" (I'd call "foul" on that one - there would be no reason to have a "secret" console - my guess is he used the actual corporate vCenter console or used a direct client against ESX, which you can download from any ESX server, so he had rights there as well) then proceeded to delete a large part of the company infrastructure (88 servers in the story I read). The company was offline for "a number of days", and Jason is now facing charges. This diary isn't about the particulars of this case, it's much more of a common occurrence than you might think. We'll talk a bit about what to do, a bit about what NOT to do, and most important, we'd love to hear your insights and experiences in this area. First of all, my perspective ... Separation of duties is super-critical. Unless you are a very small shop, your network people shouldn't have your windows domain admin account, and vice versa. In a small company this can be a real challenge - if you've only got 1 or two people in IT, we generally see a single password that all the admins have. Separation of duties is simple to do in vmWare vSphere - for instance, you can limit the ability to create or delete servers to the few people who should have that right. If you have web administrators or database administrators who need access to the power button, you can give them that and ONLY that. Hardening your infrastructure is also important. Everything from Active Directory to vSphere to Linux have a "press the enter key 12 times" default install. Unfortunately, in almost all cases, this leaves you with a single default administrator account on every system, with full access to everything. Hardening hosts will generally work hand-in-hand with separation of duties, in most cases the default / overall administator credentials are left either unused or deleted. In the case of network or virtual infrastructure, you'll often back-end it to an enterprise directory, often Active Directory via LDAP (or preferably LDAPs), Kerberos or RADIUS. This can often be a big help if you have audits integrated into your change control process (to verify who made a particular change, or to track down who made an unauthorized change) HR processes need to be integrated with IT. This isn't news to most IT folks. They need to know when people are hired to arrange for credentials and hardware. But much more important, IT needs to be involved in termination. They need to collect the gear, revoke passwords and the like, in many cases during the exit interview. When an IT admin is layed off, fired or otherwise terminated, it's often a multi-person effort to change all the passwords - domain admin credentials, passwords for local hosts, virtual infrastructure admins, and the myriad of network devices (routers, switches, firewalls, load balancers, etc). If you've integrated your authentication back to a common directory, this can be a very quick process (delete or disable one account). In this case, a known disgruntled employee was kept on after termination as a consultant with admin rights. You would think that if HR as aware of this, or any corporate manager knew of it for that matter, that common sense would kick in, and the red flags would be going up well before they got to the point of recovering a decimated infrastructure. Yea, I know the proverb about common sense not being so common, but still .... Backups are important. It's ironic that I'm spelling this out in the diary adjacent to the one on the fallout from the 2003 power outage where we talk about how far we've come in BCP (Business Continuity Planing), but it's worth repeating. Being out for "a number of days" is silly in a virtual environment - it should be easy to recover, that's one of the reasons people virtualize. It's very possible, and very often recommended, that all servers in a virtual infrastructure (Hyper-V, XEN, vmWare, KVM whatever), be imaged off to disk each day - the ability and APIs for this are available in all of them. The images are then spooled off to tape, which is a much slower process. This would normally mean that if a server is compromised or in this case deleted, you should be able to recover that server in a matter of minutes (as fast as you can spin the disks). This assumes that you have someone left in the organization that knows how to do this (see the next section). Don't give away the keys. Organizations need to maintain a core level of technical competancy. This may seem like an odd thing for me to say (I'm a consultant), but you need actual employees of the company who "own" the passwords, and have the skills to do backups, restores, user creation, all those core business IT tasks that are on the checklist of each and every compliance regulation. In a small shop, it's common for IT to give consultants their actual administrative credentials, but it's much more common these days to get named accounts so that activity can be tracked, these accounts are often time limited either for a single day or the duration of the engagement. I'd very much like to see a discussion on this - what processes do you have in place, or what processes have you seen in other organizations to deal with IT "root level" users - how are they brought on board, how are they controlled day-to-day, and how are things handled as they leave the organization? I'm positive that I've missed things, please help fill in the blanks ! If I'm off-base on any of my recommendations or comments above, by all means let me know that too ! =============== Rob VandenBrink Metafore	Rob VandenBrink 579 Posts ISC Handler Aug 17th 2011
Thread locked Subscribe	Aug 17th 2011 1 decade ago
I don't have a problem with what you wrote, it's really what you didn't write I have to question. In my opinion, allowing any former employee continued access to your systems is fundamentally wrong, I don't care why they separated. Unfortunately, it's behavior I have seen more than a few times with clients and former employers. If you accept the premise that your greatest risk comes from the person(s) you hire -- even the "gruntled" ;) -- then allowing an individual who left on less than genial terms to continue consulting is insane! Proper termination procedures will ensure prompt removal of access rights and a general cleanup such as you mentioned.	Anonymous
Quote	Aug 17th 2011 1 decade ago
I agree on all counts. Unfortunately, I see the same as you - people leave and management leaves hiring their replacement until much too late, or worse, gives their responsibilities to someone already on staff, then never trains them. Hiring the person who left back in as a consultant is a common "solution" to these - fixing one bad idea with another bad idea. We're seeing this week what the risk is of this approach.	Rob VandenBrink 579 Posts ISC Handler
Quote	Aug 17th 2011 1 decade ago
Another area that is commonly neglected is controlling the information that not only consultants remove from the company but also full time employees. Although it is stated in our company policy that information is the sole property of the company, employees do run reports and export them maybe to portable media. What happens to those reports when an employee leaves? Even if the equipment was removed, they can still have printouts, copied media, items saved in personal PCs, etc. It is very difficult to contain information in smaller shops.	Rob VandenBrink 3 Posts
Quote	Aug 17th 2011 1 decade ago
I got laid off last year, then got called back a few weeks later because the PKI I implemented melted down. They wanted me to fix it as a consultant, which would have required Enterprise Admin. No way, not a chance - way too much risk to me, and they should have thought things through before deciding I was expendable. I want nothing to do with that company ever again, and I'm certainly not going to take the risk of something going wrong while I have access and getting blamed. That may even be the case here, who knows? So, stupidity on both sides here, but mostly the company's - he wouldn't have been disgruntled in the first place if they hadn't fired him, then they compounded the error by using him after that. He erred by accepting the work, but he may have done so because he wanted to use the access to get them back. Don't, just don't. How much will this mess cost that company? I bet a lot more than it would have cost to find a consultant rather than use their ex-employee.	Rob VandenBrink 4 Posts
Quote	Aug 17th 2011 1 decade ago
The central tenant of the problem of who watches the watch-keeper is trust. Restrict access to the superuser accounts and the IT job becomes more difficult. Provide access to superuser accounts you must trust. Establishing the infrastructure that limits access without using superuser takes time and effort. Separation of duties to use the accounts requires bodies. All of the above takes money. Superuser access provides the most privileges with the least amount of time, effort, bodies and money. You now trust the watch-keeper to tell you the time. Who is watching the watch-keeper? Unconditional trust is a risk. To mitigate risk requires and investment. The balancing act of investing money across low risk off occurring verses large impact or high risk of occurring verses small impact is a balancing act of risk management decision making. It can be unnerving if you reflect on the number of IT people who have grown-up in basements of home soothed by the gentle purring of their server fans while they honed the computer skills so valued by companies and stunted social and emotional skills needed by society. I will speculate the problem of watch-keepers out of sync happens more often than disclosed in the media. If it was a rampant problem the time, effort, bodies and money need to establish the infrastructure to support the unconditional risk would be a solution in the toolboxes of MBA that manage companies instead of “are you done yet”.	Rob VandenBrink 1 Posts
Quote	Aug 18th 2011 1 decade ago
For all I know when they say "secret VMware console", it could mean that he SSH'd into the servers, and used the command line to shutdown and delete things. He was a consulant... it could very well be that he was in a position of trust and needed this access. Suggestions like enforcing some kind of specific assignment 'separation of duties' are burdensome with not a clear proven benefit. Separation of duties implies imposing limits on each team members' area of responsibilities, which means more employees are required If someone's duty is to troubleshoot fix, or preen a VMware environment, then as a result, they're in a position of trust, and you're still screwed if they misbehave at any time. The only assured saving grace is to have good backups....	Mysid 146 Posts
Quote	Aug 19th 2011 1 decade ago
The best thing to do is have all of your access tied to some sort of central Identity and Access Management solution so that as soon as you are flagged as terminated in the system, your accounts are disabled.	Jasey 93 Posts
Quote	Aug 19th 2011 1 decade ago
I have to disagree with the idea of not hiring an ex-employee as a consultant, or a previous consultant again. The assumption is that in both cases, the termination was on good terms. I know a lot of companies in the US force retirement on employees when they turn 65, and some opt to retire earlier. It is common practice to have them at the retirement party on Friday afternoon as a departing employee, and back at their desk again on Monday morning as a consultant. This separates them from the company's employee benefits system -- especially the health insurance, since they are now elligible for goverment funded medicare. I am a consultant myself -- for the past 25 years -- and not because of retirement. I try to help these retiring employees understand how to set themselves up as a business, with liability insurance, to protect themselves from accusations fo misdeeds, as well as accidents, etc.	Moriah 133 Posts
Quote	Aug 19th 2011 1 decade ago

The recent story about Jason Cornish, a disgruntled employee of pharmaceutical company Shionogi is getting a lot of attention this week. In a nutshell, he resigned after a dispute with management, and was kept on as a consultant for a few months after.

The story then goes that he logged into the network remotely (ie - VPN'd in using his legitimate credentials), then logged into a "secret vSphere console" (I'd call "foul" on that one - there would be no reason to have a "secret" console - my guess is he used the actual corporate vCenter console or used a direct client against ESX, which you can download from any ESX server, so he had rights there as well) then proceeded to delete a large part of the company infrastructure (88 servers in the story I read). The company was offline for "a number of days", and Jason is now facing charges.

This diary isn't about the particulars of this case, it's much more of a common occurrence than you might think. We'll talk a bit about what to do, a bit about what NOT to do, and most important, we'd love to hear your insights and experiences in this area.

First of all, my perspective ...
Separation of duties is super-critical. Unless you are a very small shop, your network people shouldn't have your windows domain admin account, and vice versa. In a small company this can be a real challenge - if you've only got 1 or two people in IT, we generally see a single password that all the admins have. Separation of duties is simple to do in vmWare vSphere - for instance, you can limit the ability to create or delete servers to the few people who should have that right. If you have web administrators or database administrators who need access to the power button, you can give them that and ONLY that.

Hardening your infrastructure is also important. Everything from Active Directory to vSphere to Linux have a "press the enter key 12 times" default install. Unfortunately, in almost all cases, this leaves you with a single default administrator account on every system, with full access to everything. Hardening hosts will generally work hand-in-hand with separation of duties, in most cases the default / overall administator credentials are left either unused or deleted. In the case of network or virtual infrastructure, you'll often back-end it to an enterprise directory, often Active Directory via LDAP (or preferably LDAPs), Kerberos or RADIUS. This can often be a big help if you have audits integrated into your change control process (to verify who made a particular change, or to track down who made an unauthorized change)

HR processes need to be integrated with IT. This isn't news to most IT folks. They need to know when people are hired to arrange for credentials and hardware. But much more important, IT needs to be involved in termination. They need to collect the gear, revoke passwords and the like, in many cases during the exit interview. When an IT admin is layed off, fired or otherwise terminated, it's often a multi-person effort to change all the passwords - domain admin credentials, passwords for local hosts, virtual infrastructure admins, and the myriad of network devices (routers, switches, firewalls, load balancers, etc). If you've integrated your authentication back to a common directory, this can be a very quick process (delete or disable one account). In this case, a known disgruntled employee was kept on after termination as a consultant with admin rights. You would think that if HR as aware of this, or any corporate manager knew of it for that matter, that common sense would kick in, and the red flags would be going up well before they got to the point of recovering a decimated infrastructure. Yea, I know the proverb about common sense not being so common, but still ....

Backups are important. It's ironic that I'm spelling this out in the diary adjacent to the one on the fallout from the 2003 power outage where we talk about how far we've come in BCP (Business Continuity Planing), but it's worth repeating. Being out for "a number of days" is silly in a virtual environment - it should be *easy* to recover, that's one of the reasons people virtualize. It's very possible, and very often recommended, that all servers in a virtual infrastructure (Hyper-V, XEN, vmWare, KVM whatever), be imaged off to disk each day - the ability and APIs for this are available in all of them. The images are then spooled off to tape, which is a much slower process. This would normally mean that if a server is compromised or in this case deleted, you should be able to recover that server in a matter of minutes (as fast as you can spin the disks). This assumes that you have someone left in the organization that knows how to do this (see the next section).

Don't give away the keys. Organizations need to maintain a core level of technical competancy. This may seem like an odd thing for me to say (I'm a consultant), but you need actual employees of the company who "own" the passwords, and have the skills to do backups, restores, user creation, all those core business IT tasks that are on the checklist of each and every compliance regulation. In a small shop, it's common for IT to give consultants their actual administrative credentials, but it's much more common these days to get named accounts so that activity can be tracked, these accounts are often time limited either for a single day or the duration of the engagement.

I'd very much like to see a discussion on this - what processes do you have in place, or what processes have you seen in other organizations to deal with IT "root level" users - how are they brought on board, how are they controlled day-to-day, and how are things handled as they leave the organization? I'm positive that I've missed things, please help fill in the blanks !

If I'm off-base on any of my recommendations or comments above, by all means let me know that too !

===============
Rob VandenBrink
Metafore

Rob VandenBrink

579 Posts
ISC Handler

Aug 17th 2011

I don't have a problem with what you wrote, it's really what you didn't write I have to question. In my opinion, allowing any former employee continued access to your systems is fundamentally wrong, I don't care why they separated. Unfortunately, it's behavior I have seen more than a few times with clients and former employers.

If you accept the premise that your greatest risk comes from the person(s) you hire -- even the "gruntled" ;) -- then allowing an individual who left on less than genial terms to continue consulting is insane! Proper termination procedures will ensure prompt removal of access rights and a general cleanup such as you mentioned.

Anonymous

I agree on all counts. Unfortunately, I see the same as you - people leave and management leaves hiring their replacement until much too late, or worse, gives their responsibilities to someone already on staff, then never trains them.

Hiring the person who left back in as a consultant is a common "solution" to these - fixing one bad idea with another bad idea. We're seeing this week what the risk is of this approach.

Rob VandenBrink

579 Posts
ISC Handler

Another area that is commonly neglected is controlling the information that not only consultants remove from the company but also full time employees. Although it is stated in our company policy that information is the sole property of the company, employees do run reports and export them maybe to portable media. What happens to those reports when an employee leaves? Even if the equipment was removed, they can still have printouts, copied media, items saved in personal PCs, etc. It is very difficult to contain information in smaller shops.

Rob VandenBrink
3 Posts

I got laid off last year, then got called back a few weeks later because the PKI I implemented melted down. They wanted me to fix it as a consultant, which would have required Enterprise Admin.

No way, not a chance - way too much risk to me, and they should have thought things through before deciding I was expendable. I want nothing to do with that company ever again, and I'm certainly not going to take the risk of something going wrong while I have access and getting blamed. That may even be the case here, who knows?

So, stupidity on both sides here, but mostly the company's - he wouldn't have been disgruntled in the first place if they hadn't fired him, then they compounded the error by using him after that. He erred by accepting the work, but he may have done so because he wanted to use the access to get them back.

Don't, just don't. How much will this mess cost that company? I bet a lot more than it would have cost to find a consultant rather than use their ex-employee.

Rob VandenBrink
4 Posts

The central tenant of the problem of who watches the watch-keeper is trust. Restrict access to the superuser accounts and the IT job becomes more difficult. Provide access to superuser accounts you must trust.

Establishing the infrastructure that limits access without using superuser takes time and effort. Separation of duties to use the accounts requires bodies. All of the above takes money.

Superuser access provides the most privileges with the least amount of time, effort, bodies and money. You now trust the watch-keeper to tell you the time. Who is watching the watch-keeper?

Unconditional trust is a risk. To mitigate risk requires and investment.
The balancing act of investing money across low risk off occurring verses large impact or high risk of occurring verses small impact is a balancing act of risk management decision making.

It can be unnerving if you reflect on the number of IT people who have grown-up in basements of home soothed by the gentle purring of their server fans while they honed the computer skills so valued by companies and stunted social and emotional skills needed by society.

I will speculate the problem of watch-keepers out of sync happens more often than disclosed in the media. If it was a rampant problem the time, effort, bodies and money need to establish the infrastructure to support the unconditional risk would be a solution in the toolboxes of MBA that manage companies instead of “are you done yet”.

Rob VandenBrink
1 Posts

For all I know when they say "secret VMware console",

it could mean that he SSH'd into the servers, and used the command line to shutdown and delete things.

He was a consulant... it could very well be that he was in a position of trust and needed this access.

Suggestions like enforcing some kind of specific assignment 'separation of duties' are burdensome with not a clear proven benefit.

Separation of duties implies imposing limits on each team members' area of responsibilities, which means more employees are required

If someone's duty is to troubleshoot fix, or preen a VMware environment, then as a result, they're in a position of trust, and you're still screwed if they misbehave at any time.

The only assured saving grace is to have good backups....

Mysid

146 Posts

The best thing to do is have all of your access tied to some sort of central Identity and Access Management solution so that as soon as you are flagged as terminated in the system, your accounts are disabled.

Jasey

93 Posts

I have to disagree with the idea of not hiring an ex-employee as a consultant, or a previous consultant again. The assumption is that in both cases, the termination was on good terms. I know a lot of companies in the US force retirement on employees when they turn 65, and some opt to retire earlier. It is common practice to have them at the retirement party on Friday afternoon as a departing employee, and back at their desk again on Monday morning as a consultant. This separates them from the company's employee benefits system -- especially the health insurance, since they are now elligible for goverment funded medicare. I am a consultant myself -- for the past 25 years -- and not because of retirement. I try to help these retiring employees understand how to set themselves up as a business, with liability insurance, to protect themselves from accusations fo misdeeds, as well as accidents, etc.

Moriah

133 Posts