Watching the Watchers –Guarding the Keys to theKingdomVersion 1.3Released: April 26, 2012Securosis, L.L.C.515 E. Carefree Highway Suite #766 Phoenix, AZ [email protected] www.securosis.comT 602-412-3051

Author’s NoteThe content in this report was developed independently of any sponsors. It is based on material originallyposted on the Securosis blog, but has been enhanced, reviewed, and professionally edited.Special thanks to Chris Pepper for editing and content support.Licensed by XceediumXceedium, Inc., is the leading provider of Privileged Identityand Access Management solutions. The company’saward-winning Xsuite platform protects organizations fromthe numerous threats that privileged users and trustedinsiders pose to networks and data.Xsuite is used globally by large enterprises and government agencies to meet stringent security andcompliance requirements. Xsuite enables organizations to implement a zero trust security model — ensuringthat privileged users are able to access only explicitly authorized systems and commands. The enterpriseclass solution provides organizations with a unified platform for controlling, auditing, and continuouslymonitoring all privileged access. Xsuite also vaults and manages privileged credentials, fully records usersessions, and provides real-time alerting. The system extends security controls across IT infrastructurelocated on premise, in public or private cloud environments, or any combination thereof.Xceedium’s products are FIPS 140-2 Level 2 and Common Criteria EAL4 certified. For more information,visit For more on Xsuite, watch the Xsuite 2-Minute Explainer.CopyrightThis report is licensed under Creative Commons Attribution-Noncommercial-No Derivative Works .0/us/Securosis — Watching the Watchers: Guarding the Keys to the Kingdom2

Table of ContentsIntroduction4The Privileged User Lifecycle8Restrict Access11Protect Credentials14Enforce Entitlements17Monitor Privileged Users19Clouds Rolling In21Integration25Summary27About the Analyst28About Securosis29Securosis — Watching the Watchers: Guarding the Keys to the Kingdom3

IntroductionIt’s about time we documented our research on privileged user management (PUM), which you overlook atyour own risk. Administrators (those privileged users) have the keys to your kingdom. A sysadmin withmalicious intent can mean a very bad day for you and your organization.And no, this isn’t just another recycled attempt to bring the insider threat back into vogue – much to thechagrin of the DLP vendors, who built their first wave of growth on it. First of all, privileged users (P-Users)don’t necessarily need to be insiders. And most insiders have limited access and authorization entitlements,whereas administrators can basically give themselves whatever access want — that old privilege escalationthing. That’s why we call this paper Watching the Watchers – because if not properly managed,administrators are Above the Law.Business Imperatives Changing PrivilegesWe live in a brave new world of technology. What used to be within your site, in your data center, or runningon your big iron, now may or may not be in any or all of those places. Even for stuff runs in your data center,you might not know exactly where and it may not be under your control. It might not be running on anoperating system you understand. You might not control the pipes to the data. And you certainly can’t tellbusiness users and business partners that they need to go back to the old model, where you had visibilityfrom the bare metal all the way to the data layer. Times have changed.Even better, you might not even know who is responsible for managing those specific systems. With layers ofvirtualization abstracting just about all physical networks, storage, and servers, different folks assumeresponsibility for managing the pieces of what we call an application. Even the term ‘application’ is really amisnomer – applications can be almost anything, processing in numerous places, accessing data fromanywhere, and presenting information to anyone anywhere. So let’s start by defining a privileged user:Privileged User: Anyone with admin (or root) access to a device.By that definition every user is privileged on some device. But that’s a bit broad, so we’ll restrict ourdiscussion, and this research, to users who manage critical devices – running applications, hostingdatabases, or pushing packets to the places they need to be. Sure, it’s problematic if the P-user in charge ofthe receptionist’s device (you know, the receptionist) is compromised. But it’s much more serious if anadministrator of your customer database host gets compromised.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom4

Let’s be a bit more specific about the business drivers and impact on privileged users: Reduce Cost – Virtualization/Cloud: Many organizations are under intense pressure to continuereducing costs wherever possible. That means embracing technologies such as virtualization to makebetter use of physical hardware and cloud computing to make due with less data center real estate. Theimpact of this driver is scale. Now you have a lot more things to manage, and they can be spun up andtorn down at the click of a button or via script. Throw in the unbounded number of instances that can berun in the public cloud, and the only thing you can be sure of is a massive change management headache. Reduce Cost – Outsource: While data centers are virtualizing, organizations are contracting with (lower)cost management to do their (alleged) commodity work. You know, like managing databases and email. Allkidding aside, it’s common to see third parties managing wide swaths of organizations’ IT infrastructure –giving nameless, faceless folks (perhaps on the other end of a SAML link) access to critical stuff. Agility – New Apps: If you think about a typical web app, it’s more ‘assembled’ nowadays than built fromthe ground up. Parts may be yours, they could be pieces you got from someone else, and they mightinclude data from somewhere else, integrated into your environment via a foreign API. It’s hard to knowwhat an application is nowadays. It’s difficult to manage what we don’t understand.There are plenty more business drivers but you get theAnyone with access tomanage a device that runssomething important is aprivileged user, and the rushto become more efficientand leveraged can result inerrors, shortcuts, andgeneral violation of goodoperational practices.picture. Anyone with access to manage a device that runssomething important (or is a component of somethingimportant) is a privileged user, and the changemanagement issues inherent in this escalating complexityrequire administrators continue becoming more efficientand leveraged. Which can result in errors, shortcuts, andgeneral violation of good operational practices. Let’s lookat some specific threats presented by these privilegedusers.P-User Risk AssessmentWe’re old school. We still like to assess risk, or at leastrun through a quick mental exercise to figure out howmany ways we can get killed. So let’s do that with thisexplosion of devices managed by privileged users. Ofcourse this isn’t an exhaustive list – more a back-of-the-envelope exercise to uncover some of the biggest threats to our environment if privileged users arecompromised. And while we are at it, let’s define a new term, PUpwnage, for compromise of privileged user.It just rolls off the tongue, right?1. Compromised devices: This one is obvious. If a privileged user is compromised (PUpwned), theattackers gain access to any device they manage, and the fun begins.2. Data leakage: PUpwnage can result in any and all data being stolen from the devices they control.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom5

3. Create accounts: PUpwnage allows attackers to create both user and admin accounts on devices, andto pivot through the environment moving from one compromised device to another – stealing data asthey march along.4. Pollute applications and/or data: PUpwnage also results in application attacks, such as changingcode to break functionality, creating backdoors, deleting or changing data, and otherwise breaking yourapplications.5. Operational mayhem (shut down devices): Yup, PUpwnage allows attackers to shut down devices,max out device utilization (effectively a denial of service), wipe disks, or pretty much anything else.6. Fiasco in the clouds: If you run a bunch of stuff in the cloud and someone gains access to your cloudmanagement console, they can do everything else we mentioned under operational mayhem. But a newattack, unique to the cloud, is an economic denial of service. Spinning up instances and consumingcloud resources, costs you real money and can continue until you either max out your credit or theprovider shuts you down. Fun, right?As enjoyable as it would be to come up with another 20 problems posed by compromised privileged users,someone very wise once told us “never sell past the close”, so we’ll assume you are convinced of thedangers of losing control of your P-Users. But we need to discuss the other issue, which is that P-Usersaren’t really that unique because many administrators share accounts and credentials. They justify thisbehavior with many reasonable sounding excuses such as 1) we need access if Admin1 gets hit by a bus, or2) it’s too time consuming to create admin credentials for each admin on each device. The end result isn’tmuch different than a crack den: pretty messy, and at some point the cops show up to turn out the lights.Compliance has remains a major driver of all privilegeduser management. Specifically, PCI-DSS demands uniqueIDs and that account credentials aren’t shared. SOXP-Users aren’t really uniquerequires separation of duties to ensure no one (not evenbecause manyadministrators) has the ability to change an organization’sfinancial controls without oversight. And we see the needadministrators shareto audit database administrators as a clear driver forDatabase Activity Monitoring. That’s one specific use casefor privileged user management; we prefer to focus on thesecurity risks and complexity drivers for the technology,but we can’t minimize the old reliables: compliance andaccounts and credentials.The end result isn’t muchdifferent than a crack den:pretty messy, and at someaudit.point the cops show up toTo reiterate: privileged user management involvesturn out the lights.ensuring (only) the right folks access the right resources atthe right times. And that you know exactly what they did.Easier said than done, but that’s why we are writing thispaper – to map out a specific set of activities that can help reduce the risk of P-User pwnage.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom6

The Elephant in the RoomThe cold hard reality is that most organizations don’t formally manage privileged users. They may do someidentity management, but that’s for everyone. Given the potential issues, why doesn’t everyone watch thewatchers? Inevitably a higher priority project, or a new device with shinier lights, shows up to divert attentionand budget away from this non-revenue-generating and non-cost-saving activity. Until a rogue admin locksyou out of your data center or your forensics guys find the back door into your logistics system, anyway. Butwithout throwing around a bunch of FUD we realize there needs to be a logical, phased approach to solvingthe problem. Big bangs don’t work with large organizations with lots to do. So this paper will lay out alifecycle for managing your privileged users.We will go beyond the identity management aspects (provisioning, entitlements, and authentication) ofmanaging users to also focus on the operational requirements and issues. This means keeping admins (andother folks) from establishing management sessions with devices they aren’t authorized to manage. If theyare authorized you need to make sure they can’t share or otherwise misuse credentials. And finally you wantto audit what they do on the devices – critical both for forensics after an attack and to substantiate controlsfor compliance.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom7

The Privileged User LifecycleAs we described above, organizations can’t afford to ignore the issue of privileged users (P-Users) any more.A compromised P-user can cause all sorts of damage and so needs to be actively managed. Let’s now talkabout solutions. Most analysts favor models to describe things, and we call ours the Privileged UserLifecycle.But pretty as the lifecycle diagram is, first let’s scope it to define beginning and ending points. Our lifecyclestarts when the privileged user receives escalated privileges, and ends when they are no longer privileged orleave the organization, whichever comes first. Here is the whole lifecycle:Securosis — Watching the Watchers: Guarding the Keys to the Kingdom8

Provisioning EntitlementsThe Privileged User Management lifecycle starts when you determine someone gets escalated privileges.That means you need both control and an audit trail for granting these entitlements. Identity Management isa science all by itself, so this research cannot tackle it in depth – we will just point out the connectionsbetween (de-)provisioning escalated privileges and the beginning and end of the lifecycle. These privilegedusers have the keys to the kingdom so you need tight controls over the provisioning process, includingseparation of duties and a defined workflow with adequate authorization.Identity management is repository-centric, so any controlsPrivileged users have thekeys to the kingdom youneed tight controls over theprovisioning process,including separation ofduties and a definedworkflow which includesadequate implement throughout the lifecycle need nativeintegration with the user repository. It doesn’t work well tostore user credentials multiple times in multiple places.Another aspect of the provisioning process involvesdefining the roles and entitlements for each administrator,or more likely for groups of administrators. We favor adefault deny model, which basically denies anymanagement capabilities to administrators, assignscapabilities by explicit authorization to manage device(s),and defines what they can do on each specific device.Although the technology to enforce entitlements can becomplicated, as we will discuss later, defining the rolesand assigning administrators to the proper groups can beeven more challenging. This typically involves gainingsignificant consensus among the operations team, which is always fun but also critical for P-Usermanagement.Now we get to the fun stuff: actively managing what specific administrators can do. In order to gainadministrative rights to a device an attacker (or rogue administrator) needs access, entitlements, andcredentials. So the next aspect of our lifecycle address these issues.Restrict AccessLet’s first tackle restricting access to devices. The key is to allow administrators to establish managementsessions only to devices they are entitled to manage. Managing any other device should be blocked to thatP-User. That’s the meaning of default deny in this context. Network isolation is one of the oldest defensetactics you know. If a P-User can’t logically get to a device, they can’t manage it nefariously.There are quite a few ways to isolate devices, both physically and logically, including proxy gateways anddevice-based agents; we will discuss a number of these tactics later. When restricting access you also needto factor in authentication, as logging into a proxy gateway and/or managing particularly sensitive devicesshould require multiple factors.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom9

Obviously integrating private and public cloud instances into the P-User management environment requiresdifferent tactics, as you don’t necessarily have control over the network to govern access. But the cloud istoo enticing to simply avoid. We will also delve into tactics to restrict access to cloud-only and hybridenvironments later.Protect CredentialsOnce a P-User has network access to a device, they still need credentials to manage it, so administratorcredentials need appropriate protection. The next step in the lifecycle typically involves setting up a passwordvault to store administrator credentials and provide a system for one-time passwords. There are a number ofarchitectural decisions involved in vaulting privileged user passwords that impact the other controls in place:restricting access and enforcing entitlements.Enforce EntitlementsIf an administrator has access and the credentials, the final aspect of controls involve determining what theycan do. Many organizations opt for a carte blanche policy, providing root access and allowing P-Users todo whatever they want. Others take a finer-grained approach, defining the specific commands the P-Usercan perform on any class of device. For instance, you might allow the administrator to update the device orinstall software/applications, but not delete a logical volume or load an application. This all depends on howmuch granularity you use when provisioning the entitlements.Technically, this approach requires some kind of agent capability on the managed device, or running sessionsthrough a proxy gateway which can intercept and block commands as necessary. We will discussarchitectures later when we dig into this control.Privileged User MonitoringFinally, keep a close eye on what all the P-Users do when they access devices. That’s why we called thispaper “Watching the Watchers” — the lifecycle doesn’t end after implementing the controls. Privileged UserMonitoring can mean a number of different things, from collecting detailed audit logs on every transaction, toactually capturing video of each session. There are multiple benefits to detailed monitoring, includingforensics and compliance.We should also mention the deterrent benefit of privileged user monitoring. Human nature dictates thatpeople are more diligent when they know someone is watching. So Rich can be happy that human naturehasn’t changed. Yet. When administrators know they are being watched they are more likely to behaveproperly – not just from a security standpoint but also from an operational standpoint.No PanaceaOf course the privileged user lifecycle is not a panacea. A determined attacker will find a path to compromiseyour systems, regardless of how tightly you manage privileged users. No control is foolproof, and there areways to gain access to protected devices and to defeat password vaults. So we will examine theweaknesses in each of these tactics later, as well. As with everything else in security, you aren’t looking forperfection – but to make it a bit harder for attackers to gain root on your critical devices.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom10

Restrict AccessWe recommend an initial focus on Restricting Access, mostly because it reduces your attack surface, andthe earlier you can eliminate risk to your key servers, the better. You achieve the general objective ofrestricting access by implementing controls to ensure administrators only access devices they haveauthorization to manage. There are a few ways to handlerestriction:1. Device-centricity (Status Quo): Far too manyorganizations rely on their existing controls, whichinclude authentication and other server-based accesscontrol mechanisms. But given the number of ways toescalate privileges or bypass the operating systemdefenses with known or unknown vulnerabilities, it’sclear that the status quo doesn’t cut it.2. Network Isolation: Tried and true networksegmentation approaches enable you to isolateYou achieve the generalobjective of restrictingaccess by implementingcontrols to ensureadministrators only accessdevices they haveauthorization to manage.devices (typically by group) and only allow authorizedadministrators access to the networks on which theyreside.3. Privileged User Management (PUM) Proxy Gateway: This entails routing all managementcommunications through a privileged user management proxy server or service, which enforces accesspolicies. The devices only accept management connections from the proxy server, and do not allowdirect management access.There are benefits and issues with each approach, so ultimately you need to choose an appropriatecompromise. Let’s dig into each approach and highlight what’s good and what’s not so good.Device-centricity (Status Quo)There are really two levels within this category; the first is common authentication, which in this context is noteffectively restricting access. Obviously you could do a bit to make the authentication more difficult, includingstrong passwords and/or multi-factor authentication. You would also integrate with an existing identitymanagement platform (IDM) to keep entitlements current. But ultimately you are relying on credentials as away to keep unauthorized folks from managing your critical devices. And basic credentials can be defeated.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom11

The other level under this category is server access control capabilities, which are fairly mature. This involvesloading an agent onto each managed device and enforcing the access policy on the device. The agentbased approach offers fairly solid security – the risk shifts to compromise of the agent. Of course there ismanagement overhead to distribution and management of the agents, as well as the additionalcomputational load on the device imposed by the agent.But any device-based approach is in opposition to one of our core philosophies: “If you can’t see it, it’s muchharder to compromise.” Device-centric access approaches don’t affect visibility (to attackers) at all. This issuboptimal, because in the real world new vulnerabilities appear every month on all operating systems – andmany of them can be exploited via zero-day attacks. Those attacks provide “back doors” into servers, givingattackers control without requiring legitimate credentials – regardless of agentry on the device. Any devicebased method fails if the device is rooted somehow.Network IsolationThis entails using network-layer technologies such as virtual LANs (VLANs) and network access control(NAC) to isolate devices and restrict access based on who can connect to specific protected networks. Thegood news is that many organizations (especially those subject to PCI) have already implemented some levelof segmentation. It’s just a matter of building another enclave, or trust zone, for each group of servers toprotect.As mentioned earlier, it’s much harder to break intoSegmentation requires thesomething you can’t see. Segmentation requires theattacker to know exactly what they are looking for andattacker to know exactlywhere it resides, and to have a mechanism for gainingwhat they are looking foraccess to the protected segment. Of course this ispossible – there have been ways to defeat VLANs forand where it resides, and tohave a mechanism forgaining access to theprotected segment.years – but vendors have closed most of the very easyloopholes.More problematic to us is that this relies on thenetworking operations team. Managing entitlements andkeeping devices on the proper segments in a dynamicenvironment such as your data center, can bechallenging. It is definitely possible, but it’s also difficult,and it puts direct responsibility for access restriction in the hands of the network ops team. That can anddoes work for some organizations, but organizationally it is complicated and somewhat fragile.The other serious complication is cloud computing – for both private and public clouds. The cloud is key andeverybody is jumping on the bandwagon, but unfortunately it largely removes visibility at the physical layer. Ifyou don’t really know where specific instances are running this approach becomes difficult or completelyunworkable. We will discuss this in detail later when we discuss the cloud in general.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom12

PUM ProxyThis approach routes all management traffic through a proxy gateway (PUM proxy). Administratorsauthenticate to the proxy, hopefully using strong authentication. The authenticated administrator gets a viewof the devices they can manage and establishes a management session to the desired device. Anotherpossible layer of security involves a lightweight agent on every managed devices to handle the handshakeand mutual authentication with the PUM proxy, and to block management connections from unauthorizedsources.This approach is familiar to anyone who has managed cloud computing resources via vCenter (in VMwareland) or a cloud console such as the one for Amazon Web Services. You log in and see the devices/instances you can manage, and proceed accordingly.This satisfies our preference for providing visibility to onlythose devices that can legitimately be managed. It alsoprovides significant control over granular administrativeThe PUM proxy satisfies ourfunctions, as commands can be blocked in real time (it ispreference for providinga man in the middle, after all). Another side benefit is thedeterrent effect: administrators know all their activity isvisibility to only thoserunning through a central device and typically heavilymonitored (as we will discuss later).devices that can legitimatelybe managed. It alsoBut any proxy presents issues, including a possible singleprovides significant controlpoint of failure, and additional latency for managementsessions. Some additional design and architecture work isover granular administrativerequired to ensure high availability and reasonableefficiency. It’s a bad day for the security team if theyprevent ops from doing their jobs. And periodic latencytesting is called for to make sure the proxy doesn’t impairproductivity. Finally, as with virtualization and cloudfunctions, as commandscan be blocked in real time(it is a man in the middle,after all).consoles, if you own the proxy server, you owneverything in the environment. So the proxy’s security isparamount.Each of these approaches is best in different environments, and each entails its own compromises. For thosejust starting to experiment with privileged user management, a PUM proxy is typically the path of leastresistance for getting started. It’s a question of what works best for you, based on the sophistication ofrequired controls and IT culture.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom13

Protect CredentialsThe next part of the lifecycle addresses protection of the keys or credentials of these privileged users (PUsers), to prevent them from falling into the wrong hands. The best access and entitlement security controlsfail if someone can impersonate a P-User. But the worst risk isn’t even compromised credentials — it’s nothaving unique credentials in the first place. You must have seen the old admin password sharing scheme,right? It was used, mostly out of necessity, many moons ago. Administrators needed access to the devicesthey managed. But at times they needed help, so they asked a buddy to take care of something and justgave him/her the credentials. What could possibly go wrong?We have already mentioned that shared administrative credentials open Pandora’s Box. Once the credentialsare in circulation you can’t get them back – which is a problem when an admin leaves the company or nolonger has those privileges. You can’t deprovision shared credentials, so you need to change them.PCI, as the low bar for security (just ask GlobalPayments), recognizes the issues with sharing IDs, soRequirement 8 is all about making sure anyone withYou can’t deprovisionaccess to protected data uses a unique ID and that theiruse is audited – so you can attribute every action to ashared credentials, soparticular need to changeBut that’s not all! (in our best infomercial voice). Whatabout the risk that some endpoints could bethem.compromised? Even administrative endpoints — whichwould make sending admin credentials to that endpointunsafe. And what happens when developers hard-code credentials into applications? Why go through thehassle of secure coding – just embed the password right into the application! That password never changesanyway, so what’s the risk? We need to protect credentials just as much as whatever they control.Credential LockdownHow can we protect these credentials? Locking them away in a vault satisfies many of the requirements.First, if the credentials are stored in a vault, it is harder for admins to share them. We won’t put the cartbefore the horse, but a vault makes it fairly easy (and transparent) to change passwords after every access,eliminating the sticky-note-under-keyboard risk.Going through the vault for every administrative credential access creates an audit trail of who used whichcredentials and when, and often which specific devices they were managing. That kind of stuff makesauditors happy.Securosis — Watching the Watchers: Guarding the Keys to the Kingdom14

Depending on the deployment of the vault, the administrator may never even see the credentials, as they canbe automatically entered on the server under the proxy approach to restricting access. This also providessingle sign-on to all managed devices, as the administrator authenticates (presumably using multiple factors)to the proxy, which interfaces directly to the vault again, transparently to the user. So even an administrationdevice teeming with malware cannot leak critical credentials.Similarly, an application can make a call to the vault, rather than hard-coding credentials into the app. Yes,the credentials still end up on the application server, but that’s still much better than hard-coding thepassword. So are y

Watching the Watchers – Guarding the Keys to the Kingdom Version 1.3 Released: April 26, 2012 Securosis, L.L.C. 515 E. Carefree Highway Suite #766 Phoenix, AZ