Securing Sensitive Passwords with Hitachi ID Privileged Password Manager

Abstract

Hitachi ID Privileged Password Manager is a system for securing access to privileged accounts. It works by regularly randomizing privileged passwords on workstations, servers, network devices and applications. Random passwords are encrypted and stored on at least two replicated vaults. Access to privileged accounts may be disclosed:

To IT staff, after they have authenticated and their requests have been authorized.
To applications, replacing embedded passwords.
To Windows workstations and servers, which need them to start services.

Password changes and access disclosure are closely controlled and audited, to satisfy policy and regulatory requirements.

Privileged Password Management

Privileged passwords must be protected more vigorously than any other data in an organization:

Sensitive data:
Privileged passwords are arguably the most sensitive data in an organization, since they unlock all other data. Inappropriate disclosure can be catastrophic.
Business interruption:
Loss of access to privileged passwords means that the systems which the privileged passwords control cannot be managed, at least until they are powered down and "hacked into." Consider the impact on IT support of a disaster where every root or Administrator password in a company is permanently lost.
Constant change / data backup:
If privileged passwords are changed regularly, then scheduled backups will contain mostly historical data rather than current passwords and so provide little value.

Privileged Password Manager enables organizations to secure privileged passwords:

It periodically randomizes every privileged password, with support for more than 100 types of servers and workstations.
IT users must sign into Privileged Password Manager when they need to access a sensitive password.
The password change and disclosure process creates strong, personal authentication, authorization over which passwords are visible to each user and an audit history of all access attempts (AAA).

Technical Challenges

The obvious solution to the security vulnerability of static and shared privileged passwords is to change these passwords so that each one is unique and changes regularly. Doing this can be technically challenging, however:

There are thousands of privileged passwords:
Clearly automation is required to manage them.
There are passwords on many kinds of systems:
The automation must include many integrations, with different kinds of systems (Windows, Unix, SAP, mainframe, Oracle, etc.).
The majority of privileged passwords are on workstations.
Workstation passwords present special challenges:
- Workstations may be powered down.
- Workstations may be disconnected from the network.
- Workstations may not be reachable from a central data center because they are behind firewalls.
Connectivity to servers.
- Servers may not be up 100% of the time.
- Servers may not be reachable from a single data center network segment. Specifically, they may be on different network segments, blocked off from the password management system by one or more firewalls.
Secure, reliable storage.
Once automation is implemented to regularly change passwords, technical challenges regarding their storage must be addressed. The password storage system must:
- Be secure. An insecure storage system, if compromised, would allow an intruder to gain administrative access to every device in the IT infrastructure.
- Be reliable. A disk crash or facility interruption affecting the password storage system would make every administrator ID unavailable.
- Include fine-grained access controls. Only the right administrators should get access to the right passwords, after proving their identity.
- Log password disclosure. Access to privileged passwords must be logged, to create accountability.

Functional Requirements

A privileged password management system needs a set of well-integrated features to function:

It must randomize passwords regularly -- sensitive passwords should be unique and short-lived.
It must be able to disclose passwords to or inject passwords into sessions on behalf of appropriate users and software agents, but only under the right circumstances:
1. To IT staff, if they have been assigned appropriate access rights.
2. To IT staff who have not been assigned permanent access rights, but have been granted one-time permission.
3. To programs that start services (Windows Service Control Manager, Scheduler, IIS and others) so that they can start services after a password change.
4. To applications, to replace embedded passwords in programs and scripts.
Both a static access control model and a dynamic authorization workflow are required.
The system must log both password updates and disclosure. Failed updates can be used to identify infrastructure problems while logs of password disclosure create accountability.
The system should be able to control concurrent disclosure of a given password -- for example to limit the number of people concurrently able to manage a server.

Randomizing Passwords

Privileged Password Manager secures sensitive passwords by periodically randomizing them:

On push-mode servers and applications:
1. Periodically -- for example, every night between 3AM and 4AM.
2. When users check passwords back in, after they are finished using them.
3. When users request a specific password value.
4. In the event of an urgent termination of a system administrator.
On pull-mode workstations and other devices:
1. Periodically -- for example, every day.
2. At a random time-of-day, to prevent transaction bursts.
3. Opportunistically, whenever network connectivity happens to be available from the workstation to a central server.

Privileged Password Manager can enforce multiple password policies. There is a global policy and local, override policies applied to resource groups. Password policies specify the complexity of both randomly chosen and manually selected passwords. In addition to mandating character types (lowercase, uppercase, digits, punctuation), the policy can specify minimum and maximum password lengths and other characteristics, especially relevant to manually-chosen passwords, such as dictionary and history checks.

Password Disclosure

Privileged Password Manager is designed to not only randomize and securely store, but also to disclose privileged passwords to people and programs after appropriate authentication and authorization. It includes the following password disclosure capabilities:

To users, via a web interface, subject to access control policy.
To users who do not have pre-programmed password disclosure rights, after approval by pre-defined authorizers.
To applications, in order to replace embedded passwords, using an API (application programming interface) where applications authenticate using an OTP (one time password) and may only connect from a pre-defined range of IP addresses.
To service launching programs, such as the Windows Service Control Manager, by writing new password values to the appropriate locations after a successful password change.

Note that all disclosure is subject to SSL encryption, strong, personal authentication, access controls or workflow approval and audit logs.

Frequent Users: Access Controls

The most common form of access control in the Privileged Password Manager is based on resource groups. Resource groups are named collections of devices where privileged passwords are managed and to which policies are applied.

Resources can either be attached to a group explicitly (e.g., "attach workstation WKSTN01234 to resource group RGWKSTNS") or implicitly, using an expression. Expressions may be based on the operating system type, IP address, MAC address or workstation name (e.g., "attach every workstation running Windows XP in subnet 10.1.2.3/24 to resource group X")

Policies applied to resource groups include:

Which accounts' passwords to randomize.
How often to change passwords.
How to compose random passwords (e.g., length, complexity, etc.).
What actions to take after successful or failed attempts to disclose a password.
What access disclosure methods to allow (e.g., launch RDP, launch SSH, temporary group membership, display password, etc.).

Privileged Password Manager users are likewise grouped into user groups, either explicitly or implicitly (i.e., via membership in a user group on a target system, such as Active Directory). Groups of console users are granted specific rights to resource groups. Rights include the right to display member devices, to establish login sessions to privileged accounts, etc.

Business policies, such as segregation of duties between different groups of IT administrators, can be enforced by assigning users to distinct user groups, each with access to different (non-overlapping) resources and accounts.

Occasional Users: Workflow Approval

Privileged Password Manager includes the same authorization workflow engine as is used in other Hitachi ID Systems products -- Hitachi ID Identity Manager, Hitachi ID Access Certifier and Hitachi ID Group Manager. Workflow enables one user to request release of a given password. When this happens, one or more other users are invited (via e-mail) to review and approve the request. Approved requests trigger an e-mail to the password recipient, including a URL to Privileged Password Manager where he or she can re-authenticate to display the requested password or launch a login session to the device in question.

The workflow process is illustrated by the following series of steps:

User UA signs in and requests that the then-current password to login account LA on system S be made available to user UB at some later time T. UA may or may not be the same person as UB.
Privileged Password Manager looks up authorizers associated with LA on S.
Privileged Password Manager may run business logic to supplement this authorizer list, for example with someone in the management chain for UA or UB. The final list of authorizers is LA. There are N authorizers but approval by just M (M <= N) is sufficient to disclose the password to AZ.
Privileged Password Manager sends e-mail invitations to authorizers LA.
If authorizers fail to respond, they get automatic reminder e-mails.
If authorizers continue to fail to respond, Privileged Password Manager runs business logic to find replacements for them, effectively escalating the request and invites the replacement authorizers as well.
Authorizers receive invitation e-mails, click on a URL embedded in the e-mail invitation, authenticate themselves to the Privileged Password Manager web login page, review the request and approve or reject it.
If any authorizers reject the request, e-mails are sent to all participants (UA, UB and AZ) and the request is terminated.
If M authorizers approve the request, thank-you e-mails are sent to all participants. A special e-mail is sent to the recipient -- UB with a URL to a password disclosure page.
UB clicks on the e-mail URL and authenticates to Privileged Password Manager and displays the password.
UB clicks on a button to "check-out privileged access."
UB then may click on a button to do one of the following (the options available will vary based on policy):
1. Display the password.
2. Place a copy of the password in the operating system copy buffer.
3. Launch an RDP, SSH or similar remote control session to the server in question.
In other words, display of a sensitive password is not a mandatory part of the solution.

Concurrency Controls -- Checkin/Checkout

Privileged Password Manager can be configured to track and control the number of people to whom a given password is disclosed at any given time. This is done using the concept of password checkout and checkin -- in a manner similar to checking a book out of a library and checking it back in later.

Rather than simply granting access to a privileged account, a user may be required to check out access. Checkout is subject to policy control:
1. A counter is incremented whenever access is checked out, indicating that one more person is allowed to sign into the account in question.
2. The number of users who may concurrently access an account is limited -- for example, up to two at a time.
3. The time interval during which a user may be allowed to sign into an account is limited -- for example, no more than two hours.
Users are asked to check-in access rights when they are done using a privileged account.
1. The account's checkout counter is decremented.
If the maximum allowed checkout time has elapsed, Privileged Password Manager may automatically perform a checkin. This normally causes the account's password to be re-randomized.
Checkin and checkout supports coordination among IT workers:
1. Privileged Password Manager can notify users who have already checked out an account of new checkouts (e.g., via e-mail or SMS).
2. Privileged Password Manager can notify users who are newly checking out an account of existing checkouts (e.g., on the web UI).
Passwords are normally randomized whenever the checkout counter returns to zero.

Alternatives to Password Disclosure

Privileged Password Manager controls access by users and programs to privileged accounts on systems and applications. By default, that means randomizing and disclosing current password values. Display of password values is not a recommended part of the process, however:

IT staff can directly launch Terminal Services (RDP), SSH (PuTTY) and other connections to target systems from the Privileged Password Manager web user interface, without displaying a password value.
IT staff can use an ActiveX control embedded in the Privileged Password Manager web UI to place a copy of a sensitive password into their OS copy buffer, again without displaying the passwords. This password is automatically cleared from their copy buffer after a few seconds.
Privileged Password Manager can dynamically attach a recipient's Active Directory domain login ID to a local security group on a target system and later remove it. This eliminates the need to disclose passwords even to a software agent on the recipient's workstation.
Where password display is required (e.g., a target system is currently offline), JavaScript in the Privileged Password Manager web UI removes it from the screen after a few seconds.

A policy defined for each group of resources in Privileged Password Manager determines which of these access disclosure mechanisms is available to each group of users. For example, password display may be allowed for Windows workstations, since they may be inaccessible over the network, but only RDP sessions (with no possibility of disclosure) may be allowed for Windows servers.

API for Progammatic Disclosure

Privileged Password Manager includes an API that enables applications to disclose passwords and eliminates the storage of static, plaintext passwords. Privileged Password Manager periodically randomizes service passwords, while applications use the API to retrieve passwords as/when required.

The Privileged Password Manager API is accessed using SOAP over HTTPS.

For example, Privileged Password Manager may randomize an Oracle DBMS login password every 24 hours. Web applications which use the password to establish database connections can periodically sign into Privileged Password Manager with their own credentials (see below) and retrieve the current Oracle login password.

An important design consideration when implementing a privileged password retrieval API is how the client which requests password disclosure (the web application in the above example) authenticates itself to the API service. Privileged Password Manager secures this process with a combination of ACLs, one-time passwords and IP subnets:

API clients have their own IDs, used to sign into Privileged Password Manager.
These IDs are attached to console user groups and assigned ACLs, allowing them to disclose certain passwords.
API client login IDs are assigned one-time passwords (OTPs). In effect, the password used by the service to sign into the Privileged Password Manager API changes to a new, random string on each API connection.
API client login IDs are bound to IP subnets. They can only sign into the API from given IP ranges.

Wrapper code is provided for the SOAP API for a variety of platforms / programming languages, such as .NET, Java, etc. This wrapper code manages several functions:

Storing the one time password used to authenticate to the API.
Keeping cached copies of passwords retrieved from the API, along with data about how long to retain those copies and how long they should be assumed to be valid.
Encrypting the above, sensitive data so that it's not visible -- even to locally privileged users.

Encryption of the OTP and of cached passwords implies an encryption key. The API wrappers support a variety of methods to produce this key, including:

A static key (e.g., embedded into the application or configuration file) -- useful during development or debugging.
A key generated from characteristics of the machine on which the application runs, such as its MAC addresses, IP addresses, hostname, etc.
A key generated from characteristics of the program which is calling the API (i.e., a cryptographic hash of the program itself).

Hitachi ID Systems is happy to add platform bindings for this wrapper code based on customer demand (i.e., we add support for the programming language and runtime that customers need as required, and usually at no additional cost).

Updates to Service Passwords

On the Windows operating system, service programs are run either using the SYSTEM login ID, which possesses almost every privilege on the system (and consequently can do the maximum harm) and which has no password or using a real user's login ID and password, in order to execute with reduced privileges. This means that on each Windows workstation and server there are a number of service accounts, each with its own password, which are used to run service programs such as web servers, backup agents, anti-virus software, etc.

Service account passwords differ from administrator passwords in that they are stored in at least two places:

Hashed, in the security database -- e.g., the local SAM database or Active Directory, just like all users.
Reversibly encrypted, in the registry or elsewhere, where the program that starts the service (e.g., Service Control Manager or similar) can retrieve it when it needs to start the service.

Other Windows components besides the Service Control Manager also store passwords twice:

Virtual directories used to access web content from the IIS web server.
Programs scheduled to be run by the Windows Scheduler.

Third party programs may also require passwords to be stored outside the Security Accounts Manager (SAM) database.

Of the above passwords, all but those used in IIS are static and may represent a security vulnerability.

Privileged Password Manager can be configured to secure service account passwords. This means two things, depending on the mode of operation:

In pull mode, the Privileged Password Manager workstation service periodically scrambles service account passwords locally, in coordination with the central Privileged Password Manager server cluster.
In push mode, Privileged Password Manager servers periodically connect to Windows servers in order to change the passwords of service accounts.

In both cases, Privileged Password Manager notifies the program that launches service accounts of the new password value, so that it can successfully launch the service at the time of the next system restart or when an administrator manually stops and restarts the service in question.

In push mode, Privileged Password Manager runs an exit program which remotely connects to the server in question and updates the secondary storage of the service password. Exit programs are provided with the base Privileged Password Manager software distribution to remotely update:

The Windows Service Control manager.
The Windows Scheduler.
The IIS web server.

Privileged Password Manager implementers, Hitachi ID Systems and integrators can write additional exit programs to update service passwords used by other programs, stored in other locations. These are typically command-line programs (Windows executable or script) that run on the Privileged Password Manager server.

In pull mode, the Privileged Password Manager workstation service can use a DLL to update local passwords. DLLs are provided for the same Windows components as the push-mode exits above and implementers can write new DLLs to update passwords for other types of accounts.

Strong Authentication

(1)Privileged Password Manager can be configured to take advantage of an existing directory of users for authentication and authorization:

Users may sign into Privileged Password Manager with their Active Directory or LDAP login ID and password.
Users may be required to authenticate with a two-factor technology, such as an RSA SecurID token.
User membership in Privileged Password Manager security groups and consequently user privileges, may be based on user membership in AD or LDAP groups.

Externalizing user identification, authentication and authorization can significantly reduce the administrative overhead of managing a Privileged Password Manager deployment and is recommended.

Starting with version 7.1, Privileged Password Manager will support multi-step authentication. For example, a user may be required to type their AD password and then a PIN which was sent to their mobile phone via SMS.

(2)Administrators (IT staff) authenticate to the Privileged Password Manager web GUI as follows:

By typing a current network OS or directory password.
By typing a password and validating it against a password hash stored inside Privileged Password Manager itself.
Using a security token (e.g., SecurID pass-code or similar).
Using a PKI certificate or smart card.

Auditing and Regulatory Compliance

Privileged Password Manager logs all attempted and completed password updates. This data can be used to track not only current privileged passwords for workstations and servers, but also device IP addresses and network status.

Privileged Password Manager also logs all attempts by users to look up devices and to display passwords. This creates a chain of accountability, making it clear who accessed what device and when and also who attempted to access a device and was blocked by policy or failure to get approval.

Exit traps can be used to forward copies of Privileged Password Manager log entries to another system (e.g., an SIEM) for analytics and tamper-proof archive.

Privileged Password Manager includes event reports, which make it possible to see, among other things:

Who disclosed passwords to given resources.
How often any given password was disclosed.
When and how often passwords were changed on target systems.
How often users attempted to sign into Privileged Password Manager.
What the results of those authentication attempts were.

Reports are also included to examine the set of discovered / managed systems and accounts.

The Privileged Password Manager schema is well documented and the database is a standard, relational SQL back-end. This makes it possible for Hitachi ID Systems customers to write custom reports using off-the-shelf programs such as Crystal Reports or Cognos BI.

By recording administrative access to key systems and in some cases by requiring multiple people to approve such access before it happens, Privileged Password Manager can both limit and record access to sensitive systems that contain privacy-protected or financial data. These controls assist in complying with regulations such as HIPAA, SOX, PCI and more.

Privileged Password Manager Architecture

Network Architecture

Figure [link] illustrates the network communication paths in a typical Privileged Password Manager deployment, where Privileged Password Manager pushes passwords to fixed target systems -- servers, applications, network devices, etc.

Privileged Password Manager Push-Mode Network Architecture Diagram (3)

In the diagram:

Three distinct physical sites are shown, each surrounded by a dotted-line border.
Two Privileged Password Manager servers are deployed, to two different sites. Real-time replication provides for resiliency in the event of a hardware failure on a single server or a complete outage at either site.
The Privileged Password Manager servers run on Windows 2003 or Windows 2008. This platform provides the widest possible range of client software, making Privileged Password Manager easy to integrate with many kinds of target systems.
Stored passwords are encrypted (using AES). The encryption key is kept in the registry of each Privileged Password Manager server and is itself encrypted using a key embedded in the Privileged Password Manager software.
Each Privileged Password Manager server has a complete, local copy of the entire password database along with all configuration information.
Data replication traffic between the two servers is encrypted, making it resistant to snooping or tampering by a man-in-the-middle attacker.
Periodically, each Privileged Password Manager server connects to target systems and pushes new passwords to them. The protocol used depends on the type of target system, with two examples shown: LDAPS or NTLM for Windows servers, SSH to Unix or Linux servers and an encrypted TCP/IP connection to Unix targets that do not have an SSH service but do have a local Privileged Password Manager listener.
Some target systems may be unreachable directly, because of intervening firewalls. These may be contacted indirectly using a Privileged Password Manager proxy server, co-located with the target system. In this scenario, communication from the primary Privileged Password Manager server to the target system is via an arbitrarily-numbered TCP/IP connection and AES encryption using a shared key. The connection is forwarded to the target system by the proxy, using that target system's native protocol.
Privileged Password Manager clients, such as IT workers or applications that use Privileged Password Manager in place of embedded passwords, connect to Privileged Password Manager over HTTPS. Since multiple Privileged Password Manager servers are available and each of them contains a full data set, this connection can be load balanced.

Push and Pull Modes

Privileged Password Manager supports both server passwords, in "push mode," and workstation passwords, in "pull mode:"

When managing passwords on servers, Privileged Password Manager normally operates in "push mode." This means that periodically the Privileged Password Manager server will initiate communication with each target system, using connectors installed on the Privileged Password Manager server and randomize privileged passwords on that target system.

The new password(s) will be encrypted and archived in the Privileged Password Manager server's replicated storage, where IT staff may retrieve them.

When managing passwords on workstations, Privileged Password Manager normally operates in "pull mode." This means that a local agent is installed on each workstation and this agent software periodically contacts the central Privileged Password Manager server, over HTTPS, to request new administrator passwords.

Once the local password has been set, a confirmation is sent to the Privileged Password Manager server, which stores the new value. The new password(s) are encrypted and archived in the Privileged Password Manager server's replicated storage, where IT staff may retrieve them.

The same approach can be used not just on workstations, but also to any type of device which is numerous (e.g., tens of thousands of devices) and/or which is only sporadically reachable over the network. Examples include "server farms" of Windows or Unix servers or target systems at remote locations.

Privileged Password Manager Host Platform

Privileged Password Manager must be installed on a Windows 2003 or Windows 2008 server.

Installing on Windows 2003 or Windows 2008 allows Privileged Password Manager to leverage client software for most types of target systems, which is available only on the "Wintel" platform. In turn, this makes it possible for Privileged Password Manager to manage passwords and accounts on target systems without installing a server-side agent.

The Privileged Password Manager server must also be configured with a web server. Since the Privileged Password Manager application is implemented as CGI executables, any web server will work. The Privileged Password Manager installation program can detect and automatically configure IIS or Apache web servers, but other web servers can be configured manually.

Privileged Password Manager is a security application and should be locked down accordingly. Please refer to the Hitachi ID Systems document about hardening Privileged Password Manager servers to learn how to do this. In short, most of the native Windows services can and should be removed, leaving a very small attack surface, with exactly one inbound TCP/IP port (443):

IIS is not required (Apache is a reasonable substitute).
No ASP, JSP or PHP are used, so these engines should be disabled.
.NET is not required on the web UI, so should be disabled on IIS.
No ODBC or DCOM are required inbound, so these services should at least be filtered.
File sharing should be disabled.
Remote registry services should be disabled.
Inbound TCP/IP connections should be firewalled, allowing only port 443 and possibly terminal services (if required for some configuration tasks).

Privileged Password Manager is designed to be secure. It is protected using a multi-layered security architecture, which includes running on a hardened OS, using file system ACLs, providing strong application-level user authentication, filtering user inputs, encrypting sensitive data, enforcing application-level ACLs and storing log data indefinitely.

Privileged Password Manager never requires plaintext passwords to be stored in configuration files or scripts and does not store plaintext passwords anywhere. Privileged Password Manager does not ship with a default administrator password -- one must be typed in at installation time.

These security measures are illustrated in Figure [link].

Network architecture security diagram (4)

Supported Target System Types

Privileged Password Manager supports management of passwords on workstations, which may be mobile, have dynamic addresses, get unplugged, etc. This client software works by "pulling" new, local administrator passwords from the Privileged Password Manager server cluster. Client software is available for:

Windows 2000, XP, Windows 7 and Vista, 2003, 2008 and 2008R2.
Unix (various vendors) and Linux (IA86).

The Windows pull-mode service includes plug-ins to update passwords on the Windows Service Control Manager, Windows Scheduler and IIS. Other plug-ins can be added with minimal effort.

Push mode agents, installed on the Privileged Password Manager server and designed to write new passwords to fixed-address target systems, are included for:

(5)

Directories:	Servers:	Databases:
Any LDAP, AD, NDS, eDirectory, NIS/NIS+.	Windows 2000, 2003, 2008, Samba, Novell, SharePoint.	Oracle, Sybase, SQL Server, DB2/UDB, ODBC.
Unix:	Mainframes:	Midrange:
Linux, Solaris, AIX, HPUX, 24 more.	z/OS with RAC/F, ACF/2 or TopSecret.	iSeries (OS400), OpenVMS.
ERP:	Collaboration:	Tokens, Smart Cards:
JDE, Oracle eBiz, PeopleSoft, SAP R/3, Siebel, Business Objects.	Lotus Notes, Exchange, GroupWise, BlackBerry ES.	RSA SecurID, SafeWord, RADIUS, ActivIdentity, Schlumberger.
WebSSO:	Help Desk:	HDD Encryption:
CA Siteminder, IBM TAM, Oracle AM, RSA Access Manager.	BMC Remedy, BMC SDE, HP Service Manager, CA Unicenter, Assyst, HEAT, Altiris, etc.	McAfee, CheckPoint.