TCP Port Conflict Between Sun Cluster for OPS/RAC and Solaris Secure Shell Server, and Possible Denial of Service Attack by Unprivileged Users Upon Sun Cluster

---

TCP Port Conflict Between Sun Cluster for OPS/RAC and Solaris Secure Shell Server.

Impact

During a cluster reconfiguration event (e.g. when another node joins or leaves the cluster), a Sun Cluster 3.x node can panic, and a Sun Cluster 2.2 node can abort from the cluster. Also, if a local unprivileged user is allowed to run client applications on a Sun Cluster system, they may be able to run an application which utilizes a TCP port which is also used by the DLM and thus trigger this issue, causing a Denial of Service.

Contributing Factors

This issue can occur in the following releases:

SPARC Platform

• Sun Cluster 2.2 (for Solaris 2.6, Solaris 7, and Solaris 8)
• Sun Cluster 3.0 (for Solaris 8, Solaris 9, and Solaris 10)
• Sun Cluster 3.1 (for Solaris 8, Solaris 9, and Solaris 10)
• Sun Cluster 3.2 (for  Solaris 9 and Solaris 10)
  

This issue can only occur if all of the following are true:

• The Sun Cluster Oracle OPS/RAC packages ORCLudlm and SUNWudlm are installed
• The Solaris Secure Shell server daemon (sshd(1M)) is running
• The system is configured to enable X11 forwarding

A TCP port conflict may occur for the above Sun Cluster releases which have the OPS/RAC packages installed (specifically the ORCLudlm and SUNWudlm packages) and the Solaris Secure Shell server daemon (sshd(1M)) running on the system configured to allow X11 forwarding. Sun ships Solaris Secure Shell with Solaris 9 only. Sites using earlier versions of Solaris may have installed a third party version of Secure Shell.

It is recommended that a Sun Cluster system should not be used for client applications as described in:

Sun Cluster 3.0 Release Notes

• http://docs.sun.com/db/doc/806-1428/

Sun Cluster 2.2 Software Installation Guide

• http://docs.sun.com/db/doc/806-1008/

Symptoms

For Sun Cluster 3.x

During an SC3.x cluster reconfiguration event (e.g. when another node joins or leaves the cluster), one or more of the active cluster nodes can panic with the following message:

    panic[cpu0]/thread=2a100045d20: Failfast: Aborting because "ucmmd" died 30 
    seconds ago

Note: This same panic can also occur for a number of other reasons. In this particular case, the system messages file (/var/adm/messages) will contain the relevant message:

    Oct 13 15:08:36 vha-2a ID[SUNWudlm.udlm]: Unix DLM initiating cluster abort.

The DLM log file (/var/cluster/ucmm/dlm_<node-name>/logs/dlm.log) will also contain the relevant message:

    15:08:36-00000-00620- BIND ERROR: 'Address already in use', family=2, port=6007,
    in=172.16.193.1

For Sun Cluster 2.2

During an SC2.2 cluster reconfiguration event (e.g. when another node joins or leaves the cluster), one or more of the active cluster nodes can abort from the cluster. The system messages file (/var/adm/messages) will contain the messages:

    Nov  4 10:40:03 vha-2a ID[SUNWcluster.udlm.4004]: Unix DLM initiating cluster
    abort.
    Nov  4 10:40:03 vha-2a ID[SUNWcluster.clustd.transition.4008]: transition
    'step3' failed. Child exit status 15
    Nov  4 10:40:13 vha-2a ID[SUNWcluster.clustd.transition.4010]: cluster aborted
    on this node (vha-2a)

The DLM log file (/var/opt/SUNWcluster/dlm_<node-name>/logs/dlm.log) will also contain the relevant message:

    10:40:03-00000-00858- BIND ERROR: 'Address already in use', family=2, port=6004,
    in=204.152.65.33

Workaround

If Solaris Secure Shell X11 forwarding is not required, disable it by editing the server configuration file on all cluster nodes as follows:

    # cd /etc/ssh
    # cp sshd_config sshd_config.save
    # vi sshd_config

Change:

    X11Forwarding yes

To:

    X11Forwarding no

Then restart the sshd daemon:

    # /etc/init.d/sshd stop
    # /etc/init.d/sshd start

For Solaris 10, restart the sshd daemon as follows:

    # svcadm disable network/ssh
    # svcadm enable network/ssh

If Solaris Secure Shell X11 forwarding is required, the DLM should be reconfigured to use an alternate, unused range of TCP port numbers. The system administrator should use the "/etc/services" file and their knowledge of the TCP applications that run on the cluster nodes when choosing a new port range for the DLM.

In the following three examples (for Sun Cluster versions 3.1 update 1, 3.1/3.0 and 2.2), the DLM's port range is changed to the currently unassigned range of 1124-1154 (inclusive).

Example 1: Sun Cluster version 3.1 update 1

The following procedure for changing extension properties is also described in the Sun Cluster 3.1 Data Service for Oracle Parallel Server/Real Application Clusters Guide, in the section "How to Modify an Extension Property That Is Tunable Only When a Resource Is Disabled".

First, shutdown all Oracle OPS/RAC instances running on all nodes in the cluster.

Disable all RAC framework resources:

    # scswitch -n -j rac_cvm
    # scswitch -n -j rac_udlm
    # scswitch -n -j rac_hwraid
    # scswitch -n -j rac_framework

Take the RAC framework resource group offline, and change it to the unmanaged state:

    # scswitch -F -g rac-framework-rg
    # scswtich -u -g rac-framework-rg

Reboot all the nodes in the cluster. During the restart, the "rac-framework-rg" resource group will not be started automatically on any node, which allows you to alter the port range:

    # scrgadm -c -j rac_udlm -x Port=1124 -x Num_ports=30

Start the "rac-framework-rg" resource group:

    # scswitch -Z -g rac-framework-rg

All Oracle OPS/RAC instances can now be restarted on all nodes in the cluster.

Example 2: Sun Cluster versions 3.0 and 3.1

Reconfigure the DLM's port range by editing the DLM configuration file on all cluster nodes as follows:

    # cd /opt/SUNWudlm/etc
    # cp udlm.conf udlm.conf.save
    # vi udlm.conf

Change:

    udlm.port               : 6000
    udlm.num_ports          : 32

To:

    udlm.port               : 1124
    udlm.num_ports          : 30

Note: This operation must be performed on ALL nodes in the cluster, and ALL nodes must be configured to use exactly the same port range. When the changes have been made, all cluster nodes must then be rebooted simultaneously for the changes to take effect.

On one node run:

    # scshutdown -y -g0

Example 3: Sun Cluster version 2.2

Reconfigure the DLM's port range by editing the cdb configuration file on all cluster nodes as follows:

    # cd /etc/opt/SUNWcluster/conf
    # cp <clustername>.cdb <clustername>.cdb.save
    # vi <clustername>.cdb

Change:

    udlm.port               : 6000
    udlm.num_ports          : 32

To:

    udlm.port               : 1124
    udlm.num_ports          : 30

Note: This operation must be performed on ALL nodes in the cluster, and ALL nodes must be configured to use exactly the same port range. When the changes have been made, all cluster nodes must then be removed simultaneously from the cluster and then rejoined for the changes to take effect.

On all nodes, run:

    # scadmin stopnode

Then on one node:

    # scadmin startcluster <nodename> <clustername>

And on the remaining nodes:

    # scadmin startnode

The three examples provided above avoid the problem of a port conflict between the DLM and Secure Shell, but do not protect against malicious denial of service attacks from unprivileged users logged into the cluster nodes.

To avoid possible denial of service attacks on the DLM, it's port numbers should be changed to a privileged range (i.e. a range below 1024). Again, the system administrator should use the /etc/services file and their knowledge of the TCP applications that run on the cluster nodes when choosing a new port range for the DLM. The range 918-949 may be suitable for most systems, as it is currently unassigned by the Internet Assigned Numbers Authority (IANA) e.g.

    udlm.port               : 918
    udlm.num_ports          : 32

Resolution

There is no resolution to this problem incorporated into the issued product(s). Please change the DLM's port range using the methods described above to values that suite your own systems.

This Sun Alert notification is being provided to you on an "AS IS" basis. This Sun Alert notification may contain information provided by third parties. The issues described in this Sun Alert notification may or may not impact your system(s). Sun makes no representations, warranties, or guarantees as to the information contained herein. ANY AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE HEREBY DISCLAIMED. BY ACCESSING THIS DOCUMENT YOU ACKNOWLEDGE THAT SUN SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES THAT ARISE OUT OF YOUR USE OR FAILURE TO USE THE INFORMATION CONTAINED HEREIN. This Sun Alert notification contains Sun proprietary and confidential information. It is being provided to you pursuant to the provisions of your agreement to purchase services from Sun, or, if you do not have such an agreement, the Sun.com Terms of Use. This Sun Alert notification may only be used for the purposes contemplated by these agreements.

Copyright 2000-2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, CA 95054 U.S.A. All rights reserved.

Modification History

27-Apr-2004: Updated the Relief/Workaround section
13-Aug-2008: Updated Contributing Factors section
04-Sep-2008: Updated Workaround section

Attachments

This solution has no attachment