Test Suite

Description

Solaris Versions and Architectures Tested

Install / Backout Tests

IBIB

Install / Backout / reInstall / reBackout (IBIB) tests. The reinstallation of the patch is designed to catch subtle patch corruption issues (e.g. to the pkg database or as a result of a non-reversed action in a patch install / backout script). IBIB removes the patch from the Test System to leave it "blank" for the next patch. On Solaris 10 a basic startup test of Oracle 10g is performed. During IBIB testing BSM auditing is tested after the initial patch install to ensure its continued operation.

Alternate Boot Environment IB

Alternate Boot Environment Install Backout testing is designed to mimic real world patch installation scenarios, including relocatability testing (see patchadd's -R option). Relocatability is critical for the correct operation of Live Upgrade and Diskless Clients. Patches required by the target patch are applied to the system and the target patch is then installed. From a separate partition, the target patch is then backed out and reinstalled to test relocatability (using luupgrade ) Finally, from the original partition, the target patch and its required patches are backed out. This leaves the system back in its original state. During the relocatability test (luupgrade ) , an interposition library is used to catch any common system calls (open(), kill(), creat (), etc.) from the patch scripts which are not relocatable. That is, calls from the patch scripts which erroneously act on the current partition rather than the boot environment specified on the command line. On Solaris 10 this is implemented using Dtrace rather than an interposition library.

Container IB

Container IB is designed to test the functionality of patch installation and removal on a system using Solaris Zones. A system with a sparse root zones and a whole root zone is used. Each patch is added to the system in the global zone, this also causes the patch to be installed in all local zones, and then backed out. As with the IBIB testing this procedure is repeated.

Patch Package Auditing

Patch Attributes Auditing

Audit tool to check for changes to the attributes of an object (such as permissions, object type, owner, group) since the last Solaris Update Release or FCS (First Customer Shipment) of the Marketing Release.

sun_patchchk

Audit tool to programmatically check for various potential patch packaging issues.

Application Compatibility Tests

Sun ONE Studio 7 C Compiler

In-house developed test suites to test that the Sun ONE Studio 7 C compiler functionality continues to work on a patched Solaris system

Sun ONE Studio 7 C++ Compiler

In-house developed test suites to test that the Sun ONE Studio 7 C++ compiler functionality continues to work on a patched Solaris system

MySQL 3.23.39

Test suite bundled with MySQL. Used to test that MySQL functionality continues to work on a patched Solaris system.

OCE for Oracle 10g 32-bit & 64 bit

Oracle's Oracle Certification Environment. Used to test that Oracle functionality continues to work on a patched Solaris system.

OCE for Oracle 9.2.0 32-bit & 64 bit

Oracle's Oracle Certification Environment. Used to test that Oracle functionality continues to work on a patched Solaris system.

OCE for Oracle 8.1.7 64-bit

Oracle's Oracle Certification Environment. Used to test that Oracle functionality continues to work on a patched Solaris system.

OCE for Oracle 8.1.7 64-bit

Oracle's Oracle Certification Environment. Used to test that Oracle functionality continues to work on a patched Solaris system running SunCluster 3.0.

OCE for Oracle 8.0.5 32-bit

Oracle's Oracle Certification Environment. Used to test that Oracle functionality continues to work on a patched Solaris system.

Perl 5.8.3

Test suite bundled with Perl. Used to test Perl functionality continues to work on a patched Solaris system.

Perl 5.6.1

Test suite bundled with Perl. Used to test Perl functionality continues to work on a patched Solaris system.

Perl 5.00503

Test suite bundled with Perl. Used to test Perl functionality continues to work on a patched Solaris system.

WebTest SunONE WebServer 6.0 SP 4

In-house developed test suite to test WebServer functionality continues to work on a patched Solaris system.

WebTest SunONE WebServer 6.0 SP 4

In-house developed test suite to test WebServer functionality continues to work on a patched Solaris system running SunCluster 3.0. A node failover is triggered during the test suite execution to check that SunCluster failover is unaffected by the Solaris patches.

WebTest for Apache WebServer 2.0.39

In-house developed test suite to test WebServer functionality continues to work on a patched Solaris system.

WebTest for Apache WebServer 1.3.12

In-house developed test suite to test WebServer functionality continues to work on a patched Solaris system.

WebTest Sun ONE Web Server 6.1 (as bundled in the JavaES)

In-house developed test suite to test WebServer functionality continues to work on a patched Solaris system.

SCATE

Sun Cluster Automated Test Environment. It consists of low-level cluster functionality tests, and application tests, which verify that clustered data services (such as Oracle) operate correctly on patched systems.

gtkvts

Gtkvts verifies that GTK+-2.0 function behavior is compliant to the GTK+ API specification on patched Solaris systems.

JES_MQ Longevity

This test suite tests the integrity of the Sun Message Queue Broker on patched Solaris systems.

JCK-runtime

The JCK-runtime ensures that the JDK components conform to the Java specifications on patched Solaris systems.

Solaris Functional Tests

x11perf

X11 Performance Benchmark used to test that X11 functionality continues to work on a patched Solaris system. This tests the Sun X server as well as the Xorg X server.

Zones

In house developed test suite to test the functionality of Solaris Zones. Used to test that zones function correctly on a patched Solaris system.

ZFS

In house developed test suite to test the functionality of ZFS. Used to test that ZFS pools can be created and modified on a patched solaris sytem.

Trusted Solaris

In house developed test suite to test the functionality of Trusted Solaris. The testsuite tests that Trusted Solaris functions correctly after applying patches to the system.

Load and Stress Tests

Liverpool

In-house developed Load and Stress test suite which mimics a University environment. Used to test a patched Solaris system.

LST

In-house developed Load and Stress Test used to test the robustness of a patched Solaris system.

Interoperability Tests

Interop

In-house developed test suite to test rsh, cpio, and rcp functionality continues to work between systems running different patched releases of Solaris, both SPARC, amd64 and x86.

Largefile

In-house developed test suite to test that the transfer of very large files (>2GB) continues to work between patched systems running different releases of Solaris on SPARC, amd64 and x86.

Sensitive Date Test

JES Testing

Hivemind

The methodology used to test JES is "scenario-based" testing. "Scenario-based testing" is a methodolgy that describes a test as a series of independent actions that are executed to achieve an end result. Such actions (tasks) are small, manageable and independent. These are the building blocks for the tests (scenarios). A large array of tests can be constructed by "piecing" together a set these building blocks. In the context of testing, this "piecing together" is done by describing where the action should be executed and when that action should be started and/or stopped in the testing environment. The HiveMind framework currently being used for the JES System Level Testing is based and supports this methodology of testing. PST use HiveMind to test JES Release 3 and later.

Performance QA Benchmarking

CDEbench

CDEbench is a benchmark developed by the CDE team in Solaris Software.

SpecWeb 99

SPECweb99 is a software benchmark designed to measure a system's ability to act as a web server for static and dynamic pages.

SpecWeb SSL 99

SPECweb SSL 99 is a software benchmark designed to measure a system's ability to act as a web server for static and dynamic pages over secure connections.

SpecJAppserv2002

Industary Standard Application Server Benchmark see http://www.spec.org for further details.

Netbench

NetBench is a portable benchmark program that measures how well a file server handles file I/O requests from 32-bit Windows clients, which pelt the server with requests for network file operations. NetBench reports throughput and client response time measurements. To run NetBench, you need a file server, a PC running Windows XP (called the controller) to start and monitor the tests, and clients to actually run the tests.

SpecJBB2000

SPECjbb2000 is a Java program emulating a 3-tier system (the most common type of server-side Java application today), with emphasis on the middle tier. Random input selection represents the 1st tier user interface. It fully implements the middle tier business logic and object manipulation. The 3rd tier database is replaced by binary trees.

Volano

VolanoMark is a 100% Pure Java server benchmark characterized by long-lasting network connections and high thread counts.

DirectoryMark

LDAP Benchmark, see http://www.mindcraft.com/directorymark for more details.

Kenbus

Kenbus is a multi-tasking workload which executes UNIX commands along with several compiles of C programs. The commands used are typical of those used in a research and development environment. Each Kenbus script executes 18 UNIX commands including cc, cat, grep, mkdir, and rm. The performance metric reported is the peak throughput in scripts/hour.

NetPerf

NetPerf is a networking performance benchmark / tool. The current version includes throughput (bandwidth) and request / response (latency) tests for TCP and UDP.

NTTCP

NTTCP is a more convenient version of the TTCP program. It uses inetd (or simulates its behavior) to start off the remote side program which will send / receive data. It measures both TCP and UDP transmissions and gives individual statistics for each.

Oracle Database Build Time

This benchmark measures the time taken to build a medium sized TPC-C style Oracle database with various disk configurations. The build process is made up of the following phases: Tablespace and Object Creation, Data Load, Index Creation, and Analysis of Tables and Indexes. The benchmark is run using Oracle 9i on Raw Devices, UFS with various tunable parameters, and VxFS.

SPECglperf

GLperf is an OpenGL primitive-level performance measurement tool distributed by the OpenGL Performance Characterization (OPC) Committee. It measures performance of specific OpenGL primitives, like triangle strips and quads, under a set of specifiable states.

TTCP

TTCP times the transmission and reception of data between two systems using the UDP or TCP protocols.

XEngine

Xengine is a reciprocating engine for X. Its results are based on the number of cycles per second of a recurring function are completed.

XMar

Excerpts from "Xmark93 Provides Standardized X Performance Measurement" by Susan Maxwell, NCD, and David Cooper, HP, The XPC Journal, The GPC Quarterly, 4th Quarter 1993. Xmark93 provides a standardized way to measure the performance of computer systems running the X Window System.

Boottime

Homegrown benchmark for measuring system boottime.