Release Process

From Crypto++ Wiki
(Redirected from Release Testing)
Jump to: navigation, search

This page describes the process used to release a version of the Crypto++ library. The process is a mashup of development, testing and then independent confirmations. The testing occurs on various CPUs, compilers, platforms, using different configurations. In addition, analysis tools are used as a security gate. The testing favors platforms that are used in practice today, like desktop, server and mobile operating systems.

The CPUs include commodity IBM PC compatible processors, like AMD and Intel; and mobile processors like ARM and MIPS. Platforms are selected based on availability and functionality (for example, CentOS 5 and OpenBSD are often used when security is a high priority). Optimizations include both -O2 and -O3 because GCC vectorizes at -O3.

Configurations include Debug (-DDEBUG -Og) and Release (-DNDEBUG -O{2|3|5|s}); C++ standard versions C++03, C++11, C++11, C++17; and C++ standard libraries include GNU's libstdc++ and LLVM's libc++. In addition it includes various combinations of defines, like NO_OS_DEPENDENCE, PREFER_BERKELEY_STYLE_SOCKETS and PREFER_WINDOWS_STYLE_SOCKETS.

A related topic is Release Versioning, and it discusses how the project names a version of Crypto++, what to expect when a version number changes, and how to locate a particular state of the source code in Subversion or Git.

Related pages are cryptest.sh and cryptest.nmake, and it discusses the test scripts for Linux, Unix and Windows used in the release process.

The Process

The process has six broad steps:

  1. Identify objectives
  2. Perform the work
  3. Measure performance
  4. Independent cross-validation
  5. Prepare the release
  6. Announce the release

Identify objectives

The community generally sets the course the library is taking. Features can be sensitive to the release cycle, depending on the impact to exiting users. For example, a change that requires all client code to be recompiled cannot occur at a maintenance release (a release where the last number changes). Changes that require recompilation usually occurs at a major version bump (a release where the first number changes). See Release Versioning for more details.

Sometimes a release slips a bit as planned features are added. Sometimes features are removed if they could cause us to miss a date. Other times there may be spare cycles to add more than was planned.

Also see the Crypto++ Roadmap for the approximate list of features and when they could be added to the library.

Perform the work

The work is performed by folks who have the spare cycles. Sometimes its easy on Wei and the delegates because the only thing needed is a pull or merge request. Other times its more difficult, like adding additional functionality from the ground up. At other times, the work is monotonous, like rounding out test cases and test scripts to ensure a stable release.

The documentation (if any :) is added during the development phase while its fresh. Its not as bad as it sounds, though. At Crypto++ 5.6.3, which was a maintenance release, nearly 17,000 lines of Doxygen documentation was added. In addition, wiki pages covering new classes was added.

Measure performance

Throughout the process we measure progress to ensure we are meeting goals and objectives. Work is performed offline, outside of master. On occasion, a minor commit may occur that precedes a full release check-in.

Measuring performance includes security testing and evaluations. In this step, we ensure specific tools are applied to the work in progress to avoid latent and not so readily apparent bugs. This step often depends on a complete set of positive and negative test cases, and is carried out by tools like Undefined Behavior Sanitizer, Valgrind, Enterprise Analysis and Coverity.

At this stage, the script cryptest.sh (Unix, Linux, OS X) and cryptest.cmd (Windows) is run often on multiple platforms. cryptest.sh and cryptest.cmd are comprehensive testing scripts that ensure the library is well exercised.

The process can take one of two paths at this point. If there are problems or the work is incomplete, then the process loops back to the perform the work stage. If the work is complete and scripts are successful, the process can move towards release approval.

Independent cross-validation

Independent validation is important to ensure biased and myopic views are avoided. At any point in time, anyone is free to test the code and report issues or make suggestions. The mailing list and bug reporter are not a hostile environment to those doing the reporting.

All bugs are available to anyone who wants to see them. In addition, results from tools like Coverity are available to anyone who wants to see them. We take the position because risk cannot be mitigated when critical information is withheld from the folks who need it.

Prepare the release

When the interested parties and stakeholders feel the library is ready for release, a "gold" ZIP is built from sources. The ZIP is made available for download on the website, and the same sources used to build the ZIP are committed to source control.

There may be a few errata items against the ZIP at this point, but all the major work has been completed. If there are errata items then they are cleared, a new ZIP is uploaded and the change is checked into source control.

Once the library is stable in the various environments, we ask the "Crypto++ Primaries" to approve a release. The primaries are like a Board of Directors with Wei Dai as the Chairman and CEO. Folks on the board often have limited cycles to share, so we try not to disturb them too often. We also solicit community feedback.

Announce the release

After approval is provided by the primaries and users, we perform two final tests. First, we test the download of the ZIP and check-out of the code from version control and ensure there are no material differences between them. Second, we ensure the library builds and executes successfully with a make check. We do this for both the ZIP and the check-out to detect potential irregularities that can occur at the "ZIP and commit" stage.

Once the library passes the final validations, the hash of the ZIP is published, the source code is tagged in version control and the release announcement is made. The hash of the ZIP is created with a second, distinct security-centric program such as OpenSSL. The release announcement is made on three lists. The three lists are Crypto++ Announce list, Crypto++ Users list and sci.crypt Usenet group. In addition, an email is sent to a number of distributions and package maintainers.

The OpenSSL commands are openssl {sha1|sha256|sha512|whirlpool} cryptoppXXX.zip

Increment version numbers

After releasing the library the project's compatibility should be bumped. For example, if Crypto++ 5.6.4 was released, then the compatibility number should be changed from 4 to 5 in in 5.6.5. And if Crypto++ 5.7.0 was released, then the version should be changed to 5.7.1.

This change helps with user questions that start with "I'm using Crypto++ 5.6.3 and I'm having problems with ...". When the project encounters the question, its unclear if its the 5.6.3 release ZIP, or Master without the next version bump. The change also helps determine the actual compatibility level during development. That is, if Master advertises version information X.Y.Z, then it better be compatible with X.Y.

Also see Release Versioning and Post Release Increment on the wiki and the discussion Post-release version number increment on the user mailing list.

Self Tests

Crypto++ relies on two sets of self tests to test compatibility and provide functional evaluations. Both are invoked using cryptest.exe.

During comprehensive testing, cryptest.sh repeatedly builds the library and runs the self tests using different configurations.

cryptest.exe

The first set of self tests are the validation tests, and they are run with ./cryptest.exe v. The validation tests include testing the basic configuration, like the word size and endianess, and unaligned data access. The tests also include basic algorithm tests.

The second set of test are the test vectors, and they are run with ./cryptest.exe tv all. The test vectors are more comprehensive, and include a complete set of tests when available, like Brian Gladman's test suite for AES.

The library attempts to gather test vectors from independent sources to avoid false validations. However, its not always possible since not every standard and algorithm publishes test vectors. If a test vector is not available, then the library will create its own set and list the Source as Crypto++.

Analysis Tools

The source code is subjected to six different phases of analysis. Loosely speaking, a clean compile with reasonable warnings is a security gate, as is additional static and dynamic analysis. To help ensure a trouble free release, Crypto++ is tested with the following:

  • Compiler warnings
  • Clang and GCC Undefined Behavior Sanitizer (UBsan)
  • Clang and GCC Address Sanitizer (Asan)
  • Clang and GCC Bounds Sanitizer (Bsan)
  • Valgrind at -O1 optimizations
  • Visual Studio Enterprise Analysis
  • Coverity (anyone can view the defects)

Compiler Warnings

"Reasonable warnings" is subjective, but its roughly -Wall -Wextra under modern GCC (and compatible compilers), and /W4 under Visual Studio and the VC++ compilers. Older GCC compilers, like GCC 4.2.1 on OpenBSD, are not tested with elevated warnings because there's nothing new to find (it should have already been found under a modern GCC compiler), and its hard to manage false positives (older GCC compilers lack the pragma GCC diagnostic ignored).

Clang and GCC Sanitizers

The GNUmakefile has targets ubsan and asan, so testing support is built in. If you are using Clang 3.2 or GCC 4.8 (or higher), then issue make ubsan or make asan. The makefile will set the appropriate compiler options and Crypto++ defines. Once built, issue ./cryptest.exe v or ./cryptest.exe tv all to execute under the sanitizers.

GCC 6.0 offer -fsanitize=bounds-strict, and cryptest.sh enlists it if its available. Clang does not appear to offer it at the moment.

Valgrind

To test the library under Valgrind, build the library as normal using -O1 optimization level:

export CXXFLAGS="-DNDEBUG -g2 -O1"
make -j 4

Once built, issue ./cryptest.exe v or ./cryptest.exe tv all to execute under Valgrind.

A related page is Profilling on the Crypto++ wiki. It discusses some known leaks on different platforms, like OS X and Microsoft.

Enterprise Analysis

Under Visual Studio when Enterprise Analysis is available, the the library tests both the solution's cryptlib and cryptest projects.

Coverity

Coverity is an offline code analysis service. The project regularly tests under Unix, Linux, OS X and Windows. See the Coverity Scan page for the recipes we use and other details, like OS X workarounds.

The Crypto++ project reports are world readable, so you should be able to view them directly. Crypto++ currently enjoys a 0.01 defect rate based on about 92,800 lines of code.

False Positives

Compiler warnings and Coverity produce false positives on occasion. Sometimes that means Crypto++ has to work around a problem that does not really exist. For example, the following was recently checked-in because Clang flagged k as possibly uninitialized even though Decode initialized the value.

-   unsigned int k, count, repeater;
+   unsigned int k = 0, count = 0, repeater = 0;
    bool result = codeLengthDecoder.Decode(m_reader, k);
    if (!result)
        throw UnexpectedEndErr();

This is a known issue with Clang's inter-procedural analysis engine, and we simply view it as "playing and working nicely with the tools". If the compiler feels inclined, then it can discard the unneeded initialization during optimization.

UBsan, Asan and Valgrind operate on real data, and they do not produce false positives. That has two implications for Crytpo++. First, the library values complete sets of self tests because its an opportunity to help vet trouble spots. Second, UBsan, Asan and Valgrind are security gates, and the findings must be cleared (or the risk documented) to move beyond them.

The Mashup

The following is the mashup various CPUs, compilers and platforms tested before a release. Keep in mind the mashup implicitly includes variations, like Debug (-DDEBUG -Og) vs Release (-DNDEBUG -O{2|3|s}), C++03 vs C++11, and C++ standard libraries like GNU's -stdlib=libstdc++ vs LLVM's -stdlib=libc++. And these platforms are in addition to those tested by way of GNU's GCC compile farm.

The Mashup Matrix
Platform CPU Details or Comments
AIX 7.1 Power7 (PowerPC) via GNU's GCC compile farm
Android ARM and ARM64 Cross-compile, R10 NDK using STLPort C++ standard library
Android MIPS and MIPS64 Cross-compile, R10 NDK using STLPort C++ standard library
BeagleBone ARMv7 Debian 8.2, GCC 5.x compiler
CentOS 5 x86_64 GCC 4.1 compiler
Cygwin & Cygwin64 i686 and x86_64 Running on Windows 7
Debian 7 MIPS ci20 dev-board
Debian 7 & 8 i686 and x86_64 Modern GCC compiler
Debian Sid x86_64 Bleeding edge GCC compiler
Debian 8 ARMEL QEMU, embedded ARMEL chroot, potential issue with 5.2 cross-compiler
Debian 8 ARMHF QEMU and embedded ARMHF chroot (Beagleboards and compatible boards)
Debian 8 S/390 and S/390x QEMU and embedded S/390 chroot
Debian 8 X32 X32 chroot, potential issue with GCC 5.2 cross-compiler
Fedora 1 i386 GCC 3.2 compiler (requires GNU Make 3.81)
Fedora 22* i686 and x86_64 Latest GCC compiler
Mustang ARM64 ARM64 server board from Robert Nelson's test environment
LeMaker HiKey ARM64 ARMv8, Aarch64, CRC+Crypto
Pine64 ARM64 ARMv8, Aarch64, CRC+Crypto
ODROID C2 ARM64 ARMv8, Aarch64, lacks Crypto
Raspberry PI 3 ARHHF Wasted A53 CPU being operated as ARMHF
iOS 6 & iOS 8 ARM and ARM64 Cross-compile, modified Clang
MinGW i686 Running on Windows 7
FreeBSD x86_64
NetBSD x86_64
OpenBSD 5.7 x86_64 GCC 4.2.1 compiler
OpenBSD 5.8 x86_64 GCC 4.6 compiler
OS X 10.5 PowerPC GCC 4.0.1 compiler
OS X 10.7 & 10.8 x86_64 Ancient C++ standard library that claims to be C++11, modified Clang
OS X 10.9 x86_64 Modern C++ standard library, modified Clang
OS X 10.10 x86_64 Modern C++ standard library, Xcode 6.4, (native) Clang
Solaris Sparc and Sparc64 via GNU's GCC compile farm
Solaris 11.3 x86_64 DL360 G5
Ubuntu 12 & 14 i686 and x86_64 Long Term Support, modern GCC compiler
Ubuntu 14 x86_64 Intel ICC compiler
Ubuntu 14 arm64 via GNU's GCC compile farm
Windows XP x86_64 Visual Studio 2003 and 2005
Windows Vista x86_64 Visual Studio 2008 and 2010
Windows 7 x86_64 Visual Studio 2010 and 2012
Windows 7 x86_64 Visual Studio 2012 and Enterprise Analysis (/analyze)
Windows 8 x86_64 Visual Studio 2012 and 2015
Windows Phone 8 & 10 ARM Command line build using cryptest.nmake
Windows Store 8 & 10 ARM Command line build using cryptest.nmake
Windows Universal Platform ARM & x86 Command line build using cryptest.nmake

* Though Fedora 22 is listed, the latest Fedora is used because it provides the latest tools (and latest GCC) conveniently packaged into a distribution.

Notably missing from the list is Embarcadero (formerly Borland) C++ Builder. The project asked for a license, but the company decided against providing one. Also missind is the Comeau compiler and the Dinkumware Standard C++ library. The Crypto++ project wrote to both companies and asked for the tools to use for testing, but the emails went unanswered.

Also missing is MinGW (both i686 and x86_64). MinGW i686 appears to be abandoned. Attempts to test it result in constant stream of errors, like "sh: command not found" and similar. MinGW x86_64 appears to lack a download for its setup program, and clear instructions on how to setup the environment.

CPUs

Crypto++ is tested under the following CPUs:

  • i686, x86_64
  • ARM, ARM64
  • MIPS, MIPS64
  • Sparc, Sparc64
  • PowerPC

i686 and x86_64 are commodity PCs. They are implicitly tested during the course of platform testing.

ARM, ARM64, MIPS and MIPS 64 are tested under cross-compiles for iOS and Android.

The PowerPC is a big endian processor provided by a Apple PowerMac running OS X 10.5. The machine only exists for testing Crypto++ (the OpenSSL and Cryptlib projects use it on occasion).

In addition, the Crypto++ project uses GNU's GCC compile farm.

Compilers

Each compiler has its own personality, and each has different strengths and weaknesses. Microsoft's compiler is very forgiving or accommodating, while Intel's compiler is ruthless when it comes to undefined behavior. To help ensure a trouble free release, Crypto++ is tested under the following compilers:

  • Microsoft's VC++
  • LLVM's Clang
  • GNU's GCC
  • Oracle SunCC
  • Intel's ICC

Platforms

Each platform has its own personality, just like each compiler, and each platform provides a unique environment. To help ensure a trouble free release, Crypto++ is tested under the following platforms:

  • Microsoft Windows
  • OpenBSD, NetBSD and FreeBSD
  • Ubuntu and Fedora
  • Solaris 10 and 11
  • Apple OS X and iOS
  • Solaris
  • Android

In addition, the Crypto++ project uses GNU's GCC compile farm and a number of Debian QEMU/Chroot environments.

Testing Gaps

Testing suffers a few known gaps. The gaps are enumerated below. If you can help with them, please contact us.

  1. "ports" on OS X are not well tested. In particular, only MacPorts is tested; and Fink, Gentoo/Alt, Homebrew, Pkgsrc and Rudix are not tested.
  2. MinGW is not well tested. In particular, MinGW-i686 is tested on a Windows 7 machine. MinGW-64 is not tested on Windows 7 or 8, and MinGW-i686 simply does not work on a Windows 8 machine.
  3. Android and iOS mobile platforms are only tested under a test case. The test case is a default configuration provided by GNUMakefile-cross.
  4. Windows Phone and RT mobile platforms are not tested because the platforms are crippled to the point the library is probably insecure and nearly useless. Its probably insecure because the random number generator does not work. (Don't believe the Channel 9 videos on porting a DLL to Windows Phone).
  5. Bleeding edge distributions, like Debian Unstable, may need -DCRYPTOPP_DISABLE_ASM, -fno-inline and/or -fno-devirtualize due to problems caused by unstable optimizations.

Downloads

cryptest-sh.zip - archive with script to test Crypto++ using various configurations. The configurations include Debug, Release, -O1, -O2, -O3, -Os, C++03, C++11, C++14, C++17, GNU's libstdc++, LLVM's libc++, Undefined Behavior sanitizer, Address sanitizer and Valgrind.