PEM Pack

PEM Pack
Documentation
`#include <cryptopp/pem.h>`

The PEM Pack is a partial implementation of message encryption which allows you to read and write PEM encoded keys and parameters, including encrypted private keys. The additional files include support for RSA, DSA, EC, ECDSA keys and Diffie-Hellman parameters. The pack includes five additional source files, a script to create test keys using OpenSSL, a C++ program to test reading and writing the keys, and a script to verify the keys written by Crypto++ using OpenSSL.

PEM encrypted private keys use OpenSSL's key derivation algorithm EVP_BytesToKey (OpenSSL documentation at EVP_BytesToKey). The function is PKCS#5 v1.5 compatible for derived material up to and including 16 bytes. If the required key is over 16 bytes (for example, AES-256 needs 32 bytes), then a non-standard extension is engaged. Regardless of the key size, Crypto++ and OpenSSL will interop as expected because both libraries implement the same derivation algorithm. You can find the Crypto++ reimpmentation at OPENSSL_EVP_BytesToKey.

PEM encryption is an old format specified in Privacy Enhancement for Internet Electronic Mail: Part I: Message Encryption and Authentication Procedures. If given a choice, you should prefer a newer standard like PKCS #8.

There is no standard list of PEM objects or names, though a request was made for one on the IETF's PKIX mailing list. The PEM Pack tries to provide support for most of those provided by OpenSSL. You can get OpenSSL's list from the project's pem.h source file (located at <openssl src>/crypto/pem/pem.h).

There is a separate page on using X.509 certificates with the Crypto++ library. See the X509Certificate article for details.

Note well: this class is not part of the Crypto++ library. You must download the files below.

End of Lines

Around 1992 the IETF published RFC 1421, Privacy Enhanced Mail (PEM). The RFC specified a standard format with a BNF grammar, and stated the end-of-line characters as the carriage return/line feed pair CRLF. Over time many developers did not follow the standard. Instead the developers just used a newline — whatever it happened to be on the platform they were working on. On Windows a newline is a CRLF, on Linux a newline is a line feed LF, and on OS X a newline is a carriage return CR.

Around 2015 the IETF published RFC 7468, Textual Encodings of PKIX, PKCS, and CMS Structures. RFC 7468 legitimized the non-conforming practices by saying just about anything with an encapsulation header and Base64 encoding is a valid PEM encoding. This is not the first time the IETF has done this crap. In the past the IETF collected nearly everything developers were doing with X.509 certificates and published a standard that legitimized non-conforming practices in RFC 5280. RFC 5280's KeyUsage (KU) and ExtendedKeyUsage (EKU) is such a mess that it is nearly worthless as a control.

The PEM Pack will write certificates, keys and parameters according to RFC 1421 using CRLF as end-of-line. The PEM pack will also limit lines to 64 characters when writing an object. The PEM Pack will attempt to read certificates, keys and parameters using CRLF, CR, LF and missing end-of-lines. The PEM Pack does not limit the length of a line during a read operation.

Compiling and Testing

To compile the source files, simply drop them in your cryptopp folder and then execute Crypto++'s GNUmakefile. The library's makefile will automatically pick them up. Windows users should add the header and source files to the cryptlib project in their respective folders.

There are five C++ source files in the pack to add to Crypto++. The files are:

pem.h - the include file for the PEM routines used by applications
pem_common.h - the internal include file for common PEM routines not exposed to the application
pem_common.cpp - the source file for the common PEM routines
pem_read.cpp - the source file for the internal PEM load and read routines
pem_write.cpp - the source file for the internal PEM save and write routines

An additional file is provided for testing. The file is pem_test.cxx, and it should be compiled like any other cpp file and linked against the updated library.

The default behavior is to validate keys and parameters after reading them in Debug builds (but not Release builds). If you want the keys and parameters validated after reading them, then open pem_common.h and uncomment the define PEM_KEY_OR_PARAMETER_VALIDATION. Once compiled, changing PEM_KEY_OR_PARAMETER_VALIDATION has no effect.

When using PEM_KEY_OR_PARAMETER_VALIDATION, an OS random number generator must be available, like AutoSeededRandomPool. Do not build the library and the PEM Pack with -DNO_OS_DEPENDENCE because the define removes the OS random number generators.

The easiest way to test the PEM source files is drop them in your cryptopp directory, and then issue the following commands. The scripts will build and test for you.

$ cd cryptopp
$ make distclean
$ ./pem_create.sh && ./pem_verify.sh

Public API

The public API primarily consists of PEM_Load and PEM_Save routines. The read and write routines are overloaded to accept a BufferedTransformation and a public or private key type. The supported systems are RSA, DSA, EC (both ECP and EC2N) and Diffie-Hellman.

There are two helper routines to extract a PEM object from a stream, PEM_NextObject, and identify the PEM object, PEM_GetType.

PEM_Load

Each system has three read functions. For example, the routines to read RSA keys:

void PEM_Load(BufferedTransformation& bt, RSA::PublicKey& rsa);
void PEM_Load(BufferedTransformation& bt, RSA::PrivateKey& rsa);
void PEM_Load(BufferedTransformation& bt, RSA::PrivateKey& rsa, const char* password, size_t length);

When reading an encrypted key, you must specify the password and its length in case there's an embedded NULL character in the string. The encapsulated header includes the encryption algorithm, so there's no need to specify it.

In general, reading is deferred to Crypto++ and its various BERDecode* routines. On occasion, the PEM Pack needs to provide a modified routine. For example, for DSA keys, Crypto++ uses {version, x} (where x is the private key) from Certicom's ECC-1 while OpenSSL provides {version, p,q,g,y,x} (where y is the public element):

void PEM_LoadPrivateKey(BufferedTransformation& bt, DSA::PrivateKey& key)
{
    BERSequenceDecoder seq(bt);
    
      word32 v;                // check version
      BERDecodeUnsigned<word32>(seq, v, INTEGER, 0, 0);
    
      Integer p,q,g,y,x;
    
      p.BERDecode(seq);
      q.BERDecode(seq);
      g.BERDecode(seq);
      y.BERDecode(seq);
      x.BERDecode(seq);
    
    seq.MessageEnd();
    
    key.Initialize(p, q, g, x);
}

When there's a problem, the PEM pack will throw a Crypto++ exception; and not a custom exception. Be prepared to catch Exception, InvalidArgument and InvalidDataFormat exceptions (in addition to anything Crypto++ might throw, like a DER decode error or bad padding exception).

PEM_Save

Each PEM_Load routine has a corresponding write routine, so there are three write functions for each system. For example, in the case of RSA:

void PEM_Save(BufferedTransformation& bt, const RSA::PublicKey& rsa);
void PEM_Save(BufferedTransformation& bt, const RSA::PrivateKey& rsa);
void PEM_Save(BufferedTransformation& bt, const RSA::PrivateKey& rsa,
              RandomNumberGenerator& rng, const string& algorithm,
              const char* password, size_t length);

The RandomNumberGenerator is needed for initialization vectors used with some modes of operation. If you use a mode that does not need an initialization vector, they you can pass NullRNG().

When writing an encrypted key, you must specify the password, the password's length and the algorithm. The recognized algorithms are listed below.

AES-256-CBC
AES-192-CBC
AES-128-CBC
CAMELLIA-256-CBC
CAMELLIA-192-CBC
CAMELLIA-128-CBC
DES-EDE3-CBC
DES-EDE2-CBC
DES-CBC
IDEA-CBC

As with the read routines, the write routines defer to DEREncode* routines but sometimes need to provide an OpenSSL compatible key. For example OpenSSL expects {version, p,q,g,y,x} for DSA, so an override is provided:

void PEM_DEREncode(BufferedTransformation& bt, const DSA::PrivateKey& key)
{
    const DL_GroupParameters_DSA& params = key.GetGroupParameters();
    
    DSA::PublicKey pkey;
    key.MakePublicKey(pkey);
    
    DERSequenceEncoder seq(bt);

      DEREncodeUnsigned<word32>(seq, 0);
      params.GetModulus().DEREncode(seq);
      params.GetSubgroupOrder().DEREncode(seq);
      params.GetGenerator().DEREncode(seq);

      pkey.GetPublicElement().DEREncode(seq);
      key.GetPrivateExponent().DEREncode(seq);
    
    seq.MessageEnd();
}

If you need a new algorithm, then modify PEM_CipherForAlgorithm in both pem_read.cpp and pem_write.cpp. Be sure to test the new algorithm against OpenSSL since OpenSSL only provides the algorithms listed above in its various commands (like openssl genrsa). However, OpenSSL should recognize anything EVP_get_cipherbyname understands.

When there's a problem, the PEM pack will throw a Crypto++ exception; and not a custom exception. Be prepared to catch Exception, InvalidArgument and InvalidDataFormat exceptions (in addition to anything Crypto++ might throw).

PEM_NextObject

There's also a function that allows you to read the first key or parameter called PEM_NextObject. The function locates the first PEM object in src and places it in dest. PEM_NextObject essentially peeks into the stream, so the source buffer is unchanged if there is no PEM object present.

void PEM_NextObject(BufferedTransformation& src, BufferedTransformation& dest);

PEM_NextObject will silently discard any characters that proceed the PEM Object. The destination BufferedTransformation will have one line ending if it was present in source.

Internally, the various PEM_Load functions call PEM_NextObject. That means you can call PEM_NextObject and then PEM_Load; or you can simply call PEM_Load alone (and PEM_Load will call PEM_NextObject).

PEM_NextObject will parse an invalid object. For example, it will parse a key or parameter with -----BEGIN FOO----- and -----END BAR-----. The parser only looks for BEGIN and END (and the dashes). The malformed input will be caught later when a particular key or parameter is loaded.

On failure, InvalidDataFormat is thrown.

PEM_GetType

The final function attempts to classify a PEM object. The function is PEM_GetType, and its also called internally after PEM_NextObject.

PEM_Type PEM_GetType(const BufferedTransformation& bt);

The function returns one of the following enums:

PEM_PUBLIC_KEY
PEM_PRIVATE_KEY
PEM_ENC_PRIVATE_KEY
PEM_RSA_PUBLIC_KEY
PEM_RSA_PRIVATE_KEY
PEM_RSA_ENC_PRIVATE_KEY
PEM_DSA_PUBLIC_KEY
PEM_DSA_PRIVATE_KEY
PEM_DSA_ENC_PRIVATE_KEY
PEM_ELGAMAL_PUBLIC_KEY
PEM_ELGAMAL_PRIVATE_KEY
PEM_ELGAMAL_ENC_PRIVATE_KEY
PEM_EC_PUBLIC_KEY
PEM_ECDSA_PUBLIC_KEY
PEM_EC_PRIVATE_KEY
PEM_EC_ENC_PRIVATE_KEY
PEM_EC_PARAMETERS
PEM_DH_PARAMETERS
PEM_DSA_PARAMETERS
PEM_REQ_CERTIFICATE
PEM_X509_CERTIFICATE
PEM_CERTIFICATE
PEM_UNSUPPORTED

PEM_GetType peeks at the underlying stream. It does not consume the stream in the BufferedTransformation.

The function only looks for the header (i.e., pre-encapsulated boundary), and does not probe for the footer (i.e., the post-encapsulated boundary). Its possible that PEM_GetType will return a type but later the PEM object will be rejected.

PEM_X509_CERTIFICATE and PEM_CERTIFICATE require the X509Certificate class. Though PEM_REQ_CERTIFICATE is recognized, there is no corresponding PEM_Load or PEM_Save routine.

PEM_ENC_PRIVATE_KEY is recognized, but there is no code to handle the decryption. PEM_ENC_PRIVATE_KEY is PKCS#8 encryption, and the header (i.e., pre-encapsulated boundary) is -----BEGIN ENCRYPTED PRIVATE KEY-----.

If an unknown type or bogus PEM object is presented, like -----BEGIN FOO----- and -----END BAR-----, then PEM_UNSUPPORTED is returned.

Sample Code

Using the PEM pack is straight forward. Include pem.h, and then use either PEM_Load or PEM_Save. For example:

#include <cryptopp/pem.h>
...

// Load a RSA public key
FileSource fs1("rsa-pub.pem", true);
RSA::PublicKey k1;
PEM_Load(fs1, k1);

// Load a encrypted RSA private key
FileSource fs2("rsa-enc-priv.pem", true);
RSA::PrivateKey k2;
PEM_Load(fs2, k2, "test", 4);

// Save an EC public key
DL_PublicKey_EC<ECP> k16 = ...;
FileSink fs16("ec-pub.new.pem", true);
PEM_Save(fs16, k16);

// Save an encrypted EC private key
AutoSeededRandomPool prng;
DL_PrivateKey_EC<ECP> k18 = ...;
FileSink fs18("ec-enc-priv.new.pem", true);
PEM_Save(fs18, prng, k18, "AES-128-CBC", "test", 4);

Depending on PEM_NO_KEY_OR_PARAMETER_VALIDATION, you may need to validate the key. If you have to validate keys, then the code would looks similar to:

FileSource fs("rsa-pub.pem", true);
RSA::PublicKey key;
PEM_Load(fs, key);

AutoSeededRandomPool prng;
bool result = key.Validate(prng, 2);
if(!result)
    throw std::runtime_error("Failed to validate public key");

Input and Output

Below is an example of the OpenSSL and Crypto++ keys. Keys written by OpenSSL lack "xxx", while keys written by Crypto++ include "xxx" in the filename.

OpenSSL:

$ cat rsa-pub.pem 
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCl/OKZWiBSG+kyJLdMWbK81VEt
7kMBM2FZyAH1siPEyqkjfCV3zWj7zAQtzJUlIP5KrPhezb9agYo+Xwuj3ODnS20h
FxQzPuPwzdfCUoy9khs2NY7vu8KECIVNwUi4tJCaom9otKHnRbn5ZfMicLFV/bHr
mGQqXNeMYr2i1kPGVwIDAQAB
-----END PUBLIC KEY-----

$ cat rsa-enc-priv.pem 
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,CDCA2C5DC5084410C33F2FD6439F910A

AOX63PM/YJqcCfLi3NCkInuUBgV07MK8vyKJJSvBig1iFZy0VWDZPkgYUae7iZPX
O8tBLgdunN+k/S8K/bksG8m0/iU3v0B3iTsTLmLQQg0pJYjXQyY16faouJVaovqi
hp/Zohri49giJ/49Lee6dFoZomebY6fXiI33KVqv1o91i++y/Qp7d4Djwp5aKHDd
gseB5lU/kMBAVt5Q1CllBMPD0vCbvCFCyHpLrC5RwFxFEs5Cdhhny95na1i12Khm
2kPtF6hVhGBKFzCiFexOMgLfw6VDXQJqb6MRnnxbc1igCZ2ZzBlnstXK0esd4HfY
NCBt9Z1jUUoYdQ8QXPN/cN8Xp8XMfahvmoihJNk3xWmnQFVfl+azGSaD7FG7xSmS
y9KbqQCXAJCO51nXW2m2L9u8Brf4sf1Tltc+S4gJWsGFzzNpIOZ7um7dst4ArgJT
gMY6H7nqxgCNYaZNLTT1t4ALyfCWH6eI9q7clygQMsG07ncVKkIO9dRV/ElozCwR
iV/cOYdmlt4dVXUc5uSmHHvqTRUT2T5HVw5m6Gh9uJaEt9CDdeXQg/JrUeHvQ5HT
7cs3hr6cxhSlifECUKYmunXd3bbmbWTI7yb2U0uJkoDUsCkvgJJisFjdYSwAI0yZ
zKisZrftthH8OouXjFB+VVRwpAhAMqLbe1l8EP25nGJ7e+TtVLo4fHIpxr2WdF/e
NfjqnLeReHw6W4UsObqvCu1kGZygeMRSN+6mkb3dm+u2buI2SVRvIFiHkWYD3g/l
3Lk6vfaxnTYCXkSXDax0j+ckJTTAK2i9lkcH2wqMv05FoHVi7aVnR8IIV00TnC7t
-----END RSA PRIVATE KEY-----

And Crypto++:

$ cat ec-pub.new.pem 
-----BEGIN PUBLIC KEY-----
MFYwEAYHKoZIzj0CAQYFK4EEAAoDQgAExcZuVIOO7UZSCaZYgP+faEVrvMWeBYtD
ul2u5lrpxoCEMxXlICGCwKaB1WgkJUK5sjGk45eCMv8B/78lhjVVVw==
-----END PUBLIC KEY-----

$ cat ec-enc-priv.new.pem 
-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,483C6901DE64A75A3CF218BE0A8F3AF6

u0IGXknqwne3WOz/nkgnXcfhMdRqXxjqhOmk3X1lqLwDnBBZXAytidP7b+8Teufq
xtFs9jrPoeKwWiyQ8jF6v14dQ3wDHI5LHSAJWQvcPtHZ9h5UtoPNuJktEi+eWRvt
99DCTYbC7RAx2MQnaFAFwsvUJPoJvtS8tykxXQSHqj0=
-----END EC PRIVATE KEY-----

Testing Keys

You should test the code before you use it. To do so, two scripts and a cxx source file are provided. The scripts depend on OpenSSL and PERL to create a set of PEM encoded test keys, and to verify the keys written by Crypto++ are OK. PERL is used to chop and chomp a good key into one that should cause an exception.

The steps to test the keys are as follows.

Compile the library as normal
Run pem_create_keys.sh to create the OpenSSL keys and compile pem_test.cxx
Run pem_verify_keys.sh to verify the library wrote the keys correctly and OpenSSL can read them

One of the artifacts of testing is pem_test.exe. It is run during pem_verify_keys.sh. You can run pem_test.exe stand-alone, if desired.

Expected Output

The expected output of pem_test.exe is:

$ ./pem_test.exe
Load RSA public key
  - OK
Load RSA private key
  - OK
Load encrypted RSA private key
  - OK
Load DSA public key
  - OK
Load DSA private key
  - OK
Load encrypted DSA private key
  - OK
Load ECP parameters
  - OK
Load ECP public key
  - OK
Load ECP private key
  - OK
Load encrypted ECP private key
  - OK
Save RSA public key
  - OK
Save RSA private key
  - OK
Save encrypted RSA private key
  - OK
Save DSA public key
  - OK
Save DSA private key
  - OK
Save encrypted DSA private key
  - OK
Save ECP parameters
  - OK
Save ECP public key
  - OK
Save ECP private key
  - OK
Save encrypted ECP private key
  - OK
Load malformed key 1
  - OK
Load malformed key 2
  - OK
Load malformed key 3
  - OK
Load malformed key 4
  - OK
Load malformed key 5
  - OK
Load malformed key 6
  - OK
Load malformed key 7
  - OK
Load malformed key 8
  - OK

[X.509 test cert omitted]

Load malformed X.509 certificate
  - OK

Load root certificates from Mozilla
  - OK (145 certificates)

Load root certificates from Google
  - OK (36 certificates)

OpenSSL 3.0

OpenSSL 3.0 changed the on-disk format of DSA private keys. Rather that writing Traditional keys, PKCS#8 keys are now written. There does not seem to be a way to revert to Traditional keys. Also see Issue 23497, Can't create a traditional DSA encoded private key using OpenSSL 3.0.

If OpenSSL 3.0 is expected or encountered, then DSA private key tests will be skipped:

Load DSA public key
  - OK
Load DSA private key
  - Skipped due to OpenSSL 3.0
Load encrypted DSA private key
  - Skipped due to OpenSSL 3.0

Automated Testing

You can automatically test the PEM Pack with Crypto++ by using cryptest-pem.sh script in the Crypto++ TestScripts directory. The script downloads the individual files for the PEM Pack, builds the library with the PEM Pack, generates the keys using OpenSSL, and verifies the keys using Crypto++. The script also downloads Mozilla's cacerts.pem and verifies the certificates, and downloads Google's roots.pem and verifies the certificates.

Perform the following actions to test the PEM Pack with Crypto++.

cd cryptopp
cp -p TestScripts/cryptest-pem.sh .
./cryptest-pem.sh

The output of the script should be the same as Testing Keys above.

Downloads

cryptopp-pem - Additional source files which allow you to read and write PEM encoded keys, including encrypted private keys and X.509 certificates. The collection includes a script to build test keys and certificates with OpenSSL, a small C++ test program to test reading and writing the keys and certificates, and a script to verify the keys written by Crypto++ using OpenSSL.

PEM Pack

Contents

End of Lines

Compiling and Testing

Public API

PEM_Load

PEM_Save

PEM_NextObject

PEM_GetType

Sample Code

Input and Output

Testing Keys

Expected Output

OpenSSL 3.0

Automated Testing

Downloads

Navigation menu

PEM Pack

End of Lines

Compiling and Testing

Public API

PEM_Load

PEM_Save

PEM_NextObject

PEM_GetType

Sample Code

Input and Output

Testing Keys

Expected Output

OpenSSL 3.0

Automated Testing

Downloads

Navigation menu

Search