# How TLS Communication between Services Work # How TLS Communication between Services Work If you're running a software company and have many web services, and some of them have public facing endpoints, and some have private, internal endpoints, then you can have your own private CA, and have it sign intermediate CAs which can then sign the individual leaf certs for the endpoints of your services. This tutorial explains how such a thing can be implemented. As a starting point for this experiment, create these dirs first: ```sh mkdir -p pki/{root,intermediate,server} && cd pki ``` ### Creating the Private Root CA Create a strong private key that'll be used to generate the private root CA: ```sh openssl genrsa -out root/root.key.pem 4096 ``` This creates a pem file that has the `---BEGIN PRIVATE KEY---` thingy. And so it means it's a secret. In fact it's the most important secret in your whole security infra. Keep it offline and safe. You only need it to sign the intermediate CAs mostly. You can inspect this key file with `openssl rsa -in root/root.key.pem -noout -text`. But I wouldn't know the meaning of the things printed. Use it to create the root certificate `root.cert.pem`, by passing in the details needed for the cert (org name, country, state, org unit, common name) etc. Add whatever extensions you need: ```sh openssl req -x509 -new -nodes \ -key root/root.key.pem \ -sha256 -days 3650 \ -out root/root.cert.pem \ -subj "/C=US/ST=CA/O=Pras Corp/OU=Security/CN=Pras Root CA" \ -addext "basicConstraints=critical,CA:TRUE" \ -addext "keyUsage=critical,keyCertSign,cRLSign" \ -addext "subjectKeyIdentifier=hash" ``` You can inspect this cert file with: `openssl x509 -in root/root.cert.pem -noout -text`. You can see all the deets you passed in. More importantly, this cert contains the public key and the details of the root's Org. This root cert and the root key above... both will be used to sign the intermediate CA's CSR (certificate signing request) (covered later below). ### Create the Intermediate CA, signed by the Root CA Create a strong private key that'll be used to generate the intermediate CA: ```sh openssl genrsa -out intermediate/intermediate.key.pem 4096 chmod 400 intermediate/intermediate.key.pem ``` Use it to create the CSR (certificate signing request) file that'll be sent to the org of the private CA where the security people at the Org will sign it: ```sh openssl req -new \ -key intermediate/intermediate.key.pem \ -out intermediate/intermediate.csr.pem \ -subj "/C=US/ST=CA/O=Pras Intermediate Corp/OU=Security/CN=Pras Intermediate CA" ``` You can inspect this intermediate csr file with `openssl req -in intermediate/intermediate.csr.pem -text -noout`. The root-signed intermediate CA will have metadata in the form of extensions. Those are embedded in separate file `intermediate/intermediate.ext`. Create it with this content: ```txt basicConstraints = critical, CA:TRUE, pathlen:0 keyUsage = critical, keyCertSign, cRLSign subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer ``` These extensions define some properties. For example, it defines pathlen as 0. It means this intermediate CA can only sign leaf certs. Not other CA type certs like itself. You'd need pathlen 1 for that. Now's the moment. Use the CSR given by your Org's intermediate unit, and your own private root key and cert to create the intermediate cert: ```sh openssl x509 -req \ -in intermediate/intermediate.csr.pem \ -CA root/root.cert.pem \ -CAkey root/root.key.pem \ -CAcreateserial \ -out intermediate/intermediate.cert.pem \ -days 1825 -sha256 \ -extfile intermediate/intermediate.ext ``` You'll see the success output: ``` Certificate request self-signature ok subject=C=US, ST=CA, O=Pras Intermediate Corp, OU=Security, CN=Pras Intermediate CA ``` This cert file, `intermediate.cert.pem`, it has the root's details as well as the intermediate CA's details. This is what You can verify it with `openssl verify -CAfile root/root.cert.pem intermediate/intermediate.cert.pem`. ### Create the Leaf Certs, signed by the Intermediate CA Create a private key first, as usual, for the specific endpoint you want to generate the cert for: ```sh openssl genrsa -out server/mysite_com.key.pem 2048 ``` The leaf's cert too needs metadata in the form of extensions. Create the extension file `server/api.txt` with this content: ```txt basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth, clientAuth subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer subjectAltName = @alt_names [alt_names] DNS.1 = api.mysite.com DNS.2 = api.internal.mysite.local DNS.3 = localhost IP.1 = 127.0.0.1 ``` Create a CSR for the specific endpoint with: ```sh openssl req -new \ -key server/mysite_com.key.pem \ -out server/api.csr.pem \ -subj "/C=US/ST=CA/O=Pras Corp/OU=Services/CN=api.mysite.com" ``` Finally, sign the leaf's CSR with the intermediate's CA: ```sh openssl x509 -req \ -in server/api.csr.pem \ -CA intermediate/intermediate.cert.pem \ -CAkey intermediate/intermediate.key.pem \ -CAcreateserial \ -out server/api.cert.pem \ -days 365 -sha256 \ -extfile server/api.ext ``` ### Deploy the Leaf+Intermediate certs to the Leaf endpoint's servers (ELB's mostly) ```sh cat server/api.cert.pem intermediate/intermediate.cert.pem > server/api.fullchain.pem ``` ### Verify it ```sh openssl verify \ -CAfile root/root.cert.pem \ -untrusted intermediate/intermediate.cert.pem \ server/api.cert.pem ``` You'll see `server/api.cert.pem: OK`. verify with `api.fullchain.pem` too. This is likely how the client (browser, or the service calling mysite.com in its code) might be verifying the fullchain cert presented by mysite.com. --- ## tidbits - `ca-certificates.crt` is the concatenation of public certificate PEM file content of popular trusted CA root certificates. If you're adding a root CA that's known only to your Org, it should be added to this list - cmd to verify if a specific root CA is in the instance: ```sh cd /etc/ssl/certs openssl x509 -noout -subject -in filename.pem ``` Usually the filenames are enough to identify the Org name. So get that/them with `ls *orgname*`, and then use the cmd above on individual files. ## questions - how is your private root CA imprinted onto the ec2 nodes? ```sh sudo cp my-internal-ca.crt /usr/local/share/ca-certificates/my-internal-ca.crt sudo chmod 644 /usr/local/share/ca-certificates/my-internal-ca.crt # regenerates /etc/ssl/certs/ca-certificates.crt bundle sudo update-ca-certificates ```