two generated certificates have the same serial number [Concurrency issue]

[zappee]

I use EasyRSA in my Docker environment to generate certificates for my containers that are signed by my root CA. Recently a strange thing happened and I have no idea why and how to fix it.

Inside the Docker containers I run applications with web servers (Spring Boot/Tomcat). Behind the web servers I use certificates issued by EasyRSA. Today, when I tried to open web content from different containers, the web browser complained about SEC_ERROR_REUSED_ISSUER_AND_SERIAL.

URLs that I opened:

• https://localhost:14114/api/user/1
• https://localhost:14124/api/user/1

As I see it, the problem is that the two certificates generated have the same serial number. I used the same EasyRSA environment to issue the certificates. The issued certificates have different serial numbers except for these two: user-service-1 and user-service-2:

[link to the screenshot](https://drive.google.com/file/d/1fm9u6kpzhCSYbkzABRJ2QSOLeXMpAefW/view?usp=sharing

I use the following commands to generate certs:

# generating a certificate request
$ ./easyrsa --batch --passout="pass:$EASYRSA_PASS" --req-cn="$domain" gen-req "$domain"
    
# signing the certificate request
./easyrsa --batch --passin="pass:$EASYRSA_PASS" sign-req "$cert_type" "$domain"

# creating a new PKCS#12 keystore and import the certificate into it
./easyrsa --passin="pass:$EASYRSA_PASS" --passout="pass:$EASYRSA_PASS" export-p12 "$domain"

where:

• domain=user-service-1.hello.com
• cert_type="serverClient"

EasyRSA version:

$ ./easyrsa version
EasyRSA Version Information
Version:     3.2.1
Generated:   Fri Sep 13 13:04:18 CDT 2024
SSL Lib:     OpenSSL 3.3.2 3 Sep 2024 (Library: OpenSSL 3.3.2 3 Sep 2024)
Git Commit:  3f60a68702713161ab44f9dd80ce01f588ca49ac
Source Repo: https://github.com/OpenVPN/easy-rsa

The creation date of the two certs with the same serial number:

$ pwd
/opt/easy-rsa/pki/issued
$ ll
-rw------- 1 root root 2902 Jan 4 22:52 echo-service-1.hello.com.crt
-rw------- 1 root root 2824 Jan 4 22:52 pki.hello.com.crt
-rw------- 1 root root 2892 Jan 4 22:52 user-service-1.hello.com.crt
-rw------- 1 root root 2895 Jan 4 22:52 user-service-2.hello.com.crt

file dates:

$ stat user-service-1.hello.com.crt
File: user-service-1.hello.com.crt
Size: 2892      	Blocks: 8          IO Block: 4096   regular file

 Birth: 2025-01-04 22:52:52.685615758 +0000
Modify: 2025-01-04 22:52:52.693615758 +0000
Change: 2025-01-04 22:52:52.695615758 +0000
Access: 2025-01-04 22:52:52.697615758 +0000


$ stat user-service2.hello.com.crt
File: user-service-2.hello.com.crt
Size: 2895      	Blocks: 8          IO Block: 4096   regular file

 Birth: 2025-01-04 22:52:52.686615758 +0000
Modify: 2025-01-04 22:52:52.695615758 +0000
Change: 2025-01-04 22:52:52.697615758 +0000
Access: 2025-01-04 22:52:52.698615758 +0000

Why the two generated certs have the same serial number and how to avoid this situation?
Is this a bug?

[TinCanTech]

It is difficult to validate your claim from a screenshot.

Please share your certificates.

You can share your certificates here on github or
via the OpenVPN security mailing list.

[zappee]

Thanks for trying to help.
I uploaded the certs to a public Git repo here:
https://github.com/zappee/easyrsa-cert-issue/tree/master

[zappee]

I updated the sample project to help you understand better the situation and the EasyRSA issue/bug.

Project that show the situation: https://github.com/zappee/easyrsa-cert-issue/tree/master

Please let me know if you need anything else.

[TinCanTech]

Thanks for your detailed analysis, it looks like you have isolated the problem.

I could use your setup to develop a working locking mechanism but that will not be done for some time.

I think you will need to accept that Easy-RSA is a simple tool and not designed for your use case. Which means you will need to ensure that you do not access the PKI multiple times, concurrently.

If you want to hack a lock-file into easyrsa then I would consider a PR.

[zappee]

Thanks for the response and the comments.

Finally, I found an acceptable solution that ensures that EasyRSA runs only once at the same time and the other certificate-generating requests wait until it has finished.

#!/bin/bash -ue

# ------------------------------------------------------------------------------
# EasyRSA is a simple tool and is not designed to be run multiple times
# concurrently. This means that you must ensure that you do not run the tool
# more than once in parallel.
#
# Running multiple EasyRSA jobs in parallel cause that the same 'serial number'
# is reused multiple times by EasyRSA and the certificates generated will have the
# same 'serial number'. This causes various problems, such as a
# SEC_ERROR_REUSED_ISSUER_AND_SERIAL error in the web browser.
#
# This function checks whether EasyRSA is running or not and only continues if
# it is not.
# ------------------------------------------------------------------------------
function wait_for_easyrsa_has_completed() {
  local pid

  while : ; do
    pid=$(pidof easyrsa) || true
    if [ -n "$pid" ]; then
      printf "%s | [DEBUG] EasyRSA is running, pid: %s\n" "$(date +"%Y-%m-%d %H:%M:%S")" "$pid"
      printf "%s | [DEBUG] wait for EasyRSA to finish its work...\n" "$(date +"%Y-%m-%d %H:%M:%S")"
      sleep 0.5
    else
      printf "%s | [DEBUG] EasyRSA is not used by any other client, so let's continue\n" "$(date +"%Y-%m-%d %H:%M:%S")"
      break
    fi
  done
}

Then I added this function call at the beginning of the script that generates my server certificates:

wait_for_easyrsa_has_completed
generate_cert_req_and_key ...

# no more concurrency issue after that point
signing_cert_req ...
export_to_keystore ...
import_to_keystore "ca-cert" ...

As far as I can see, only the `easyrsa gen-req' command has a problem with parallel execution.

That would be great if you could add this check somehow to the tool as a new feature in the future.

Thanks for helping me and for this nice EasyRSA tool.

log:

2025-01-06 23:12:37 | [DEBUG] ====> executing the "/docker.init/30602_generate-server-cert.sh" script...
2025-01-06 23:12:37 | [INFO]  generating a server certificate...
2025-01-06 23:12:37 | [DEBUG]        PKI_HOST: "pki.hello.com"
2025-01-06 23:12:37 | [DEBUG]        SSH_USER: "root"
2025-01-06 23:12:37 | [DEBUG]    SSH_PASSWORD: "password"
2025-01-06 23:12:37 | [DEBUG]       cert_type: "serverClient"
2025-01-06 23:12:37 | [DEBUG]          domain: "user-service-2.hello.com"
2025-01-06 23:12:37 | [DEBUG]             san: "DNS:user-service-2.hello.com,DNS:localhost,IP:127.0.0.1"
Warning: Permanently added 'pki.hello.com' (ED25519) to the list of known hosts.
2025-01-06 23:12:37 | [INFO ] generating a certificate using easy-rsa...
2025-01-06 23:12:37 | [INFO ] running "generate-cert.sh" script on "pki.hello.com" machine...
2025-01-06 23:12:37 | [DEBUG]           EASYRSA_HOME: "/opt/easy-rsa"
2025-01-06 23:12:37 | [DEBUG]         EASYRSA_REQ_CN: "hello.com"
2025-01-06 23:12:37 | [DEBUG]    EASYRSA_REQ_COUNTRY: "DK"
2025-01-06 23:12:37 | [DEBUG]   EASYRSA_REQ_PROVINCE: "Region Hovedstaden"
2025-01-06 23:12:37 | [DEBUG]       EASYRSA_REQ_CITY: "Copenhagen"
2025-01-06 23:12:37 | [DEBUG]        EASYRSA_REQ_ORG: "Hello Software"
2025-01-06 23:12:37 | [DEBUG]      EASYRSA_REQ_EMAIL: "[email protected]"
2025-01-06 23:12:37 | [DEBUG]         EASYRSA_REQ_OU: "IT Department"
2025-01-06 23:12:37 | [DEBUG]           EASYRSA_ALGO: "ec"
2025-01-06 23:12:37 | [DEBUG]         EASYRSA_DIGEST: "sha256"
2025-01-06 23:12:37 | [DEBUG]       EASYRSA_KEY_SIZE: "2048"
2025-01-06 23:12:37 | [DEBUG]           EASYRSA_PASS: "changeit"
2025-01-06 23:12:37 | [DEBUG] arguments of the "generate-cert.sh" script:
2025-01-06 23:12:37 | [DEBUG]       certificate type: "serverClient"
2025-01-06 23:12:37 | [DEBUG]                 domain: "user-service-2.hello.com"
2025-01-06 23:12:37 | [DEBUG]                    san: "DNS:user-service-2.hello.com,DNS:localhost,IP:127.0.0.1"

!!!!

2025-01-06 23:12:37 | [DEBUG] EasyRSA is running, pid: 434
2025-01-06 23:12:37 | [DEBUG] wait for EasyRSA to finish its work...
2025-01-06 23:12:38 | [DEBUG] EasyRSA is running, pid: 668
2025-01-06 23:12:38 | [DEBUG] wait for EasyRSA to finish its work...
2025-01-06 23:12:38 | [DEBUG] EasyRSA is not used by any other container, so let's continue

!!!!

2025-01-06 23:12:38 | [INFO ] generating a certificate request and key...
2025-01-06 23:12:38 | [DEBUG]    EASYRSA_HOME: "/opt/easy-rsa"
2025-01-06 23:12:38 | [DEBUG]    EASYRSA_PASS: "changeit"
2025-01-06 23:12:38 | [DEBUG]          domain: "user-service-2.hello.com"
2025-01-06 23:12:38 | [DEBUG]             san: "DNS:user-service-2.hello.com,DNS:localhost,IP:127.0.0.1"
2025-01-06 23:12:38 | [DEBUG]        work_dir: "/opt/easy-rsa"
2025-01-06 23:12:38 | [DEBUG] generating a certificate request with SAN: DNS:user-service-2.hello.com,DNS:localhost,IP:127.0.0.1...

[TinCanTech]

For your own clarity, it is command sign-req which assigns a certificate serial number not gen-req. However, your use case is best served by the check before gen-req (Which will cascade down to sign-req) because the private key generation could be degraded by the same concurrency issue.

[TinCanTech]

@zappee btw, nice example! 👍

Edit: Re-opened, OpenSSL warn of this concurrency issue.