is there a problem with the repos for erlang

[dmastrop]

hi all

Is there an issue with the repos for erlang8/9 (centos8/9) during rabbitmq installation. I am getting this error and have tried several different things. I tried both sets of scripts for centos 9 and 8
https://www.rabbitmq.com/docs/install-rpm

I am using the scripts from the official website and have this running on debian/ubuntu. I am migrating some of the servers to centos and hence the issue with the dnf/rpm install. All scripts are basically unmodified from the official website.

I am using with Cloudformation nested stack, but isolated the issue to the server installation itself as manual installation via SSH is failing as well.

The /etc/yum.repos.d/rabbitmq.repo looks good as well.

I did the standard browser test to the urls below and they are all 404s.
I am not using VPN or behind any type of proxy/firewall

[root@ip-172-31-81-210 yum.repos.d]# sudo dnf update -y
CentOS Stream 9 - BaseOS 48 kB/s | 3.0 kB 00:00
CentOS Stream 9 - AppStream 39 MB/s | 19 MB 00:00
modern-erlang-el9 364 B/s | 153 B 00:00
Errors during downloading metadata for repository 'modern-erlang':

• Status code: 404 for https://yum2.novemberain.com/erlang/el/9/repodata/repomd.xml (IP: 104.21.27.7)
• Status code: 404 for https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/rpm/el/9/repodata/repomd.xml (IP: 108.138.64.83)
• Status code: 404 for https://yum1.novemberain.com/erlang/el/9/repodata/repomd.xml (IP: 172.67.140.63)
  Error: Failed to download metadata for repo 'modern-erlang': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[centos@ip-172-31-93-33 yum.repos.d]$ sudo dnf update -y
modern-erlang-el8 188 B/s | 153 B 00:00
Errors during downloading metadata for repository 'modern-erlang':

• Status code: 404 for https://yum2.novemberain.com/erlang/el/8/repodata/repomd.xml (IP: 104.21.27.7)
• Status code: 404 for https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/rpm/el/8/repodata/repomd.xml (IP: 108.138.64.31)
• Status code: 404 for https://yum1.novemberain.com/erlang/el/8/repodata/repomd.xml (IP: 172.67.140.63)
  Error: Failed to download metadata for repo 'modern-erlang': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

dig NS for novemberain.com and cloudsmith.io are ok but the hosted zones yum2 and yum and dl are not.
Have the zones changed on the repos?

thanks and warm regards
Dave

[dmastrop]

i ended up doing manual script installation via rpms with erlang 25 and rabbitmq 3.12.14 and it is coming up fine on centos stream8
The repos have issues....
warm regards
Dave

[malsssh]

bump, still not possible to use this repository.

Maybe u could add rpm for el9 to releases assets??

[dmastrop]

hi malssh

<<< Maybe u could add rpm for el9 to releases assets??

That would help so that we can access erlang9.

If you are interested here is the manual UserData block (using rpms) from the cloudformation yaml template that works for latest rabbitmq/erlang install on centos stream8

warm regards
Dave

UserData:
        Fn::Base64:                             
          !Sub |
              #!/bin/bash

              ## MANUAL INSTALLATION OF RABBITMQ
              #sudo yum install socat -y
              sudo dnf install socat -y
              cd /tmp/
              curl -sfL -O https://github.com/rabbitmq/erlang-rpm/releases/download/v25.3.2.12/erlang-25.3.2.12-1.el8.x86_64.rpm
              sudo rpm -Uvh erlang-25.3.2.12-1.el8.x86_64.rpm 
              sudo rpm --import 'https://github.com/rabbitmq/signing-keys/releases/download/3.0/rabbitmq-release-signing-key.asc'
              sudo rpm -Uvh https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.12.14/rabbitmq-server-3.12.14-1.el8.noarch.rpm
              sudo  systemctl start rabbitmq-server
              sudo systemctl enable rabbitmq-server
              sudo systemctl status rabbitmq-server
              sudo -i
              echo "[{rabbit, [{loopback_users, []}]}]." > /etc/rabbitmq/rabbitmq.config
              rabbitmqctl add_user test test
              rabbitmqctl set_user_tags test administrator
              systemctl restart rabbitmq-server

[michaelklishin]

FYI,

echo "[{rabbit, [{loopback_users, []}]}]." > /etc/rabbitmq/rabbitmq.config

is a catastrophically bad security practice. The docs very explicitly say "DANGER ZONE".

On top of that, RabbitMQ 3.12.x is out of community support.

[michaelklishin]

According to mirror logs and metrics, both mirrors are fine. They have served millions of requests and tens of GiBs of data in the last ≈ 24 hours.

[lukebakken]

@dmastrop @malsssh please see this response from the maintainer of those mirrors. The issue is in your environment. Also note that we do NOT provide 32-bit packages.

[michaelklishin]

As well as aarch64 packages via mirrors. Those are built manually and uploaded to GitHub releases.

[michaelklishin]

A longer period of time (these are Cloudflare stats):

[michaelklishin]

Nearly all 404s our mirrors get come from the fact that some try to use them on 32-bit operating systems, which we do not and will not provide packages for.

Another set of environments we do not support is aarch64 packages of Erlang. They are built manually locally and uploaded to GitHub releases but not to Cloudsmith and thus not to the mirrors.

According to the logs and metrics an absolute majority of requests are 2xx.

If mirrors were down, we'd be flooded with similar questions.

[michaelklishin]

Another data point: the absolute majority of requests served are for xml, text or binary files and not html (served for 404s), which for RPM repositories that change about once a month is the expected state. These are successful dnf metadata updates.

[dmastrop]

hi Michael, yes mine are xml metadata requests.

I am doing everything on 64 bit centos stream

I found the source of the error.
The nginx was throwing the error in the packet traces, so it was a URL link routing issue

The updated links should be the following (see below)

I will redo the prototyping that i was working on with the updated links to mitigate the security risk in manual scripts

Proper links:
https://yum1.novemberain.com/erlang/el/8/x86_64/repodata/repomd.xml
https://yum1.novemberain.com/erlang/el/9/x86_64/repodata/repomd.xml
https://yum2.novemberain.com/erlang/el/9/x86_64/repodata/repomd.xml
https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/rpm/el/9/x86_64/repodata/repomd.xml

These are the nonfunctional links below (for example):
https://yum1.novemberain.com/erlang/el/9/repodata/repomd.xml
https://yum1.novemberain.com/erlang/el/8/repodata/repomd.xml
https://yum2.novemberain.com/erlang/el/9/repodata/repomd.xml
https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/rpm/el/9/repodata/repomd.xml

This is covered with the scripts in the rabbitmq page https://www.rabbitmq.com/docs/install-rpm
baseurl=https://yum1.novemberain.com/erlang/el/8/$basearch
https://yum2.novemberain.com/erlang/el/8/$basearch
https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/rpm/el/8/$basearch

warm regards
Dave

[malsssh]

Erlang repo works fine but both rabbitmq-el9 and rabbitmq-el8 doesnt work

yum -v repolist

Repo-id : rabbitmq-el9
Repo-name : rabbitmq-el9
Repo-revision : 1683290246
Repo-updated : Fri 05 May 2023 02:37:26 PM CEST
Repo-pkgs : 0
Repo-available-pkgs: 0
Repo-size : 0
Repo-baseurl : https://yum2.rabbitmq.com/rabbitmq/el/9/x86_64, https://yum1.rabbitmq.com/rabbitmq/el/9/x86_64
Repo-expire : 300 second(s) (last: Tue 20 Aug 2024 01:40:47 PM CEST)
Repo-filename : /etc/yum.repos.d/rabbitmq.repo

Repo-id : rabbitmq-el8
Repo-name : rabbitmq-el8
Repo-revision : 1683290246
Repo-updated : Fri 05 May 2023 02:37:26 PM CEST
Repo-pkgs : 0
Repo-available-pkgs: 0
Repo-size : 0
Repo-baseurl : https://yum2.rabbitmq.com/rabbitmq/el/8/x86_64, https://yum1.rabbitmq.com/rabbitmq/el/8/x86_64
Repo-expire : 300 second(s) (last: Tue 20 Aug 2024 01:56:14 PM CEST)
Repo-filename : /etc/yum.repos.d/rabbitmq.repo

Repo-id : modern-erlang
Repo-name : modern-erlang-el9
Repo-revision : 1720647069
Repo-updated : Wed 10 Jul 2024 11:31:09 PM CEST
Repo-pkgs : 27
Repo-available-pkgs: 27
Repo-size : 297 M
Repo-baseurl : https://yum1.rabbitmq.com/erlang/el/9/x86_64, https://yum2.rabbitmq.com/erlang/el/9/x86_64
Repo-expire : 300 second(s) (last: Tue 20 Aug 2024 01:53:36 PM CEST)
Repo-filename : /etc/yum.repos.d/rabbitmq.repo

[michaelklishin]

@malsssh we cannot commend on "it does not work". If the Erlang repo "does not work" how come according to the logs it serves multiple requests per second and we are not flooded with similar complaints?

[michaelklishin]

@malsssh I do not observe any anomalies w.r.t. RPM repo operations, neither from metrics nor from logs nor from a test on Fedora 40 with the following configuration:

##
## RabbitMQ Server
##

##
## CentOS Stream 9
##

[rabbitmq-el9]
name=rabbitmq-el9
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/9/$basearch
repo_gpgcheck=1
enabled=1
# Cloudsmith's repository key and RabbitMQ package signing key
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
gpgcheck=1
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md

[rabbitmq-el9-noarch]
name=rabbitmq-el9-noarch
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/9/noarch
repo_gpgcheck=1
enabled=1
# Cloudsmith's repository key and RabbitMQ package signing key
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
gpgcheck=1
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md

[rabbitmq-el9-source]
name=rabbitmq-el9-source
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/9/SRPMS
repo_gpgcheck=1
enabled=1
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
gpgcheck=0
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md


##
## CentOS Stream 8
##

[rabbitmq-el8]
name=rabbitmq-el8
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/8/$basearch
repo_gpgcheck=1
enabled=1
# Cloudsmith's repository key and RabbitMQ package signing key
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
gpgcheck=1
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md

[rabbitmq-el8-noarch]
name=rabbitmq-el8-noarch
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/8/noarch
repo_gpgcheck=1
enabled=1
# Cloudsmith's repository key and RabbitMQ package signing key
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
gpgcheck=1
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md

[rabbitmq-el8-source]
name=rabbitmq-el8-source
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/8/SRPMS
repo_gpgcheck=1
enabled=1
gpgkey=https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key
gpgcheck=0
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
pkg_gpgcheck=1
autorefresh=1
type=rpm-md

We will ignore such "issue reports" without any details. Either provide an step-by-step set of instructions for us to reproduce or seek help elsewhere.