IPv6 only, cannot join cluster: nxdomain

[zhangguanzhang]

Describe the bug

standalone and join cluster

$ docker exec -ti rabbitmq-2 bash -c "rabbitmqctl stop_app && rabbitmqctl  join_cluster rabbit@rabbitmq-cluster1 && rabbitmqctl start_app"
rabbit@rabbitmq-cluster1:
  * unable to connect to epmd (port 4369) on rabbitmq-cluster1: nxdomain (non-existing domain)

but rabbitmq-diagnostics resolve_hostname works:

$ docker exec -ti rabbitmq-2  rabbitmq-diagnostics resolve_hostname rabbitmq-cluster1 --address-family IPv6 
Asking node rabbitmq-cluster2@rabbitmq-cluster2 to resolve hostname rabbitmq-cluster1 to IPv6 addresses...
2408:8656:22df:ff01::13:181

$ docker exec -ti rabbitmq-2   cat /etc/hosts
::1 localhost rabbitmq-cluster2
::1     ip6-localhost ip6-loopback
2408:8656:22df:ff01::13:181 rabbitmq-cluster1 rabbitmq-1
2408:8656:22df:ff01::13:182 rabbitmq-cluster2 rabbitmq-2
2408:8656:22df:ff01::13:183 rabbitmq-cluster3 rabbitmq-3

$ docker exec -ti rabbitmq-2 getent hosts rabbitmq-cluster1
2408:8656:22df:ff01::13:181 rabbitmq-cluster1

addition info:

$ rabbitmqctl status
Status of node rabbitmq-cluster2@rabbitmq-cluster2 ...
Runtime

OS PID: 20
OS: Linux
Uptime (seconds): 376
Is under maintenance?: false
RabbitMQ version: 3.12.14
RabbitMQ release series support status: out of support
Node name: rabbitmq-cluster2@rabbitmq-cluster2
Erlang configuration: Erlang/OTP 25 [erts-13.2.2.11] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.1.7 3 Sep 2024
Erlang processes: 366 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * accept
 * prometheus
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbitmq-cluster2@rabbitmq-cluster2
Raft data directory: /var/lib/rabbitmq/mnesia/rabbitmq-cluster2@rabbitmq-cluster2/quorum/rabbitmq-cluster2@rabbitmq-cluster2

Config files

 * /etc/rabbitmq/rabbitmq.conf
 * /etc/rabbitmq/conf.d/10-defaults.conf
 * /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf

Log file(s)

 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1691 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 27.0192 gb

reserved_unallocated: 0.0995 gb (58.83 %)
code: 0.0355 gb (20.97 %)
other_system: 0.022 gb (13.0 %)
other_proc: 0.0186 gb (11.02 %)
metrics: 0.0017 gb (1.02 %)
atom: 0.0015 gb (0.87 %)
other_ets: 0.0014 gb (0.83 %)
msg_index: 0.0004 gb (0.21 %)
plugins: 0.0003 gb (0.18 %)
mnesia: 0.0001 gb (0.04 %)
binary: 0.0001 gb (0.04 %)
quorum_ets: 0.0 gb (0.02 %)
connection_other: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
mgmt_db: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 1048479
Sockets: 0, limit: 943629

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 468.7445 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 15672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Reproduction steps

docker-compose.yml

services:
  rabbitmq:
    image: 'rabbitmq:3.12'
    container_name: "rabbitmq-2"
    network_mode: host
    hostname: "rabbitmq-cluster2"
    restart: always # unless-stopped
    working_dir: '/etc/rabbitmq/'

    environment:
      - RABBITMQ_NODENAME=rabbitmq-cluster2
      - RABBITMQ_ERLANG_COOKIE=CWQFVXMAJUTZRDHPQCCH
      - RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
      - CTL_ERL_ARGS="-proto_dist inet6_tcp"

    volumes:
      - ./data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
      - ./rabbitmq-env.conf:/etc/rabbitmq/rabbitmq-env.conf
      - ./erl_inetrc:/etc/rabbitmq/erl_inetrc

rabbitmq-env.conf

$ cat rabbitmq-env.conf 
# these flags will be used by RabbitMQ nodes
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
# these flags will be used by CLI tools
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"

erl_inetrc

{inet6,true}.

rabbitmq.conf

$ cat  rabbitmq.conf 
loopback_users.guest = false
cluster_partition_handling = pause_minority
default_pass = xxxxxx
default_user = root

management.tcp.port = 15672

distribution.listener.interface = ::
listeners.tcp.default = :::5672
listeners.tcp.management = :::15672

Expected behavior

resolve host on IPv6 only

Additional context

No response

[michaelklishin]

@zhangguanzhang 3.12.x is out of community support and close to being entirely out of support.

We will not troubleshoot networking for non-paying users. RabbitMQ does not control how hostname resolution is set up on your hosts. An nxdomain is a result of a standard library call returned by the kernel.

The Erlang runtime has plenty of options that control hostname resolution besides everything that's available in the Linux kernel, on Kubernetes, and so on. RabbitMQ or its community Docker image do no touch any of those.

If this is on Kubernetes, there is a Core DNS caching setting mentioned in the Kubernetes Peer Discovery docs.

Please take it from here.