MOVED 3999 127.0.0.1:6381 (and ASK) in Redis Cluster

What it means

Redis Cluster splits the keys over several primaries. Every key belongs to one of 16,384 hash slots (CRC16(key) mod 16384), and each primary owns some of the slots. A node only runs commands for keys in its own slots. For any other key it replies with a redirection instead of an answer:

MOVED 3999 127.0.0.1:6381

That reads: the key is in slot 3999, and the node at 127.0.0.1:6381 serves that slot. Nothing ran. A cluster-aware client sends the command again to that node, and remembers which node owns which slot so it goes to the right one first next time.

ASK looks the same and means something narrower:

ASK 5061 127.0.0.1:6382

The slot is being moved from this node to another (resharding) and this key has already gone. The client should send this one command to the new node, preceded by ASKING, and keep sending everything else for that slot to the old node until the move finishes. Sent to the new node without ASKING, the command gets a MOVED back to the old one.

You see these as errors when the client doesn’t follow redirections: redis-cli without -c, a library’s plain (non-cluster) client, or a tool that connects to one node.

Common causes

1. A cluster opened with a non-cluster client. About two-thirds of the keys in a three-node cluster live on the other nodes, so most commands fail.
2. A cluster-mode endpoint used like a single server. On AWS ElastiCache (cluster mode enabled) the configuration endpoint needs a client that supports Redis or Valkey Cluster, AWS’s docs say. On Redis Cloud, the default endpoint routes commands for you, but with the OSS Cluster API turned on you need a cluster client.
3. Addresses the client can’t reach. The node in MOVED is the address it announces. In Docker or behind NAT that’s often an internal address, so even a cluster client follows the redirect and then fails to connect.
4. Resharding in progress (ASK): adding or removing nodes, or rebalancing slots. A cluster client handles it; a plain client fails for keys already moved.
5. An out-of-date slot map in a client after a failover or resharding: it gets MOVED once, then updates. A steady stream of MOVED from a cluster client means it isn’t refreshing its map.

How to fix it

Use a cluster-aware client

From the command line, add -c; redis-cli then follows MOVED and ASK and says so:

redis-cli -c -h <host> -p <port>

-> Redirected to slot [5061] located at 127.0.0.1:6381

In code, use the library’s cluster client rather than the single-server one, for example redis-py’s RedisCluster, node-redis’s createCluster, go-redis’s NewClusterClient or Jedis’s JedisCluster. Give it one or more nodes; it reads the slot map from them.

Look up where a key lives

redis-cli -h <host> -p <port> CLUSTER KEYSLOT user:1000
redis-cli -h <host> -p <port> CLUSTER SHARDS

CLUSTER KEYSLOT gives the slot; CLUSTER SHARDS (or the older CLUSTER SLOTS and CLUSTER NODES) lists which node serves which range. For a one-off look you can connect to the node the error names.

Make the announced addresses reachable

If redirects point at addresses your machine can’t reach, set what each node announces in its config: cluster-announce-ip (or cluster-announce-hostname with cluster-preferred-endpoint-type hostname), cluster-announce-port and cluster-announce-bus-port. Or run the client where those addresses resolve, inside the same Docker network.

Use a single-server endpoint if your tool needs one

If a tool can’t speak cluster, give it something that isn’t sharded: a database with clustering off, an ElastiCache replication group with cluster mode disabled, or a provider endpoint that routes commands for you (Redis Cloud without the OSS Cluster API).

Reproduce it

One temporary Redis 8.10.2 container running three nodes on ports 6381, 6382 and 6383 (each started with --cluster-enabled yes), joined into a cluster with no replicas:

redis-cli --cluster create 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 --cluster-replicas 0 --cluster-yes

>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
…
[OK] All 16384 slots covered.

CLUSTER KEYSLOT put bar in slot 5061, user:1000 in 1649 and foo in 12182. Asking the wrong nodes:

$ redis-cli -p 6383 SET bar 1
(error) MOVED 5061 127.0.0.1:6381
$ redis-cli -p 6383 GET user:1000
(error) MOVED 1649 127.0.0.1:6381
$ redis-cli -p 6381 SET foo 1
(error) MOVED 12182 127.0.0.1:6383

With -c, the same SET bar on port 6383 followed the redirect and printed -> Redirected to slot [5061] located at 127.0.0.1:6381 before OK.

For ASK, slot 5061 was moved by hand from the node on 6381 to the one on 6382 (CLUSTER SETSLOT 5061 IMPORTING … on 6382, CLUSTER SETSLOT 5061 MIGRATING … on 6381), and the key moved with MIGRATE 127.0.0.1 6382 "" 0 5000 KEYS bar. Then:

$ redis-cli -p 6381 GET bar
(error) ASK 5061 127.0.0.1:6382
$ redis-cli -p 6382 GET bar
(error) MOVED 5061 127.0.0.1:6381

The new node sent the client back because it hadn’t sent ASKING. With ASKING first, on the same connection, it returned "2". After CLUSTER SETSLOT 5061 NODE … finished the move on all three nodes, the old node answered (error) MOVED 5061 127.0.0.1:6382.

A temporary Valkey 8.1.10 cluster built the same way gave the same MOVED replies, for example (error) MOVED 5061 127.0.0.1:6381 (we didn’t repeat the ASK test there).

In Inlet

Inlet connects to one node and doesn’t support cluster mode yet. When a key is on another node it doesn’t follow the redirection; it explains it, naming the slot and the node:

This key is on another node of the cluster (slot 5061, on 127.0.0.1:6381). Inlet connects to one node: connect to that one to use it.

The error links to this page. On AWS, Inlet works with ElastiCache clusters that have cluster mode disabled (MemoryDB and ElastiCache Serverless always use the cluster protocol); see connecting to ElastiCache.