Can't write to slave

[tallesl]

I have a slave with slave-read-only set to no but when I attempt to do a SADD to it I got a No connection is available to service this operation.

at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 1857
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 80
at StackExchange.Redis.RedisDatabase.SetAdd(RedisKey key, RedisValue[] values, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 921

When I issue SMEMBERS it works fine, I'm assuming it's a write problem.

Inspecting the variables while debugging, I noticed that the Server object has SlaveReadOnly set to false as expected, but there's AllowSlaveWrites also set to false. I don't know how those are set (or what exactly they affect), but I forcefully changed AllowSlaveWrites to true with no luck.

Setting abortConnect=false on configuration string didn't help either.

If I set the instance to master everything works fine.
It's reproducible in my development machine where the redis configuration is pretty much vanilla.
This is not an intermittent error, I get it as soon I start my application.

[tallesl]

Looks like that is IsMasterOnly fault.

Is writing to a slave even supported? If not, why not?

I've changed the flag at runtime (while debugging) to the original PreferMaster (instead of DemandMaster) and everything went fine...

[mgravell]

I will need to revisit the code for determining when slave writes are
legal. At the moment IIRC the code assumes the "safe" default of
master-only writes. But : this is indeed configurable in Redis (and before
it was configurable: slave writes were allowed).

So: yes, I agree we should look at this. But: a part of me will always
wonder why people would want out-of-step slaves :)

On Fri, 5 Feb 2016 13:15 Talles L notifications@github.com wrote:

Looks like that is IsMasterOnly
https://github.com/StackExchange/StackExchange.Redis/blob/567cdf7/StackExchange.Redis/StackExchange/Redis/Message.cs#L382
fault.

Is writing to a slave even supported? If not, why not?

—
Reply to this email directly or view it on GitHub
#354 (comment)
.

[tallesl]

If my master goes down (for whatever reason) the slave should take place while someone (human) goes on putting the master back online. The service that uses redis does both reads and writes on it (and it doesn't care if it's talking to the master or a slave).

I'm aware that just turning the master back on wouldn't sync the writes performed on the slave (it's a slave after all) and the master would have outdated data, but since this is supposed to be exceptional I could easily deal with it manually (swapping the dump files or swapping the master and slave roles and syncing for instance).

Thanks for the response by the way!

P.S.: Maybe this worked before?

[NickCraver]

@tallesl Maybe this will help: we simply change which one is master in that situation, and easily sync the changes back by slaving the other direction. It's a very simple and very solid approach to handling the failure and losing as little data as possible with a bonus of little effort as possible.

[tallesl]

Yeah, that's my idea too but, until some human goes there and swap the roles, all the writes performed to the slave will error using StackExchange.Redis (that's my problem).

(Did I get what you say right?)

[micdenny]

Sorry to kick-in, I'm new to redis replication, and I'm configuring my actual redis production to have a replica, and was wondering how to best maintain redis master-slave, and I was trying to use slave-read-only with se.redis, to do scheduled maintenance, with no success as @tallesl shown.

In other words, which are the steps to do a scheduled maintenance on redis servers (like updating the OS, adding memory, cpu, ...)?

From the documentation I read this where in fact it's saying to set slave-read-only to on and re-route all the clients traffic to the salve node before cutting off the master.

How do you guys do maintenance?

-------------- UPDATE ------------------

Waiting until the support for slave-read-only, I would like to share my solution to fail-over a node with a minimum write interruption, maintaining data consistency, reducing (eliminating?) possible false write acknowledgment (on the node that is failing). You can lose some write because there's a window where both nodes are slaves. I feel that you can still have some false write ack because you can receive an ack while setting the master as slave and that write could be lost.

https://gist.github.com/micdenny/2e62ff26b8aeeb1edd6cabe3d302104f

[tallesl]

Any progress on this issue?

Since I need the ability to write on slave (on an eventual failure), I ended making a private fork of the project which hacks IsMasterOnly to allow me to. I'm not 100% sure this would be the right way to do it, but I've been using it successfully in production for slightly over a year.

It's a struggle to keep my private NuGet package updated. I would love to make a pull request it but I'm afraid I'm not that comfortable with the codebase.

[mgravell]

You should be able to turn this on via IServer:

var server = conn.GetServer(...);
server.AllowSlaveWrites = true;

[NickCraver]

I'll add a test for this in master shortly.

[NickCraver]

Whelp, it's indeed busted, example test:

[Fact]
public void WriteToSlave()
{
    using(var mutex = Create(allowAdmin: true))
    {
        var server = mutex.GetServer(TestConfig.Current.SlaveServerAndPort);
        Assert.True(server.IsSlave);
        var db = mutex.GetDatabase();
        var ex = Assert.Throws<RedisCommandException>(() => db.StringSet(Me(), "Test", flags: CommandFlags.DemandSlave));
        Assert.Equal("Command cannot be issued to a slave: SET", ex.Message);

        server.AllowSlaveWrites = true;
        Assert.True(db.StringSet(Me() + 2, "Test", flags: CommandFlags.DemandSlave)); // still throws
    }
}

The problem is in Message's constructor:

bool masterOnly = IsMasterOnly(command); // return true for all modification ops
Db = db;
this.command = command;
this.flags = flags & UserSelectableFlags;
if (masterOnly) SetMasterOnly(); // throws from the flags mismatch (demanding a slave here)

Similarly, this also fails due to automatic-flags detection:

[Fact]
public void WriteToSlaveDirectly()
{
    using (var mutex = ConnectionMultiplexer.Connect(TestConfig.Current.SlaveServerAndPort))
    {
        var server = mutex.GetServer(TestConfig.Current.SlaveServerAndPort);
        Assert.True(server.IsSlave);
        var db = mutex.GetDatabase();
        var ex = Assert.Throws<RedisConnectionException>(() => db.StringSet(Me(), "Test"));
        Assert.StartsWith("No connection is available to service this operation:", ex.Message);

        server.AllowSlaveWrites = true;
        Assert.True(db.StringSet(Me() + 2, "Test"));
    }
}

Due to ServerSelectionStrategy.Select() determining "only master" the same way.

@mgravell this would be a fairly low-level change in places that don't currently know the about the ServerEndPoint and what's allowed (in the Message constructor case). Thoughts?

I pushed the above tests up in a branch for an easy repro in craver/write-to-slave.