If you want to put haproxy in front of mongodb cluster, haproxy needs to know the status of mongodb nodes. I’ve found this config, which uses mongos native ping command. But I needed to know, which node is master (because I use simple replication, not a shard) so I can send write queries. For this purpose, I’ve modified that config, so it uses isMaster() function and expects “ismaster\x00\x01” as result.
Feel free to use or modify.
listen mongodb_cluster bind 10.0.0.10:27017 option tcp-check # MongoDB Wire Protocol tcp-check send-binary 3a000000 # Message Length (58) tcp-check send-binary EEEEEEEE # Request ID (random value) tcp-check send-binary 00000000 # Response To (nothing) tcp-check send-binary d4070000 # OpCode (Query) tcp-check send-binary 00000000 # Query Flags tcp-check send-binary 61646d696e2e # fullCollectionName (admin.$cmd) tcp-check send-binary 24636d6400 # continued tcp-check send-binary 00000000 # NumToSkip tcp-check send-binary FFFFFFFF # NumToReturn # Start of Document tcp-check send-binary 13000000 # Document Length (19) tcp-check send-binary 10 # Type (Int32) tcp-check send-binary 69736d617374657200 # ismaster: tcp-check send-binary 01000000 # Value : 1 tcp-check send-binary 00 # Term tcp-check expect binary 69736d61737465720001 #ismaster True option tcpka option tcplog server mongo-node-01 10.0.0.11:27017 check inter 2000 server mongo-node-02 10.0.0.12:27017 check inter 2000
very good article. thank you very much
Hi Danman,
I configured haproxy like that and really works fine.
Although there is a communication problem when I want to connect with an external client:
MongoDB shell version: 3.2.8
connecting to: 192.168.2.45:27017/test
2016-07-22T00:09:42.300+0200 D NETWORK [thread1] creating new connection to:192.168.2.45:27017
2016-07-22T00:09:42.300+0200 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG
2016-07-22T00:09:42.301+0200 D NETWORK [thread1] connected to server 192.168.2.45:27017 (192.168.2.45)
2016-07-22T00:09:42.301+0200 I NETWORK [thread1] recv(): message len 1347703880 is invalid. Min 16 Max: 48000000
2016-07-22T00:09:42.302+0200 D – [thread1] User Assertion: 6:network error while attempting to run command ‘isMaster’ on host ‘192.168.2.45:27017’
2016-07-22T00:09:42.302+0200 D – [thread1] User Assertion: 1:network error while attempting to run command ‘isMaster’ on host ‘192.168.2.45:27017’
2016-07-22T00:09:42.302+0200 E QUERY [thread1] Error: network error while attempting to run command ‘isMaster’ on host ‘192.168.2.45:27017’ :
connect@src/mongo/shell/mongo.js:231:14
@(connect):1:6
2016-07-22T00:09:42.303+0200 D – [thread1] User Assertion: 12513:connect failed
2016-07-22T00:09:42.303+0200 I QUERY [thread1] MozJS GC prologue heap stats – total: 1589501 limit: 0
2016-07-22T00:09:42.306+0200 I QUERY [thread1] MozJS GC epilogue heap stats – total: 1229 limit: 0
2016-07-22T00:09:42.306+0200 I QUERY [thread1] MozJS GC prologue heap stats – total: 3653 limit: 0
2016-07-22T00:09:42.306+0200 I QUERY [thread1] MozJS GC epilogue heap stats – total: 5 limit: 0
2016-07-22T00:09:42.307+0200 D – [main] User Assertion: 12513:connect failed
exception: connect failed
Have you had this problem in your configuration?
Maybe haproxy is sending tcp over http to mongodb master server although it is configured in tcp mode.
Thanks in advance,
Dani
Hey, I am a bit confused behind what architecture have you used. May be i could explain you my point of view of implementing you so that you could help me out asap.
Thanks.
Ok, go ahead…
above configuration worked for me
Why do we need to use haproxy? Should not driver with replica connection string set know which server is primary automatically and redirect traffic?
This works perfectly if all the mongo nodes are online.
However, my primary node and any offline mongo nodes (with nothing running on port 27017) are appearing as healthy. HAProxy 1.5.8.
You might want to use a newer version
I can confirm it works very well (see my post below) with version 1.7 and mongo 3.2.x 3 nodes
Awesome article!
I used it for an application which was not compatible with replicaset and works well.
One suggestion: in my case, the application remained stuck on the slave server in case of failover (rs.reconfigure).
On the backend definition I used:
“on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions”
and worked like a charm.
Bye!
This is really helpful, thanks!
Hey, I have a question –
I have a proxy which forwards traffic to certain db, including Mongo
I changed Mongo to mode requireTLS, and now this health check that I added does not work. I get an error:
Server mongo/mongo-replica-set-172.16.0.1 is DOWN, reason: Socket error, info: ” at step 16 of tcp-check (expect binary ‘69736D61737465720001’)”
This is my code:
backend mongo
option tcp-check
tcp-check comment “Check mongo replica set node is primary”
# MongoDB Wire Protocol
tcp-check send-binary 3a000000 # Message Length (58)
tcp-check send-binary EEEEEEEE # Request ID (random value)
tcp-check send-binary 00000000 # Response To (nothing)
tcp-check send-binary d4070000 # OpCode (Query)
tcp-check send-binary 00000000 # Query Flags
tcp-check send-binary 61646d696e2e # fullCollectionName (admin.$cmd)
tcp-check send-binary 24636d6400 # continued
tcp-check send-binary 00000000 # NumToSkip
tcp-check send-binary FFFFFFFF # NumToReturn
# Start of Document
tcp-check send-binary 13000000 # Document Length (19)
tcp-check send-binary 10 # Type (Int32)
tcp-check send-binary 69736d617374657200 # ismaster:
tcp-check send-binary 01000000 # Value : 1
tcp-check send-binary 00 # Term
tcp-check expect binary 69736d61737465720001 #ismaster True
# Mongo server (using DNS)
server mongo mongodb.service.gc.guardicore:27017 check port 27017 resolvers consul_resolver inter 2000 fall 20
# Mongo replica set. Configure a server for each node. servers will be turned off if not primary.
# Check every 2000 seconds. If server fails to be primary for 20 consecutive checks, it’ll be marked as down
server mongo-replica-set-172.16.0.1 172.16.0.1:27017 check port 27017 inter 2000 fall 20
Can you please help me understand what do I need to add in order to get this health check to work?
Not sure if that’s relevant but I have a service that is connecting to Mongo successfully using tls, so the forwarding works, it does not terminate ssl, but the health check itself does not work.
I tried to add “tcp-check connect ssl”, so it was:
backend mongo
option tcp-check
tcp-check connect ssl
tcp-check comment “Check mongo replica set node is primary”
…….
But then I got an error when trying to run the ssl-proxy:
[ALERT] (35) : config : [/local/ssl_proxy.conf:188] : ‘server mongo/mongo-replica-set-172.16.0.1’ : verify is enabled by default but no CA file specified. If you’re running on a LAN where you’re certain to trust the server’s certificate, please set an explicit ‘verify none’ statement on the ‘server’ line, or use ‘ssl-server-verify none’ in the global section to disable server-side verifications by default.
So I tried in the server commands to add at the end of the line “check ssl verify none” / “check-ssl verify none” and I get an error:
Server mongo/mongo-replica-set-172.16.0.1 is DOWN, reason: Layer7 invalid response, info: “TCPCHK got an empty response at step 16”
Interesting but I have no idea how to solve this. Do you see something in logs on mongo side?