Troubleshooting MongoDB Connection Timeout Error between DataHub and EC2 Instance

Original Slack Thread

Hi , we are getting below error while connecting datahub with mongo db
Failed to configure the source (mongodb): Could not reach any servers in [(‘ip-x-x-x-x’, 27017)]. Replica set is configured with internal hostnames or IPs?, Timeout: 30s, Topology Description: <TopologyDescription id:6xxxxxxx91, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('ip-x-x-x-x, 27017) server_type: Unknown, rtt: None, error=NetworkTimeout(‘ip-x-x-x-x:27017: timed out’)>]

<@U05JJUZ6DQF>
networking issue, where your DataHub is deployed ?

Hi <@U0348BYAS56>, Datahub is deployed on AWS EC2 instance , platform ubuntu

<@U05JJUZ6DQF> mongo Db is on same EC2? If it is on different EC2 then please check EC2 security group policy, port need be allow

Hi <@U0348BYAS56>, Mongo DB is on different EC2, I will try to allow port in security group policy

Hi <@U0348BYAS56>, which port we should allow in security group?

27017

<@U0348BYAS56>, it has been already added, please see below screenshot

Ok still it is not working, <@U03MF8MU5P0> might help you

The error is definitely a timeout. Is mongo listening on the expected IP or perhaps mistakenly only on loopback? Are the ec2 instances in the same vpc? You can use normal tools like telnet/curl/netcat to check for the expected open port from one instance to another. I suspect that despite the security group inbound rule, there is something else with the network configuration which is dropping traffic to your mongo instance.

Hi <@U03MF8MU5P0>, just to give you more information, when we disable the replica on mongo server then ingestion works fine without giving any error

I am not 100% familiar with mongodb, it looks like the connection url should be a bit different then normal. Are you using a url string like this? <mongodb://mongodb1.example.com:27317>,<http://mongodb2.example.com:27017/?replicaSet=mySet&amp;authSource=authDB|mongodb2.example.com:27017/?replicaSet=mySet&amp;authSource=authDB> with the replicateSet name? Or have DNS configured with the extra TXT record as indicated in the mongodb https://www.mongodb.com/docs/manual/reference/connection-string/|docs?

I don’t see how not using that version of the connection url would result in a timeout though.

Is it possible to debug using the mongodb shell, can you connect fine using that cli tool?

<@U03MF8MU5P0>: Mongo server and datahub are hosted in different VPCs

Are the VPCs peered? https://docs.aws.amazon.com/vpc/latest/userguide/route-table-options.html#route-tables-vpc-peering

Hi <@U03MF8MU5P0>, I tried on mongo shell , getting below error
Please enter a MongoDB connection string (Default: mongodb://localhost/): mongodb://x.x.x.x:27017/?replicaSet=myrep
Please enter a MongoDB connection string (Default: mongodb://localhost/): mongodb://x.x.x.x:27017/?replicaSet=myrep
Current Mongosh Log ID: xxxxxxxaaaassssdddd
Connecting to: mongodb://x.x.x.x:27017/?replicaSet=myrep&amp;appName=mongosh+1.10.6
MongoNetworkError: getaddrinfo ENOTFOUND ip-y-y-y-y
Press any key to exit:

That error indicates that you do not have working DNS in your environment. Check if you have <https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html|DNS setup> in your VPCs and whether your ec2 hosts are properly configured to use it.

Hi <@U03MF8MU5P0>, Do you think it is mandatory to configure DNS since when we are able to connect fine when replica is not enabled by using IP (not DNS) in url e.g. ‘mongodb:192.1.1.85:27017’/dbname/’ ?