Troubleshooting MongoDB Connection Timeout Error between DataHub and EC2 Instance

Original Slack Thread

Hi , we are getting below error while connecting datahub with mongo db
Failed to configure the source (mongodb): Could not reach any servers in [(‘ip-x-x-x-x’, 27017)]. Replica set is configured with internal hostnames or IPs?, Timeout: 30s, Topology Description: <TopologyDescription id:6xxxxxxx91, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('ip-x-x-x-x, 27017) server_type: Unknown, rtt: None, error=NetworkTimeout(‘ip-x-x-x-x:27017: timed out’)>]

networking issue, where your DataHub is deployed ?

Hi <@U0348BYAS56>, Datahub is deployed on AWS EC2 instance , platform ubuntu

<@U05JJUZ6DQF> mongo Db is on same EC2? If it is on different EC2 then please check EC2 security group policy, port need be allow

Hi <@U0348BYAS56>, Mongo DB is on different EC2, I will try to allow port in security group policy

Hi <@U0348BYAS56>, which port we should allow in security group?


<@U0348BYAS56>, it has been already added, please see below screenshot

Ok still it is not working, <@U03MF8MU5P0> might help you

The error is definitely a timeout. Is mongo listening on the expected IP or perhaps mistakenly only on loopback? Are the ec2 instances in the same vpc? You can use normal tools like telnet/curl/netcat to check for the expected open port from one instance to another. I suspect that despite the security group inbound rule, there is something else with the network configuration which is dropping traffic to your mongo instance.

Hi <@U03MF8MU5P0>, just to give you more information, when we disable the replica on mongo server then ingestion works fine without giving any error

I am not 100% familiar with mongodb, it looks like the connection url should be a bit different then normal. Are you using a url string like this? <mongodb://>,<;authSource=authDB|;authSource=authDB> with the replicateSet name? Or have DNS configured with the extra TXT record as indicated in the mongodb|docs?

I don’t see how not using that version of the connection url would result in a timeout though.

Is it possible to debug using the mongodb shell, can you connect fine using that cli tool?

<@U03MF8MU5P0>: Mongo server and datahub are hosted in different VPCs

Are the VPCs peered?

Hi <@U03MF8MU5P0>, I tried on mongo shell , getting below error
Please enter a MongoDB connection string (Default: mongodb://localhost/): mongodb://x.x.x.x:27017/?replicaSet=myrep
Please enter a MongoDB connection string (Default: mongodb://localhost/): mongodb://x.x.x.x:27017/?replicaSet=myrep
Current Mongosh Log ID: xxxxxxxaaaassssdddd
Connecting to: mongodb://x.x.x.x:27017/?replicaSet=myrep&amp;appName=mongosh+1.10.6
MongoNetworkError: getaddrinfo ENOTFOUND ip-y-y-y-y
Press any key to exit:

That error indicates that you do not have working DNS in your environment. Check if you have <|DNS setup> in your VPCs and whether your ec2 hosts are properly configured to use it.

Hi <@U03MF8MU5P0>, Do you think it is mandatory to configure DNS since when we are able to connect fine when replica is not enabled by using IP (not DNS) in url e.g. ‘mongodb:’/dbname/’ ?