[FIXED] How does connection pooling work in the following cases specific to MongoDB-MongoClient driver?

Issue

A connection pool is a pool of db connections that is loaded when the application starts. When a transaction comes in, the driver will pick a connection from the connection pool to serve the transaction. A connection to the database happens when both ends open a socket over tcp for exchanging messages. Will that socket connection be active at the time the connection is in the pool? or a new socket connection will start once picked up from the connection pool?. If the latter then what's really stored in the connection pool?. I have an application configured with the following mongo values:

KeepAlive=true&poolSize=20&autoReconnect=true&socketTimeoutMS=5000&connectTimeoutMS=30000&maxTimeMS=5000

The application starts with a pool with 20 db connections.

If the sockettimeoutms is 5000, what will happen to the connections if they are not used within a timeframe of 5 seconds in the pool? or will socketTimeoutMs just be used in a transaction context?

For connectTimeoutMS I believe it is used at the time the application attempts to create that socket connection. When a transaction comes in, will the server attempt to create a socket connection or will that be already done at the pool level?.

For maxTimeMS, I attempted to test this value by setting it to 1 ms and trying to send a query to mongo with the following statement:

    @Autowired
    protected MongoTemplate mongoTemplate;

 mongoTemplate.findAll(Myclass.class);

It didn't fail. However it does fail when I set either connectTimeout or socketTimeout to 1ms which is expected.

Regarding autoReconnect, I didn't any information on it. I believe it is used along with KeepAlive in case the driver detects an issue with the network it will attempt to reconnect.

Solution

For a very detailed description of how connection pools are supposed to work in MongoDB drivers, see this spec. Keep in mind the following caveats:

• Some drivers have existing connection pool behavior that may contradict the requirements in this specification. This behavior may be on by default for compatibility reasons. Consult your driver documentation to see if this is the case.
• Drivers not maintained by MongoDB generally are slower to implement driver specifications such as this one and hence are more likely to have different pool behavior.
• This specification deals principally with connection pools and even though it defines behavior of individual connections to some extent, some of the individual connection behavior isn't covered.

A connection to the database happens when both ends open a socket over tcp for exchanging messages

Technically the server socket is opened much earlier. Strictly speaking, only the client socket is opened when a connection is made.

Will that socket connection be active at the time the connection is in the pool?

Yes. Unless the connection was torn down by the operating system or a network device between the client and the server and the driver hasn't learned this. TCP keepalive is supposed to inform the driver and the server of such things happening but it doesn't detect problems instantly.

Furthermore, drivers aren't currently obligated to perform liveness checking on connections prior to using them, hence they may receive errors because they think a connection is good but actually it isn't. Some drivers may perform liveness checking and be immune to this issue.

or a new socket connection will start once picked up from the connection pool?

This was legacy behavior in Ruby driver. It is not compliant with current CMAP spec.

If the latter then what's really stored in the connection pool?

A lightweight object referencing the connection, but not a TCP socket that is connected to the server.

If the sockettimeoutms is 5000, what will happen to the connections if they are not used within a timeframe of 5 seconds in the pool?

Socket timeout only applies when an operation is performed. It is not applicable to an idle connection.

Once an operation is sent, the driver must receive a response in 5 seconds or it will fail the operation with a network error.

If the operation takes more than 5 seconds, the server may well be working on performing it but the driver will report a network error. For this reason a low socket timeout is generally not very usable. Instead use TCP keepalive (which drivers configure automatically if possible).

or will socketTimeoutMs just be used in a transaction context?

Transactions have nothing to do with connections, you are misusing the term.

For connectTimeoutMS I believe it is used at the time the application attempts to create that socket connection.

Right.

When a transaction comes in, will the server attempt to create a socket connection or will that be already done at the pool level?.

For the purposes of the present discussion, there is no such thing as a "socket connection". There are sockets and there are connection objects. As mentioned earlier, the server does not create sockets. The server would create a connection object corresponding to the received connection.

One can say that a socket establishes a "socket connection" on the operating system level, but this is on a level below connection pools and connection objects as they are being discussed in this answer.

For maxTimeMS, I attempted to test this value by setting it to 1 ms and trying to send a query to mongo with the following statement:

Two possible reasons:

1. You did not set the option correctly. Use command monitoring to see the actual commands the driver is sending to the server, verify maxTimeMS is there and it is used as per server documentation.
2. maxTimeMS is enforced by the server. It may well be checked at specific points in query execution and if query can be executed quickly it may technically exceed the specified timeout and succeed. Reference server documentation for any caveats and test with an actual long running query (you can use $where and sleep for example).

Answered By - D. SM