[FIXED] MongoDB loading +3000 entries data pagination

Issue

I have to serve an HTML tables with more than 3000 results, getting MachineID, Username, Data from my MongoDB. BUT, i'm having a problem when im rendering the data to datatables.

The Data and MachineID comes from model/table Logs, and the Username comes from User. The relation between them will be user_id from Logs and the _id object from User.

As you can see on my query, im trying to compare pc.user_id with the object _id of parent table, to get the username of that process.

My query:

Pincode.find({}).exec(async (err: any, pincodes: any) => { 
        for (var pc of pincodes){
            var uid = pc.user_id;
            var res = await User.findById({ _id:ObjectId(uid) },'username').exec();
            if(res!= null){
                pc.user_id = res.username
            }else{
                pc.user_id = "Unknown"
            }
        }

But this is slowing SO much my query because im looping trhougt all +3000 documents.

Thanks for your answers,

Solution

You can try this:

let getData = async () => {
  try {
    let pincodeData = await Pincode.find({}).exec()
    if (pincodeData.length < 1) {
      return res.send({"error": true, "message": "No data found"});
    }
    await Promise.all(
      pincodeData.map(async (x) => {
        let getUserData = await User.findOne(
          { _id: ObjectId(x.user_id) },
          { _id: 0, username: 1}
        ).exec();
        x.user_id: 'Unknown';
        if (getUserData) {
          x.user_id: getUserData.username;
        }
      })
    )
    return res.send({ "error": false, "message": "Data found", "data": pincodeData });
  } catch (error) {
    return res.send({ "error": true, "message": "No data found" });
  }
}
getData()

Answered By - Aks