"nonlinear mixed model estimation done with the original data set, but not converge with some selected sub data sets"

Leaving 20% data out, is still validation process. Yes, it's part of the model selection.
Also if you are doing 10,000 times simulation, and as you said "not convergent with some selected sub-data sets"  if only "some" were not convergent, I don't think it is a major concern anways.
The reasoan to do the simulation is to assess the robustness of measurements of models' coefficients. If "some" not convergent, then it may tell that the models done using those subsamples were not robust enough to the true model.