Hi @shinya.sakae - Yes, RDS proxy is helpful when you want to control number of connections from client. You can configure it from AWS management console → RDS page. Once you created it, you can connect using the RDS proxy end point.
Can you please try this, if you are facing any error let us know.
Regards - Sanjeeb