Quicksight - Opensearch high Memory comsumption

Hi there.

I’ve been working with quicksight and opensearch integration, I created a SPICE dataset wich consumes data from my Opensearch index.

A few day ago it started to have a intermitent fail. I’ve been looking into my Opensearch cluster and it’s hiting 99% memory usage while refreshing. I read that quick supports incremental refresh, but only with SQL based datasets.

Does anyone ever had problem like this? My Cluster is 4vCPU/32GB RAM, and my index have about 4M docs with 20k growth per day. Tip: I recently updated my cluster from 2/16, and 99% reamains.

I have a few lambdas that do a ANN (Approximate Search), maybe this is caching registers and bring me trouble? This ANN search happens in separated index with not more than 1000 docs.

I thinking about to create subindexes that have fewer data indexed by dates, 1 Month, 2 Months… but sometimes i have the necessity of compare YEAR growth for my clients.

@Pedro_Henrique ,

Maybe post the question here to get help on your opensearch question.

Kind regards,
Koushik

1 Like