Scale test Quicksight Dashboards Load Time

Question & Answer
, ,
1

Hello all,

IHAC who would like to scale test Quicksight Dashboards to simulate up to 50 concurrent users as the expected number of concurrent users during the peak time > 100. The customer has just migrated from Oracle/Qlik. Currently, Quicksight Dashboards are configured in Direct Query to Amazon Redshift, retrieving Billions of rows.

1/ How to invoke the Quicksight dashboards programmatically to scale test the dashboards and simulate concurrent users ?
2/ How the caching in Quicksight Direct Query with Amazon Redshift works, if any ?
3/ I do not see all the data points of CloudWatch DashboardViewLoadTime when opening the Dashboards in parallel containerized sessions in my browser. Is it expected ?
4/ Is there any reference customer using both Quicksight and Redshift at scale with the number of users, the size of Amazon Redshift cluster and data volume/number of rows ?

Any help is much appreciated.

Many thanks!
Hassen

2

@hriahi Hopefully this gives some direction to you. QuickSight auto scales on the website layer so 100 concurrent users is quite small a load.
1/QuickSight doesn’t have a mechanism where you can set up a load test. If you do build your own programmatic mechanism, do realize that there is throttling that can kick in based on the rate of calls.
2. When using RedShift with direct query, caching is really happening at the RedShift layer. You would need to configure RedShift and enable cache there (Performance - Amazon Redshift)
3. That is not expected - do you not see them in near real time or historically too?
4. If you don’t get help from rest of the community, I would suggest to work with your QuickSight account contacts to seek help on the set up based on your particular usecase

3

Can you embed your dashboard and do your load testing by accessing the dashboard through your web application?

4

Many thanks @Shekhar and @David_Wong for your support!

@Shekhar for 3/, I do not see them neither in real-time nor historically. Maybe it does count only for individual users ? In my case, I am using the same user in containerized parallel sessions in my web browser. I reported the issue here as well https://t.corp.amazon.com/D50622991. If it does count only for individual users, then in my case I need to create 100 individual users in QuickSight.

@David_Wong I can try to setup a web application and embed the dashboard into it. As I do not have experience with the embedding, will the dashboard load automatically by loading the main web page where the dashboard is embedded ?

Many thanks!

5

@hriahi Yes, your embedded dashboard will load automatically when the web page loads.

6

Thank you @David_Wong ! I am giving it a try.

7

@hriahi @David_Wong

I am so glad that this article existed. We are trying to solve something similar. Team has successfully embedded the dashboard onto a webpage. We are trying to learn and understand ways to do performance testing of the dashboard. If you have any information then please share

8

@ssy1
You can take a look at DashboardViewLoadTime and VisualLoadTime on this page:

You can use CloudWatch to see how long your dashboards and visuals take to load.

2 Likes
9

I tried K6 for browser based load tests.

Refer: Running browser tests

Once we have embed URL generated for dashboard, we can use this for performing load tests.

import { browser } from 'k6/experimental/browser';

export const options = {
    scenarios: {
        browser: {
            executor: 'constant-vus',
            vus: 5,
            duration: '1m',
            options: {
                browser: {
                type: 'chromium',
                },
            },
        },
    },
    thresholds: {
        checks: ["rate==1.0"]
    }
}

export default async function () {
  const context = browser.newContext();
  const page = context.newPage();

    try {

        await page.goto("YOUR_DASHBOARD_EMBED_URL");

        // Sheet 1 has 1 chart. Wait for default sheet's chart element to be available
        page.waitForSelector('#block-50e95175-xxxx-xxxx-a52c-18596b3a3880_a34a2b28-310e-4fcb-9747-450537a0b04f div[data-automation-id="resizer_horizontal"]');

        // Taking sheet 1's screenshot
        page.screenshot({ path: 'screenshots/sheet1.png' });

        // Waiting for tabs / dashboard sheet's links to show up
        page.waitForSelector('.sheet-tab[aria-label="sheet2"]');

        // Clicking on sheet 2
        page.click('.sheet-tab[aria-label="sheet2"]')

        // Sheet 2 has 3 charts. Wait for all 3 sheet's element to be available
        page.waitForSelector('#block-50e95175-xxxx-xxxx-a52c-18596b3a3880_edb3f7e3-9525-4082-b64c-7eca1c55b6ee div[data-automation-id="resizer_horizontal"]');
        page.waitForSelector('#block-50e95175-xxxx-xxxx-a52c-18596b3a3880_390d20ce-2524-4bd6-95aa-0161d10171f3 div[data-automation-id="resizer_horizontal"]');
        page.waitForSelector('#block-50e95175-xxxx-xxxx-a52c-18596b3a3880_2bdbf13a-5fca-4dc9-82e0-de6f6f9de374 div[data-automation-id="resizer_horizontal"]');

        // Taking sheet 2's screenshot
        page.screenshot({ path: 'screenshots/sheet2.png' });
      

  } finally {
    page.close();
  }
}

Powershell Command to execute the tests:

PS C:\Program Files\k6> .\k6 run browser_tests.js

This produces output with following information:

iteration_duration.............: avg=10.59s   min=9.13s    med=10.2s    max=14.08s   p(90)=13.03s   p(95)=13.23s

I tried Http protocol based tests (web recording & running tests in K6/Jmeter) but since all QuickSight dashboard contents are loaded in browser via web socket communication, this was not straightforward.

Load testing using recordings : Create tests from recordings

10

Regarding Redshift Direct caching, the link provided in accepted answer covers it - Performance - Amazon Redshift

The 2 main factors were:

  1. Instance type of cluster. On smaller sized instance like DC2.Large, the cache was not created. Similar query on RA3.xLarge was cached
  2. If we use functions like getdate, the query was not getting cached. Workaround like maintaining latest date in a table and referring it in queries works