Row limit issue

Question & Answer
,
1

Hello I just want to understand even after creating partition. How can I join these datasets so data can be represented in single dashboard. Please help

2

Hi @deepa.singh
how many rows are we talking about?
BR

3

More than billion rows as of now

4

Around 4 billion rows

5

SPICE or Direct Query?

6

Spice , I know the limit but if we create dataset for different region : the row limit is still maintained I want to know how to join those datasets so it can be displayed in one dashboard

7

This question has an answer to pull data in multiple set but not sure how to join them

8

Hi @deepa.singh
so you mean you have partitioned by e.g. year (one for 2023 and one for 2022) and want to join it into one dataset?
BR

9

Yes correct , Can this be done

10

@ErikG - Please share an update here

11

But do you want to join or merge?
Because join would create more columns and merge more rows.

12

I want to merge so we can have more rows

13

@deepa.singh ,

if the row limit at a dataset level is what you are talking about, then we currently support 1 billion records / 1 TB which ever comes first. So you cannot have more rows.
You can have multiple datasets which can be used in the same dashboard.

Kind regards,
Koushik

14

Hi @deepa.singh
as @Koushik_Muthanna said. And it would work anyway to merge two datasets each 0,6 billion records as you would reach 1,2 billion.

Any chance to reduce the rows, e.g. by aggregation or somethingelse?

BR

15

Hello @ErikG and @Koushik_Muthanna
The row limit applies to one spice dataset at a time what if i want to merge multiple dataset each one of them is not breaching row limit but together they will this still apply?

Because as per comments on ROW_SIZE_LIMIT_EXCEEDED - what to do next? - #4 by chridoms

We can join multiple datasets to fix the issue.

16

So you want to merge two SPICE datasets (each 0.4 billion rows) into one Direct Query (on SPICE) dataset?

I’m not aware of a merge/union function.

@thomask wrote about COALESCE

17

@erik . What if we want to merge 10 dataset with similar row rows the overall row count will increase right… will this still work

18

Hi @deepa.singh - If your overall row count > 1billion, it will throw the error. I believe, you should perform some level of aggregation or filter and reduce number of rows and then bring the data to QS for reporting.

Regards - Sanjeeb