QS / Salesforce Datasource and Incremental Refresh Issue

Q&A
,
1

Hello,

I have setup Datasets pulling data from different Salesforce Objects (Custom and native).

The full refreshes are working properly however once I setup Incremental refreshes (Using SysModStamp with different window sizes) then my issues began…=

  1. The Ingested # of rows always matches the number of records contained in Salesforce and the Duration is always about the same as the Full Refresh. It seems to me like there is no filtering applied on the data being pulled from SF e.g. it pulls everything out whether it’s a full or an incremental. I’ve tried different datetime fields (LastModifiedDate and SysModStamp) and different Window size and scheduled. Can someone confirm this?
  2. My bigger issue is that for 2 of the 5 datasets I have created, they ingest the entire row sets daily and also APPENDS them to the spice dataset on EVERY incremental refresh. So a dataset which started with 3 millions rows a few days ago is now growing by roughly 3millions each day. As you can see below…
    image
    image1539×417 56.7 KB

I have Googled, youtube etc.. and enlisted the help of a few AI and could not find. I’m hoping someone with QS/SF experience can pitch in.

I did not run into these issues setting up datasets from PGSQL so suspecting it is a salesforce specific issue.

This is the best video I have found explaining how incremental work: https://www.youtube.com/watch?v=jl_91ahT-P8

Thank you

2

Hi @steve.rochefort

Welcome to the Quick Suite community!

As per the official documentation on Refreshing SPICE data, incremental refresh is explicitly supported for SQL-based data sources only: For SQL-based data sources, such as Amazon Redshift, Amazon Athena, PostgreSQL, or Snowflake, you can refresh your data incrementally within a look-back window of time.

Salesforce is listed as a SaaS (OAuth) connector rather than a SQL-based data source, and it is not included in the incremental refresh section of the documentation. You can confirm this on the Supported Data Sources page, where Salesforce is listed separately under Software as a Service (SaaS).

This also aligns with your observation that you did not encounter these issues with PostgreSQL, because PostgreSQL is a supported SQL-based source where the lookback window filter is applied as expected.