Derived output schema is no longer compatible with the expected output columns

I have a dataset that uses another dataset as source and all of a sudden the refresh it failing with the below.
But the field was there both in the parent and in the child datasets.

I tried to remove and then readd the field and then I got the below error.

This happens only when the parent dataset is in direct query mode. When I put it in SPICE all refreshes are done correctly.
How can I fix this?

Hi. Thanks you for your question! :slight_smile: Many of our community Experts and folks from our team are on holiday right now. I asked “My Assistant” (universal chat) in Quick Suite, and got this back.
@Brett Could you validate the answer below on Monday after you return from holiday?

”Here’s what this error typically means and how to resolve it:

What This Error Means

This error occurs when there’s a mismatch between the schema that QuickSight expects and the actual schema of your dataset. This commonly happens when:

  • Column names have changed in your data source

  • Columns have been added or removed

  • Data types have changed

  • Custom SQL queries have been modified

Resolution Steps

1. Refresh the Dataset Schema

  • Navigate to your QuickSight dataset

  • Click “Edit dataset”

  • QuickSight should automatically detect schema changes

  • Click “Save & Publish” to apply the changes

    1

2. For Incremental Refresh Datasets If you’re using incremental refresh, the issue may be that the refresh schedule retains old metadata. The fix is to:

  • Go to your dataset settings

  • Find the “Reset Incremental refresh” option

  • Click to reset it

    2

3. Check for Filter Issues This error commonly occurs when filters reference columns that no longer exist or have been renamed:

  • Remove any filters that reference old column names

  • Update filters with the new column names

    3

4. Verify Data Source Connection

  • Ensure your data source is accessible

  • Check that permissions haven’t changed

  • Verify the underlying data hasn’t been corrupted

    4

Prevention Tips

  • When making schema changes, edit the dataset in QuickSight immediately after

  • Test schema changes in a development environment first

  • Document any column name changes for your team”

Hi @Kristin,

Nothing of the above has happened in my case.

All fields are there, nothing has changed and all of a sudden I get this error and nothing works whatever I do.

Thanks

Hi @pantelis,
Does it have anything to do with a calculated field being involved? And if so, could you share a bit more about what the calculated field is doing?
If it’s working when in SPICE but not working correctly with Direct Query, my thoughts are that something is relying on the output of a calculated field; so when in SPICE, the output is pre-ran, therefore making them easier to utilize. Where with Direct Query, it has the re-run the query each time and may affect.

Hi @Brett

The calculated field is just a simple field that checks if the type is final and if the currency is USD and applies a multiplication.

ifelse({Type}='FINAL',
     ifelse({Currency}='USD',
          {epfi_discount_amount},
          {epfi_discount_amount}*{Rate})
,NULL)

The epfi_discount_amount comes from a parent dataset and it is not a calculated field. It is a column from a table extracted directly from our db.

The parent dataset is in direct query with absoletuly no problem and the child dataset when it is refreshed it shows me the below error.
It says “Didn’t find field epfi_discount_amount”.

I have deleted the calculated field and excluded the epfi_discount_amount field from child and parent datasets and the include it again but the issue persists.

Hi @pantelis,

Thanks for providing further clarification regarding your issue! I believe that because the parent dataset is in Direct Query, the child dataset runs into an error as it is likely running into reference issues when querying between both of them. Would it be possible to combine both datasets into one (i.e. putting data from each into different sheets within book)? If doable, this workaround should hopefully bypass this refresh error.

Let me know if this suggestion helps!