Best practice or approach to manage delta log files of Delta Tables

Question & Answer
,
1

What is the best practice or approach to manage delta log files to control the sizing e.g. using optimize and vacuum?

Thanks.

2

Hi @lbl,
I’ve included a couple articles below that have additional information on best practices for optimize and vacuum.

To my understanding, optimize can help consolidate files where vacuum will help remove unused data files.

I believe that the best practice will depend on what works best for your scenario.

Let me know if you have any additional questions!

3

hi @Brett
The 2 articles shared are on data file. I am referring to files in _delta_log folder.

image
image582×1258 52.6 KB

Thanks.

4

Hello @lbl !

Can you clarify the issue that you are having with QuikSight regarding these log files/tables?

Are you having trouble ingesting them into QuickSight?

5

hi @duncan

Not issue with ingesting into QuickSight.
Housekeeping of files inf _delta_log folder is handled using delta.logRetentionDuration and vacuum.

Thanks.

6

Hey @lbl

I’m not sure if this is something that can be handled on the QuickSight community if it doesn’t relate to a QS specific issue.

I would recommend posting your query in the following community: