Using this suggestion: minOver(min({date}), [{date}])
And filtering to D-1, displays only D-1 while having my AvgOver calculate correctly.
I am not sure why this works though, although this is end of day and my brain may be mush.
After further testing as well, my previous parameter is not making my avgover excluded to just the past 7 days, and does change if I further the date filter. This is a bit concerning if I aim for larger time frames.