Custom Sort of X-Axis Values in Bar Graph

Are you sorting by the Max or Min of the calculated field? On the sort by side panel, after selecting more sort options, you want to select Max or Min from this dropdown:

image