It was definitely the hub inside the bearing and it was pretty bad, I'd say about 1-2mm of play at the end of the hub. Once I replaced the bearing the play was gone. I thought for sure that was going to fix the noise but it didn't even make a difference.
The noise is just like what you are describing, maybe the other bearing is bad too - there was a slight clicking when wiggling the left hub.