That’s an accurate expectation. Only about 1-2% of users choose to explicitly add location data to their Tweets, and the statuses/filter endpoint is capped to 1% of the firehose, so you’re looking at a very small number of Tweets that might match your filter and receive those fields.
You can read more about when and why these fields are populated here.
For higher fidelity on volume, you’d need to look at enterprise PowerTrack, but even then, the limitation on the number of Tweets that carry location information remains.