The point of Matrix, as the name implies, is to link together and decentralise conversations between different silos. I can guarantee that people you know use Matrix, given there are an ever-increasing number of networks which it links together into the resulting meta-network - for instance, all of Freenode, Moznet, W3C irc, and any other bridge you or others choose to run and expose to the wider Matrix ecosystem.

In other words, you don't have to be running a native matrix client like Vector. There are probably people you already interact with today via (say) Freenode who are bridged in via Matrix - they typically have an M- prefix on their IRC nick, although we're shortly changing to an [m] suffix.