I noticed that there's a Speedo brand brief with design that's inspired by some underwear
http://www.speedousa.com/3-active-brief.shtml
with a waist band on the top, (I recall one web site did say it was styled like underwear)
Then there are many men's brief style underwear that's made similar to a swim brief -- made with nylon/poly plus lycra. For example, Calvin Klein has many and other fashion underwear brands like 2(x)ist.
The main difference seems to be that swimsuits have drawstrings while underwear has an elastic waistband, and that swimwear typically has a lining and underwear/underpants tend to be made with slightly thinner fabric.
So for people who like to wear swimbriefs for purposes other than for swimming, it seems that the underwear manufacturers are already making pants that meet that need . They are smooth and somewhat stretchy. Very unlike typical cotton briefs.
And for people who swim, it's interesting that at least one Speedo model is made to look like underwear/underpants.