The base of your Regency outfit is a shift or chemise, next comes the stays, then petticoat, chemisette (if daytime) and gown; and if outside, spencer or pelisse and bonnet.

For stays (note this is different to the Victoria tight-lacing corset), you have half stays and full length stays.
For examples of where to buy half stays please see Redthread and Nemuro.
For examples of where to buy full stays please see Redthread and Nemuro.

Chemisette
For example of a chemisette please see eBay.

Cheaper Regency gowns, which do not require stays underneath them:
Blue gown on Amazon
Green gown on Amazon
Striped dress on Amazon
Striped dress on Light in the Box