Saggy Bottom Pants

I'm in the process of making a pair of green pants. I have not made pants in a really long time, so I'm at a little bit of a loss on how to fix the saggy back side. The rear-end area and through the thighs are really big. I have triple checked my measurements and I made the right size. I don't think the pattern measurement really line up with the garment, I've already taken in an inch on each side seam. The only thing I can think of is to take the pants completely apart and cut out a smaller size and start all over. I'm hoping that you ladies will have a simpler and or faster fix for me.

Here are some pictures of my problem...

Thanks for your help!

Aimie