In this talk, I will discuss the relative $K$-stability and modified $K$-energy associated to the Calabi's extremal metrics on toric manifolds. I will show a sufficient condition in the sense of polyhedrons associated to toric manifolds for both relative $K$-stability and modified $K$-energy. In particular, our result holds for toric Fano manifolds with vanishing Futaki invariant. We also verify our result on toric Fano surfaces.