The following is an example of estimation of an ANST-GARCH model (Asymmetric Non-linear Smooth Transition GARCH model). This has a two branch GARCH model with a logistic transition based upon the value of the lagged residual. (The related GJR asymmetric model has a "sharp" transition at 0).
Note that the full ANST-GARCH, which allows all the GARCH model parameters to shift, in practice probably won't work well, mainly because of the behavior of the "GARCH" (lagged variance) parameter. González-Rivera(1998) does a very similar model, but only allows the "ARCH" (lagged squared residual term) to shift. This example pegs the GARCH parameter in the transition branch to 0, as the model produces unreasonable results (two very ill-behaved branches) when all three parameters are included. Instead, it allows just the the "ARCH" and the variance constant to shift, so it's a mix of the ANST and the model from Gonzalez-Rivera. As with the Gonzales-Rivera model, there is no real evidence that this is better than the simpler GJR asymmetry.