Technology helps us in many activities and keeps growing, so it makes activities more efficient, time-saving, using fewer resources and also information and entertainment are accessible. Machine Learning technology is the fastest-growing field in computer science that is used in many areas such as marketing, healthcare, manufacturing, information security, and transportation. One of the machine learning methods is the Ensemble of Regression Tree (ERT) which has succeeded in detecting facial features on the eyebrows, eyes, nose, and lips. However, utilization ERT method has not been found to detect specific areas such as lips only for gaining optimization. Then this research will be conducted to extract the facial feature annotation dataset from the iBUG 300W dataset with 68 facial features to 20 lip area points. The results of the extraction are reduced error rate, resources saving, lip features still detected and lip coloring simulation was successfully carried out using the configuration of hyperparameter values, tree = 4, regularization = 0.25, cascade = 8, feature pool = 500, oversampling = 40 and translation jitter = 0. From observations also discovered optimization that hard disk resource savings are 69.36%, RAM 30.8%, and CPU 3.8%; reduce the error rate by 0.058%; and increase inference speed by 39%.