Face detection and tracking algorithms mainly suffer from low accuracy, slow processing speed, and poor robustness when meet with real-time setup. The problem becomes crucial in real-time situations such as in human robot interactions (HRI) or video analysis. A margin-based region of interest (ROI) hybrid approach that combines Haar cascade and template matching for face detection and tracking is proposed in this paper to improve the detection accuracy and processing speed. To speed up the processing time, region of interests (ROIs) with fixed and dynamic margin concepts are used. A dataset comprising of ten RGB video streams of fifteen seconds have been created from real-life videos containing a person in lecture delivering environment. In each video, there exists person’s movement, face turning and camera movements. An accuracy of 97.96% with processing time of 10.76 ms per frame has been achieved. The proposed algorithm can detect and track faces in sideway orientation apart from frontal face. The proposed approach can process the video streams at the speed above 90 frames per second (FPS). The proposed approach reduces processing time by ten times and with a boost to accuracy in comparison to the conventional full frame scanning techniques.