Facial attractiveness has been an interesting topic in cognitive psychology due to its key role in human communication and experience. The evaluation of attractiveness is adjusted by many factors including gender differences and cultural biases. In this paper, event-related potential (ERP) activity was recorded in an oddball paradigm from 10 Chinese men and 10 Chinese women who judged attractiveness of faces. Participants were told to detect faces with neutral expression and judge their attractiveness among a train of neutral objects that were presented more frequently than the faces. The ERP analyses showed that there was enhanced detection over early (P1, N170, P2, N300) and late (P3b) components in both genders. This suggests that a biased electrophysiological response to attractive faces compared to unattractive faces could indicate the involvement of emotion and reward pathways in judging facial attractiveness. Specifically, there were delayed P1 and P3b latencies in response to attractive faces with slower response times in men compared to women. From an evolutionary perspective, this may suggest that men attribute more value to facial appearances, especially attractive features, than women do, as evidenced by their cognitive load while processing attractive faces compared to unattractive faces.