Idk about the notation of your points so Im not sure if they got more structure you can use but for any list of points Id just project them down to your direction vector and sort them according to that
thats literally what i tried to implement