Webcam Eye Tracker

Building your own eye tracker for dirt cheap. How hard can it be? Turns out the basics are surprisingly simple! On this page, I’ll try to keep you posted on how the project advances. If you want to play around with the source code, feel free to grab if off GitHub.

First steps

11 October 2013

This video shows the result of my first attempt at pupil tracking. A lot of work needs to be done before this is actually useable, but I’m already quite happy with the results. The software is, of course, based on PyGaze. It uses the new webcam library. Image analysis is done on the fly, using PyGame. The webcam I use is a Trust Cuby model (retail price: 15 Euro), from which I removed the infrared filter. All regular lights in my office were off during tracking; I used a bunch of infrared LEDs to illuminate my face.

Progress

25 November 2013

I’ve added two new videos, showing the current state of the project. As you can see, I made quite a bit of progress! A Graphical User Interface (GUI) has been added, to allow users to set their own system up in an easier way (previously, it was done via code and keyboard shortcuts). Furthermore, the pupil’s edges (indicated by the green rectangle) are now detected, which means the software can now be used for pupilometry (the science of measuring pupil size). The pupil centre is still indicated by the red dot, but now there is the option to only locate a potential pupil inside of a user-defined enclosure. This is useful, because the pupil detection is done based on a rather simple principle: “find the largest dark bit in the picture, this must be the pupil!”. As you can see, my eyebrows and hair are pretty dark as well, therefore oftentimes falsely recognized as a pupil. The blue rectangle you see in the screen that says “select pupil and set pupil detection bounds” is the enclosure outside of which no pupil detection is attempted. As you can see in the video, this pupil enclosure moves along with the pupil, so you don’t have to worry about moving your head. Note that the lighting conditions were pretty normal in the current videos: I was simply sitting in my office, with all the lights and two monitors on, without the infrared LEDs that I used for the previous video.

Software explained

My software works in a relatively straightforward way. Every image that the webcam produces (30 per second!) is analyzed to find the dark bits in the image. This makes sense, because your pupil usually is one of the darkest parts of an image of your face (go check for yourself by looking through your Facebook profile pics). Of course, how dark exactly your pupil is can differ depending on the environment you are in. Therefore, you will have to tell my software what ‘dark’ actually is. You do so by setting a threshold value: a single number below which everything is considered ‘dark’. If you are now wondering “how can any part of an image be ‘lower than a single number’?”, you’re on the right track!

A computer does not see a picture in the way we humans do. Quite frankly, it doesn’t really ‘see’ the picture at all! For your computer, your profile pic is simple a collection of pixels, the tiniest parts of an image. If you zoom in real close, you should be able to see them in any image: small squares, consisting of only a single colour. To a computer a colour isn’t a colour in the way we perceive colours. To a computer, a colour consists of three numbers: one value for the amount of red, one for the amount of green, and one for the amount of blue. In our case, these numbers range from 0 to 255, where 0 means ‘none of this colour at all, please’ and 255 means ‘maximal colour!’. To give you some examples: (0,0,0) is black (no colour for all) and (255,255,255) is white (maximal colour for all). By now, you might be able to guess what the brightest red is to a computer: (255,0,0).

Now you know how a computer reads out the webcam’s images, it’s time to return to pupil detection. As I mentioned before, you can select a single threshold value for what my software must consider ‘dark’ (and therefore potentially the pupil!). My software then looks at every single pixel in the image, and checks if any of the values for red, green and blue are below that threshold. Only if ALL the values for that pixel are below the threshold value, that pixels is considered ‘dark’. In the movies below, you can see a blue-black screen, where all the dark parts of the webcam’s video stream are black and all other pixels are blue. As you can see: the pupil is one of the darkest parts in the video stream. In essence, this is what we want. But what about all other dark parts of the images? We don’t want the software to mistake my eyebrows for my pupil!

The easiest way to prevent incorrect pupil detection, is by specifying where in each image my software is allowed to look. Basically, it needs to know where the pupil is, and how big the area around the pupil is in which it can look for the pupil. The easiest way to achieve this, is by directly telling my software where your pupil is. If you’re thinking “Wait, hang on Mr. Genius. Your software is supposed to tell me where the pupil is, not the other way around!”, you might have a point. Luckily, you will only have to tell my software where your pupil is once. After that, it’s perfectly capable of telling you where your pupil is. Alternatively, I could try to write some sophisticated face detection algorithm, that finds your face in an image, and then knows where to look for your pupil. This, however, has the disadvantage that an entire face would have to be present in the image, which is not necessarily the case (see the videos under First steps and Progress). On top of this, even the world’s best programmer wouldn’t be able to write face detection software that surpassed your ability to recognize an eye in an image, because you’re just so darn good at it! Therefore, I chose to let you tell the software where your pupil is, by clicking on it with the mouse.

After indicating the pupil location, you can increase or reduce the size of the ‘pupil bounding rect’, the enclosure outside of which my software ignores everything. You can set its limits to anything (and you can even deactivate it), but I’d recommend a bounding rect that encloses the entire eye, and maybe even a bit around it. The larger the bounding rect, the higher the risk of false pupil detection, but the smaller the bounding rect, the higher the risk of losing pupil detection if you move too fast. After setting the rect, you can test if your settings are good by moving and gazing around a bit, and you can adjust the threshold if needed (or go back to any earlier step). Please see the videos above for some nice demo’s!

35 Comments:

  1. i need python opencv2 code gaze tracking……

  2. sir, i m working on pupil detection and found your code helpful but i have no idea about the language python so kindly tell me the procedure to run this code

    • Assuming you’re on windows, the easiest way is to download the following things:

      1) Anaconda, version 2.7 (a Python distribution that includes a lot of cool stuff, including the Spyder editor)
      2) PyGame (choose the version for Python 2.7)
      3) The webcam-eyetracker source code (click on the ‘DOWNLOAD ZIP’ button)

      Now run the Anaconda installer, and then run the PyGame installer. Afterwards, unzip the webcam-eyetracker source code archive. You can open ‘GUItest.py’ in the Spyder editor, and run it from there. A better alternative is to create a batch file, in which you write: "C:\Anaconda\python.exe" "GUItest.py". Then run that batch file.

      PS: To create a batch file, create a new and empty plain text document, and change the .txt extension to .bat. Then right-click the batch file, and choose ‘Edit’.

      • thanks for your reply

        i m using linux mint and also tell me about the calibration is it necessary to do when ever we use this code??

        • If you’re on Linux, Python should already be installed. Downloading the necessary packages is as easy as running sudo apt-get install python-numpy pygame

          The code requires a calibration, but you could implement this in an automated way. It’s there to illustrate the basic principles of pupil tracking, and definitely needs more work to fit with whatever application you have in mind.

          Good luck!

      • I am getting this error:
        Webcam Eyetracker>"C:\Anaconda\python.exe" "GUItest.py"
        Traceback (most recent call last):
        File "GUItest.py", line 3, in
        from camtracker import Setup
        File "C:\Users\hazzj\OneDrive\Stuff\Webcam Eyetracker\camtracker.py", line 25, in
        raise Exception("Error in camtracker: PyGame could not be imported and initialized! :(")
        Exception: Error in camtracker: PyGame could not be imported and initialized! :(

  3. Hi Edwin,
    I’m trying to run your software on Windows but I got this “Error: Cannot set capture resolution.” at line 52 of the file “_camera_vidcapture.py” which is a file of the pygame folder … I tried to comment this line, the result was interesting because I was able to access your Welcome page in the pygame window and when I press a key my webcam turns on and the software opens the ActiveMovie Window ! However, after less than a second, the image stops as the python.exe stops working …
    If you have any idea of what could be the source of the problems, I take. I read somewhere that pygame only handles the camera on Linux but it’s likely out of date .

    • I’ve been trying to figure out what was the problem more precisely.
      I found out that the software stops on line 865 of camtracker.py :
      pygame.transform.threshold(thimg, image, self.settings[‘pupilcol’], th, self.settings[‘nonthresholdcol’], 1)

      • In my case the software also stops at that particular line. I get the following error:

        TypeError; must be pygame.Surface, not None.

        I checked some of the variables that are used in the pygame.transform.threshold() function, and it seems that the image variable (passed as a parameter) is the problem. It is defined as img = self.get_snapshot() at the end of the file. However, the get_snapshot() function tries to return a self.cam.get_image(), which returns None on my pc.

        Then, on stackoverflow – http://stackoverflow.com/questions/25711028/typeerror-must-be-pygame-surface-not-none – I found something that turned out to be the solution for me.

        I added pygame.Surface((640,480)) as a parameter to get_image() in the get_snapshot() function, so:

        def get_snapshot(self):

        “””Returns a snapshot, without doing any any processing

        arguments
        None

        keyword arguments
        None

        returns
        snapshot — a pygame.surface.Surface instance,
        containing a snapshot taken with the webcam
        “””

        pygame_surface = pygame.Surface((640,480))
        return self.cam.get_image(pygame_surface)

        After this, everything worked according to plan.

  4. Hi Edwin,
    im trying to run my code on my raspberry pi but unable to track pupil efficiently can y suggest me the web cam for raspberry pi or what you r using
    thank you

    • Hi,

      Sounds like a cool project! On a Pi, you could use the Pi NoIR camera. You will have to change the code, as that one is not compatible with PyGame. The first webcam I used was a Trust Cuby (at the time, it was 15 Euros). Since I have also used others. They were all super cheap webcams, from which I manually removed the infrared filter (you don’t want to do that with an expensive webcam, as it mucks up your regular image quite bad).

      Good luck!

    • Hai adeeth,
      Do you succeed in running this code without any error?

  5. hi there,
    i need help to run this code,could you tell me the procedure to run the code
    properly because when i open the guitest.spy

  6. Hi Edwin Dalmaijer ,
    Is the webcam eye tracker only on GUI. Is is possible to get the location of retina(eye) in x and y in command line.
    Can I use the code without executing guitest.py ? if yes how can i do it ?

    • Yes, you can use it without the GUI. In fact, the GUI setup is only there to return a ‘calibrated’ (set pupil threshold) tracker instance. Look up the underlying code on GitHub, and it should be self-evident :)

  7. hi Edwin Dalmaijer ,,
    thank’s for your project ,, can i run this code with Windows surface pro webcam ?

    i have this
    Exception: Error in camtracker: PyGame could not be imported and initialized! :(

    pygame and anaconda is instaled in my pc .

    thanks

  8. ABHISHEK SACHAN

    Hello
    How can I run your PyOpenCV code. Which all files I need to run. what all dependcies i need to install.

  9. I want to make one eye tracking device…the camera you mentioned trust cuby webcam not available right now in market…so which camera I can use as a replacement for that which will better work to track pupil?

  10. Hi Edwin, I have a problem.
    You can help me?
    I installed everything as you explained, but after starting the application, when the program calls the webcam, I get a message of error.

    “This application has requested the runtime to terminate it in an unusual way”.

    I can’t see in which part of program is the error.

    What can be?

    Thank you!

  11. rajarajan elango

    i am getting error ” videocapture module not found ” while running GUItest.py in linux ( ubuntu ). how to install that library in ubuntu. kindly help. Installed pil but having same error

  12. just a beginner question, what is the difference between openCV and PyGaze?

    I have a save mp4 video of the eye/pupil, and I want to track the movement.

    I guess that I don’t need face recognition?(seems to be a feature of openCV)?

    • PyGaze is a software library that you can use to create experiments for psychological research. It allows you to display things on the monitor, to interface with external devices (keyboards, mice, joysticks, EEG equipment, etc.), and also to interface with existing eye trackers (EyeLink, EyeTribe, GazePoint, SMI, and Tobii). See here for the source code: https://github.com/esdalmaijer/PyGaze

      However, what you’re likely referring to is my webcam eye tracker. This allows you to track pupils and glints in a webcam stream. The old codebase used PyGame for this, but the newer version uses OpenCV (including some of its face detection functionality). For the source code, see here: https://github.com/esdalmaijer/webcam-eyetracker

      For your purpose, I think you might want to use my OpenCV implementation. This offers a generic class that handles the tracking, and all you have to do to allow it to process your videos rather than a webcam stream is to slightly adjust the existing code. Basic steps:

      1) Inherit the generic class, EyeTracker, in a child class for your video analysis.
      2) Use my webcam tracker as inspiration. https://github.com/esdalmaijer/webcam-eyetracker/blob/master/PyOpenCV/pygazetracker/webcam.py
      3) Use my image implementation as inspiration: https://github.com/esdalmaijer/webcam-eyetracker/blob/master/PyOpenCV/pygazetracker/images.py
      4) Write your own class for your videos. Make sure it inherits the generic EyeTracker class, and that you define a ‘connect’, a ‘_get_frame’, and a ‘_close’ function.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">