SIAM News Blog

Changing the Way We See Sports – with Computer Graphics

By Kai Berger

Kai Berger is a computational scientist at Genius Sports Media, a company that offers state-of-the-art, data-driven solutions in sports data technology for leagues and federations. In this issue’s Career Column, he discusses his professional trajectory and current role at Genius Sports.

I work as a machine vision researcher in the areas of motion capture, image registration, and uncalibrated camera synchronization at Genius Sports Media. My role entails developing prototypes in MATLAB or deriving formulae in Mathematica to read and manipulate unsynchronized camera footage from sporting events.

However, as a young student I had envisioned a very different future. In middle and high school, I considered a career in music. But shortly before college, I realized that images and graphics can convey the same beauty as music. While assessing musical flair is an instantaneous act that requires attention to each specific note, chord, or phrase, evaluation of an image is to some extent more timeless. The eye can move back and forth over the same picture multiple times, uncovering increasingly more facets.

This realization motivated me to reach out to the Computer Graphics Laboratory at the Technische Universität Braunschweig in pursuit of internships for high school students. The department allowed me to work on a three-week “practicum,” during which I learned some OpenGL programming. The professor teaching the course encouraged me to study computer science and revisit his department upon entering college and declaring a major.

When I returned two years later, I found that my contact had left the university and a young professor named Marcus Magnor had recently taken over the Computer Graphics Laboratory, which he was looking to grow. I stopped by one day and asked him for a job as a student assistant; he agreed, contingent upon my successful completion of the computer graphics and computer vision classes offered in his lab. I accepted and spent the next year taking classes and pursuing undergraduate studies in computer science.

During this time, TU Braunschweig elected to participate in the Defense Advanced Research Projects Agency’s urban challenge with a self-driving car. Magnor decided to form undergraduate and graduate student teams to help with the vision/environmental sensing aspects of the car. I became quite involved in the project and ended up publishing a paper about my work, which was accepted at a conference in New Zealand. To secure travel funding for the conference—and due to university stipulations for such grants—I began working on my master’s thesis earlier than normal on the condition that I spend a month conducting research in a laboratory in Auckland, New Zealand. I was able to present my paper, spend time collaborating with the lab, and finish my thesis on time. Magnor then offered me a Ph.D. position in his lab, and I worked on several exciting projects related to computer graphics, visualization, and computer vision. During visits to partner universities, I met other researchers in institutions like the Max Planck Institute for Informatics in Saarbrücken and the University of British Columbia in Vancouver.

Kai Berger explains an optical flow algorithm to his group in Genius Sports Media’s Santa Monica, Calif., office. Photo credit: Pareen Kalia.
My first postdoctoral appointment was with the Oxford e-Research Centre, where I provided computer-vision-based solutions for large data visualizations. Together with the research team, I developed a camera-based traffic monitoring toolkit to overlay on Google Maps. I also worked on a project that visualized anomalies in cardiovascular magnetic resonance images (MRIs), which aimed to compute image analysis operations on time-varying MRI data and comprehensibly overlay it on the gray-level video.

Next, I accepted an appointment with a group that focused on particle imaging velocimetry at the French Institute for Research in Computer Science and Automation (INRIA). Here I presented new solutions that incorporated Compute Unified Device Architecture/graphical processing unit computing in online visualizations of flow reconstructions on MacBooks. This reduced computation time for flow data from hours to seconds, thus increasing productivity.

Following my time at INRIA, I worked at NASA’s Jet Propulsion Laboratory as a robot vision scientist. I sought solutions to help robots navigate environments that challenge their perception, such as off-road terrains, forests, and urban settings with many glass windows and reflective surfaces. I provided new means of visualizing a robot’s vision, perception, and subsequent interpretation in three dimensions.

In my current position at Genius Sports, I am also tasked with simulating various situations, such as sensor placement scenarios for potential sporting venues. Simulation relies on quick-to-evaluate, highly parallelizable formulae for camera projection and scene reprojection. Furthermore, the outcome of such simulation may be a 10+ dimensional dataset of millions of data points. To envision the benefits and tradeoffs of certain placement setups, I create new visualization techniques that facilitate interactive dataset exploration. Users can choose up to three dimensions for a layered type of visualization. Scalar color coding then allows them to assess the optimality of a certain visualized subset with respect to the entire dataset’s overall value range. Doing so enables us to derive optimal placements for setup capture and efficiently communicate our findings to customers in a way that is visually understandable.

To ensure that uncalibrated and unsynchronized camera sensors work together effectively, I devise new methods for proper calibration. In a pending patent, I have derived a method that allows one to use a laptop or tablet of his/her choice to display a defined series of checkerboard patterns individually into each camera. A postprocessing algorithm then permits detection of the series in the camera footage, helping to distinguish each video stream from others in temporal space. The output is a set of cameras calibrated to our sports court and synchronized to the individual frame that provides two-dimensional information about court activity. I utilize this information, along with the cameras’ calibration, to reconstruct court action in three dimensions.

Many of my daily computations and algorithms involve the use of algebraic vision, linear algebra, and projective geometry. Ready-built functions like MATLAB’s stereo vision toolbox help with the calibration and reconstruction process. I like to design prototypes as independent graphical user interfaces (GUIs) in MATLAB to provide colleagues and customers with maximal freedom in interacting with our datasets and defining ways to process them. As a personal maxim, my desire is that the GUIs and visualized content be as intuitive and self-explanatory as possible. Shapes and colors must optimally convey the datasets’ underlying meaning and the associated applied processing. In other words, I aim for results that meet the highest level of visual appeal. Therefore, I often include recent findings from color science and statistics to increase the data plots’ comprehensibility.

During the course of my career, I have continually sought understandable, intelligible, and appealing ways to visualize data. Recognizing how my algorithms interpret and compute that data is equally important. The meaningful use of shapes and colors is key to rendering a successful solution that not only works on data but also tells a coherent story.

Kai Berger is a senior machine vision scientist at Genius Sports Media. He received his Ph.D. from the Technische Universität Braunschweig. As a member of the Institute of Electrical and Electronics Engineers and the Association for Computing Machinery, he regularly reviews for Eurographics, the Eurographics Symposium on Rendering, 3DV, and Multimedia Tools and Applications.

blog comments powered by Disqus