Picture a two-dimensional grid, like a chessboard or a piece of graph paper. Each square on the grid contains a tower of blocks. These towers come in different heights, ranging from zero to 255. “Towers” with zero blocks are black, towers with 255 blocks are white, and every number in between is a shade of grey.
The towers aren’t random: they form a landscape, a desaturated city of shining skyscrapers and empty black demolition sites. A line of white towers marches around the centre of the grid in a pointed oval. Further out, four curves arc away into darkness, their trailing ends fading into ever blacker greys and ever smaller stacks of blocks.
But not every tower is part of a larger structure. Many are free-standing, isolated spires nearing 255 while their neighbours hunch in the 10s and 20s. The entire grid is dotted with these individual white towers.
What you’re picturing is the image below:
A computer sees this image as a bunch of pixels (the squares on the grid) with values from zero (black) to 255 (white). The image structures – like the nebula – are connected areas of bright pixels. The stars are the isolated tall towers. We’d like to tell the computer to get rid of the stars, but leave the nebula untouched. Although distinguishing stars from nebulae is pretty easy for a human, it’s a surprisingly subtle problem for a computer. Brightness isn’t a good criterion, since both stars and nebula are bright. Location doesn’t work either, since the stars overlap with the nebula in much of the image. At first glance, there isn’t much other easily accessible information the computer can use.
Why do we want to get rid of the stars in the first place? There are a variety of reasons for this, but it’s mainly to make it easier to do other processing tasks on the image, like automatically detecting satellite or meteor streaks, galaxies, and other interesting stuff. If computers can be taught to accurately detect astronomical objects, they can sort through huge numbers of images much faster than astronomers can. Unfortunately, computer vision is often very sensitive to interference from other visual features like the stars in this picture.
There are already a number of methods for removing background stars, mostly involving subtracting similar images from each other to erase permanent features (stars) and leave only transient ones. One downside of this, in addition to the difficulty of aligning the images, is that it requires multiple images, preferably taken close together in time. (In one of my PhD interviews, the interviewer asked if the goal could be achieved using neural networks. I have no idea if that would be a good approach for this task, but in any case, it would also require training a neural network on a body of similar images before it could be used.) What if we only have one image? How can we teach the computer to distinguish the stars from the nebula?
In the algorithm proposed in my recent paper, we do it by measuring the shape of the star.
A star is basically a very tall tower of blocks surrounded by much shorter towers. Inside the nebula or at its edges, on the other hand, tall towers are surrounded – at least on some sides – by other tall towers. To decide whether a pixel is a star or a nebula, the computer calculates the difference in height between the pixel and each of its four nearest neighbours. Then it compares the drop-off in all four directions: if it’s similar all around (symmetrical), the central pixel is classified as a star, but if there are differences between the drop-off on different sides, the pixel is classified as part of the nebula (asymmetrical). “Star” pixels get shrunk bit by bit to be more similar to their surrounding pixels, while “nebula” pixels are left alone. The process is repeated until the star has been shrunk to the level of its neighbours and vanishes into the background.
The result is all right, but we thought we could do better, so we added a second step to the algorithm – quality assurance, in a sense. At each repetition, the program compares the new modified image to the original image by subtracting them. If the first, asymmetry-measuring step were perfect, the image difference (“residue”) would contain only stars. In reality, after a few iterations, some bits of the nebula we’re trying to preserve also appear in the image residue. The computer monitors the residue for groups of pixels that are connected to each other. Once a group like this reaches a certain size, the program stops modifying the pixels in that group – the algorithm selectively stops there, but continues to whittle away at other pixels in the image. Some areas get processed more and some less. This vastly improves the final result:
To check how well the algorithm performed, we used a starless nebula image (right) and added stars (left) so we could compare the star removal algorithm’s output (centre) to the original, starless image. The reconstruction isn’t perfect, but compares well to other attempts to use a similar, single-image based algorithm to remove stars. We then tested the algorithm’s performance on a few of my favourite nebulae, Henize 70 (on the left) and the Veil nebula (on the right). These test images are particularly difficult and fun because the nebula structures are thin, wispy, and easily eroded.
If we’d had more time, I would have tried to find a way to design the algorithm to automatically stop repeating when the image is “finished” – a goal many others working on similar iterative algorithms have attempted to reach. It’s difficult because there’s no clear way to know on the pixel level when the overall picture is “finished” and the best-looking result has been achieved. People might not even agree what the best-looking result is! In any case, my Master’s programme was over before we could get that far.
I was lucky enough to have four semesters to work part-time on this project, much more than most Master’s students get. The physics department at my university, Heriot-Watt, had just changed the structure of its degree programme when I was entering my fourth year. Previously, students had worked on a one-year group project in their fourth year, and then an individual Master’s project in their fifth. The innovation was to allow students already committed to a Master’s year to start their project a year early, for a total project length of two academic years. This change made a huge difference. I spent most of the fourth year just learning the basics of image processing and Python and only managed to generate a few original ideas at the end – but I had a whole fifth year to actually develop the project. I was also lucky to have a very involved and conscientious supervisor, Dr Weiping Lu, who not only took my project seriously, but also trained me how to do research in the future.
Doing research was by far my favourite part of the degree programme. So much so that I’ve wondered if it would be possible to get rid of lectures and exams all together and have physics education consist only of labs, projects, and problem-solving workshops. The gap between learning physics and doing physics (or engineering, or computer science, or whatever) seems much larger to me than the gap between learning subjects in the humanities and practising them. In the humanities, you start out reading simple texts and writing simple analysis and you move on to more complex but still secondary texts and more complex analysis, and finally primary sources and original research papers. In some fields, you also collect data, but there’s a substantial overlap between the skills you use even as an undergrad and those you use as a researcher. In contrast, doing a research project in physics was vastly different (and much more fun) than physics lectures and exams. If my Master’s hadn’t switched focus to research in the last two years, I would have felt a lot more lost on starting my PhD.
Dayside Stories 