A fruit, botany tells us, is any body that develops as the result of a flower’s ovary being fertilised. Botany also tells us that a vegetable is any edible segment of a plant that is not a fruit. A tomato is then, botanically speaking, a fruit. However, tomatoes’ longstanding culinary use alongside vegetables has led to their common labelling as a vegetable. In this essay-interview between designer and researcher Simone C Niquille and professor of psychology Eleanor Rosch, this seemingly mundane example of understanding and negotiating categories becomes a wormhole for exploring the limits and fuzzy borders of categorisation in the world. Seeking to make legible the otherwise hidden frameworks of knowledge production embedded in machine learning, Niquille turns to Rosch, whose prototype theory involves a “graded degree of belonging rather than strict boundaries”. Together, in a conversation that centres on Niquille’s video work Homeschool, they refuse “categorisation as a tool to control a system of relations” and instead propose “a network of objects, things, and concepts that are all connected and yet more or less alike.”

‘When I use a word,’ Humpty Dumpty said, in a rather scornful tone, ‘it means just what I choose it to mean, neither more nor less.’ Lewis Carroll, Through the Looking-Glass⁠1

Eyes don’t know what they are looking at. Cameras don’t know what they are looking at. To make sense of visual input, the brain (or neural network) organises these signals into categories. Chairs are ‘furniture,’ an apple is ‘fruit.’ Like a paint-by-number landscape, the world is constantly segmented into digestible parts by machines and humans alike. To the human mind, the scientific accuracy of such mental sorting isn’t crucial. In the world of botany, a tomato is classified as fruit. In the supermarket, to most nutritionists, and in my mind, it is a vegetable. Organising what one sees into mental groups is essential to understanding the surroundings. Categorisation simplifies complexity by drawing comparison and pointing out similarity. In doing so, it also defines what belongs – and what doesn’t. Without categories, there is no anomaly, no possibility to fall outside of their scope. Where a category’s borders are drawn is deeply personal and communal; it reveals cultural context, a set of references and personal beliefs. Categories are a reflection of someone’s worldview.

While communicating with others, categories are constantly negotiated, allowing for approval, synchronisation, disagreement. I refuse to be (mis)gendered based on my hair length. All food can be a snack. My friend’s cat is a human. It is in systems of control and organisation that categories are fixed containers. I am thinking of predefined drop-downs to select one’s gender in online profiles, the taxonomy of plants and animals, the classification of ethnicity through race with a few commonly used labels. In computer vision, with whom does the machine negotiate? The definition of categories and where a category’s border is drawn matters. In supervised machine learning, categories are clearly defined in a labelled training data set. Huge sets of images are assembled to teach computers what they are looking at; these images are segmented and labelled by content, and then they are sorted into categories. This grouping produces an average that simplifies complexity: one does not need to know all of the possible content of a category to process the world. One does not need to have seen all of the chairs of the world to understand what a chair is. Yet, what a chair is depends not on what you use for sitting, but what was defined as such in the training data set. What – and more importantly, whose – world model is used to categorise training data? It takes a position of lazy comfort to make sense of the world by subtracting everything from what one already knows, rather than wanting to understand a thing for what it reveals.

The use of an object and its surrounding rituals and associated behaviours cannot be detached from the thing itself. In my video piece, Homeschool (2018–ongoing), the virtual camera displays a first-person view as it moves through a domestic space, attempting to orient itself among the objects. The scenography of Homeschool is constructed with the content of a synthetic training data set for indoor navigation. The data set is composed of thousands of 3D meshes classified as domestic items: furniture, cutlery, laptops, weapons, and lamps. All sorted into categories. The film’s voice-over uses these categories to try to call the objects by their names. It wonders: Is this decoration a toy, a gun, or all of the above? Paraphrasing Wittgenstein, the voice concludes: “The limits of my categories mean the limits of my world. The limits of my data mean the limits of my world.” Not an issue, unless such a training data set is treated as a universal home, representative of all domesticity. A desirable resource when developing a product for a global market, but a room-mate to reject in a home that breathes ambiguity.

In the following conversation with psychologist Eleanor Rosch, we discuss such limits and fuzzy borders of categorisation. Author of the ‘prototype theory’ as defined in Natural Categories,[fn]E. H. Rosch, “Natural categories,” Cognitive Psychology, vol. 4, 3, 1973, pp. 328–350. [/fn] Rosch proposes a graded degree of belonging rather than strict boundaries. A prototype, a prime example, serves as the anchor point of a category, while other members are related on a spectrum. This theory of categorisation is less about belonging and excluding than about proposing a network of objects, things, and concepts that are all connected and more or less alike. As such, prototype theory offers a departure from categorisation as a tool to control a system of relations.

Simone C Niquille: I am fascinated by computer vision, the desire to make cameras know what they are looking at and the pursuit to implant perception in autonomous machines. My interest is in how the development of computer vision reveals human mechanisms and worldviews that are deeply personal, instinctive, and seem so obvious they are not disputed but certainly should be. Seeing technology unveils the way humans perceive the world. It destabilises agreed-upon definitions: What is a chair, how to describe an apple, and what is a weed? To ‘teach’ a machine to ‘see,’ representations of the world are created by accumulating large amounts of images. These training data sets rely on language and, more precisely, categorisation to describe the image’s content by segmenting and labelling various parts, organising images into groups based on their content. All images labelled ‘chair’ are gathered into the chair category, and so on. I would like to talk about categorisation with you. It seems straightforward enough at first but quickly falls apart when speaking about more than just an array of objects. In my experience, the world’s complexity does not fit within neat boundaries, as the classical view of categorisation proposes. For this reason, your ‘prototype theory’ struck a chord: What would a computer vision system based on prototype theory look like? And more importantly, how would it perceive the world?

Eleanor Rosch: The thing about prototypes is that they are multisensory. In some research, we had people describe what they do with a chair, like the muscle movements involved in sitting, getting up, etc. This multisensory experience is part of one’s image – their mental image – of what a chair is. That image draws upon the entire knowledge set, the encyclopedia, of what you do with chairs, as well as on your memories of past chairs, your preferences. One’s idea of a chair may even involve the smell of furniture, the utensils you use at the kitchen table, the food you eat. The experience and the object can’t be separated, just as the object and the abstract knowledge base in which it is located can’t be separated. That’s how you get a good representative of the category – a prototype. Members of a category with really different sensory experiences or a different conceptual context from more prototypical members will be judged more peripheral to a category although still related. That’s how a turkey is judged – far from a typical bird but still a bird.

SCN: Would you argue that an image of a chair can’t represent a category because it doesn’t document the sensory experience? An image evokes memory in the human brain; it triggers and accesses one’s stored data, if you will. Without this ‘data’, a computer vision system lacks context and misses the larger world of the thing depicted based upon an image. In the SceneNet indoor training data set assembled to teach a computer vision system what a domestic environment looks like, the category ‘chair’ is a good example of this: in it are common chairs, but also a gynaecological examination chair and an electric chair. These are chairs, but likely not the kind expected in a domestic context.

ER: That’s a problem for a computer; if all it has is an image, it needs its programmer to give it more. But images are great at representing all sorts of things for people. That’s how we get art. Images are the stuff of just about everything from advertising to religious practices. And imagery is how we learn things; every time you hear “for example, x, y, z, …,” a specific case of what was just described abstractly is being given, a case concrete enough that you can imagine it and incorporate it. By the same token, a visual image in a book or on a computer screen is not something two-dimensional, static and lifeless; it’s part of us, of the whole living event that’s going on. The thing that we call ourselves or our own experience is not only multisensory, it’s also all of this stuff around us. We say, oh, that’s just the environment. Well, not really. What is a home? It is, in some ways, a part of your outer skin. People identify with their homes. I know I do. The entire street you live on or the woods that might surround your place are part of it. When you’re driving in a car, the car is an extension of you. You can’t possibly learn to park unless you can feel the exterior part of the car in some sense. We’re not as centralised as we think we are.

SCN: A driverless car then is a very strange concept. According to your hypothesis, if there’s no driver, there’s no one sensing the surroundings. Instead, the sensing is outsourced to other systems embedded in the car itself, including computer vision. As far as computer vision is concerned, I am fascinated by the attempt to replace or teach sensory decision-making via two-dimensional images. How does complexity become so flat? Beyond the images in training data sets, the language used to sort them into groups and label their content reveals a limitation. The fact that the category ‘chair’ only contains objects that have four legs, for example. Why isn’t there also a toilet, a pillow? Why aren’t the box, the floor, other objects, or parts of the environment that you can sit on also part of the chair category?

ER: Few, if any, objects are classified based on a single attribute, even if it’s an important one, such as what they’re used for or could be used for. There’s a longstanding argument in developmental psychology about whether young children use form or function to classify. You should see the ingenious set-ups experimenters use to try to pull these apart. But of course, in the real world, the form and function of objects usually go together, and that’s (probably) what children are learning most. In the real world, there are also a variety of cues, such as the location of an object, that trigger one’s expectation of what one is going to find. I think that was at play in differentiating a toilet from a chair, even if the chair has a circular cushion on it. When you are in a particular room with an object, its use is clearer. In your video Homeschool, you had a scene of a toilet-like object in the living room, and my mind went, Aaah! Of course, this is based on human customs and culture. A dog doesn’t see why it’s alright to drink water out of the little bowl on the floor in the kitchen but not out of the big raised bowl in the little room at the other end of the house.

SCN: During the research for Homeschool, I spoke with the scientists of the SceneNet RGB-D indoor training data set. They told this story about how there were too many images of toilets in the initial training data set. The algorithm got overfitted with ‘toilets’. As a result, the whole world looked like a toilet to the computer vision system. It tried to identify everything based on this one object. To me, this is beautifully absurd. It’s another version of looking at clouds and trying to discover animals, faces – the familiar. Do you find categorisation a useful way of organising the world for computer vision?

ER: I have wondered that. Computers don’t have to do it exactly how people do it, especially since there aren’t any adequate models of how the mind organises the world. One of the things that watching your video Homeschool provokes in me is the wondrousness of what the brain is doing, just looking around and knowing so much. This poor little, perhaps incipient, vacuum cleaner is wondering and struggling. Watching it had this great emotional impact. I was curious – because it also has a soundtrack – how much emotion came from the music and how much from the vision. I closed my eyes and only listened to the music, which had a lot of force. Then I watched it for a while with the sound turned off. The music was really important for the impact. It fit just so beautifully, and I got lost in this world. Sound is an important aspect in our ability to recognise; sounds go with objects.

SCN: Absolutely. The resolution adds definition. I don’t just mean image resolution, but the resolution of detail in terms of input signals: temperature, smell, and sensory stimuli that add definition to the world. When I exhibit Homeschool, there are four different speakers, one in each corner of the space. Each of the speakers has an individual sound channel. Some of the sounds you hear, like the toilet flushing, or the dog barking, only play on one of the speakers. It adds a simulated volume to the projected video. You start turning around, you start orienting yourself, even though you know there is no dog in the back left corner. None of this is real, yet the sound, the signifier, is there. The experience is real.

ER: Orientation is so automatic. It is a very low-level, basic process. When there’s movement in the periphery of the eyes, they move towards it. You can’t not do that. It’s a reflex. It does not matter if it is a simulated periphery; it’s gonna happen anyway.

SCN: What would your response be if you were contacted by a company building a robot trying to sort trash? The company would get in touch as they are struggling to figure out how to categorise trash to differentiate between what to keep and what to throw out.

ER: I can imagine the usefulness of a little robot that would go around the streets picking up trash. But you don’t want it to pick up a little dog along with the trash or people’s flowers that are growing along the road. So we need to consider what trash is. That’s so interesting. Notice that trash is what I call a ‘superordinate category.’ Superordinate categories are abstractions that have few, if any, visual or other perceptual distinguishing attributes. Still, they are composed of collections of more basic-level categories rich in informative sensory characteristics. For example, furniture is a superordinate for which people can list almost no attributes. Still, it is composed of chairs, tables, beds, lamps, and so on, that have myriad list-worthy and potentially machine-recognisable characteristics. Trash is a massively inclusive category. It can refer to any material object or piece of an object in its death spiral, that is, where it’s no longer considered useable and needs to be disposed of.

So, what would I say to the company wanting to design a trash-collecting robot? The first is to forget about the word ‘trash’ and to go down a level of abstraction to the actual objects whose broken or discarded forms will be present in the locations where you want the robot to work. Then the robot can be programmed to distinguish the objects in their pariah form from kindred objects that are still functional and from other objects in that environment that look like the targets but are something else.

Or, perhaps, one could go down the abstraction ladder even further and forget about what the objects are altogether. Go out to all the sites where you’d want the robots to work – roads, streets, parks, playgrounds, schoolyards, beaches, public spaces in buildings, backyards, etc. – gather a huge data set of examples of trash and non-trash for each location, have people rate how well each object fits their idea or image of trash, and train the robots in a neural net learning design to discriminate trash from non-trash, giving them the best-rated best examples first (children learn new concepts most easily that way). With neural nets, you’re letting the robots build their concepts and heuristics about trash, bottom-up. In either way of proceeding, I think you could avoid both over- and under-extension of the concept of trash. You don’t want robots that go around snatching half-eaten sandwiches out of the hands of picnickers, but you don’t want your robot to ignore an odiferous sandwich lying in mud because it doesn’t look enough like its training image of a crumpled cigarette package.

SCN: The technology in terms of computer vision is always trying to be very general or universal. That might be where the challenge lies. If it’s about cigarette butts, it’s hyper-specific, and specificity can be defined. “Only collect cigarette butts with a Marlboro logo.” Such a computer vision system would rely on a data set containing only images of cigarette butts with Marlboro logos.

The big challenge within computer vision is the politics within data sets. The fact is that having to make certain choices is discriminating. I think that when we speak of plants, it’s one thing. If we speak of something like facial recognition and how to categorise people, it’s a whole different story. I was curious if that plays a role in the research you’ve done. I think it’s useful to hear from your experience how much categorisation and discrimination or exclusion are linked.

ER: In any categorisation that people make, either of each other or of the rest of the world that concerns them, it’s political all the way down. We’ve been talking about trash. But what gets discarded like trash in one culture may be repaired and used in another. Think about planned obsolescence in manufacturing, rapid change in clothing in fashion, recycling versus dumping, garbage versus compost. All political!

Of course, what we usually mean by political categorisations are gross partitions of humans in terms of race, ethnicity, gender, and religion. Look at the main issue psychology has found to study: stereotypes, i.e., prototypes in social categorisation. Stereotypes are often approached as a unique social phenomenon, an outgrowth of prejudice and other societal problems that require specialised training to overcome. The study of categorisation can contribute to the knowledge that cognitive representation of categories naturally has that form, i.e., representation by concrete, quasi-imaginable good members. However, this only becomes visible and an object of discussion and research when there is social disagreement about certain kinds of categorisations. So, the objective of changing socially objectionable stereotypes will fail if it aims to eliminate the concreteness and imaginative quality of representations. Instead, it should aim to educate people about the contents of those representations, for example, by broadening knowledge about members of the category and offering alternative images to adopt for the most representative category members. Think of the emotional impact it would have if the stereotype of ‘welfare queen’ were to slowly morph into something more like the image of Dorothea Lange’s photograph Migrant Mother (1936).

Finally, there is the quasi-political issue of what research gets conceived, funded, carried out, written, and published. The research should be new but not too much so; easy to understand and write about with an older mindset but not completely a popular cliché. Best if it’s at a level of abstraction where it can generate many new studies that add to or contradict it. It helps if it can be studied with whatever tools are in fashion at the time. By this token, it’s easier to study categorisation from the viewpoint of classical categories than from any alternatives and to describe them with mathematical models. Mathematical models of cognition are all the rage now, as is using large data sets available on the internet. This is where we are right now.

SCN: This reminds me of the theories of William Labov and his proposal of a mathematical function to calculate what a cup or what a vase is. It sounds like concrete poetry but was presented as a theory of categorisation to prove that objects can be clearly defined – as much as the categories they belong to. Labov’s theory is in line with the craze of making things calculable and measurable. Do you disagree with Labov’s cup calculation?

ER: Well, it works for the given data set that he was working with. But of course, what counts as a cup changes pretty rapidly. Cups, with saucers, without handles, tall cups, vessels the shape and height of a glass but not transparent.

SCN: So you’re saying that the definition of a cup depends on the data set from which it is derived. I guess we’re back to the computer vision problem, right? Where you can prove something works if it’s functioning within your data set versus, well, in the world? If I never leave my house, its contents make perfect sense to me. My house is my training data set. Once I open the front door, I will have to start to compare everything to this knowledge. Some things will make sense, and some won’t. Some mistakes will be hilarious, others disastrous. There were probably no cars inside my house, and I won’t know how to interact with them.

If you could have an academic career without mathematical models, maybe the technology produced would change. Is there a model you’d rather see, if not a mathematical model, to compute the world?

ER: If you’re doing computer vision – or computer anything – you are working on getting machines to do things that humans do – or wish they could do. The problem is that you have to figure out how to do that. You can feed the algorithms with large data sets; you can input an encyclopaedia of sorts. But then you have to figure out what parts or what type of encyclopaedia. (We’re like the failing student who wants to copy some work from a good student, but they have to know enough in order to know what to copy.) It’s mind-boggling, how to do that, and you surely want to get it ‘right’. If you’ve got a self-driving car, you really want it to not run down people or crash into other cars.

There are at least three kinds of researchers. Many of them are just trying to get a machine to solve a problem. They don’t care how humans do it; it is not about imitation. Maybe the machines can pilot an aeroplane or make a medical diagnosis much better than the human mind does. More power to them! Then there are the researchers who want to imitate the human structure and see themselves as doing brain research. Finally, some seem most interested in machine learning itself: How will the machine structure itself to learn and perform tasks? These are the people more likely to take a prototype view on categorisation, perhaps because that is how concepts form in naturalistic learning situations. Just look at one of the basic experiments in psychology that launched psychology as a scientific research field: an experiment in animal learning by Ivan Pavlov. Pavlov gives a tone, and the dog learns to respond. The dog starts salivating, knowing he’s going to get food. With tones slightly different than the training tone, he salivates, but less. This is the entire prototype structure. It’s engrained in very low levels of the brain, this ability to judge based on what you’ve learned. You always learn from something specific unless you’re in school and taught some high-level abstract ideas. But even then, even in mathematics, there have to be specific examples you’re taught to help illustrate those abstract ideas. What you’re trained on, what you learn first, is what you start judging similarity against.

SCN: You’re sorting based on an initial model picked up in childhood.

ER: Well, you modify the categories throughout your life. But note that it’s notoriously hard to get any change in people’s ideas through abstract reasoning. It takes specific experiences with objects or situations – your own or those you hear, see, or read. That’s what brains operate on naturally. And they’re more open to thinking of categorisation in a prototype structure where there are no clear boundaries but degrees of similarity or relation.

There’s another problem with strict mathematical models and computers as models of the mind. I’ve been spending great amounts of time with Eastern teachings and practices because I believe that there are aspects of what minds are and do that just don’t fit into a Western model of the mind, the personality, or the rest of psychology. The more rigid you are in your research, the less able you are to drop your preconceptions enough to relax into other ways of understanding and doing. It’s those other modes that may be needed to solve the world’s pressing problems – something we’re certainly not doing very well right now.

SCN: I wonder to what extent the toys and the songs that teach children categorisation and sorting influence or produce a specific worldview. Is it a model they use for a lifetime? Is it very difficult to unlearn? If sorting is instinctive, the rules still need to be learned. To compute has a mathematical connotation, which makes it seem objective – but this is not always true. Rather than trying to replace the human or imitate its brain, an attempt could be made to figure out what a machine can do on its own.

ER: Yes, to actually respect it for what it is and what it can do. I heartily endorse that. The job then is to think like the machine, the way you were doing in Homeschool. You have to put your human empathy into the machine and work things out from its point of view.

I just had an idea about the vacuum cleaner. It would be very expensive, but rich people would probably like it. You would send a video of your home or the place that the vacuum cleaner will work in – your living room, your kitchen, your bathroom, wherever – to the developers of the machine. They then have a designer train the machine to work in that environment with all of the objects it contains. Anytime you get a new couch or whatever, you send in an update, and the system adapts.

SCN: Well, Roomba vacuum cleaners collect data of the rooms they clean, essentially creating a data set of their clients’ homes, producing the robot’s future domestic environment based on the present. A prospective owner will have a better functioning product if their home is similar to the past robot owners. The question arises if this spiral of data collection eventually creates a feedback loop where homes start to look a certain way for domestic robots to function properly. A chair needs to look a certain way to qualify as such for the system. It needs to pass according to the robot’s ‘mental image’ of a chair. As an amalgam of domestic spaces and objects, the data set aspires to be the image of a home. Houses ordered from a Sears Modern Homes catalogue in 1908 are a precursor of the domestic reduced to an image or an aesthetic, rather than a ritual. Instead of being designed for a specific place and inhabitants, the house was purchased based on visual attributes. A modern-day continuation of this trend might be IKEA’s steady transition since 2014 from a photographic product catalogue to one nearly entirely filled with 3D-rendered images. Instead of gathering data via cleaning robots and feeding it back into bettering the technology, IKEA’s warehouse of 3D furniture is the high-resolution data set that computer vision development has been missing. By essentially cutting out the end user and training on products that are not yet purchased but globally distributed, the question arises if the house is ultimately built for the robots’ needs or for the human inhabitants. I’m interested in the architectural consequences of the home that these cohabiting technologies generate. What a chair is can easily be taught to computer vision when everyone sits on an IKEA Poäng chair. Global generalisation ensures that technology functions.

ER: What an awful idea, but you’re right. I imagine a science-fiction story where you’re limited to the things that cleaning machines can recognise, and you’re limited to recipes that cooking appliances can produce.

Wittgenstein’s paraphrased words echo still: The limits of my categories mean the limits of my world. The limits of my data mean the limits of my world.