People who can’t see know the world through touch, hearing, smell and taste. Hearing the World lets you lift your phone, take a picture of what’s around you, and have the app recognise what’s in it and tell it as a short, vivid story read aloud to you.
It was a quarterly project from my time as a Microsoft student ambassador. Four people, three months, from choosing the topic to Demo Day, and it won that quarter’s championship.
Where the idea came from
In early 2023 I was reading Madame Bovary in a reading-notes course. I’d heard Flaubert built the novel out of a short newspaper report, adding detail and using language to put you right there. Could a photo become a story the same way?
Language is exactly what large models are good at. A machine could recognise what’s in a photo, and a model could tell it as a story, a “new pair of eyes” for people who can’t see. The introduction I wrote for the project then:
Narrative is powerful. Novelists create novels with stories and narration; historians write histories with sources and narration; philosophers write philosophy with ideas and narration. The power of narrative, of words, may help people with visual impairments feel the world around them. So our team designed Hearing The World, an app that describes photos to users as stories.
Four people, not in one country
Each quarter the Student Ambassadors program runs a round of Social Impact Projects in teams. Ours was a Healthcare prototype sprint, and our Teams channel was called “Team-3”. My teammates and I weren’t in the same country, so we met on India time.
| When | What happened |
|---|---|
| Jan 25 | Channel created. Someone asked the lead, “what are we building this time?” |
| Jan 26 | First meeting; I sent the invite. The same day I reached Gold as a student ambassador, and the lead congratulated me in the channel |
| Feb 4 | Brainstorming |
| Feb 12 | Topic meeting |
| Feb 18 | The lead: “We must start building today” |
| mid-March | The back end worked: send a photo, get a story back; the Android app had an installable build |
| Mar 28 | Demo Day |
| Apr 26 | The program announced the champions in the community channel |
Who it’s for
At Demo Day we described the challenge in two lines: without someone’s help, it’s hard to understand what’s around you; and to learn about the objects and happenings nearby, you have to ask someone.
Our storyboard imagined a blind user named Dan:
- Dan wants to know what the world around him looks like;
- he opens the app and takes a photo;
- the app processes it for a few seconds;
- when it’s done, it begins: “Hi, Dan. This is the world around you…”
- Dan listens and pictures his surroundings in his mind.
What we wanted it to do
We wrote the goal in one sentence: help people with visual impairments understand and explore the world with a single touch of the screen. That meant three things:
- Recognise the surroundings: say clearly what’s in the picture and where;
- Reassure: let the user know their surroundings are safe;
- Be worth listening to: not a flat list of object names, but something told like a story.
The third is what set it apart from ordinary “describe this image” tools, and it was the part I cared about most.
How it works
- The phone takes a photo and sends it to the server;
- the server uses Azure Cognitive Services to find what’s in it: “dog”, “yellow”, “sun”…
- those words go to OpenAI, which writes a story about the photo;
- the story goes back to the phone and is read aloud.
The Android app is written in Kotlin and the back end in Python, running on an Azure virtual machine.
What I did
- Came up with the idea and settled it with the team;
- wrote the user stories and thought through features. Halfway through, Sijan wrote to me in the channel: “How about thinking of more features and writing market research? go for it!”
- wrote the project introduction (quoted above);
- set up the first few meetings. Sijan and Mritul wrote the Android app and back end; I worked on the product side.
Demo Day
Each team got only 3 to 5 minutes. Mritul asked in the channel, “So we can never show a complete project? :(”
The app wasn’t stable yet either: it “does not work all the time”. The lead asked us to record a short demo, and that recording is what we showed.
On April 26 the program announced in the Teams community that Hearing the World was that quarter’s champion, and more than twenty people replied: “Congratulations Team Hearing the World!🎉”
After
After winning, the team wanted to keep going and enter Imagine Cup. Sijan told me, “Actually it needs 3-6 months to make a good imagine cup project.” We were still finding times to discuss new ideas at the end of 2023, but it didn’t come together.
At the end of that year I nominated our lead, Deepthi, for Gold. In February 2024 she told me she’d made it and thanked me for always supporting her. Of the four of us, three went on to reach Gold. We kept helping each other along, and I think they’re all amazing.
Looking back
Later I thought about the next step: people who were born blind and people who lost their sight later need different descriptions. Those born blind may be more at home with the language of touch; for those who lost their sight later, it could adapt to how old they were when it happened.
I remember being really happy then.