AI assistance for seniors with vision Impairment

My wife is a senior with vision impairment. Her preferred electronic display device is a large TV screen and to a lesser extent the large Windows desktop screen. She very much prefers interacting with devices with smaller screens and screenless devices like remotes primarily through AI assisted voice, when possible, especially when away from home. The Ipad Mini has a bigger screen and acts as a alternate talk/text device when mobile to the IPhone with the same phone number. Alternatively, when home, she can project the small apple device screens onto the large TV screen with Apple airplay. On the desktop, she can cast a chrome tab onto the large TV screen. 

She consumes video content only through her large TV screen. When the internet is functioning, all our video content is through streaming. She uses a single partly voice controlled large button Xumo stream box remote to control the Xumo stream box and the TV. The box has all her spectrum cable TV channels in the spectrum TV app (and she can record there to her hearts content with unlimited spectrum cloud DVR). It also has other streaming apps she uses frequently like Netflix and YouTube.

She interacts with her people calling systems (phone calls, facetime video calls, WhatsApp video calls) largely through Siri on the apple mobile devices. She sometimes uses the desktop for WhatsApp calls. For text messaging she uses the Windows desktop. Windows Narrator can be used for voice rendering of text messages. Wont discuss that here. 

She queries her google calendar and yahoo mail and Gmail information primarily through Google Gemini voice commands (I maintain the calendar for her). She gets her morning briefing on Kailua air quality and Kailua weather from her AI assistant completely by voice interaction. Key questions in the morning are “Tell me the air quality in Kailua Oahu today especially particulate matter in air” and “what is the weather like today in Kailua Oahu” and "give me a five-day weather forecast for Kailua Oahu”. She interacts with other AI assistants for other purposes primarily through voice.

For accessing online accounts and shopping she uses the windows desktop where an easier experience to visit online sites has been implemented.

This essay focuses primarily on two AI systems she uses – OpenAI’s ChatGPT and Google’s Gemini – and how it assists the vision impaired. I had already touched on her use of Siri. I will also discuss the voice enabled big button Xumo stream box remote. A few AI assistants constantly by your side are invaluable for a senior, especially a vision impaired one.

While you can access ChatGPT and Google Gemini from a browser, I will only focus here on the App versions available in the apple app store because they are better.

It is very easy and straight forward to install the two apps from the apple app store in your iPhone and iPad mini. Those are the two apple mobiles we have. You then enter your login information into the apps.  We have separate ChatGPT accounts for Dee and me. We have multiple Google accounts. We use one of our Google accounts for Gemini - same one as the one with the Google calendar we use. If you are willing to pay you can upgrade to Gemini Pro.

To enable voice on chat GPT, tap the microphone icon. The first time around you allow microphone access (before that, make sure it is enabled in your device setting). You then start talking. The words it hears are displayed (you can verify) before the response. You can pick a voice in ChatGPT. Once a voice is picked, ChatGPT will respond both verbally and textually to what you say.

To enable voice on Gemini is similar. Tap the microphone icon. The first time around you allow microphone access (before that, make sure it is enabled in your device setting). You can then start talking. The words it hears are displayed (you can verify) before the response. Like chat GPT you can pick a voice. Gemini will respond both verbally and textually to what you say.

ChatGPT and Gemini understands natural language English. You can converse with them back and forth. Gemini Live icon option (wave like icon) is especially good for that and for exploring complex topics. On an android phone, Gemini can be set up for completely hands-free operation like Siri. However, on an apple mobile, you still need to press the microphone icon or the google live wave looking icon today. 

To allow Gemini to access your Calendar or Gmail, do the following.

• Click your profile picture → Settings
• Go to “Connected apps”
• Turn on Google Workspace / Google Calendar access
• You may need to set up appropriate permissions in various places for Gemini access. Gemini will lead you through that.

Now you can ask questions about your Gmail (or gmailified yahoo mail) or calendar. You can switch to a different google account by clicking your profile picture and switching google accounts. Gemini Live however does not currently have access to calendar or emails but Vanilla Gemini does.

Now I will focus on the kinds of calendar questions you can ask. I will explore later the kinds of Gmail (and indirectly yahoo mail) questions you can ask (not in this essay).

The most important questions to ask in the morning is "what are my appointments and tasks today" and  “give a summary of unread emails received yesterday and overnight that are likely not spam or marketing messages”. 

<< to be added>>

 

Lastly, I will discuss the voice enable Xumo stream box remote and the kind of voice commands you can give there.

 

<< to be added >>

As a extra effort project, exploring in the background two things.

1. Voice rendering of text messages and emails

2. Voice diction of text messages and email response to someone. .