This is the biggest project I’ve worked on so far, and I’ve learned a lot while building it.
What can it do?
Right now Jarvis has quite a few tools/features:
- Voice interaction
- I can interrupt Jarvis while it is speaking
- Create files and folders, Read files, Write/update files
- Open websites
- Open applications
- Create new projects from basic template
- React template
- Express/Node.js server template
- OCR extract information from screenshots
- Play songs through Spotify
Run terminal commands (I don't allow every command because safety comes first)
RAG based personal memory
- Search my personal vector database
- Retrieve relevant information
- Use that information to answer questions
- Continuous monitoring
- Face detection and identification
- Identify someone from a newly captured picture when continuous monitoring is off
- Identify the current person directly from currentPerson.json when continuous monitoring is running
- Face registration using multiple poses: Center , Left , Right , Up , Down
- Manual face registration is also supported
For registration, I calculate the pose using face landmarks from the eyes and nose rather than just asking the person to move randomly.
It's definitely not perfect One of the problems I'm currently dealing with is the smaller local LLM I'm using. I'm running Qwen3 1.7B because my hardware is pretty limited. Sometimes it makes mistakes when deciding which tool it should call. For example, in video when i ask Who is in front of you? instead of actually calling my face identification tool, it will simply respond that the person in front of it has been identified.
So there are definitely still plenty of things that need fixing.
But honestly, I'm pretty happy with it considering this is my first project of this size.
There are a lot more things I want Jarvis to do.
One of my ideas is to make it respond differently depending on who is standing in front of it. For example, if Jarvis recognizes someone, it could use the personal memory associated with that person when answering.
The problem is that running:
Continuous vision + LLM + Whisper STT + TTS
at the same time is already pushing my PC pretty hard.I have a GTX 1650 4GB, so there's only so much I can throw at it.
So for now i can only improve the UI and polishing what I already have.
For example, I'm currently working on things like:
- A camera panel that comes down from the top when face registration starts
- Displaying what Jarvis is currently saying as text
Eventually, when I have better hardware, I'd really like to come back to this project and push it much further.
For now, I'm just enjoying building it and learning along the way.
Would you actually use an offline assistant like this? And what feature would you add next?