Two incoming streams
Speech input and Player Logger game-state capture supply the interaction. Structured parsing and validation prepare state for the LLM/API response.
← Back to timeline
Team project · Real-time voice system
Minecraft Voice Assistant Grounded in Live Game State
Jan 2025 – May 2025Connected speech, structured live game state and an LLM/API response to parallel voice and interface output.
Role: Live-state integration, structured validation and responsiveness testing.
Problem
Ground a voice assistant’s responses in structured, live Minecraft game state.
System representation
Real-time Interaction Loop
A helper that hears the question and knows what is happening in the game
The assistant combines a player's question with game events. It can explain a crafting recipe through text and speech. My contribution connected live-state capture and parsing, structured validation, API integration and responsiveness testing.
WOODEN AXE · REPORT EXAMPLE
A useful answer includes where to put each ingredient
The report's example uses three planks and two sticks to make one wooden axe. Ingredients, a 3×3 layout and instructions turn the answer into steps a player can follow.
See the actual interface screenshot from the report
PLAYER REQUEST
Speak or type
Gradio accepts text or audio. Google speech recognition turns recorded speech into text.
GAME CONTEXT
Events → structured state
Player Logger records events such as actions, coordinates and timestamps. Parsing turns CSV records into structured events.
CRAFTING KNOWLEDGE
Recipe lookup
For crafting requests, the report describes looking up a recipe and adding its ingredients and layout to the prompt.
DeepSeek-Chat generates a response
Pydantic-AI connects the model. Pydantic checks the response structure for general, crafting or game-state answers.
Read it · Gradio
The interface displays the response, including structured crafting information.
Hear it · gTTS
The speech output reads the reply. If speech output fails, text remains available.
↺ A new question and updated game events provide the next turn's context.
What structured validation does—and what it cannot promise
A response can match the expected fields while still being factually wrong. Recipe lookup supplies a reference, while the game-state input supplies context. The report also records response delays and the need to switch between the game and the assistant window.
Technical walkthrough
Response and parallel output
The response feeds gTTS speech and Gradio UI output in parallel.
The next interaction
Further speech and updated game state supply the next turn.
Core Ideas / Concept Primer
The ideas behind the implementation
Grounding in live state
A game-state-aware response depends on what is happening in the game as well as what was spoken. My contribution included live-state capture and parsing.
Structured validation
Pydantic-AI and structured-state validation prepare game state for the response workflow.
Responsiveness across two outputs
Parallel TTS and UI output were tested for synchronisation and responsiveness. The captioned demo shows the interaction in-game.
Implementation details
Speech input
STT
Live game state
Player Logger and game-state parsing
Structured validation
Pydantic-AI
LLM / API
LLM / API integration
Speech and UI output
Parallel gTTS and Gradio output
Results and Demonstrations
Connected speech, structured live game state and an LLM/API response to parallel voice and interface output.
Ownership
Implemented
Real-time state capture and parsing, structured-state validation and API integration.
Evaluated and team context
Controlled synchronisation and responsiveness testing. The demo was recorded by a teammate.