Define oversight for one routine and one KOKO build
For any proposed KOKO routine, define who observes it, who approves a consequential action, and who can intervene. Homebot One is still testing the platform. Labels such as autonomous or supervised cannot establish the required human attention or the controls available on the unit you receive.
Make the expected human attention, responsibility, and recovery work visible before approving a routine. Oversight that exists only as a label does not tell a household who is watching, what they can see, or how intervention works.
Assign accountability before assigning monitoring
NIST's AI Risk Management Framework calls for clear roles and responsibilities for human-AI configurations and oversight. Begin with the person accountable for deciding that a routine may operate. Then name the operator, affected residents, technical support, safety or privacy reviewer, and person who can pause or stop the activity. Watching a dashboard is not the same as owning the decision.
For a household, one person may fill several roles, but write the roles anyway. For a care, research, or developer pilot, separate program approval, day-to-day operation, participant or resident support, technical changes, incident handling, and independent review when practical. Make clear that an affected person can raise a concern even if they are not an administrator.
- Accountable owner: approves the routine and accepts residual risk
- Operator: starts, observes, pauses, and records the activity
- Affected person: receives information, choices, and a complaint path
- Technical owner: manages build, configuration, updates, and support
- Reviewer: examines evidence, exceptions, incidents, and continuation
Choose an intervention level that matches consequence
A low-consequence routine may need periodic review, while movement near a person, handling an object, sharing sensitive information, or operating in a changing environment may justify active supervision or approval. Build an intervention ladder: observe, ask for confirmation, slow or restrict, pause, take manual control if supported, enter a safe state, and escalate. Ask Homebot One which controls actually exist for the current build and how quickly they take effect.
Define the trigger for each step. Examples may include an unexpected person or pet, blocked path, unrecognized object, conflicting command, uncertain identity, sensor obstruction, lost connectivity, repeated failure, unusual account event, or a resident asking the robot to stop. These are planning examples, not statements about KOKO's current sensors or behavior. The plan should state what happens when the system cannot determine the condition reliably.
- Normal operation and information visible to the operator
- Confirmation required before a sensitive or physical action
- Automatic or human-imposed limits on space, time, speed, or task
- Supported pause, stop, manual-control, and safe-state actions
- Escalation when the operator cannot resolve the condition
Give the overseer usable information and enough time
Human oversight can fail when the person receives an unclear alert, too many alerts, stale state, or too little time to act. Ask what the operator can see about the robot's task, location, current state, command source, uncertainty, connectivity, and recent changes. Test whether the information is understandable under realistic noise, lighting, distance, accessibility, and workload conditions.
NIST's AI RMF Playbook suggests evaluating oversight and user feedback, including the frequency of overrides. Build a record suited to the setting: a simple household journal or controlled pilot logs. Track errors, complaints, response times, and escalation decisions relevant to the routine. Collect only what is necessary and protect the record.
- Missed, late, false, and ambiguous alerts
- Pauses, overrides, manual recoveries, and aborted routines
- Operator response time and whether the available control worked
- Resident objections, complaints, and accessibility barriers
- Changes made after review and evidence that they helped
Review oversight when the context or system changes
Run a controlled rehearsal before ordinary use. Include the normal routine, a request to pause, a blocked or changed condition, an unavailable operator, lost connectivity, and recovery. Verify who notices, which control is used, what feedback the affected person receives, and how the event is recorded. Do not simulate dangerous motion or use an occupied care context merely to test a plan.
Set change triggers: a new software build, sensor, integration, room, user group, task, data use, or operating schedule. The accountable owner should decide whether prior evidence still applies and whether consent, training, controls, or approval need revision. A good plan can conclude that a routine needs constant supervision, a narrower scope, or no deployment until the build provides adequate intervention and evidence.
- Routine, environment, users, build, and approved operating limits
- Named roles, alternates, training, and communication path
- Intervention ladder with tested controls and response expectations
- Metrics, review cadence, stop conditions, and decision owner
- Change-control events that require renewed approval
Prepare a KOKO demonstration around real conditions
Plan a response to suspected security incidents
Frequently asked questions
Does human oversight mean someone must watch KOKO continuously?
Not for every possible routine. The appropriate level depends on the task, build, environment, affected people, uncertainty, consequence, available controls, and evidence. Define it explicitly rather than assuming.
Who should be accountable for a KOKO routine?
Name the person authorized to approve the routine, accept residual risk, and decide whether it continues. Operators and technical staff may support that person without owning the final decision.
What should an intervention plan contain?
Include observable triggers, confirmation rules, restrictions, pause or stop steps, supported manual or safe-state controls, escalation contacts, feedback to affected people, and return-to-service criteria.
When should the oversight plan be reviewed?
Review it after incidents or repeated overrides and whenever the build, task, room, integration, user group, data use, schedule, or available control changes.
Sources & further reading
From Homebot One, the team building KOKO in Fremont, California.
