Server system

The perception server (fp_perception::PerceptionServer) is a ROS 2 node that loads perception "drivers" as pluginlib plugins and exposes a simple set of ROS interfaces (topics and services) for audio, vision, transcription, speech synthesis, sentiment, and image analysis.

This document focuses on:

• the main concepts
• how devices are acquired and used
• the server’s ROS interfaces and configuration

Main concepts

Drivers are plugins

• Drivers implement fp_perception::DriverBase.
• The server loads drivers at runtime using pluginlib::ClassLoader<fp_perception::DriverBase>.
• Each driver is selected by a fully-qualified class name parameter (for example fp_perception::MicrophoneAudioDriver).
• Loaded plugins are cast to typed role interfaces such as AudioSourceDriver, AudioSinkDriver, TranscriptionDriver, SpeechSynthesisDriver, SentimentAnalysisDriver, VisionSourceDriver, and ImageAnalysisDriver.

Typed data exchange

• Runtime workflows use typed interfaces and shared structs such as fp_perception::audio_data, fp_perception::text_data, transcription_request, sentiment_request, and image_analysis_request.
• DriverBase remains the pluginlib base and lifecycle surface; its legacy untyped data hooks are kept only for compatibility with old callers.
• The server converts between ROS messages (fp_perception_msgs/*, sensor_msgs/*) and internal typed structs at the node boundary.

Device acquisition lives in the plugins

The server orchestrates and routes data, but the actual device I/O happens in the plugins:

• Microphone / speaker use PortAudio and resolve devices by name.
• Non-ROS vision uses OpenCV device capture (cv::VideoCapture).
• ROS vision subscribes to a topic via image_transport.

What the server runs

Background threads

If enabled by parameters, the server starts periodic loops:

• publishAudio(): pulls microphone audio chunks and optionally maintains a rolling buffer for transcription; can also publish audio to a ROS topic.
• publishVideo(): pulls frames from a vision driver and optionally publishes them as sensor_msgs/msg/Image.

Services

If enabled by parameters, the server provides:

• transcription (PerceptionTranscribe)
• speech synthesis (PerceptionSpeech)
• sentiment analysis (PerceptionSentiment)
• image analysis (PerceptionImageAnalysis)

Parameters

Feature enable flags

The server uses boolean flags to decide which drivers to load:

• use_ros_vision_driver
• use_non_ros_vision_driver
• use_microphone_driver
• use_speaker_driver
• use_transcription_driver
• use_speech_driver
• use_sentiment_driver
• use_image_analysis_driver

And:

• use_diagnostics: if true, enabled drivers publish standard ROS 2 health updates on /diagnostics via diagnostic_updater.
• run_tests: if true, calls test() on each loaded driver during startup.

Plugin selection (which class to load)

These parameters select which pluginlib class gets loaded:

• ros_vision_driver (default fp_perception::DefaultDriver)
• non_ros_vision_driver (default fp_perception::OpenCVDriver)
• microphone_driver (default fp_perception::MicrophoneAudioDriver)
• speaker_driver (default fp_perception::SpeakerAudioDriver)
• transcription_driver (default fp_perception::OpenAIDriver)
• speech_synthesis_driver (default fp_perception::OpenAISpeechDriver)
• sentiment_driver (default fp_perception::SentimentDriver)
• image_analysis_driver (default fp_perception::OpenAIImageAnalysisDriver)

Note: if both use_ros_vision_driver=true and use_non_ros_vision_driver=true, the current server code loads both vision drivers and the publish loop can emit frames from both sources.

ROS interface configuration

Audio input topic (optional publish)

• interface.audio_input.publish (bool)
• interface.audio_input.topic (string)
• interface.audio_input.frame_id (string)
• interface.audio_input.frequency (int)
• interface.audio_input.audio_retention_window (int, seconds)
• interface.audio_input.default_audio_request_window (int, seconds)

If enabled, the server publishes fp_perception_msgs/msg/PerceptionAudio at the configured rate.

Audio output topic (optional subscribe)

• interface.audio_output.subscribe (bool)
• interface.audio_output.topic (string)

If enabled, the server subscribes to PerceptionAudio and forwards audio samples into the speaker driver.

Transcription service

• interface.transcription.provide_service (bool)
• interface.transcription.service (string)

The rolling microphone buffer is sized as:

$$\text{max_samples} = sample_rate \times channels \times buffer_duration$$

where audio_retention_window comes from interface.audio_input.audio_retention_window.

When the buffer exceeds this size, the server drops the oldest samples and keeps the latest window.

Speech service

• interface.speech.provide_service (bool)
• interface.speech.service_name (string)

Sentiment service

• interface.sentiment.provide_service (bool)
• interface.sentiment.service_name (string)

Image analysis service

• interface.image_analysis.provide_service (bool)
• interface.image_analysis.service_name (string)

Vision publish

• interface.vision_input.publish (bool)
• interface.vision_input.topic (string)
• interface.vision_input.frame_id (string)
• interface.vision_input.frequency (int)

If use_ros_vision_driver=true, the server can republish frames from the ROS image subscriber driver. If use_non_ros_vision_driver=true, the server can also publish frames from the OpenCV driver.

The checked-in config currently enables the ROS vision path and publishes on perception/camera.

End-to-end flows

Device microphone -> transcription service

1. Microphone plugin acquires audio from PortAudio.
2. Server appends audio into the shared rolling AudioBuffer.
3. A client calls PerceptionTranscribe with use_device_audio=true.
4. Server calls the transcription driver with the buffer and returns the text.

Text -> speech

1. Client calls PerceptionSpeech with PerceptionText.
2. Server calls the speech driver.
3. If use_device_audio=true, server forwards synthesized audio to the speaker driver.
4. Otherwise, server returns synthesized audio in the service response.

Diagnostics

• When use_diagnostics=true, instrumented drivers publish diagnostic_msgs/msg/DiagnosticArray updates on /diagnostics.
• Audio drivers report callback overflow, underrun, queue, and buffer state.
• REST-backed drivers report request counts, failure counts, last HTTP code, and last error state.
• Vision drivers report subscription or capture health and recent frame activity.

Device microphone -> sentiment

1. Client calls PerceptionSentiment with use_device_audio=true.
2. Server transcribes the rolling audio buffer.
3. Server runs sentiment on the transcription and returns (label, score).

Device vision or external image -> image analysis

1. Client calls PerceptionImageAnalysis with a prompt.
2. If use_device_vision=true, server captures a frame from the configured vision driver; otherwise it uses the request-provided sensor_msgs/Image.
3. Server calls the image analysis driver and returns the model output text.