Theater Vision System

The theater control system includes a camera system.

For 2024 we are using the Blackfly S camera on a short tripod at the back of the room, looking slightly up. It is fitted with a zoom lens adjusted to see all the A11 windows (from the inside). The capture region is programmed to ignore the bottom and top quarter of the frame to limit view to the potential audience.

Hardware

  • FLIR Blackfly S Mono 1.6 MP GigE Vision (Sony IMX273)

  • Power over Ethernet Injector, Phihong POE20U-560, 56V, 19.6W

  • TP-Link USB3 Gigabit Ethernet Adapter

Lenses

  • Tamron 13FM22IR 1/3” 2.2mm F/1.2 Compact CS-Mount Lens

  • Tamron C-Mount 4 to 12mm Varifocal Manual Iris Lens (uses C to CS adapter)

  • computar Megapixel Series M12 Mount 1.05mm Fisheye Lens (uses M12 to CS adapter)

The native resolution of the camera is 1440x1080 monochrome. The sensor is specified as 1/2.9” which is assumed to be 4.96 x 3.72 mm. At this size, the 2.2 mm lens is a wide-angle lens with 97 degrees view width. The 4-12 mm lens ranges from a medium 64 degree view at the short length but narrows to a 23 degree telephoto view width at the long length.

Software

The camera system uses the Spinnaker SDK to interface with the camera over Gigabit Ethernet. The SDK is installed on the Ubuntu Linux 22.04 theater controller. In practice we use the PySpin API to write computer vision code using Python, Spinnaker, and OpenCV.

demo-vision.py

A demonstration performance script which reacts to audience presence by changing the lighting color. The communication with the vision system is handled in the stage.network module.

 1#!/usr/bin/env python3
 2"""demonstrate vision system interaction
 3"""
 4import argparse, time, logging
 5import numpy as np
 6
 7from pythonosc import osc_message_builder
 8from pythonosc import udp_client
 9
10import stage.config
11import stage.network
12
13# import common logging functions
14import stage.logconfig
15
16# initialize logging for this module
17log = logging.getLogger('demo')
18
19#================================================================
20def idle_show_sequence(network):
21    network.lights.send_message("/fixture", ['rgba1', 0, 0, 128, 0])
22    network.lights.send_message("/fixture", ['rgba3', 0, 0, 128, 0])
23    time.sleep(2)
24    network.lights.send_message("/fixture", ['rgba1', 0, 0, 0, 0])
25    network.lights.send_message("/fixture", ['rgba3', 0, 0, 0, 0])
26    time.sleep(1)
27
28#================================================================
29def right_show_sequence(network):
30    network.lights.send_message("/fixture", ['rgba1', 0, 255, 0, 0])
31    time.sleep(2)
32    network.lights.send_message("/fixture", ['rgba1', 0, 0, 0, 0])
33    time.sleep(1)
34
35#================================================================
36def left_show_sequence(network):
37    network.lights.send_message("/fixture", ['rgba3', 0, 255, 0, 0])
38    time.sleep(2)
39    network.lights.send_message("/fixture", ['rgba3', 0, 0, 0, 0])
40    time.sleep(1)
41
42#================================================================
43if __name__ == "__main__":
44
45    # set up logging
46    stage.logconfig.open_log_file('logs/vision-protocol-test.log')
47    log.info("Starting vision-protocol-test.py")
48
49    parser = argparse.ArgumentParser( description = "Demonstrate the vision system protocol.")
50
51    parser.add_argument("--ip", default=stage.config.theater_IP,
52                        help="IP address of the remote OSC receiver (default: %(default)s).")
53
54    parser.add_argument("--recv", default='0.0.0.0',
55                        help="IP address of the local OSC receiver (default: %(default)s).")
56
57    stage.logconfig.add_logging_args(parser)
58    args = parser.parse_args()
59
60    # Modify logging settings as per common arguments.
61    stage.logconfig.configure_logging(args)
62
63    # create OSC clients to send messages to the theater system
64    network = stage.network.TheaterNetwork(args)
65
66    # start a local server to receive messages from the theater vision system
67    network.start_vision_client()
68
69    # start event loop
70    try:
71        while True:
72            if network.motion_detected[0]:
73                log.info("seeing motion at arm window, modifying performance...")
74                right_show_sequence(network)
75
76            elif network.motion_detected[1]:
77                log.info("seeing motion at wheel window, modifying performance...")
78                left_show_sequence(network)
79            else:
80                log.info("running idle performance...")
81                idle_show_sequence(network)
82
83            time.sleep(1)
84
85    except KeyboardInterrupt:
86        log.info("user interrupt, shutting down.")
87
88    log.info("Exiting demo-vision.py")

vision_server.py

The camera server accepts requests from clients and then begins streaming image tiles or detection events. Streams time out after an interval, so requests should be periodically renewed (typically every five seconds).

  1#!/usr/bin/env python3
  2# Obtain and process images from a Blackfly GigE machine vision camera using the FLIR Spinnaker PySpin API.
  3
  4cmd_desc = "Stream FLIR camera images using OSC UDP messages."
  5
  6import os
  7import sys
  8import logging
  9import argparse
 10import datetime
 11import threading
 12
 13# OpenCV
 14import cv2 as cv
 15
 16# NumPy
 17import numpy as np
 18
 19# python-osc
 20from pythonosc import osc_message_builder
 21from pythonosc import udp_client
 22from pythonosc import dispatcher
 23from pythonosc import osc_server
 24
 25# theater space configuration
 26from .config import camera_roi, vision_UDP_port, client_UDP_port
 27
 28# camera interface
 29from .camera import SpinnakerCamera
 30from .vision import VideoRegion
 31
 32# common logging functions
 33from .logconfig import add_logging_args, open_log_file, configure_logging
 34
 35# initialize logging for this module
 36log = logging.getLogger('server')
 37
 38#================================================================
 39def elapsed(time0, time1, interval):
 40    return (time1-time0).total_seconds() >= interval
 41
 42#================================================================
 43class ClientManager:
 44    def __init__(self, args):
 45
 46        self.args = args
 47
 48        # empty list of clients to start
 49        self.clients = {}
 50
 51        # Create the networking input and output.
 52        self.init_networking(args)
 53
 54        return
 55    #---------------------------------------------------------------
 56    def _get_or_add_client(self, ip, port):
 57        client = self.clients.get(ip)
 58
 59        if client is None:
 60            # Add a new client.
 61            # Create an OSC socket to communicate back to the client.
 62            socket = udp_client.SimpleUDPClient(ip, port)
 63            log.info("Opened socket to send to client at %s:%d", ip, port)
 64
 65            # create a new client record
 66            client = {'timestamp': datetime.datetime.now(),
 67                      'socket': socket,
 68                      'motion': False,
 69                      'images': False,
 70                      }
 71            self.clients[ip] = client
 72
 73        return client
 74
 75    #---------------------------------------------------------------
 76    def init_networking(self, args):
 77        # Initialize the OSC message dispatch system.
 78        self.dispatch = dispatcher.Dispatcher()
 79        self.dispatch.map("/request/motion", self.motion_request, needs_reply_address=True)
 80        self.dispatch.map("/request/images", self.images_request, needs_reply_address=True)
 81        self.dispatch.set_default_handler(self.unknown_message, needs_reply_address=True)
 82
 83        # Start and run the server.
 84        self.server = osc_server.OSCUDPServer((args.recv, vision_UDP_port), self.dispatch)
 85        self.server_thread = threading.Thread(target=self.server.serve_forever)
 86        self.server_thread.daemon = True
 87        self.server_thread.start()
 88        log.info("started OSC server on port %s:%d", args.recv, vision_UDP_port)
 89
 90    def motion_request(self, client_address, msgaddr, *args):
 91        # client_address is a tuple (ip address, udp port number)
 92        log.info("received motion data request from %s: %s %s", client_address, msgaddr, args)
 93        client_ip = client_address[0]
 94
 95        # fetch the client record if it exists or make a new one
 96        client = self._get_or_add_client(client_ip, client_UDP_port)
 97        client['motion'] = True
 98        client['timestamp'] = datetime.datetime.now() # update the timestamp
 99        return
100
101    def images_request(self, client_address, msgaddr, *args):
102        log.info("received images data request from %s: %s %s", client_address, msgaddr, args)
103        client_ip = client_address[0]
104
105        # fetch the client record if it exists or make a new one
106        client = self._get_or_add_client(client_ip, client_UDP_port)
107        client['images'] = True
108        client['timestamp'] = datetime.datetime.now() # update the timestamp
109        return
110
111    def unknown_message(self, client_address, msgaddr, *args):
112        """Default handler for unrecognized OSC messages."""
113        log.warning("Received unmapped OSC message from %s: %s: %s", client_address, msgaddr, args)
114
115    #---------------------------------------------------------------
116    def drop_inactive_clients(self):
117        # Purge any inactive clients from the list.  Should be called periodically.
118        purge = []
119        now = datetime.datetime.now()
120        for ip, record in self.clients.items():
121            timestamp = record['timestamp']
122            if (now - timestamp).total_seconds() > 10:
123                log.info("connection to %s has timed out", ip)
124                purge.append(ip)
125
126        for key in purge:
127            del self.clients[key]
128
129#================================================================
130class EventLoop:
131    def __init__(self, camera, args):
132
133        self.camera = camera
134        self.args = args
135        self.last_capture = None
136
137        # create a manager to keep track of client requests
138        self.manager = ClientManager(args)
139
140        # create a tracker for each active image region specified in the configuration
141        self.trackers = [VideoRegion(roi) for roi in camera_roi]
142
143        self.packet_count = 0
144        self.next_tile = -1
145        self.tile_width = 144
146        self.tile_height = 54
147        self.num_tile_cols = self.camera.capture_width // self.tile_width
148        self.num_tile_rows = self.camera.capture_height // self.tile_height
149        return
150
151    #---------------------------------------------------------------
152    def write_capture_file(self, image):
153        now = datetime.datetime.now()
154        folder = os.path.expanduser("logs")
155        filename = now.strftime("%Y-%m-%d-%H-%M-%S") + "-camera.png"
156        path = os.path.join(folder, filename)
157        cv.imwrite(path, image)
158        log.debug("Wrote region image to %s", path)
159
160    #---------------------------------------------------------------
161    def send_subsampled_image(self, image):
162        # Send a complete image at low resolution.  With 2x2
163        # decimation and a height window applied, the system is
164        # returning (270, 720) images.  Dividing by 5 yields a (54,
165        # 144) image with 7776 bytes which still fits within an OSC
166        # UDP packet.
167        subsampled = image[::5, ::5]
168        height, width, depth = subsampled.shape
169        data = subsampled.tobytes()
170        msg = osc_message_builder.OscMessageBuilder(address='/image')
171        msg.add_arg(height) # number of rows of pixel data
172        msg.add_arg(width)  # number of columns of pixel data
173        msg.add_arg(self.camera.capture_height) # full image pixel rows
174        msg.add_arg(self.camera.capture_width)  # full image pixel columns
175        msg.add_arg(data, arg_type='b') # blob
176        packet = msg.build()
177        self.client.send(packet)
178        self.packet_count += 1
179        if self.packet_count % 900 == 0:
180            log.info("Sent %d packets", self.packet_count)
181
182    #---------------------------------------------------------------
183    def send_image_tile(self, socket, image, tile_row, tile_col):
184        # Send a section of an image at full resolution.
185        offset_cols = tile_col * self.tile_width
186        offset_rows = tile_row * self.tile_height
187        tile = image[offset_rows:offset_rows+self.tile_height, offset_cols:offset_cols+self.tile_width]
188        height, width, depth = tile.shape
189        data = tile.tobytes()
190        msg = osc_message_builder.OscMessageBuilder(address='/tile')
191        msg.add_arg(height) # tile pixel rows
192        msg.add_arg(width)  # tile pixel columns
193        msg.add_arg(offset_rows) # tile pixel row position in full image
194        msg.add_arg(offset_cols) # tile pixel column position in full image
195        msg.add_arg(self.camera.capture_height) # full image pixel rows
196        msg.add_arg(self.camera.capture_width)  # full image pixel columns
197        msg.add_arg(data, arg_type='b') # blob of 8-bit mono pixel data
198        packet = msg.build()
199        socket.send(packet)
200        self.packet_count += 1
201        if self.packet_count % 900 == 0:
202            log.info("Sent %d packets", self.packet_count)
203
204    #---------------------------------------------------------------
205    def send_debug_region(self, image, row, col):
206        # Write a debugging image into the remote debug buffer at the specified location.
207        height, width, depth = image.shape
208        data = image.tobytes()
209        msg = osc_message_builder.OscMessageBuilder(address='/debug')
210        msg.add_arg(height) # image pixel rows
211        msg.add_arg(width)  # image pixel columns
212        msg.add_arg(row) # image pixel row position in full image
213        msg.add_arg(col) # image pixel column position in full image
214        msg.add_arg(self.camera.capture_height) # full image pixel rows
215        msg.add_arg(self.camera.capture_width)  # full image pixel columns
216        msg.add_arg(data, arg_type='b') # blob of 8-bit mono pixel data
217        packet = msg.build()
218        self.client.send(packet)
219        self.packet_count += 1
220        if self.packet_count % 900 == 0:
221            log.info("Sent %d packets", self.packet_count)
222
223    #---------------------------------------------------------------
224    def run(self):
225        while True:
226            image = self.camera.acquire_image()
227
228            # ignore empty images
229            if image is None:
230                continue
231
232            # update the motion detection for each active region
233            now = datetime.datetime.now()
234            for i, tracker in enumerate(self.trackers):
235                changed = tracker.update(image)
236                if changed:
237                    log.info("tracker %d reports motion state %s", i, tracker.motion_detected)
238
239                    # send motion data to active clients`
240                    for record in self.manager.clients.values():
241                        if record['motion'] is True:
242                            record['socket'].send_message("/motion", [i, tracker.motion_detected])
243
244                    # capture the event image (with a rate limit)
245                    if tracker.capture_timestamp is None or elapsed(tracker.capture_timestamp, now, 30):
246                        tracker.write_capture_file()
247
248            # cycle through the image tiles to send
249            self.next_tile = (self.next_tile+1) % (self.num_tile_cols*self.num_tile_rows)
250            next_row = self.next_tile // self.num_tile_cols
251            next_col = self.next_tile % self.num_tile_cols
252
253            # send image data to active clients`
254            for ip, record in self.manager.clients.items():
255                if record['images'] is True:
256                    self.send_image_tile(record['socket'], image, next_row, next_col)
257
258            # check whether any connections have timed out
259            self.manager.drop_inactive_clients()
260
261
262#================================================================
263if __name__ == '__main__':
264
265    # Initialize the command parser.
266    parser = argparse.ArgumentParser(description = cmd_desc)
267    parser.add_argument("--recv", default="0.0.0.0",    help="IP address of the OSC receiver (default: %(default)s).")
268    add_logging_args(parser)
269
270    # Parse the command line, returning a Namespace.
271    args = parser.parse_args()
272
273    # set up logging
274    open_log_file('logs/vision-server.log')
275    log.info("Starting vision_server.py")
276
277    # Modify logging settings as per common arguments.
278    configure_logging(args)
279
280    # Open the camera.
281    camera = SpinnakerCamera(args)
282    camera.start_continuous()
283
284    if not camera.continuous_mode:
285        log.error("Camera not acquiring, quitting.")
286
287    else:
288        # Create the event loop.
289        event_loop = EventLoop(camera, args)
290
291        try:
292            event_loop.run()
293
294        except KeyboardInterrupt:
295            print ("User quit.")
296
297    log.info("Exiting vision server.")