Finish setting up an ophyd_async OAV #857

noemifrisina · 2024-10-21T12:31:22Z

Closes #716
Closes #824

Create an ophyd_async MJPG area detector plugin to replace old one
Re-write snapshot devices to inherit from new MJPG
Add snapshots to ophyd_async OAV
Use async OAV in beamline device instantiation
Add a small CAM plugin with minimum requirements for hyperion
Add unit tests, fix system tests to work with async device
Remove all obsolete code and tests

Instructions to reviewer on how to test:

Check that all the functionality of the old ophyd OAV device and MJPG plugin is covered by the new ophyd_async code
Check connection for all mx beamlines still works
Run tests and confirm they pass
Confirm tests in mx-bluesky still pass (see mx-bluesky#594)

Checks for reviewer

Would the PR title make sense to a scientist on a set of release notes
If a new device has been added does it follow the standards
If changing the API for a pre-existing device, ensure that any beamlines using this device have updated their Bluesky plans accordingly
Have the connection tests for the relevant beamline(s) been run via dodal connect ${BEAMLINE}

codecov · 2024-10-21T12:34:02Z

Codecov Report

Attention: Patch coverage is 98.55769% with 3 lines in your changes missing coverage. Please review.

Project coverage is 95.53%. Comparing base (c89282a) to head (f30a519).

Files with missing lines	Patch %	Lines
src/dodal/devices/oav/oav_detector.py	94.82%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #857      +/-   ##
==========================================
- Coverage   95.59%   95.53%   -0.06%     
==========================================
  Files         125      127       +2     
  Lines        5422     5378      -44     
==========================================
- Hits         5183     5138      -45     
- Misses        239      240       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

noemifrisina · 2024-10-24T11:59:21Z

I realised once I started updating the hyperion side that we do set some of the cam plugin. For now, I just made a small device here with the minimum requirements, but I'm wondering iff it's worth opening an issue to do it properly in the future?

Also, another pair of eyes on the codecov patch would help, please. I think I covered all the old tests and I can't figure out what's missing that makes the coverage decrease...

DominicOram

Great, thanks. Almost there but I added a few comments

DominicOram · 2024-10-25T15:55:49Z

src/dodal/devices/areadetector/plugins/CAM.py

+        self.color_mode = epics_signal_rw(ColorMode, "GC_BalRatioSelector")
+        self.acquire_period = epics_signal_rw(float, "AcquirePeriod")
+        self.acquire_time = epics_signal_rw(float, "AcquireTime")
+        self.gain = epics_signal_rw(float, "Gain")
+
+        self.array_size_x = epics_signal_r(int, "ArraySizeX_RBV")
+        self.array_size_y = epics_signal_r(int, "ArraySizeY_RBV")


Must: these aren't connecting in dodal connect i03 as they need the PV prefix

DominicOram · 2024-10-25T15:56:16Z

src/dodal/devices/areadetector/plugins/MJPG.py

+        self.x_size = epics_signal_r(int, prefix + "ArraySize1_RBV")
+        self.y_size = epics_signal_r(int, prefix + "ArraySize2_RBV")


Must: these aren't connecting in dodal connect i03 as you're missing a colon

DominicOram · 2024-10-25T15:59:15Z

src/dodal/devices/areadetector/plugins/MJPG.py

+        # TODO check type of these two
+        self.input_rbpv = epics_signal_r(str, prefix + "NDArrayPort_RBV")
+        self.input_plugin = epics_signal_rw(str, prefix + "NDArrayPort")


Should: These two PVs aren't used anyway, lets just remove them

DominicOram · 2024-10-25T16:01:25Z

src/dodal/devices/areadetector/plugins/MJPG.py

+        await self.last_saved_path.set(path, wait=True)
+
+    @AsyncStatus.wrap
+    async def trigger(self):
        """This takes a snapshot image from the MJPG stream and send it to the
        post_processing method, expected to be implemented by a child of this class.

        It is the responsibility of the child class to save any resulting images.


Nit: by calling _save_image

DominicOram · 2024-10-25T16:11:45Z

src/dodal/devices/areadetector/plugins/MJPG.py

+                if not response.ok:
+                    LOGGER.error(
+                        f"OAV responded with {response.status}: {response.reason}."
+                    )
+                    raise ClientConnectionError(
+                        f"OAV responded with {response.status}: {response.reason}."
+                    )


Should: I think we can just do response.raise_for_status() or actually ClientSession(raise_for_status=True)

DominicOram · 2024-10-25T16:35:41Z

src/dodal/devices/oav/utils.py

 ) -> np.ndarray:
-    beam_distance_px: Pixel = oav_params.calculate_beam_distance(*pixel)
+    beam_distance_px: Pixel = calculate_beam_distance((beam_centre), *pixel)


Nit: These brackets seem unrequired

DominicOram · 2024-10-25T16:37:46Z

system_tests/test_oav_system.py

+    oav_config = OAVConfig(ZOOM_LEVELS_XML, DISPLAY_CONFIGURATION)
+    async with DeviceCollector():
+        oav = OAV("", config=oav_config, name="oav")
+    await oav.connect()


Should: DeviceCollector already does the connect for you

DominicOram · 2024-10-25T16:41:02Z

tests/devices/unit_tests/oav/test_oav.py

+    autospec=True,
+)
+@patch("dodal.devices.areadetector.plugins.MJPG.Image")
+async def test_snapshot_trigger_fails_in_post_processing_withouth_raising_error(


Nit: Typo on without

Should: I also think we probably do want it to raise

DominicOram · 2024-10-25T16:44:31Z

tests/devices/unit_tests/oav/test_snapshots.py

+    # Set new directory and test that it's created
+    set_mock_value(snapshot.directory, "new_dir")
+
+    await snapshot.trigger()
+
+    mock_mkdir.assert_called_once()


Nit: I would test this in a separate test that's just testing a new folder is created i.e. like test_given_directory_not_existing_when_snapshot_triggered_then_directory_created

DominicOram · 2024-10-25T16:45:21Z

tests/devices/unit_tests/oav/test_snapshots.py

+    mock_get.return_value.__aenter__.return_value = (mock_response := AsyncMock())
+    mock_response.ok = True
+    mock_response.read.return_value = (test_data := b"TEST")
+
+    mock_aio_open = mock_aiofiles.open
+    mock_aio_open.return_value.__aenter__.return_value = (mock_file := AsyncMock())
+
+    mock_open = patch_image.open
+    mock_open.return_value.__aenter__.return_value = test_data


Should: You do this mocking a few times, I think pull it into a helper function

noemifrisina added 24 commits October 16, 2024 15:36

Set up skeleton of async mjpg

b2d7b1b

Start filling things in

bf59cce

Try using completed_status

579bac1

Add SnapshotWithBeamCentre

67e774c

Add SnapshotWithBeamCentre

dd0e707

A first try at adding snapshot to oav

2a1ed7f

Rename

75e83b5

Start adding async grid overlay

e47699b

Actually add file

6d28b69

Start adding a test - might be deleted

866addb

Move mjpg to plugins and get trigger working with async

07ffa05

Remove microns per pixels from mjpg

4d9c485

Pass an instance of soft signal beam_centre and fix imports

76eecae

Fix some typing and signals

3553bfc

A bunch of fixmes

a15765c

A bunch of fixmes with test

a0092a3

Improving file saving to make async and looking for the problem

b5f6e7c

Merge branch 'main' into 824_mjpg-async-device

233abc9

A passing test

51d45f7

get test to pass

46c4a33

Tidy up and fix tests

f6af967

Add a test for snapshot with grid

d321c16

Merge branch 'main' into 824_mjpg-async-device

d316812

Fix oav

13edeca

noemifrisina added 5 commits October 21, 2024 13:43

Start breaking things: remove old OAV

126f687

Rename oav async file

25166a4

Remove old MJPG code and move around some files

a5b800b

Rename

83429ac

Instantiate async oav on beamlines

c6a518b

noemifrisina added 18 commits October 23, 2024 11:59

Remove some old tests and add to new ones

0f44f68

Remove tests for old oav

a27afb9

Remove old OAVConfigParams

dab5697

Fix import

dc3650b

Hopefully fix system test

fe599ed

Fix more imports

caef818

Add small missing test

acc1bd0

Fix snapshot prefixes

c7b78cd

Fix snapshot prefixes - part 2

1b03b35

Merge branch 'main' into 824_mjpg-async-device

5358486

Add small test

dcaa8fd

Try to make codecov happy

b4ec7a2

Fix test

fe259d7

Add a small cam plugin for oav since we use it

eb32c5a

Try to fix test

5a03213

Add test for parameters

6541cdd

Add array sizes

9f13205

Merge branch 'main' into 824_mjpg-async-device

df612d8

noemifrisina marked this pull request as ready for review October 24, 2024 11:59

noemifrisina requested review from DominicOram and olliesilvester October 24, 2024 11:59

noemifrisina added 4 commits October 24, 2024 14:29

Add check for zoom level and test

b2d0dbe

Fix linting

9c8a3cc

Really fix linting

c0914a6

Merge branch 'main' into 824_mjpg-async-device

bffbd29

noemifrisina mentioned this pull request Oct 24, 2024

Use ophyd_async OAV DiamondLightSource/mx-bluesky#594

Draft

noemifrisina added 2 commits October 24, 2024 17:00

Fix docstring

6f19281

Change i04 oav instatiation

f30a519

DominicOram requested changes Oct 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finish setting up an ophyd_async OAV #857

Finish setting up an ophyd_async OAV #857

noemifrisina commented Oct 21, 2024 •

edited

Loading

codecov bot commented Oct 21, 2024 •

edited

Loading

noemifrisina commented Oct 24, 2024

DominicOram left a comment

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

DominicOram Oct 25, 2024

		self.x_size = epics_signal_r(int, prefix + "ArraySize1_RBV")
		self.y_size = epics_signal_r(int, prefix + "ArraySize2_RBV")

Finish setting up an ophyd_async OAV #857

Are you sure you want to change the base?

Finish setting up an ophyd_async OAV #857

Conversation

noemifrisina commented Oct 21, 2024 • edited Loading

Instructions to reviewer on how to test:

Checks for reviewer

codecov bot commented Oct 21, 2024 • edited Loading

Codecov Report

noemifrisina commented Oct 24, 2024

DominicOram left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noemifrisina commented Oct 21, 2024 •

edited

Loading

codecov bot commented Oct 21, 2024 •

edited

Loading