Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Realtime Color tracking #1680

Open
vikramms opened this issue Aug 22, 2024 · 39 comments
Open

Realtime Color tracking #1680

vikramms opened this issue Aug 22, 2024 · 39 comments

Comments

@vikramms
Copy link

Hi Marko,

I just got a real time Colortrack system installed. For now, I am just adding a few buttons to annotate the graph. But is there a plan to add Color track support? I can help with testing. I can connect to my PC via USB or BT. Right now, it is over BT.

Regards,

Vikram

@MAKOMO
Copy link
Member

MAKOMO commented Aug 23, 2024

What do you expect from a Color track support in Artisan exactly? Typing in the resulting color value in the corresponding RoastProperties field might be faster than connecting the device via USB. What am I missing?

@vikramms
Copy link
Author

Hi Marko,

The unit I have is the ColortrackRT, which tracks the change in color in real time. I have it installed with its laser aimed at the sight glass. Many industry experts(Rob Hoos, Scott Rao) have been saying that the success of roast in flavor is 60-80% the finishing color, 10-20% Time to first crack, and 10-20% development time. So, I embarked on this journey to explore that specifically and hence decided to have it done in real-time instead of using a benchtop unit for post-roasted analysis.

So, the first issue is logging color changes over time. They have an app, but it is crazy to run two loggers at the same time. I am attaching a picture of one of their logs for you to take a look at below. That is the first picture.

The second goal is to correlate color values with other coffee roasting stages we know, like turning point, marking the dry end, 1st crack, etc. Now, just like Artisan logs turning point in temperature, there is also a turning point in color, as you can see from the color log picture. I created a button that I click to annotate the BT curve in Artisan (as you see in the 2nd picture). This turning point in color is the starting point of the Maillard reaction. Color track folks call it Blanch Point. I mark it as MLSTART. It would be great to do this in real-time. Over time, as a roaster, I would like to know and modulate MLStart to 1st crack instead of TP in just temperature. It will be great to automate the recording. Also, I only have a marker on the graph right now, but I have to write down the color reading at each point of interest and add it to my notes per roast. I will be adding more EVENT buttons in Artisan for now to log the color reading at 1st crack, etc., but it will be great to have a target color, % of target reached on an LCD on the side and log it automatically using some ALARMS.

Right now, I am using the phone app with BT for convenience, but I can also use USB on the PC directly.

Those were my thoughts so far,

Thanks,

Vikram

IMG_3005
IMG_0008

@MAKOMO
Copy link
Member

MAKOMO commented Aug 30, 2024

I added initial ColorTrack support to Artisan. This one is completely untested as we do not have access to a ColorTrack device. A new continuous build with this should be available in a few minutes.

This one communicates via USB/Serial. It might be possible to connect via Bluetooth LE (BLE). For this we would need to know the Service UUID as well as the notify Characteristic UUID of the ColorTrack. You might investigate both using one of those free BLE scanner apps for your phone like LightBlue or nRF Connect.

For now, there is a new device type named "ColorTrack". Add an extra device of this type and configure its serial comm port according to your setup.

Screenshot 2024-08-30 at 15 52 57 Screenshot 2024-08-30 at 15 53 14

@vikramms
Copy link
Author

Trying USB now. But here is a picture of the Bluetooth information using nRF Connect.

Regards,

Vikram
IMG_3007

@vikramms
Copy link
Author

And here are the pictures from Light Blue app.

IMG_3010

Vikram

@MAKOMO
Copy link
Member

MAKOMO commented Aug 31, 2024

Could you please attach another picture from the LightBlue app after clicking the entry "Laser Measurements"? This should display the corresponding notify UUID. Thanks!

@vikramms
Copy link
Author

vikramms commented Sep 2, 2024

Hi Marko,

Here is the picture,

IMG_3052

Regards,

V

@MAKOMO
Copy link
Member

MAKOMO commented Sep 2, 2024

Thanks for this further image. BLE support might take some time to realize due to general difficulties of running proper BLE connections on desktop machines, but we will work on this during the next month and will ping you if there is something to test.

Did the USB version work for you?

@MAKOMO
Copy link
Member

MAKOMO commented Sep 11, 2024

Stalled. What a pity!

@MAKOMO MAKOMO closed this as completed Sep 11, 2024
@vikramms
Copy link
Author

Hi Marko,
Sorry, i have not been able to reply earlier. The USB is not based on UART-USB bridge, so I was not able to make it work. The main board uses some other chip and requires a proprietary driver from National Instruments. NI was recently bought and their driver is giving me issues on my new machine. It took me a lot of time to find the compatible driver.
I think the BLE is best path at this point. Do you meed me to get any more information on the BLE side?

Thanks,

Vikram

@vikramms
Copy link
Author

Hi Marko,
I made some progress. But it doesn't look like there is any information over USB. I also used the screen command on my MAC to test and there is no information.
Vikram
Artisan_CT_ET

@MAKOMO
Copy link
Member

MAKOMO commented Oct 17, 2024

Hm. So serial comm is not working. By now I added novel Bluetooth Low Energy support to Artisan and changed the ColorTrack extra device to connect via BLE. The problem is that I do not know what data in which format your device is sending. Thus for now it is not decoded and used, but instead logged into the artisan.log file.

Maybe you find the time to download the current Continuous Build, configure again one extra device of type ColorTrack (you can use the Dummy device as main device), press ON and check if Artisan is connecting to the device via BLE. If it says "Connected to ColorTrack" it should start logging the received data to the artisan.log file. Turn Artisan off and click on the plus icon in the upper left corner while holding the OPTION key. An email should open with the artisan.log file attached. Please send this and I will see if I can decode that data somehow. If not, we will have to ask FreshRoast (the producer of the ColorTrack devices) for some hints. THANKS!

@MAKOMO MAKOMO reopened this Oct 17, 2024
@vikramms
Copy link
Author

Thanks Marko, I am working on this today. I also did contact FreshRoast too. I expected that they might have have proprietary data packet format. But, I will send you the logs. Incase the data is obvious.

@MAKOMO
Copy link
Member

MAKOMO commented Oct 20, 2024

Excellent! Thanks a lot already for your support.

@vikramms
Copy link
Author

Hi Marko,

BTW, On another note. I modded my roaster to add 1 more temp. probe in the burner box at the inlet of the drum. I would like to log that too and one more post catalyst in my afterburner. How do I add more temp LCDs? I know their modbus registers. I have seen some Artisan curves with multiple temperatures but I am not able to get that setup using the Modbus dialog for multiple inputs. All the additional LCDs I add only show BT.
Is there a post, forum link or FAQ on this?
Thanks,
Vikram

@vikramms
Copy link
Author

Hi Marko,
I have downloaded the new continous build. Not sure if the Colortrack is connecting.

I can seem to select the port for the Colotrack in the window. See below,

image

But, I am also attaching the log file.

Thanks for all your help,

Vikram

PS: On the previous question I am guessing I should use MODBUS34 in the Extra Device setup.
artisanlog_102024.txt

@MAKOMO
Copy link
Member

MAKOMO commented Oct 21, 2024

All fine. The ColorTrack connected and was sending data which got logged. You had "Debug Logging" on (hold OPTION+COMMAND while clicking on the plus icon to toggle it off) and device logging ticked (menu Config >> Devices, flag "Logging") which was flooding the log with some other messages, but I could extract some of the ColorTrack responses.

Do you have an idea what number the ColorTrack showed? I assume it was static and reading about the same number during the captured period (18:07:23 -- 18:08:21), correct?

@vikramms
Copy link
Author

Hi Marko,
Actually it just shows "u.u".
Vikram

@MAKOMO
Copy link
Member

MAKOMO commented Oct 21, 2024

Don't worry. I worked it out. A moment and I will have something to test for you which should work.

@MAKOMO
Copy link
Member

MAKOMO commented Oct 23, 2024

At this moment a new Continuous Build is created which adds the data decoding for the ColorTrack. There are now two device types, ColorTrack Serial and ColorTrack BT. Please make sure you selected the later. There are no serial configurations needed for this one. The correct color readings should now show up in Artisans LCDs. It would be could if you could verify this and check if the data shown is about correct. THANKS!

There is one more thing (isn't there always one?): the ColorTrack seems to send out also temperature and humidity data. I am not sure how this is measured and for what it is good for (maybe it is mainly used for internal calibration?). To check on this data I added code to request this data form the device and log it to the artisan.log file. It would be cool if you could record some minutes and attach again the resulting artisan.log file here. Before you should disable Debug Logging (click the plus icon while holding the OPTION and COMMAND/Apple keys to toggle) and also disable device logging (menu Config >> Devices, "Logging flag" at the lower border.

Thanks for your support on this integration!

@vikramms
Copy link
Author

Hi Marko,
I will upload some logs tomorrow. The initial update is that the LCD shows as "UU" for a bit and then suddenly starts showing a number. Also, the number is <10, but normally it is an integer between 0-100. There too, the dark empty drum should show 80, but I am seeing 6.7-7.5. I did use 10*X for the LCD. I will be roasting some batches tomorrow and see the graph logged to see how close I get. I will have some log files for you tomorrow.

Thank you as well, I am quite excited.

Vikram

@MAKOMO
Copy link
Member

MAKOMO commented Oct 25, 2024

Thanks for your feedback. BLE connection takes a moment after ON, therefore it first shows the UU. The values are currently converted such that the app shows the correct color readings between 0-100. No idea yet why the readings are off for you. I will investigate using your log files once I got them.

@MAKOMO
Copy link
Member

MAKOMO commented Oct 25, 2024

Just checked again. The data you logged before does display as about 71. I just started a new build which displays the raw value received on the second LCD. Maybe you can still upgrade before roasting tomorrow.

@MAKOMO
Copy link
Member

MAKOMO commented Oct 31, 2024

Any news? Could you run some tests on the weekend?

@vikramms
Copy link
Author

vikramms commented Nov 1, 2024

Hi Marko,
Here are some logs for you to look at.

  • There are a few issues. Like I mentioned the connection is a little flaky and it kicks in a little late into the roast and a few graphs that I captured I only see the value past 60% in the roast.
  • Also, on the one day I was roasting I was having a problem where the values were just blinking. I will post a video.
  • Also, for sure the scale of the values is off. I think we need Fresh Roast to give some guidance. I will reach out to Michael W.

Here are some logs,

Vikram
artisan_102524.log
artisan_102524 v2.log
artisan_102624.log

@vikramms
Copy link
Author

vikramms commented Nov 1, 2024

Hi MArko,

Here is the video of the LCD flashing the reading and then going back to UU.

IMG_3461.MOV

Thanks,

Vikram

@MAKOMO
Copy link
Member

MAKOMO commented Nov 1, 2024

Could you please also attach one of the recorded Artisan profile as .alog.txt file to allow me to check for the actual readings received. Thanks for connecting to FreshRoast on this. Hope they can clarify. I will try to improve the connection issue. However, it is well known that the Windows BLE implementation is, well, weak.

@vikramms
Copy link
Author

vikramms commented Nov 1, 2024

Hi Marko,

One more from today. Let me get you an .alog.txt too.

Vikram

artisanlog_103124.txt

@MAKOMO
Copy link
Member

MAKOMO commented Nov 3, 2024

Ok, no artisan.alog file. Hm.

I revised the data interpretation according to the new data I see in your logs. Before Artisan interpreted only every second byte as the other bytes were all equal and regarded as sync characters. This interpretation was wrong as your new data shows that both bytes are carrying information.

I also removed any extra logging output thus attaching log files is not productive any longer.

Could you please test again with the latest continuous build and in the best case attach a recorded .alog file with the data?

Regarding the slow Bluetooth connect: the connection is actually using the Bluetooth variant Bluetooth Low Energy (BLE) which is known to be not well supported on desktop systems, especially Windows. Some Windows 10 versions and hardware do not work at all with BLE, others work but take several minutes to disconnect. While still connected, another connection cannot be established. I guess you suffer from this. Not much I can do on this, however, I could improve the scanning process slightly. On Windows 11 things seem to work a bit smoother, but there are also reports on long disconnect (all independent of Artisan or Python). On macOS connect and disconnect is instant and connections are rock solid due to the underlying BLE implementation is directly inherited von iOS running on iPhones and the BLE hardware is somewhat standardized. I fear we can not do anything to improve the BLE connect time on Windows from the Artisan side. Sorry. Thank god that Windows 10 is scheduled for obsolete soon.

THANKS!

@vikramms
Copy link
Author

vikramms commented Nov 4, 2024

Hi Marko,
Here is the alog.txt for the previous build version if that helps.
Vikram
Guatemala_110124.csv
24-11-01_0759_Guatemala.alog.txt

@vikramms
Copy link
Author

vikramms commented Nov 4, 2024

Hi Marko,
I tested the latest build. I checked the CT app on the phone for an empty drum, and it reports about 83, but the Artisan LCD shows 6.8. I will have the logs and an .alog.txt from it tomorrow when I roast.

I've attached an updated log file from today.

Vikram

artisan_log_110324.txt

@vikramms
Copy link
Author

vikramms commented Nov 4, 2024

Hi Marko,

From the Guatamala csv above, it looks like the column C_Raw is closer to the right values. However, the samples are bouncing a bit. I have colored some of the cells below to show possibly what the correct values should be. I don't know why the values after 11.14 start dropping. I have not dropped the coffee yet. This was the last batch on 11/1, so you should be able to see this from the last artisan log I sent.

Vikram

Guatemala_110124.xlsx

@vikramms
Copy link
Author

vikramms commented Nov 4, 2024

Hi Marko,
Here are logs from today.
Vikram
artisan_log_110424.txt

@MAKOMO
Copy link
Member

MAKOMO commented Nov 5, 2024

As said before. I removed all extra ColorTrack log out put from the Artisan logs in these later builds. So the log files are not productive.

The reason why values started dropping were that Artisan rejected all data not containing a $ as every second byte as I assumed those were sync bytes and all of your initial data was formatted that way. Your later logs showed that this assumption is not true thus I revised the implementation in the last builds to accept also data with second byte values different to $. I found that this second byte does contribute to the color value, but as lower-value data.

The raw color value logged additionally (but no longer in current builds) by the first builds supporting the ColorTrack did actual show the mapped value. I had the two values (raw, translated) assigned in the wrong order.

It would help if you could record one more roast with the current build and share the .alog profile again to check for the results. If those are still wrong, I see no way to work the correct data mapping out with this approach and we will need the support by the manufacture of those devices.

@vikramms
Copy link
Author

vikramms commented Nov 5, 2024

Hi Marko,
Understood. Here is another .alog and CSV export.
Regards,
Vikram
24-11-04_0730 Guatemala.alog.txt

Guatemala_0730_alog.csv

@vikramms
Copy link
Author

vikramms commented Nov 5, 2024

Hi Marko,
This looks correct but quite noisy. I did a moving average trendline with a period of 30 in XL to smoothen it out. But it may need a bit more statistical approach to pick out outliers within a moving window of 20-30 samples. The idea would be to find the color turn around point which would be the beginning of the Maillard reaction. As you can see from the Fresh roast app picture in the orginal ticket called Blanch Point. But definitely it looks like you have gotten the data values correct now.
Thanks,
Vikram

PS: In this XL I created a moving average column and then a +/-15% filter to accept the sample or the MAV and curve now shows 2 potential turn around points.

Guatemala_0730_alog.xlsx

@MAKOMO MAKOMO added this to the v3.0.4 milestone Nov 6, 2024
@MAKOMO
Copy link
Member

MAKOMO commented Nov 8, 2024

Dear Vikram,

I just uploaded a new Continuous Build adding 2 configurable data filters for the ColorTrack. The first ColorTrack device channel is using a mean sliding window filter and the second a median sliding window filter. I cannot really test this as I do not have enough data from the device. The ColorTrack sends about 60-70 readings per seconds in packages of 10 readings. As the data is rather noisy as you have observed yourself (and what you saw was already averaged!) it seems some heavy filtering is needed. I tried several variants of filters from the literature, but the simple mean filter seemed to outperform the others. I added the median filter as it deals better with small rare outliers, but the data looks like it carries rather uniform noise.

You can configure the two filters under Config >> Device, last tab "Networks" under "ColorTrack". The range of each window is 10-200. I set the default to 50 which might be a good starting point, but larger filter sizes might be needed up to 100 or even a bit more.

To not miss readings at the begin of your recordings you could first press ON and wait until you see the ColorTrack readings in the LCDs. There is now also a message "ColorTrack connected" on successful connect. Only then press START to start the recording and start your roast only after the recording is started. CHARGE and DROP should be detected automatically.

I am awaiting your report, thus I am a bit sceptic about that Blanch Point, which is not a very common concept, and the chance to be able to reliable determine this one based on online color measurements (after having worked on our Colorimeter Tonino for years). But I am always willing to learn...

@vikramms
Copy link
Author

Hi Marko,

I guess from FR's term of "Blanch Point" what it is trying to identify is sort of a more accurate point for "Dry End" and beginning of the Maillard reaction. This is when the colormeter is reading its lowest value after the roast starts while going from green to light yellow. After this point, the beans continue to get more "brown" towards its target point.

Attached is the new alog. Definitely the smoothing is helping but still noisy. Its is interesting to see that after 6.30mins the readings get super steady.

Vikram
24-11-08_1651_Chias_12_398.alog.txt

@MAKOMO
Copy link
Member

MAKOMO commented Nov 11, 2024

Vikram, did you keep the smoothing parameters at their defaults (50)?

Did you try different filter parameters like 70 or even 100, 150,....?

Any reason why you activated the DeltaAxis flag on the second color value? Even with this deactivated I see an offset in the data of about 24 points. Did you apply an offset using a symbolic formula? Also the end of both curves look very different. I don't understand how that can happen. The two filters should perform very similar.

Why after 6:30 the data contains less noise is unclear to me. Artisan is not doing any thing different than before so it must be the data from the device changing. Maybe the device switched into a different mode? Very strange.

Do the Color readings make sense to you?

Did FreshRoast came back to you with some insight on the data format?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants