Skip to content
This repository has been archived by the owner on Mar 19, 2023. It is now read-only.

Commit

Permalink
Merge pull request #66 from robmarkcole/drop-predictions
Browse files Browse the repository at this point in the history 
Drop predictions
  • Loading branch information
@robmarkcole
robmarkcole authored Oct 10, 2019
2 parents 03010cf + b3f3493 commit 92499d0
Show file tree
Hide file tree
Showing 6 changed files with 90 additions and 106 deletions.
22 changes: 1 addition & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Deepstack [object detection](https://python.deepstack.cc/object-detection) can i
docker run -e VISION-DETECTION=True -e API-KEY="Mysecretkey" -v localstorage:/datastore -p 5000:5000 --name deepstack -d deepquestai/deepstack:noavx
```

The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total number of `target` objects that are above a `confidence` threshold which has a default value of 80%. The class and number of objects of each object detected (of any confidence) is listed in the entity attributes, as well as the time of the last positive detection of the target object in the `last detection` attribute. An event `image_processing.object_detected` is fired for each object detected. Optionally the processed image can be saved to disk. If `save_file_folder` is configured two images are created, one with the filename of format `deepstack_latest_{target}.jpg` which is over-written on each new detection of the `target`, and another with a unique filename including the timestamp. You can use a [local_file](https://www.home-assistant.io/integrations/local_file/) camera to display the `deepstack_latest_{target}.jpg` image on the HA front-end.
The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total number of `target` objects that are above a `confidence` threshold which has a default value of 80%. The class and number of objects of each object detected (of any confidence) is listed in the entity attributes under `summary`, as well as the time of the last positive detection of the target object in the `last detection` attribute. An event `image_processing.object_detected` is fired for each object detected. Optionally the processed image can be saved to disk. If `save_file_folder` is configured two images are created, one with the filename of format `deepstack_latest_{target}.jpg` which is over-written on each new detection of the `target`, and another with a unique filename including the timestamp. You can use a [local_file](https://www.home-assistant.io/integrations/local_file/) camera to display the `deepstack_latest_{target}.jpg` image on the HA front-end.

Add to your Home-Assistant config:
```yaml
Expand Down Expand Up @@ -69,26 +69,6 @@ Configuration variables:
<img src="https://github.com/robmarkcole/HASS-Deepstack-object/blob/master/docs/object_detail.png" width="350">
</p>

## Predictions
The full predictions from Deepstack are returned in the `predictions` attribute as json in a standardised format. This makes it easy to access any of the predictions data using a [template sensor](https://www.home-assistant.io/integrations/template/). Additionally the box coordinates and the box center (centroid) are returned, meaning an automation can be used to determine whether an object falls within a defined region-of-interest (ROI). This can be useful to include/exclude objects by their location in the image.

**Note:** In the case that a unique object is present in an image (e.g. a single person or single car) then the direction of travel of the object [can in principle be determined](https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/) by comparing centroids in adjacent image captures. This tracking functionality is not implemented yet in Home Assistant.

Example prediction output:
```
[{'confidence': 85.0,
'label': 'person',
'box': [0.148, 0.307, 0.817, 0.47],
'centroid': [0.388, 0.482]},
.
.
]
```

The `box` is defined by the tuple `(y_min, x_min, y_max, x_max)` (equivalent to image top, left, bottom, right) where the coordinates are floats in the range `[0.0, 1.0]` and relative to the width and height of the image.

The centroid is in `(x,y)` coordinates where `(0,0)` is the top left hand corner of the image and `(1,1)` is the bottom right corner of the image.

#### Event `image_processing.object_detected`
An event `image_processing.object_detected` is fired for each object detected above the configured `confidence` threshold. An example use case for this is incrementing a [counter](https://www.home-assistant.io/components/counter/) when a person is detected. The `image_processing.object_detected` event payload includes:

Expand Down
63 changes: 15 additions & 48 deletions custom_components/deepstack_object/image_processing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,6 @@
)


def get_now_str():
"""
Returns now as string.
"""
return dt_util.now().strftime("%Y-%m-%d-%H-%M-%S")


def get_box(prediction: dict, img_width: int, img_height: int):
"""
Return the relative bounxing box coordinates
Expand All @@ -94,40 +87,6 @@ def get_box(prediction: dict, img_width: int, img_height: int):
return box


def get_box_centroid(box: Tuple) -> Tuple:
"""
Locate the box centroid in (x,y) coordinates where
(0,0) is the top left hand corner of the image and
(1,1) is the bottom right corner of the image.
"""
rounding_decimals = 3

y_min, x_min, y_max, x_max = box
centroid = ((x_max + x_min) / 2, (y_max + y_min) / 2)
centroid = [round(coord, rounding_decimals) for coord in centroid]
return centroid


def format_predictions(predictions: dict, img_width: int, img_height: int) -> str:
"""
Return deepstack predictions in standardised json format where confidences
are a percentage (%) and bounding boxes are in relative cordinates
"""
predictions_formatted = []
for prediction in predictions:
formatted_prediction = {}
formatted_prediction["confidence"] = ds.format_confidence(
prediction["confidence"]
)
formatted_prediction["label"] = prediction["label"].lower()

box = get_box(prediction, img_width, img_height)
formatted_prediction["box"] = box
formatted_prediction["centroid"] = get_box_centroid(box)
predictions_formatted.append(formatted_prediction)
return json.dumps(predictions_formatted)


def draw_box(
draw: ImageDraw,
box: Tuple[float, float, float, float],
Expand Down Expand Up @@ -231,7 +190,9 @@ def __init__(

def process_image(self, image):
"""Process an image."""
self._image_width, self._image_height = Image.open(io.BytesIO(bytearray(image))).size
self._image_width, self._image_height = Image.open(
io.BytesIO(bytearray(image))
).size
self._state = None
self._targets_confidences = []
self._predictions = {}
Expand All @@ -255,7 +216,7 @@ def process_image(self, image):
)
)
if self._state > 0:
self._last_detection = get_now_str()
self._last_detection = dt_util.now()
self._summary = ds.get_objects_summary(self._predictions)
self.fire_prediction_events(self._predictions, self._confidence)
if hasattr(self, "_save_file_folder") and self._state > 0:
Expand Down Expand Up @@ -286,7 +247,7 @@ def save_image(self, image, predictions, target, directory):

latest_save_path = directory + "deepstack_latest_{}.jpg".format(target)
timestamp_save_path = directory + "deepstack_{}_{}.jpg".format(
target, get_now_str()
target, self._last_detection.strftime("%Y-%m-%d-%H-%M-%S")
)
try:
img.save(latest_save_path)
Expand Down Expand Up @@ -333,17 +294,23 @@ def name(self):
"""Return the name of the sensor."""
return self._name

@property
def unit_of_measurement(self):
"""Return the unit of measurement."""
target = self._target
if self._state != None and self._state > 1:
target += "s"
return target

@property
def device_state_attributes(self):
"""Return device specific state attributes."""
attr = {}
attr["target"] = self._target
attr["target_confidences"] = self._targets_confidences
attr["summary"] = self._summary
attr["predictions"] = format_predictions(
self._predictions, self._image_width, self._image_height
)
attr["last_detection"] = self._last_detection
if self._last_detection:
attr["last_detection"] = self._last_detection.strftime("%Y-%m-%d %H:%M:%S")
if hasattr(self, "_save_file_folder"):
attr["save_file_folder"] = self._save_file_folder
return attr
55 changes: 18 additions & 37 deletions development/.ipynb_checkpoints/Deepstack object detection-checkpoint.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -611,43 +611,6 @@
" return json.dumps(predictions_formatted)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'confidence': 0.9995428,\n",
" 'label': 'person',\n",
" 'y_min': 95,\n",
" 'x_min': 295,\n",
" 'y_max': 523,\n",
" 'x_max': 451},\n",
" {'confidence': 0.9994912,\n",
" 'label': 'person',\n",
" 'y_min': 99,\n",
" 'x_min': 440,\n",
" 'y_max': 531,\n",
" 'x_max': 608},\n",
" {'confidence': 0.9990447,\n",
" 'label': 'dog',\n",
" 'y_min': 358,\n",
" 'x_min': 647,\n",
" 'y_max': 539,\n",
" 'x_max': 797}]"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predictions"
]
},
{
"cell_type": "code",
"execution_count": 30,
Expand Down Expand Up @@ -679,6 +642,24 @@
"json.loads(format_predictions(predictions, img_width, img_height))"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2019-10-10 16:39:03\n"
]
}
],
"source": [
"import datetime\n",
"print(datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down
56 changes: 56 additions & 0 deletions development/Deepstack object detection.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -642,6 +642,62 @@
"json.loads(format_predictions(predictions, img_width, img_height))"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2019-10-10 16:39:03\n"
]
}
],
"source": [
"import datetime\n",
"print(datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"a = 'test' "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"a += 's'"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'tests'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down
Binary file modified docs/object_detail.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/object_usage.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 92499d0

Please sign in to comment.