Find text from images using Google Vision AI and OCR

This walkthrough shows all the steps it takes to analyze text from images with Optical Character Recognition (OCR) using Power Automate Desktop.

WebAutomation.LaunchFirefox.LaunchFirefox Url: $'''https://unsplash.com/photos/jVcha8wHtg8''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False Timeout: 60 BrowserInstance=> Browser
SET TemporaryFile TO $'''C:\\PowerAutomate\\temporary.jpg'''
Workstation.TakeScreenshot.TakeScreenshotOfForegroundWindowAndSaveToFile File: TemporaryFile ImageFormat: System.ImageFormat.Jpg
GoogleCognitive.TextDetectionGoogle.TextDetectionFromFile APIKey: $'''AIzaSyCrubQAOucU8d-dSI5ekMlxwECKNegLS8I''' ImageFile: TemporaryFile Timeout: 30 Response=> JsonAsCustomObject StatusCode=> StatusCode
LOOP FOREACH CurrentItem IN JsonAsCustomObject['responses'][0]['textAnnotations']
    SET Word TO CurrentItem['description']
    SET X TO CurrentItem['boundingPoly']['vertices'][0]['x'] + (CurrentItem['boundingPoly']['vertices'][2]['x'] - CurrentItem['boundingPoly']['vertices'][0]['x']) / 2
    SET Y TO CurrentItem['boundingPoly']['vertices'][0]['y'] + (CurrentItem['boundingPoly']['vertices'][2]['y'] - CurrentItem['boundingPoly']['vertices'][0]['y']) / 2
    Variables.TruncateNumber.GetIntegerPart Number: X Result=> X
    Variables.TruncateNumber.GetIntegerPart Number: Y Result=> Y
    # Do whatever you like with Word, X and Y variables.
END
# Tested with Power Automate Desktop version 2.15.284.21354

WebAutomation.LaunchFirefox Url: $'''https://unsplash.com/photos/jVcha8wHtg8''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False Timeout: 60 BrowserInstance=> Browser
SET TemporaryFile TO $'''C:\\PowerAutomate\\temporary.jpg'''
WebAutomation.DataExtraction.TakeScreenshotSaveToFile BrowserInstance: Browser File: TemporaryFile FileFormat: WebAutomation.FileFormat.Jpg
Cognitive.Google.Vision.TextDetectionFromFile APIKey: $'''API_KEY''' ImageFile: TemporaryFile Timeout: 30 Response=> JsonAsCustomObject StatusCode=> StatusCode
LOOP FOREACH CurrentItem IN JsonAsCustomObject['responses'][0]['textAnnotations']
    SET Word TO CurrentItem['description']
    SET X TO CurrentItem['boundingPoly']['vertices'][0]['x'] + (CurrentItem['boundingPoly']['vertices'][2]['x'] - CurrentItem['boundingPoly']['vertices'][0]['x']) / 2
    SET Y TO CurrentItem['boundingPoly']['vertices'][0]['y'] + (CurrentItem['boundingPoly']['vertices'][2]['y'] - CurrentItem['boundingPoly']['vertices'][0]['y']) / 2
    Variables.GetIntegerPart Number: X Result=> X
    Variables.GetIntegerPart Number: Y Result=> Y
    # Do whatever you like with Word, X and Y variables.
END
# Tested with Power Automate Desktop version 2.11.51.21196

How it works

In this template, a web page screenshot is used as the OCR image, but you could use any web image or a local image file.

Firefox is used to navigate to a website. Entire web page will be captured and saved to a temporary local file (.jpg). The image is sent to Google Vision API for OCR analysis and a JSON is returned. The JSON response contains text and x, y coordinates for all words and characters in the screenshot, for example:

        ...
        {
          "description": "PIZZA",
          "boundingPoly": {
            "vertices": [
              {
                "x": 673,
                "y": 395
              },
              {
                "x": 716,
                "y": 395
              },
              {
                "x": 716,
                "y": 421
              },
              {
                "x": 673,
                "y": 421
              }
            ]
          }
        },
        ...

How to enable Google Cloud Vision API

Follow these steps to activate Google Vision AI and to use OCR with Power Automate Desktop.

Register a Google Cloud Account trial.
Open Google Cloud Vision dashboard.
Choose API & Services from top-left navigation menu and then select Credentials from left navigation menu.
Select +Create Credentials > API Key from top menu.
As the last step, enable Google Cloud Vision API.

Instructions

Line 1: Replace https://unsplash.com/photos/jVcha8wHtg8 with any website URL.
Line 2: Replace C:\PowerAutomate\temporary.jpg with a temporary path and file name on your computer.
Line 4: Replace API_KEY with your Google Cloud Vision API key.
Line 11: Words and characters found by OCR are now stored into Word, X and Y variables.

Notes

Chrome and Edge web browsers seems to have issues with Take screenshot of web page action. The action may fail with following error message if the web page height is larger than the browser viewport:

Failed to capture image (error in communication with browser)

Google Vision AI includes two annotation features that support optical character recognition (OCR): TEXT_DETECTION and DOCUMENT_TEXT_DETECTION. The built in Google Vision action in Power Automate Desktop uses only TEXT_DETECTION.
Do not expose your API key to anyone. Set API key restrictions to prevent unauthorized use and quota theft.

Changes

2022-01-01: Updated flow actions to work with Power Automate Desktop version 2.15.284.21354

Author

PAG Admin