Find Text From Images Using Google Vision AI and OCR

This walkthrough shows all the steps it takes to analyze text from images with Optical Character Recognition (OCR) using Power Automate Desktop.

WebAutomation.LaunchFirefox.LaunchFirefox Url: $'''https://unsplash.com/photos/jVcha8wHtg8''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False Timeout: 60 BrowserInstance=> Browser
SET TemporaryFile TO $'''C:\\PowerAutomate\\temporary.jpg'''
Workstation.TakeScreenshot.TakeScreenshotOfForegroundWindowAndSaveToFile File: TemporaryFile ImageFormat: System.ImageFormat.Jpg
GoogleCognitive.TextDetectionGoogle.TextDetectionFromFile APIKey: $'''AIzaSyCrubQAOucU8d-dSI5ekMlxwECKNegLS8I''' ImageFile: TemporaryFile Timeout: 30 Response=> JsonAsCustomObject StatusCode=> StatusCode
LOOP FOREACH CurrentItem IN JsonAsCustomObject['responses'][0]['textAnnotations']
    SET Word TO CurrentItem['description']
    SET X TO CurrentItem['boundingPoly']['vertices'][0]['x'] + (CurrentItem['boundingPoly']['vertices'][2]['x'] - CurrentItem['boundingPoly']['vertices'][0]['x']) / 2
    SET Y TO CurrentItem['boundingPoly']['vertices'][0]['y'] + (CurrentItem['boundingPoly']['vertices'][2]['y'] - CurrentItem['boundingPoly']['vertices'][0]['y']) / 2
    Variables.TruncateNumber.GetIntegerPart Number: X Result=> X
    Variables.TruncateNumber.GetIntegerPart Number: Y Result=> Y
    # Do whatever you like with Word, X and Y variables.
END
# Tested with Power Automate Desktop version 2.15.284.21354

WebAutomation.LaunchFirefox Url: $'''https://unsplash.com/photos/jVcha8wHtg8''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False Timeout: 60 BrowserInstance=> Browser
SET TemporaryFile TO $'''C:\\PowerAutomate\\temporary.jpg'''
WebAutomation.DataExtraction.TakeScreenshotSaveToFile BrowserInstance: Browser File: TemporaryFile FileFormat: WebAutomation.FileFormat.Jpg
Cognitive.Google.Vision.TextDetectionFromFile APIKey: $'''API_KEY''' ImageFile: TemporaryFile Timeout: 30 Response=> JsonAsCustomObject StatusCode=> StatusCode
LOOP FOREACH CurrentItem IN JsonAsCustomObject['responses'][0]['textAnnotations']
    SET Word TO CurrentItem['description']
    SET X TO CurrentItem['boundingPoly']['vertices'][0]['x'] + (CurrentItem['boundingPoly']['vertices'][2]['x'] - CurrentItem['boundingPoly']['vertices'][0]['x']) / 2
    SET Y TO CurrentItem['boundingPoly']['vertices'][0]['y'] + (CurrentItem['boundingPoly']['vertices'][2]['y'] - CurrentItem['boundingPoly']['vertices'][0]['y']) / 2
    Variables.GetIntegerPart Number: X Result=> X
    Variables.GetIntegerPart Number: Y Result=> Y
    # Do whatever you like with Word, X and Y variables.
END
# Tested with Power Automate Desktop version 2.11.51.21196

How it works

In this template, a web page screenshot is used as the OCR image, but you could use any web image or a local image file.

Firefox is used to navigate to a website. Entire web page will be captured and saved to a temporary local file (.jpg). The image is sent to Google Vision API for OCR analysis and a JSON is returned. The JSON response contains text and x, y coordinates for all words and characters in the screenshot, for example:

        ...
        {
          "description": "PIZZA",
          "boundingPoly": {
            "vertices": [
              {
                "x": 673,
                "y": 395
              },
              {
                "x": 716,
                "y": 395
              },
              {
                "x": 716,
                "y": 421
              },
              {
                "x": 673,
                "y": 421
              }
            ]
          }
        },
        ...

How to enable Google Cloud Vision API

Follow these steps to activate Google Vision AI and to use OCR with Power Automate Desktop.

Register a Google Cloud Account trial.
Open Google Cloud Vision dashboard.
Choose API & Services from top-left navigation menu and then select Credentials from left navigation menu.
Select +Create Credentials > API Key from top menu.
As the last step, enable Google Cloud Vision API.

Instructions

Line 1: Replace https://unsplash.com/photos/jVcha8wHtg8 with any website URL.
Line 2: Replace C:\PowerAutomate\temporary.jpg with a temporary path and file name on your computer.
Line 4: Replace API_KEY with your Google Cloud Vision API key.
Line 11: Words and characters found by OCR are now stored into Word, X and Y variables.

Notes

There is a free tier, where you are allowed 1,000 units per Feature Request per month, see: Cloud Vision pricing
Chrome and Edge web browsers seems to have issues with Take screenshot of web page action. The action may fail with following error message if the web page height is larger than the browser viewport:

Failed to capture image (error in communication with browser)

Google Vision AI includes two annotation features that support optical character recognition (OCR): TEXT_DETECTION and DOCUMENT_TEXT_DETECTION. The built in Google Vision action in Power Automate Desktop uses only TEXT_DETECTION.
Do not expose your API key to anyone. Set API key restrictions to prevent unauthorized use and quota theft.

Changes

2022-01-01: Updated flow actions to work with Power Automate Desktop version 2.15.284.21354

Author

PAG Admin

Also See These Flows

Flow Action

Using Bootstrap 5 Components With JavaScript

Code snippets to help building web page automation with JavaScript.

PAG Admin

365

Flow Action

Visit Web Pages Found With Duckduckgo

Run DuckDuckGo in Chrome and visit first three links. Create a variable for CSS selectors to be able to click different links using an incremental loop counter.

PAG Admin

263

Flow Action

Create Animated Doughnut in Notepad

This example shows how to read JSON data, loop items and print output to screen (Notepad in this case).

PAG Admin

345

Flow Action

PowerToys for Power Automate for Desktop

These PowerToy shortcuts are made for Windows users for the purpose of make their computing lives better.

PAG Admin

920

Flow Action

Read Google Sheets API With Power Automate for Desktop

This example uses Invoke Web Service flow action to send a request to Google Sheets API, the response contains spreadsheet values.

PAG Admin

1,578