Basic Web Scraper Built With Power Automate for Desktop

A functional web scraper with complete flow actions. Showing how to scrape websites, traverse links and download content.

Display.InputDialog Title: $'''Web scrape url''' Message: $'''Insert URL''' InputType: Display.InputType.SingleLine IsTopMost: True UserInput=> WebsiteUrl
SET TargetFilePath TO $'''c:\\PowerAutomate\\WebScraper'''
SET DownloadFileRule TO $'''\\.(ico|js|css|svg|png|jpg|jpeg|gif|webm|json|ppt)(|\\?.*)$'''
SET UserAgent TO $'''Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'''
SET MaxPageRequests TO 30
SET MaxFileRequests TO 50
# End of user variables
Text.ParseText.RegexParseForFirstOccurrence Text: WebsiteUrl TextToFind: $'''(http|https):\\/\\/([\\w_-]+(?:(?:\\.[\\w_-]+)+))''' StartingPosition: 0 IgnoreCase: False Match=> HostNameMatch
SET PageCounter TO 0
SET FileCounter TO 0
Variables.AddItemToList Item: WebsiteUrl List: LinkList NewList=> LinkList
LOOP WHILE (LinkList.Count > 0) = ($'''True''')
    IF IsNotEmpty(LinkList[0]) THEN
        IF StartsWith(LinkList[0], $'''/''', False) THEN
            SET LinkList[0] TO $'''%HostNameMatch%%LinkList[0]%'''
        END
        IF StartsWith(LinkList[0], HostNameMatch, True) THEN
            IF NotContains(FoundUrls, LinkList[0], True) THEN
                Text.ParseText.RegexParseForFirstOccurrence Text: LinkList[0] TextToFind: $'''\\.(js|css|rle|rl4|rl8|sys|giff|jif|jfi|thm|jng|mng|fax|g3f|g3n|tim|g3|xif|tga|targa|tar|afi|bpx|psd|pdd|pcd|wmf|psp|pcc|pcx|scr|dcx|jp2|j2k|jpc|jpf|jpx|jpm|j2c|mj2|pic|cel|icb|vda|win|tpic|tpi|cut|rgb|rgba|sgi|bw|int|inta|iri|iris|rla|rpf|ppm|pbm|pgm|pxm|rppm|rpgm|rpbm|rpxm|rpnm|rpp|rpg|rpb|rpx|rpn|ico|cur|ani|eps|411|wbmp|wbm|wap|crw|cr2|nef|raw|pef|raf|x3f|bay|orf|srf|mrw|dng|dcr|dicom|dcm|dic|jbg|jbig|jbi|bie|dot|dpx|fits|fit|fpx|hdf|mat|miff|mif|mtv|palm|pal|pcl|pict|pix|pwp|sun|ttf|vicar|vic|viff|vif|xbm|xcf|arw|bmq|cs1|dc2|erf|fff|hdr|ia|k25|kc2|kdc|mdc|mos|pxn|rdc|sr2|hdp|wdp|sti|xpm|mpeg|mpe|mpv|mpg|mp2|wmv|tiff|emf|mp3|xls|pdf|pub|xlt|wq1|wks|wk3|wk4|dotx|xlsx|xltx|xlsb|ppsx|docm|dotm|xlsm|xltm|pptm|ppam|ppsm|ai|lwp|ans|htm|wk5|wki|wku|wpd|psw|pwd|pab|olk|mcw|xlw|log|dxf|svg|svgz|vdw|vdx|vss|vst|vsx|vtx|slk|vsdm|vsdx|vssm|avi|qt|dv|aac|aif|aiff|mov|mp4|m4p|3gz|3gp|3g2|m2v|wma|m4v|ogg|wav|flv|webm|mpeg-1|asf|nsv|seq|mxf|gxf|avs|wve|uv2|fli|flc|cin|wc3|mpeg4|vob|mpeg-4|dvb|csv|sxc|ods|tsv|doc|docx|odt|sxw|pps|odp|sxi|pptx|jpg|jpeg|jpe|jfif|gif|png|bmp|dib|snp|ppt|tif|vsd|txt|rtf|wri|mpp|mdi|wps|potx|dwf|dwfx|swf)(|\\?.*)$''' StartingPosition: 0 IgnoreCase: False Match=> IsFileExtension
                IF IsNotEmpty(IsFileExtension) THEN
                    IF (FileCounter < MaxFileRequests) = $'''True''' THEN
                        Text.ParseText.RegexParseForFirstOccurrence Text: LinkList[0] TextToFind: DownloadFileRule StartingPosition: 0 IgnoreCase: True Match=> AllowFileDownload
                        IF IsNotEmpty(AllowFileDownload) THEN
                            Text.AppendLine Text: FoundUrls LineToAppend: LinkList[0] Result=> FoundUrls
                            IF (Folder.IfFolderExists.DoesNotExist Path: $'''%TargetFilePath%\\%AllowFileDownload%''') THEN
                                Folder.Create FolderPath: TargetFilePath FolderName: $'''files'''
                            END
                            Web.DownloadFromWeb.DownloadToFolder Url: LinkList[0] DestinationFolder: $'''%TargetFilePath%\\files''' ConnectionTimeout: 30 FollowRedirection: True ClearCookies: False UserAgent: UserAgent Encoding: Web.Encoding.AutoDetect AcceptUntrustedCertificates: False
                            Variables.IncreaseVariable Value: FileCounter IncrementValue: 1 IncreasedValue=> FileCounter
                            WAIT 1
                        END
                    END
                ELSE
                    IF (PageCounter < MaxPageRequests) = $'''True''' THEN
                        Variables.IncreaseVariable Value: PageCounter IncrementValue: 1 IncreasedValue=> PageCounter
                        Text.AppendLine Text: FoundUrls LineToAppend: LinkList[0] Result=> FoundUrls
                        IF (Folder.IfFolderExists.DoesNotExist Path: $'''%TargetFilePath%\\pages''') THEN
                            Folder.Create FolderPath: TargetFilePath FolderName: $'''pages'''
                        END
                        Web.DownloadFromWeb.DownloadToFolder Url: LinkList[0] DestinationFolder: $'''%TargetFilePath%\\pages''' ConnectionTimeout: 30 FollowRedirection: True ClearCookies: False UserAgent: UserAgent Encoding: Web.Encoding.AutoDetect AcceptUntrustedCertificates: False DownloadedFile=> WebPageFilePath
                        File.ReadTextFromFile.ReadText File: WebPageFilePath Encoding: File.TextFileEncoding.UTF8 Content=> WebPageHtml
                        # Find all urls with http(s)
                        Text.ParseText.RegexParse Text: WebPageHtml TextToFind: $'''(http|https):\\/\\/([\\w_-]+(?:(?:\\.[\\w_-]+)+))([\\w.,@?^=%%&:\\/~+-]*[\\w@?^=%%&~+#-])''' StartingPosition: 0 IgnoreCase: True Matches=> LinkMatches
                        Variables.MergeLists FirstList: LinkList SecondList: LinkMatches OutputList=> LinkList
                        Variables.RemoveDuplicateItemsFromList List: LinkList IgnoreCase: True NewList=> LinkList
                        WAIT 1
                    END
                END
            END
        END
    END
    Variables.RemoveItemFromList.RemoveItemFromListByIndex ItemIndex: 0 List: LinkList NewList=> LinkList
END
# Tested with Power Automate Desktop version 2.15.284.21354

How It Works

This is a non-interactive web scraper, meaning that it does not use browser automation (Chrome, Edge, Firefox) for scraping. Instead, all web page requests are sent with the Download from web flow action.

Instructions

Line 1: Insert the url to be scraped.
Line 2: Set a folder where files should be downloaded to.
Line 3: (Optional) Change file extensions that should be downloaded.
Line 4: (Optional) Change the custom user agent header that is sent to websites.
Line 5: (Optional) Change MaxPageRequests (default is 30)
Line 6: (Optional) Change MaxFileRequests (default is 50)

Limits

The scaper has following limits:

Scraping only static html content (so no JavaScript content).
The target website must use absolute urls (could be improved to understand relative urls).

Notes

This scraper does not obey robots.txt. If you plan to use this on a website you don’t have ownership to, then you should comply with their robots.txt. Do not scrape Disallowed Pages and obey Crawl-Delay.
Do not perform malicious activities.
Wait 1 flow action have been added after each webpage request (the flow will wait for one second after each request). Many websites do run on shared hosts with limited resources and they deserve our understanding. This is the minimum wait value that should be used.

Author

PAG Admin

Also See These Flows

Flow Action

Use Power Automate Desktop as a Web Scraper for Excel

Update existing Excel spreadsheet with information pulled from any webpage.

PAG Admin

688

Flow Action

Using Bootstrap 5 Components With JavaScript

Code snippets to help building web page automation with JavaScript.

PAG Admin

281

Flow Action

PowerToys for Power Automate for Desktop

These PowerToy shortcuts are made for Windows users for the purpose of make their computing lives better.

PAG Admin

646

Flow Action

Basic Web Scraper Built With Power Automate for Desktop

A functional web scraper with complete flow actions. Showing how to scrape websites, traverse links and download content.

PAG Admin

2,762

Flow Action

Write to Google Sheets API Using OAuth 2.0 Power Automate for Desktop

Write cell values to spreadsheets. Includes basic logic for handling refresh tokens for repetitive tasks.

PAG Admin

1,980