Cyotek WebCopy 1.9.1 Build 872 + x64

Added:
Added the ability to read cookies from an external file (#461) (User Manual)
Added the ability to read cookies from an external file (#462) (User Manual)
Test URL dialogue now allows configuring cookies
Added cookie, cookie-jar and discard-session-cookies command line parameters (User Manual)
Added support for the legacy compress (#472) and non-standard BZip2 (#473) content encodings (User Manual)
Changed:
Documentation improvements (#400, #443, #461, #462, #482, untracked)
Test URL dialogue now uses load on demand for settings pages
401 challenges no longer display credential dialogues unless the authentication type is either Basic or Digest as no other values have been tested due to lack of resource
Updated mime database
Fixed:
Posting a form did not set an appropriate content type (#437)
Custom headers were not applied when posting forms (#436)
If a URL was previously skipped but then included in future scans, the original skip reason could be retained
A blank error message was displayed for Brotli decompression errors (#447)
One-time project validation checks were ignoring the content encoding settings of the project (which by default is Gzip and Deflate) and were requesting content with Brotli compression (#446)
Brotli decompression could fail with streams larger than 65535 bytes (#448)
The URI transformation service incorrectly attempted to add prefixes to email addresses, this in turn caused a crash if the mailto: reference was malformed (#450)
A crash could occur if a content type header was malformed and was either utf or utf- (#450)
Fixed an issue where command line arguments sometimes didn't correctly process ambiguous relative arguments that could be a file name or a unqualified URI (#441). As a result of this fix, all URIs provided to command lines must be fully qualified, e.g. https://example.com over example.com
Fixed a crash that could occur when switching between empty virtual list views during a crawl and items were then subsequently added (#460)
A crash which could occur when loading localised text is no longer fatal (#465). Note that we haven't been able to reproduce this, so if you previously received a crash after setting a language other than English, please email support@cyotek.com
Speed and estimated downtime time calculations were incorrect and could cause a crash when downloading large files (#466)
A crash would occur when editing a file that didn't have a mime type (#468)
Speculative fix for a crash that could occur when finishing the New Project Wizard (#467)
Fixed a crash that occurred if a 401 challenge was received and the www-authenticate header was a bare type (#469)
If a website returns a non-standard Content-Encoding value (or one currently not supported by WebCopy), no attempt will be made to decompress the file and it will be downloaded as-is. A new setting has been added to disable this behaviour, but is currently not exposed (#471)
Crashes that occurred when applying project validation corrections (for example if the base URL redirects, WebCopy will prompt to use the redirect version) were fatal (#474)
Trying to save a CSV export with a relative filename crashed (#475)
The quick scan diagram view could crash if invalid host names were detected (#476). This is another bug reported without context, if any user has previously experienced this please email support@cyotek.com
The "Limit distance from base URL" setting now only applies to URLs that have a content type of text/html, e.g. it will prevent deep scanning whilst still allowing retrieval of all linked resources (#464)
URLs that had exclusion rules would still get requested depending on the combination of project settings (#481)
The CLI would crash if the recursive and output parameters were defined, and the specified output directory did not exist (#483)
Client is no longer marked as dpi-aware, which should resolve pretty much all the problems with the application not displaying correctly on high DPI screens. This is an interim fix until dpi-awareness can be properly introduced.
Fixed a crash that could occur when trying to query if the scan above root setting should be abled and an invalid URI was project (#493)
Fixed a crash that could occur when the scan/download progress dialog was closed (#454)
The Export CSV dialog wasn't localised correctly, resulting in seemingly two Cancel buttons (#495)
Removed:
The PDF meta data provider has been removed

Added:
Separate 32bit and 64bit setup files are now available
Fixed:
WebCopy would not copy sites using an IP address rather than a DNS name
URL validation checks were running on the base domain and ignoring any deep linking
WebCopy wouldn't always strip empty path segments from URLs
PDF and RSS will no longer be downloaded when performing a site analysis
Improvements to several windows affected when running under custom DPI scaling modes
The minimum / maximum file size editor required values to be entered as bytes, despite the labelling requesting kibibytes [#387]
Sometimes WebCopy didn't shrink a path correctly to fit within file system limitations [#393]
Default documents should no longer be named index.htm.html
Changed:
Reorganised columns in results list view to hopefully reduce confusion when URLs are skipped
Non-crawlable URLs that are skipped during an analyse are now recorded in the results list

When crawling for the first time, if the user-entered URL redirects (for example http to https or to www), WebCopy will now prompt to use the final URL [#368]
Fixed:
Changing the URL of an existing project caused further crawling to instantly fail if the domain of the new URL did not match the old [#367] (regression in 1.8.1)
Added a speculative fix for a hang when crawling a website with the experimental "use web browser" option enabled
The GUI client now correctly allows unescaped URLs to be entered
When using the "Use query string in local file names" option, WebCopy now correctly sanitises the query string
Fixed a crash that occurred when trying to empty the destination folder using the 64bit version of WebCopy
Removed:
Aero glass effects used by Windows 7 have been removed [#366]

Added:
Added new nocrashreport switch to command line clients
Command line clients can now display solution information when reporting crashes [#201]
Fixed:
Partial output is no longer printed by CLI tools when using the quiet switch
Statistics are now printed when using the statistics switch even if quiet is also specified
All output is now correctly written to log files when the log switch is used, irrespective of the quiet switch setting
Pressing Enter or Escape in the Capture Form dialog no longer closes the dialog if the embedded web browser has focus

Added:
The Origin Report setting now has a new option to embed the original URL as a comment in the body of the HTML
Changed:
The Download All Resources option is now automatically set for new projects [0000193]
The Directory Character option is now automatically set to / for new projects
The Update Local Timestamps is now automatically set for new projects
Fixed:
A crash no longer occurs opening the Options dialog and the languages folder doesn't exist [0000187]
A crash no longer occurs opening the Options dialog and duplicate languages are present [0000186]
A crash could occur when loading the sitemap and shell icons were enabled [0000190]
Fixed a number of issues that could occur after opening or saving a project and the MRU was updated (0000161, 0000176, 0000177)
WebCopy no longer aborts the crawl after trying to download a URL with the same name as a file system reserved word [0000195]
WebCopy wasn't detecting flash movies in an object tag [0000173]
Data URI's with padded data weren't processed correctly [0000197]
Fixed trying to run the CLI and specifying a project file that did not exist [0000200]
Added speculative fix for a crash generating sitemaps - this is a common issue yet we've been completely unable to reproduce it in any of our test scripts or saved projects. If anyone can supply information on how to trigger this crash it would be gratefully received! [0000160]
Fixed a regression where it was possible for redirects to get stuck in an infinite loop
Minor improvements to URI exception reporting
RSS entries would duplicate themselves depending on if the feed was accessed via HTTP or HTTPS. Note that a side effect of this fix will result all entries being marked as unread
Fixed a possible crash that could occur when trying to load a themed font [0000203]
Fixed a crash that could occur if a rule had a empty pattern [0000188]
WebCopy will now try to remove base tags after completing a crawl [0000191]

Added:
Added a new extension which allows users to easily submit websites that WebCopy isn't copying correctly [0000015]
WebCopy now warns if javascript is detected as being in use by the website being copied [0000066]
WebCopy now reports links it detected but couldn't process [0000031]
Added open output folder action to Crawl Complete dialog
Changed:
Due to some curious feedback, the checks to validate digital signatures on WebCopy binaries have been reinstated
Sitemap ordering has been changed to a simple sort order, as the natural sort took an extremely long time to run on large websites, with little benefit for the performance hit [0000004, 0000150]
External status is now stored with a link entry instead of being calculated each time it is requested [0000004]
Fixed:
Fixed an issue where WebCopy wouldn't display content properly (for example in the Test URI dialog) if the web server returned compressed content regardless of the value of the request's Accept-Encoding header
Fixed an issue where pages weren't processed correctly (e.g. corrupt titles in the UI) if the page wasn't using UTF8, didn't specify a charset in the Content-Type header but did specify the correct content type via a http-equiv meta tag in the document HTML [0000144]
Fixed an issue where remapped files were always read or written as UTF8 regardless of the original encoding [0000144]
The Test URI dialog will now automatically try and add http:// if the user just types/pastes a schema-less value
Fixed an issue where CSV export could fail
Creating an automatic rule for an external URI now creates a valid rule [0000026]
The sitemap treeview shouldn't reload itself quite as often as it previously did [0000003, 0000004]
WebCopy now correctly processes URI's above the crawl root if the Download all resources option is set [0000154]

Added:
Added a new Keep Alive setting. Setting this to false can help prevent the "The server committed a protocol violation. Section=ResponseStatusLine" crawl failure [0000002]
Added a new Prefix Mode setting. This setting allows you to force URI's to either have or remove the www prefix, useful for avoiding duplicated files when copying a website which uses a mix of prefixed and non-prefixed URI's
Added the ability to replace sections of a URI when crawling documents
Added a new report to view non-HTTP links
(Experimental) Added new Extract Data URIs setting. Enabling this option will extract inlined images using the data: protocol into separate files.
Changed:
Setup should now automatically uninstall previous versions
Numerous changes to how plugins are discovered, loaded and configured. Due to no longer storing plugin details in the Windows Registry, this will cause any disabled plugins to be re-enabled
WebCopy will now correctly report non-HTTP links such as mailto: or ftp: as skipped rather than silently ignoring them
Internal engine changes [0000062]
Removed
The project scan and repair tool is no longer included in setup
Fixed
WebCopy could incorrectly exclude some URL's believing them to be mailto: links
Fixed several occurrences when a crash could occur when invalid path characters were present in URL segments
Some HTML tags appeared as "Unknown" in list views
URI's would be incorrectly combined if the relative URI was just query string and the source URI already had a query string
Download percentages were calculated incorrectly
Fixed a crash that could occur after copying a website if the Update local time stamps option was set
Report viewer didn't show external URI's
Fixed case-sensitivity issues in some built-in reports
Fixed a crash that could occur if non-NBT files were present in report folders with the rpt extension
Fixed a startup crash if the addins folder didn't exist [0000112]
Fixed a crash that could occur when trying to calculate the depth of a URI [0000116]
Fixed a crash that occurred if a project with a blank URI was opened and the user then attempted to browse to the blank URI [0000120]
Fixed an issue where Setup sometimes wouldn't replace files

ADDED:
The original Include Subdomains option has been replaced with a new set of more comprehensive options, allowing for copying of sibling domains, linked resources, or everything
Added a new Download all resources option. When set, WebCopy will download any non-HTML linked resource, regardless of source domain. If the file would normally be excluded, it will only be downloaded, not crawled
Although the UI editor for additional hosts stated regular expressions could be used, this was never implemented. Regular expressions can now be used with additional hosts
Added a new Sitemap plugin that will generate a simple HTML sitemap of all downloaded files
Added a new Cookie Viewer plugin that allows a global view of cookies created during a crawl
Added native support for the picture element
The Capture Form window now remembers it's size and position
Added an address bar to the Capture Form tool to allow access to hidden login URI's, or for any other type of manual navigation
Added a Scan button to the Capture Form tool allowing a manual scan for forms in the page if WebCopy failed to detect them initially
The maximum redirect chain length setting can now be configured from the Advanced options group
Added a new option to control if external redirects should also be followed
Added support for the brotli compression algorithm
Added a new option to control if the results list should automatically scroll to show the active item while performing a website scan
CHANGED:
Now requires Microsoft .NET 4.6
When following redirects after posting form data, all built in skip rules are ignored, so if a post to one site directs to another to complete the post, WebCopy will now always follow the redirect
The Create Desktop Icon option in setup is no longer checked by default
The Test URL dialog now uses the proxy settings of the currently open project
The Website Links dialog has been slightly redesigned to prevent a crash when working with projects containing many thousands of links
Information dialogs accessed from list views now display the selected context in an easier to read format than plain CSV
The default maximum redirect chain length has been increased from 5 to 25
HTTP Compression options have been removed from the Advanced options group into their own dedicated group
Options for processing redirects have been removed from the Advanced options group into their own dedicated group
Minor performance improvements
Minor optimizations to reduce memory load
REMOVED:
Removed the setup option for creating a Quick Launch icon
Removed support for opening legacy XML based WebCopy projects
Due to offline help always being outdated due to the general weakness of the product manuals, the offline help files are no longer included in the setup, and requesting help will always display the online version
FIXED:
External domains that would normally be crawled were being excluded if the Download all resources option was set (nightly regression)
Sibling domains were not being crawled correctly (nightly regression)
Problems loading or saving the user agent store should no longer be fatal
Some dialogs only supported local help requests and were unable to show help if the local help file was not available
Fixed a crash that occurred downloading a file if the Content-Encoding header of the HTTP response was set to identity
Fixed a memory leak in the sitemap component
A number of HTML 5 specific tags were listed as Unknown in crawl result list views
The UI now correctly reports if part of a crawl was aborted due to too many redirects
Crawling the same project multiple times in succession reused the cookies from the preview crawl
Trying to call wcopy.exe with just the file name of an existing WebCopy project always displayed a message about unsupported protocols and refused to continue
Fixed an unauthorized access crash that could occur when using the Capture Form tool (regression)
Fixed an issue where external links could appear in some lists even when filtering options were set to exclude them
Fixed a number of issues which prevented automatically logging into websites where the post URI was different to the get URI and value merging was required, or the post returned 302 and the new location must be read to complete the login
Fixed an issue where occasionally the Capture Form tool didn't refresh available forms after navigating to a page
Fixed an issue where refreshing a page in the Capture Form tool didn't
The CLI tool no longer incorrectly reports failures to download a single file as an application exception
The CLI tool now correctly outputs the reason why a given URI failed when performing recursive downloading
The Capture Form tool now correctly detects forms that are contained within frames
Export to CSV option featured when context clicking some list views didn't correctly escape the CSV
Fixed an issue where the entire application was terminated if CSV export failed
Fixed an issue where URI's that were both invalid and very long could crash WebCopy
Fixed a crash that could occur when attempting to remap a CSS file
Fixed an issue where URI segments containing spaces (or other encoded characters) weren't correctly decoded when generating local folder names