Privoxy 4.1.0

- Bug fixes:
- Add missing client-body-tagger data to the action_type_info[] struct
so lookups based on the action index work correctly again.
Prevents assertion failures or segfaults when trying to edit
an action file with the CGI editor.
The type of failure depended on whether or not assertions
were enabled and on whether or not Privoxy had been compiled
with FEATURE_EXTERNAL_FILTERS.
Regression introduced in Privoxy 3.0.34.
Patch submitted by Aaron Li in #940.
- Bump MAX_FILTER_TYPES which should have been done in d128e6aa4
when introducing the client-body-tagger{} action.
Prevents an assertion in cgi_edit_actions_for_url() from triggering
after e32d03e0 when using the CGI editor with assertions enabled.
- is_untrusted_url(): Search the encrypted headers for the Referer
when the client is using https and https inspection is enabled.
Fixes the trust mechanism for https requests.
Reported by Laurent Caumont in #1767.
- GNUMakefile.in: Let the install target work if no group is specified.
- GNUMakefile.in: Set GROUP_T when installing configuration files as root
and there is no privoxy user available so the install target doesn't
fail. Patch by Fabrice Fontaine.
- GNUmakefile.in: Don't exit if configuration files are installed as root
as this can be considered acceptable when cross-compiling
Privoxy inside an autobuilder with only a root user.
Patch by Fabrice Fontaine.
- configure.in: Fix argument types in gmtime_r() and localtime_r() probes.
Otherwise these probes always fail with stricter compilers
even if there is C library support for these functions.
Patch submitted by Florian Weimer in SF#149.
- Fix socks4 and socks4a support under glibc's source fortification.
With glibc's source fortification, gcc offers the compilation warning
resulting in a runtime abort() when using a socks4 or socks4a upstream proxy.
Despite the warning, the strlcpy() call in question is fine: gcc
misidentifies the size of the destination buffer, estimating to hold
only a single char while in fact the buffer stretches beyond the end of
the struct socks_op.
The issue was originally reported in the NixOS issue tracker at
https://github.com/NixOS/nixpkgs/issues/265654
prompted by an upgrade of glibc from 2.37-39 to 2.38-0.
Patch submitted by Ingo Blechschmid, joint work with
@esclear and @richi235.

- General improvements:
- Allow to use wolfSSL for https inspection.
wolfSSL supports TLS 1.3 and can be significantly faster than
mbedTLS. Mainly tested on ElectroBSD amd64 where it can compete
with OpenSSL and LibreSSL:
https://www.fabiankeil.de/gehacktes/privoxy-tls-benchmarks/
To enable the support, install wolfSSL and run ./configure
with the --with-wolfssl option.
Sponsored by Privoxy project funds collected at SPI.
- Add an test framework that leverages the curl test suite.
Sponsored by Privoxy project funds collected at SPI.
- Add pcre2 support. Closes bug #935.
Initial patch submitted by Gagan Sidhu.
- Use SHA256 as hash algorithm for the certificate and key file names
instead of MD5. The known MD5 vulnerabilities shouldn't matter for
Privoxy's use case but it doesn't hurt to use a hash algorithm that
isn't deprecated.
Sponsored by: Robert Klemme
- Add support for mbedTLS 3.x. This removes a sanity check
(whether issuer key and issuer certificate match) that seems
overly cautious and fails to compile with mbedTLS 3.x as the
struct members are private. We don't have an equivalent check
in the OpenSSL or wolfSSL code either.
- Factor out newer_privoxy_version_required() and improve the logic
Previously 3.0.11 was considered newer than 4.0.0.
- init_error_log(): Include the reason for failures to open the log file.
- create_client_ssl_connection(): Don't keep the certificate lock
longer than necessary.
- Add periods to a bunch of log messages.
- normalize_lws(): Only log the 'Reducing whitespace ...' message
once per header
- log_error() Win32: Only call LogShowActivity() for debug level
LOG_LEVEL_REQUEST. As of b94bbe62a950, which was part of Privoxy 3.0.29,
LOG_LEVEL_REQUEST is used for all requests including crunched ones.
Previously LogShowActivity() was called twice for crunched
requests, (presumably) resulting in an aborted animation.
- Remove ./ prefix from tarball-dist files.
- create_client_ssl_connection(): Make it more obvious from an
error message that a function failed.
- Use stringify() instead of section_target() and remove section_target().
Like the XXX comment suggested this could be done my moving the hash
into the templates which seems preferable anyway.
- Prevent some compiler warnings.
- parse_numeric_value(): Expect a base-ten number.
- windows/MYconfigure: Have gcc diagnostics in color.

- Action file improvements:
- Block requests to .amazon-adsystem.com/
- Block requests to 0.css-load.com/
- Block requests to html-load.com/ and 1.html-load.com/
- Block requests to b.6sc.co/
- Block requests to i.clean.gg/
- Block requests to s.cpx.to/
- Block requests to track.venatusmedia.com/
- Block requests to secure-eu.nmrodam.com/
- Block requests to o2.mouseflow.com/
- Disable fast-redirects for services.akteneinsichtsportal.de/
- Disable fast-redirects for /wp-content/plugins/pdf-viewer-for-elementor
- Disable fast-redirects for syndication.twitter.com/
- Disable fast-redirects for archive.softwareheritage.org/
- Disable fast-redirects to duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
- Disable fast-redirects for .creator-spring.com/_next/image
- Disable fast redirects for accounts.bahn.de/
- Unblock .datenschmutz.de/
- Unblock requests for 'adventur*.'
- Unblock adl.windows.com/
as it is apparently required to update from Windows 10 to 11.
Reported by Sam Varshavchik.

- Privoxy-Log-Parser:
- Highlight 'Couldn't deliver the error message for [...]'.
- Highlight 'Failed to accept() incoming connection: Software caused connection abort'.
- Highlight 'Keeping chunk offset at 0 despite flushing 31 bytes.'.
- Highlight 'Not shutting down client connection on socket 8. The socket is no longer alive.'.
- Bump version to 0.9.6.

- Privoxy-Regression-Test.pl:
- Let the --min-level option increase the --max-level
if the latter is smaller than the former.
- Add --curl option to use a non-default curl binary.
- Bump version to 0.7.5.

- uagen:
- Bump BROWSER_VERSION and BROWSER_REVISION to match Firefox ESR 128.
- Bump version to 1.2.6.

- Documentation:
- Add HOWTOs for https inspection and client-tags to user-manual.
- Suggest to use the force-text-mode action when filtering binary content
with external filters.
- Declare https-inspection non-experimental.
- FAQ: Mention that Privoxy Moral Licenses are available as well.
- Fix LibreSSL URL.
- Update perlre perldoc URL.
- config: Add SOCKS 5 to the list of supported protocols.
- In the Windows build section, note that one only needs tidy
to build the docs. If you're not building the docbook stuff you
don't need tidy.
- trust: Use the words 'allowlists' and 'blocklists'
instead of "whitelists" and "blacklists" which some
people consider to be less inclusive.

- Security/Reliability:
- cgi_error_no_template(): Encode the template name to prevent
XSS (cross-site scripting) when Privoxy is configured to servce
the user-manual itself.
Commit 0e668e9409c. OVE-20211102-0001. CVE-2021-44543.
Reported by: Artem Ivanov
- get_url_spec_param(): Free memory of compiled pattern spec
before bailing.
Reported by Joshua Rogers (Opera) who also provided the fix.
Commit 652b4b7cb0. OVE-20211201-0003. CVE-2021-44540.
- process_encrypted_request_headers(): Free header memory when
failing to get the request destination.
Reported by Joshua Rogers (Opera) who also provided the fix.
Commit 0509c58045. OVE-20211201-0002. CVE-2021-44541.
- send_http_request(): Prevent memory leaks when handling errors
Reported by Joshua Rogers (Opera) who also provided the fix.
Commit c48d1d6d08. OVE-20211201-0001. CVE-2021-44542.

- Security/Reliability:
- Prevent an assertion from getting triggered by a crafted CGI request.
Commit 5bba5b89193fa. OVE-20210130-0001.
Reported by: Joshua Rogers (Opera)
- Fixed a memory leak when decompression fails "unexpectedly".
Commit f431d61740cc0. OVE-20210128-0001.

- Bug fixes:
- Fixed detection of insufficient data for decompression.
Previously Privoxy could try to decompress a partly
uninitialized buffer.

Bug fixes:
Fixed a memory leak when rejecting client connections due to the socket limit being reached (CID 66382). This affected Privoxy 3.0.21 when compiled with IPv6 support on most platforms this is the default).
Fixed an immediate-use-after-free bug (CID 66394) and two additional unconfirmed use-after-free complaints made by Coverity scan (CID 66391, CID 66376).
Actually show the FORCE_PREFIX value on the show-status page.
Properly deal with Keep-Alive headers with timeout= parameters If the timeout still can't be parsed, use the configured timeout instead of preventing the client from keeping the connection alive.
Fixes #3615312/#870 reported by Bernard Guillot.
Not using any filter files no longer results in warning messages unless an action file is referencing header taggers or filters.
Fixed a bug that prevented Privoxy from reusing some reusable connections. Two bit masks with different purpose unintentionally shared the same bit.
A couple of additional bugs were discovered by Coverity Scan. The fixes that are not expected to affect users are not explicitly mentioned here, for details please have a look at the CVS logs.
General improvements:
Introduced negative tag patterns NO-REQUEST-TAG and NO-RESPONSE-TAG. They apply if no matching tag is found after parsing client or server headers.
Add support for external filters which allow to process the response body with a script or program written in any language the platform supports. External filters are enabled with +external-filter{} after they have been defined in one of the filter files with a header line starting with "EXTERNAL-FILTER:".
External filter support is experimental, not compiled by default and known not to work on all platforms.
Add support for the 'PATCH' method as defined in RFC5789.
Reject requests with unsupported Expect header values. Fixes a couple of Co-Advisor tests.
Normalize the HTTP-version in forwarded requests and responses. This is an explicit RFC 2616 MUST and RFC 7230 mandates that intermediaries send their own HTTP-version in forwarded messages.
Server 'Keep-Alive' headers are no longer forwarded. From a user's point of view it doesn't really matter, but RFC 2616 (obsolete) mandates that the header is removed and this fixes a Co-Advisor complaint.
Change declared template file encoding to UTF-8. The templates already used a subset of UTF-8 anyway and changing the declaration allows to properly display UTF-8 characters used in the action files. This change may require existing action files with ISO-8859-1 characters that aren't valid UTF-8 to be converted to UTF-8.
Do not pass rejected keep-alive timeouts to the server. It might not have caused any problems (we know of), but doing the right thing shouldn't hurt either.
Let log_error() use its own buffer size #define to make changing the log buffer size slightly less inconvenient.
Turned single-threaded into a "proper" toggle directive with arguments. CGI templates no longer enforce new windows for some links.
Remove an undocumented workaround ('HOST' header removal) for an Apple iTunes bug that according to #729900 got fixed in 2003.
Action file improvements:
The pattern 'promotions.' is no longer being blocked.
Disable fast-redirects for .microsofttranslator.com/.
Disable filter{banners-by-size} for .dgb-tagungszentren.de/.
Add adn.speedtest.net as a site-specific unblocker. Support request #3612908.
Disable filter{banners-by-size} for creativecommons.org/.
Block requests to data.gosquared.com/. Reported by cbug in #3613653.
Unblock .conrad./newsletter/. Reported by David Bo in #3614238.
Unblock .bundestag.de/.
Unblock .rote-hilfe.de/.
Disable fast-redirects for .facebook.com/plugins/like.php.
Unblock Stackexchange popup URLs that aren't used to serve ads.
Disable fast-redirects for creativecommons.org/.
Unblock .stopwatchingus.info/.
Block requests for .adcash.com/script/.
Disable HTML filters if the response was tagged as javascript. Filtering javascript code with filters intended to deal with HTML is usually a waste of time and, more importantly, may break stuff.
Use a custom redirect{} for .washingtonpost.com/wp-apps/imrs\.php\?src=
Previously enabling the 'Advanced' settings (or manually enabling +fast redirects{}) prevented some images from being loaded properly.
Unblock "adina*." Fixes #919 reported by Morton A. Goldberg. Block '/.*DigiAd'.
Unblock 'adele*.'. Reported by Adele Lime in #1663.
Disable banners-by-size for kggp.de/.
Filter file improvements & bug fixes:
Decrease the chances that js-annoyances creates invalid javascript.
Let the msn filter hide 'related' ads again.
Remove a stray '1' in the 'html-annoyances' filter.
Prevent img-reorder from messing up img tags with empty src attributes.
Documentation improvements:
Updated the 'Would you like to donate?' section. Note that invalid forward-override{} parameter syntax isn't detected until the parameter is used.
Add another +redirect{} example: a shortcut for illumos bugs.
Make it more obvious that many operating systems support log rotation out of the box.
Fixed dead links.
Rephrased the 'Why is the configuration so complicated?' answer to be slightly less condescending.
Be more explicit about accept-intercepted-requests's lack of MITM support.
Make 'demoronizer' FAQ entries more generic.
Add an example hostname to the --pre-chroot-nslookup description.
Add an example for a host pattern that matches an IP address.
Rename the 'domain pattern' to 'host pattern' as it may contain IP addresses as well.
Recommend forward-socks5t when using Tor. It seems to work fine and modifying the Tor configuration to profit from it hasn't been necessary for a while now.
Add another redirect{} example to stress that redirect loops can and should be avoided.
The usual spelling and grammar fixes.
Mention the PCRS option letters T and D in the filter section.
Clarify that handle-as-empty-doc-returns-ok is still useful and will not be removed without replacement.
Note that security issues shouldn't be reported using the bug tracker.
Clarify what Privoxy does if both +block{} and +redirect{} apply.
Removed the obsolete bookmarklets section.