The default Helix configuration for Python is different,
so the pylint warnings aren't shown and the formatter
re-formats files by accident when you edit an existing file.
Therefore, this commit adds `python` language configuration
to ease developing SearXNG with Helix Editor [^1].
[^1]: https://helix-editor.com
html.duckduckgo.com captchas all my IPs very fast. I figured out that using
duckduckgo.com works even if html.duckduckgo.com is captcha-ed, hence adding
support for duckduckgo.com's general web search here.
This implementation fetches the link to the first API page
(i.e. ``links.duckduckgo.com/d.js?...``) from duckduckgo.com and uses the ``n``
parameter of the API to fetch all subsequent pages.
This also means that it's not possible to immediately search for the third
page - the first and the second page would need to be loaded first.
The reason why we can't just normally use the `vqd` value is that the API URLs
require an additional parameter `dp` which seems generated at server-side, so we
can't build it ourselves and must scrape it from the HTML pages.
Add support for https://tiger.ch (general, news)
It is disabled and inactive by default because it's just a metasearch engine
like SearXNG is, so it's mostly useful for bypassing rate-limits on other
engines: (it has its own German index, but it's not that great) in theory it
supports different locales, but I was too lazy to implement that (I only need
German and English results anyways, which are returned by default...)
Add support for https://seek.ninja (general)
It's very slow because the engine uses Server-side events, that incrementally
send data in their HTTP response [1].
I.e. we wait for the end of the response (7+ seconds), even though the results
data arrives within a few seconds -> it's very slow, because SearXNG wants to
get the full response body before it calls the `response(resp)` method
We could use httpx-sse [2], but I'm not sure how to integrate this into SearXNG
and if it's worth it
[1] https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/
[2] https://github.com/florimondmanca/httpx-sse
The engine is using very aggressive Cloudflare blocking for
a while now, no matter if using a normal browser like Firefox
or not.
Closes: https://github.com/searxng/searxng/issues/5976
PodcastIndex.org started using a Proof-of-Work JavaScript
challenge whose results are sent as `X-Pow-*` request headers.
Although it is technically possible to re-implement the
PoW challenge in Python, it's likely impossible to maintain
because
- the actual Proof of Concept logic might change very often
- the whole idea of the Proof of Work challenge is to use
a "big" amount of resources (about 1s on my PC); so executing the challenge
would almost block all other work on the SearXNG instance
At first glance, the challenge looks very similar to what
Anubis does, because it also uses SHA256 hashes.
- https://lite.qwant.com seems to be dead.
- The request parameters were changed to match the ones from the Qwant website.
- Qwant is now set to inactive by default due to its strict rate-limits
Changes:
- Setting the "abp" query parameter causes instant blocks, it's no longer
used at Startpage
- The safesearch map changed for both the request form and the cookies. As
we were sending invalid values, that also made it easier to detect us
I've been profiling the `/preferences` endpoint using werkzeug's
`ProfilerMiddleware` (i.e. just do `app.wsgi_app = ProfilerMiddleware(app.wsgi_app)`)
and look at the outputs in the terminal when doing `make run`.
It turns out that 95%+ of the time spent were inside babel's
Locale parsing (> 700ms on my machine). That's because, when opening the settings,
we loaded the full engine traits of each engine and checked if it matches
the user-defined search language. As we have 250+ engines, and babel is
very slow when parsing Locale's, this took a very long time.
By removing this feature that shows whether the selected search language
is supported by the engine, the load time went down from 800ms to 50ms
on my machine (which is still very slow, but well, that's future work on
optimizing).
- add button for editing docs source on GitHub
- add links to GitHub repository and public instances (searx.space)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This PR moves the `iframe` logic into a macro, so that `videos.html` and `general.html` both can benefit from the workaround to fix YouTube results by @return42 in https://github.com/searxng/searxng/pull/5858
It also fixes that only YouTube videos contained the closing `>` after `<iframe border="0" ...`, the regression has been caused by https://github.com/searxng/searxng/pull/5858
## Why is this change important?
Currently, the page breaks if there's any non-YouTube Iframe
Here, the page ends in the middle of the results and the footer and page number selector are not visible.
Notes:
- Safesearch doesn't seem to work properly?
- In theory multiple languages are supported, but even in the web UI, they don't work properly
- Possibly, we could cache the request hashes (h query parameter), I'm not sure if it ever changes
As a side effect, Cloudscraper is no longer needed.
It probably only ever worked by setting the correct request headers,
so we don't really need it since we can just set the right request
headers and ciphersuites ourselves.
Unsplash started using [Anubis](https://anubis.techaro.lol/)
for blocking crawlers. Therefore, requests using common
user agents (e.g. Firefox, Chrome) must pass a JavaScript
challenge.
However, other user agents seem unaffected for now, hence
settings the UA to something different does still work.
Since about a month, the website just says "temporarily unavailable", so it's safe to assume that it's just no longer working
Related:
- https://github.com/searxng/searxng/pull/3798
Apparently the engine statistics have been hidden from the images page when
infinite scroll was added as a preference in https://github.com/searxng/searxng/commit/56e34947a6368e6154064c52fa23d21ecda7ab4c
However, now that we also display engine statistics in the results page,
I don't see any reason why this should still be hidden from the images tab.
For example, this is very useful for seeing which engines work and which ones are broken.
Apparently, yep has been broken for a while. Measures to fix it:
- only use HTTP/1.1, because our HTTP2 client gets fingerprinted and blocked
- send the `Origin` HTTP header
there are other classes like 'site-name-content' we don't want to match,
however only using contains(@class, 'content') would e.g. also match `site-name-content`
thus, we explicitly also require the spaces as class separator
Presearch responds with a Cloudflare captcha on each request when using HTTP2.
Using HTTP1.1, everything seems to work fine.
- other engines with the same issue: pixabay, uxwing
- closes https://github.com/searxng/searxng/issues/5438
- official website: https://devicon.dev/
- the engine contains a lot of icons of popular software frameworks (e.g. pytest),
so they could for example be useful for visualizing a diagram of the tech stack used in an app
SourceHut uses a foss bot protection tool called `go-away` (which I can
recommend BTW). It blocks common crawler user agents, such as the standard
Firefox user agent. Hence, we're now using our custom SearXNG user agent to
clarify we're not a crawler.
Closes: https://github.com/searxng/searxng/issues/5270
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Apparently, setting padding on an input field and then clicking that
space created by the padding forces the seekbar cursor to jump to the
beginning of the input field instead of the actual text position.
By removing that padding, the search bar input automatically claims that
height for itself and thus clicking on the blank space moves the cursor to
the correct position.
closes https://github.com/searxng/searxng/issues/5371
Apparently, in China, Bing redirects from `www.bing.com` to `cn.bing.com`.
So in order to make Bing work for chinese users by default, we have to follow that redirect.
related: https://github.com/searxng/searxng/issues/5243
Previously, when using a search url copied from the cookies tab, clicking
at the settings icon at the top right would show the browser preferences
and not the preferences that were set and used with the search url.
Please see https://github.com/searxng/searxng/issues/5227 for more information.
To test:
- change some preferences
- copy the preferences search url in the settings' cookies tab
- reset the preferences or clear cookies
- paste the copied search url into the search bar to search for something
- press the settings icon
- you can now see/preview the actual settings that were used for the search
- by pressing 'save', you can keep these preferences
closes#5227
Repology_ monitors a huge number of package repositories and other sources
comparing packages versions across them and gathering other information.
Repology_ shows you in which repositories a given project is packaged, which
version is the latest and which needs updating, who maintains the package, and
other related information.
.. _Repology: https://repology.org/docs/about
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Format: reST
Apparently the layout of https://annas-archive.org has changed, making changes necessary.
The issue has been reported in #5146, see there for more details.
- closes#5146
The plugin uses the ``GeoLocation`` class, which is already implemented in the
context of weather forecasts, to determine the time zone. The ``DateTime`` class
is used for the localized display of date and time.
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
- the previous CDN icon list url no longer works
- a list of all icons is mirrored to the JSDelivr CDN however
- there's no reason to set the engine to inactive now that we use public CDNs
Sometimes, there's only an `adaptivestreaming` field in `streams`, which
is usually an m3u8 file. That's however not supported by the video player
of any browser, so we can't use and must build a different url instead.
Bnyro
