The code used here has always been "bad" because `about` shouldn't be used
as data source, but the engine probably broke when type checks / dataclasses
for the about parameter in engines has been added with
<https://github.com/searxng/searxng/pull/6258>.
Error log:
```
WARNING searx.engines.public domain im: ErrorContext('searx/engines/public_domain_image_archive.py', 143, '\'url\': _clean_url(f"{about[\'website\']}/images/{result[\'objectID\']}"),', 'TypeError', None, ("'EngineAbout' object is not subscriptable",)) False
ERROR searx.engines.public domain im: exception : 'EngineAbout' object is not subscriptable
Traceback (most recent call last):
File "/home/bnyro/Projects/searxng/searx/search/processors/online.py", line 253, in search
search_results = self._search_basic(query, params)
File "/home/bnyro/Projects/searxng/searx/search/processors/online.py", line 239, in _search_basic
return self.engine.response(response)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/bnyro/Projects/searxng/searx/engines/public_domain_image_archive.py", line 143, in response
'url': _clean_url(f"{about['website']}/images/{result['objectID']}"),
~~~~~^^^^^^^^^^^
TypeError: 'EngineAbout' object is not subscriptable
```
Qwant now requires a `datadome` cookie that it returns
in the first search response as a `Set-Cookie`.
This cookie has to be sent for all requests, otherwise they
will be blocked.
This means that now, the first search request is blocked (results in CAPTCHA),
and only the subsequent searches work (same happens on the Qwant website for me).
However, I don't think it's worth repeating the same
search request multiple times very quickly because
that also makes us more suspicious.
Some search engines encode domains that contain special characters
in the IDN format, i.e. by using Punycode (https://en.wikipedia.org/wiki/Punycode).
This means that domains formatted like that are not human readable
and thus very unintuive.
Also, as only some engines use Punycode, this often causes a result
to appear multiple times, once with a Punycode domain (e.g. xn--allestrungen-9ib.de),
and once with a normal domain (e.g. allestörungen.de).
Always formatting the domains nicely has the caveat that the official sites
are harder to distinguish from malicious clones that just swapped out some
characters by using Punycode, e.g. replaced `a` with `á`. I don't think
that will be much of an issue though - I don't think Punycode results
previously stopped users from clicking the link, and I also think that
most search engines filter out such bad results or don't even have them
indexed.
Changes:
- the `embed.js` request now requires a user agent header
- we include a user agent in the "actual" request (I dropped it by accident)
- we only send the first 4 decimal places of the location
instead of 7+ (not required, but harder to detect)
`engines` only contains active engines. Since `google` was set to
inactive, it's no longer part of the , making the engine crash.
With this workaround, we directly load the engine traits from the data
dictionary instead of going the intermediate step with the Google engine.
We're now shuffling the order of query arguments, as
it seems that Qwant actively blocks the query order of
SearXNG.
tgp seems to be short for "test group" (look at the XHR requests the
browser sends to "https://www.qwant.com/action/ui") - its actual value
doesn't matter, as long as it's sent and at the correct position in the
query params and doesn't change too frequently
FindFiles.net is a specialized file search engine designed to help you search
files online with precision. Unlike traditional search engines that mainly index
web pages, FindFiles focuses on finding real files on the internet - including
PDFs, documents, archives, videos, datasets, and more. [1]
[1] https://findfiles.net
The code that reads the value of variable `x` from `embed.js`, decodes
it to ASCII and based on that sets `window["tuskheader"]` and `window["tuskkey"]`
is attached below. The only real way to figure out what this is doing is
by stepping through it with the debugger, otherwise it's almost hopeless.
```js
function fe() {
const B = pe => pe.map(_e => String.fromCharCode(_e)).join(''),
ae = window,
o = ae.x;
if (o?.length) {
const pe = o.length / 2;
for (let _e = 0; _e < pe; _e++) ae[B(o[_e])] = B(o[pe + _e]);
ae.x = void 0
}
}
```
Minimal script for testing the engine:
```py
import random
from json import loads
import requests
resp = requests.get("https://api.tusksearch.com/revcontent/embed.js")
data = loads(resp.text[6:])
def _decode(text: list[int]) -> str:
return "".join([chr(x) for x in text])
header = _decode(data[3])
value = _decode(data[4])
resp = requests.get(
"https://api.tusksearch.com/Search/Web?q=test&p=1&l=center&nextArgs=&prevArgs=",
# "https://api.tusksearch.com/Search/Image?q=test&p=1&l=center",
headers={
header: value,
'x-lon': str(random.random() * 90),
'x-lat': str(random.random() * 90),
},
)
print(resp.text)
```
Category for searching personal blogs and websites.
Useful if searching for interesting articles on a topic
rather than the mainstream Wikipedia etc. results.
The results seem to be from Brave (i.e. they are exactly
the same). But it doesn't have any strict rate-limits,
so that's nice.
News support time ranges, but apart from that, unfortunately it doesn't
support any advanced features like safesearch or languages.
T-Online_ is a German news portal.
It gets its web results from Google, image results from Flickr and videos results
from YouTube.
For images and videos, it additionally returns result from its
news catalog. However, for pagination we have to specify the result
type (e.g. either videos from YouTube or from T-Online), so we use
flickr/youtube there instead of tonline because the tonline results
are usually irrelevant.
Add support for https://luxxle.com
Localization is not yet supported because it doesn't seem to work on their
website either, no matter which language I select, it only returns English web
results
RawWeb is a search engine for personal websites / blog posts.
It has its own index and the personal websites were selected
by hand. Results are quite good for what it is imo. [^1]
[^1]: https://github.com/0x2E/RawWeb.org
- adds support for https://reloado.com (german)
- as it has its own index, the results are hit or miss and mostly German,
but still worth integrating imo
Add support for https://startsiden.abcnyheter.no, a netherlandish search engine
that probably uses Google or Bing? idk it also returns English results, but
e.g. ``test`` returns mostly results from netherlands.
S1Search provides various different search services, which all seem
to be somewhat based on Google and Yahoo. The site looks kinda suspicious,
but the results are fine.
You can find a list of their engines by using a subdomain finder like
https://web-toolbox.dev/en/tools/subdomain-lookup and search for `s1search.co`.
Chatnoir is an open source search engine developed by universities, based on
CommonCrawl (and others). It's uncommented by default - we don't want to
overload the universities with bot traffic that targets SearXNG (sad truth why
we can't have nice things anymore)
e.g. when searching for "!tiger pottering github", it crashes.
not really sure why - the problem is that the HTML doesn't
really uses descriptive classes or ids, only Tailwind,
so it's very hard to select only the results HTML.
The default Helix configuration for Python is different,
so the pylint warnings aren't shown and the formatter
re-formats files by accident when you edit an existing file.
Therefore, this commit adds `python` language configuration
to ease developing SearXNG with Helix Editor [^1].
[^1]: https://helix-editor.com
html.duckduckgo.com captchas all my IPs very fast. I figured out that using
duckduckgo.com works even if html.duckduckgo.com is captcha-ed, hence adding
support for duckduckgo.com's general web search here.
This implementation fetches the link to the first API page
(i.e. ``links.duckduckgo.com/d.js?...``) from duckduckgo.com and uses the ``n``
parameter of the API to fetch all subsequent pages.
This also means that it's not possible to immediately search for the third
page - the first and the second page would need to be loaded first.
The reason why we can't just normally use the `vqd` value is that the API URLs
require an additional parameter `dp` which seems generated at server-side, so we
can't build it ourselves and must scrape it from the HTML pages.
Add support for https://tiger.ch (general, news)
It is disabled and inactive by default because it's just a metasearch engine
like SearXNG is, so it's mostly useful for bypassing rate-limits on other
engines: (it has its own German index, but it's not that great) in theory it
supports different locales, but I was too lazy to implement that (I only need
German and English results anyways, which are returned by default...)
Add support for https://seek.ninja (general)
It's very slow because the engine uses Server-side events, that incrementally
send data in their HTTP response [1].
I.e. we wait for the end of the response (7+ seconds), even though the results
data arrives within a few seconds -> it's very slow, because SearXNG wants to
get the full response body before it calls the `response(resp)` method
We could use httpx-sse [2], but I'm not sure how to integrate this into SearXNG
and if it's worth it
[1] https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/
[2] https://github.com/florimondmanca/httpx-sse
The engine is using very aggressive Cloudflare blocking for
a while now, no matter if using a normal browser like Firefox
or not.
Closes: https://github.com/searxng/searxng/issues/5976
Bnyro