Usenet is a distributed discussion system that allows users to read and post messages (called articles) to different groups (called newsgroups). Discussions are threaded and posts are stored sequentially. Usenet is distributed among large servers that store and feed messages to each other. Since all Usenet servers replicate between each other, users may read or post messages to any server such as their local ISP or a commercial Usenet provider.
Overview[edit | edit source]
Usenet is a protocol used to send and receive articles between participating servers. Like a gossip network, articles posted on one server will be forwarded to neighboring servers using a flooding algorithm so that it is propagated across the network.
The NNTP or Network News Transfer Protocol operates on TCP port 119 for unencrypted and TCP port 563 for SSL encrypted connections.
Protocol[edit | edit source]
The protocol is similar to SMTP but with different commands and usage and can be played with using
An example session:
# telnet news.newsdemon.com 119 Trying 188.8.131.52... Connected to news.newsdemon.com. Escape character is '^]'. 200 Welcome (fx38.iad) > authinfo user username 381 PASS required > authinfo pass password 281 Authentication Successful! Welcome to Usenet! (No Posting) > article <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> 220 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071 Organization: Usenet.Farm Newsgroups: alt.binaries.newznzb.bravo Date: Thu, 28 Jun 18 15:25:10 UTC Path: not-for-mail Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE----- X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> From: nEwZ[NZB] <pr3d@NET.world> X-Received-Bytes: 6481 X-Received-Body-CRC: 2720885501 =ybegin part=1 total=1 line=128 size=5071 name=serif.nfo =ypart begin=1 end=5071 74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74 =yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e . > head <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> 221 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071 Organization: Usenet.Farm Newsgroups: alt.binaries.newznzb.bravo Date: Thu, 28 Jun 18 15:25:10 UTC Path: not-for-mail Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE----- X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> From: nEwZ[NZB] <pr3d@NET.world> X-Received-Bytes: 6481 X-Received-Body-CRC: 2720885501 . > body <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> 222 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> =ybegin part=1 total=1 line=128 size=5071 name=serif.nfo =ypart begin=1 end=5071 74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74 =yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e . > stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> 223 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE> > stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484234> 430 No Such Article
Response Codes[edit | edit source]
As defined in RFC977, status response codes are 3 digit numeric codes with the first two digits defined as:
1xx - Informative message 2xx - Command ok 3xx - Command ok so far, send the rest of it. 4xx - Command was correct, but couldn't be performed for some reason. 5xx - Command unimplemented, or incorrect, or a serious program error occurred.
x0x - Connection, setup, and miscellaneous messages x1x - Newsgroup selection x2x - Article selection x3x - Distribution functions x4x - Posting x8x - Nonstandard (private implementation) extensions x9x - Debugging output
Some common response codes are:
|200||Server ready, with posting|
|201||Server ready, with no posting|
|430||Article not found|
|411||No such group|
|500||Command not recognized|
|501||Command syntax error|
|502||Access restriction or permission denied, eg. Connections exceeded|
|381||Auth password needed|
Newsgroups[edit | edit source]
There are 9 major hierarchies that newsgroups fall under. The largest hierarchy is the
alt.binaries which contains large binary posts used for sharing data or to facilitate piracy.
Binary Data[edit | edit source]
Prior to ~2001, binary content was encoded with uuencode which converts an 8-bit byte into a 7-bit ASCII representation. This was done because the underlying NNTP protocol was not designed to transmit arbitrary binary data. Later, when Usenet software allowed 8-bit extended ASCII, content began being encoded using yEnc that escapes certain characters (LF, CR, NUL, =) but otherwise encodes everything using extended ASCII character set.
Large files are typically compressed into multi-volume RAR archives along with Parchive files for error checking and to recreate missing data if files are missing from the server.
Retention[edit | edit source]
Each news server allocates a certain amount of storage space for each newsgroup. When the storage space is filled, old posts are deleted for the new content. The retention time of the server is how long a post will stay in storage before it is rolled over and deleted.
Most major Usenet providers have retention times of around 4 or more years.
Usage[edit | edit source]
Popular Newsgroups[edit | edit source]
Some of the more popular binary newsgroups are:
NZB File[edit | edit source]
A NZB file is a XML document that describes to a Usenet download client where articles making up a particular file are located. Information for each file includes the poster, groups, article name, size, and sequence number. Once all articles are downloaded, the Usenet downloader can then decode and recombine the binary data into the original file.
Usenet indexers constantly scans for new articles being posted on certain newsgroups and compiles them into NZB files for each release that is found. Indexers must also constantly scan PreDB sources (IRC, websites) for new scene release information such as name, sizes, and status (nuked or not).
PreDB and Hashed Filenames[edit | edit source]
PreDB sources include:
Filenames from scene releases are typically hashed. From nZEDb, the hashes that were used (but does not seem to be any more) are:
These hashes are then matched to the filenames that are posted on Usenet and the association is then used to generate NZB files for users to download and to obtain the scene release files.
In my vain effort to understand how the hashes are computed:
- The hashes are probably not nested MD5/SHA1/MD4/RIPEMD128/TIGER128/HAVAL128 hashes (up to 100 deep) of just the release name (reversed, upper/lower cased, XOR encrypted with the group name as the key, base64 encoded, uuencoded, rot13)
- Probably a secret key and/or date and time, concatenated with the release name. eg.
md5(key . md5($release . $time) . $time . $release)
- Not just a straight up hash of the release name, but possibly hashes of hahes of different values?
- Probably a secret key and/or date and time, concatenated with the release name. eg.
- Different hashes exist for the same release. Same hash exists for different releases (?? Is this true?)
- This means the hash includes more than just the release name
- Might also mean that the post date isn't part of the hash... perhaps a counter of some sort?
- The hashed content probably contains only information from PreDB, (ie. Release name, Size?, Date?)
- Probably does not contain the random poster name, the news group name, the unix timstamp of the post date (because... how would you time it?)
- Probably contains: PreDB release name, post date, secret key??
Hashed Examples[edit | edit source]
e0283ea766d244dbb410cae9c2ef5c8da.b.bloaf 2019-08-16 08:21:45 1xmY1W2Ovvtri@vWbDSk.7M2
6ae22cd32a0343c49bd0e20a55761ccda.b.multimedia 2019-08-16 06:51:54 HzqQ29TWq_LIGXccKrBTR@0wyQWFFhPU.pG1
4cc5e6140b3742e1a6ffadda1e612356a.b.amazing 2019-08-15 23:14:41 Z1Wm51T_eNG-AfqhnSP5riWpzQZ@r9o6dkIumq73A._60
- different poster, same date
b7eb500e245242df9e91674044948005a.b.multimedia 2019-08-07 15:04:23 kK6qv1X460DmB27-IA9AxWg0BKT408x5ryjYNq7o@5Vq_IbQs4u8E.AZ0
- Same file
- f9840aa34a274ab9aff01fc4c1734036 a.b.multimedia 2019-08-21 04:16:35 DFO2_d2k1jJ43mqLT6gfIB@UenJFDy94K_.9BB
- f6a84c99c32f4836b8507a97ef7d545c a.b.amazing 2019-08-20 21:13:49 -ZBjV3xNkDp@hVmr4.yjR
See Also[edit | edit source]
Providers[edit | edit source]
There are two main types of providers: Tier-1 provider that operates their own independent infrastructure (ie. backbone), and resellers that resell from a tier-1 provider. Understanding which backbone your provider is offering is important since it may affect:
- Download speed (due to geographical location and its resulting latency),
- DMCA (Digital Millennium Copyright Act) takedown or NTD (Notice & Take down) requests due to local laws the backbone is operated in
- Article retention period
To help with download completion and avoid missing articles, select providers from different backbones. Block accounts from alternate backbones are useful as backup providers to ensure a single DMCA/NTD on a backbone won't cause a download to fail due to incomplete articles.
A full list of Tier-1 and resellers can be found at:
Usenet providers typically have sales which are usually at a large discount. Prices below are what I've paid mostly from sales. Block accounts are typically non-expiring and I've had some for more than 4 years.
|Provider||Service and Price|
|UsenetExpress||$5.50 for 500GB Block|
|NewsDemon||$3 for 500GB Block|
|NewsDemon||$15 for 24 months, 500GB per month, 50 connections, VPN Access|
|XsUsenet||Free, requires monthly renewal, 25GB per month, 1 connection, 2MBps|
|XennaNews (test)||Free, 10 connection, 10Mbps, 60GB per month. Sometimes open for free registration.|
|NNTP.pw||Free, 1 connection at ~7Mbps, unlimited bandwidth. Was usenet.pw and usenet.cat years ago. news service at 184.108.40.206.|
|free-usenet.com||Free 3 hours of Usenet. Unlimited renewals. 1 connection, 1Mbps. ~1.2GB per 3 hours.|
Services[edit | edit source]
Usenet Indexers[edit | edit source]
Here is a list of free usenet indexers. Typically, free accounts have limited number of API and downloads per day. Use NZBHydra to consolidate and rate limit hits to the APIs.
- abnzb.com - https://abnzb.com
- miatrix - https://www.miatrix.com
- nzbfinder - https://nzbfinder.ws
- nzbnoob - https://www.nzbnoob.com
- tabula-rasa - https://www.tabula-rasa.pw
- drunken slug - https://drunkenslug.com (registrations are sometimes open)
Here are some paid indexers:
A more up to date list can also be found on the /r/usenet wiki: