Usenet

From Leo's Notes
Last edited on 14 June 2020, at 23:54.

Usenet is a distributed discussion system that allows users to read and post messages (called articles) to different groups (called newsgroups). Discussions are threaded and posts are stored sequentially. Usenet is distributed among large servers that store and feed messages to each other. Since all Usenet servers replicate between each other, users may read or post messages to any server such as their local ISP or a commercial Usenet provider.

Overview

Usenet is a protocol used to send and receive articles between participating servers. Like a gossip network, articles posted on one server will be forwarded to neighboring servers using a flooding algorithm so that it is propagated across the network.

The NNTP or Network News Transfer Protocol operates on TCP port 119 for unencrypted and TCP port 563 for SSL encrypted connections.

Protocol

The protocol is similar to SMTP but with different commands and usage and can be played with using telnet.

An example session: 

# telnet news.newsdemon.com 119
Trying 69.16.179.23...
Connected to news.newsdemon.com.
Escape character is '^]'.
200 Welcome (fx38.iad)
> authinfo user username
381 PASS required
> authinfo pass password
281 Authentication Successful!  Welcome to Usenet! (No Posting)
> article <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
220 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071
Organization: Usenet.Farm
Newsgroups: alt.binaries.newznzb.bravo
Date: Thu, 28 Jun 18 15:25:10 UTC
Path: not-for-mail
Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE-----
X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D
Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
From: nEwZ[NZB] <pr3d@NET.world>
X-Received-Bytes: 6481
X-Received-Body-CRC: 2720885501

=ybegin part=1 total=1 line=128 size=5071 name=serif.nfo
=ypart begin=1 end=5071
74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ
JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74
=yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e
.
> head <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
221 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071
Organization: Usenet.Farm
Newsgroups: alt.binaries.newznzb.bravo
Date: Thu, 28 Jun 18 15:25:10 UTC
Path: not-for-mail
Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE-----
X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D
Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
From: nEwZ[NZB] <pr3d@NET.world>
X-Received-Bytes: 6481
X-Received-Body-CRC: 2720885501
.
> body <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
222 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
=ybegin part=1 total=1 line=128 size=5071 name=serif.nfo
=ypart begin=1 end=5071
74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ
JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74
=yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e
.
> stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
223 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
> stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484234>
430 No Such Article

Response Codes

As defined in RFC977, status response codes are 3 digit numeric codes with the first two digits defined as: 

     1xx - Informative message
     2xx - Command ok
     3xx - Command ok so far, send the rest of it.
     4xx - Command was correct, but couldn't be performed for
           some reason.
     5xx - Command unimplemented, or incorrect, or a serious
           program error occurred.



     x0x - Connection, setup, and miscellaneous messages
     x1x - Newsgroup selection
     x2x - Article selection
     x3x - Distribution functions
     x4x - Posting
     x8x - Nonstandard (private implementation) extensions
     x9x - Debugging output

Some common response codes are:

Code Description
200 Server ready, with posting
201 Server ready, with no posting
220 Article Found
221 Headers follows
222 Data follows
223 Article found
430 Article not found
211 Group joined
411 No such group
215 Info follows
205 Closing connection
340 Continue posting
500 Command not recognized
501 Command syntax error
502 Access restriction or permission denied, eg. Connections exceeded
281 Auth accepted
381 Auth password needed
480 Auth required
481 Auth failed

Newsgroups

There are 9 major hierarchies that newsgroups fall under. The largest hierarchy is the alt.binaries which contains large binary posts used for sharing data or to facilitate piracy.

Binary Data

Prior to ~2001, binary content was encoded with uuencode which converts an 8-bit byte into a 7-bit ASCII representation. This was done because the underlying NNTP protocol was not designed to transmit arbitrary binary data. Later, when Usenet software allowed 8-bit extended ASCII, content began being encoded using yEnc that escapes certain characters (LF, CR, NUL, =) but otherwise encodes everything using extended ASCII character set.

Large files are typically compressed into multi-volume RAR archives along with Parchive files for error checking and to recreate missing data if files are missing from the server.

Retention

Each news server allocates a certain amount of storage space for each newsgroup. When the storage space is filled, old posts are deleted for the new content. The retention time of the server is how long a post will stay in storage before it is rolled over and deleted.

Most major Usenet providers have retention times of around 4 or more years.

Usage

Popular Newsgroups

Some of the more popular binary newsgroups are:

Group
a.b.teevee
a.b.inner-sanctum
a.b.multimedia.anime.highspeed
a.b.mp3
a.b.amazing
a.b.multimedia
a.b.bloaf
a.b.e-book.magazines
a.b.warez
a.b.triballs
a.b.ath
a.b.boneless
a.b.u4e
a.b.mom
a.b.sounds.flac
a.b.e-book.german
a.b.misc
a.b.movies.divx
a.b.etc
a.b.usenetrevolution
a.b.test
a.b.town
a.b.anime
a.b.moovee
a.b.british.drama
a.b.hdtv.x264
a.b.games

NZB File

A NZB file is a XML document that describes to a Usenet download client where articles making up a particular file are located. Information for each file includes the poster, groups, article name, size, and sequence number. Once all articles are downloaded, the Usenet downloader can then decode and recombine the binary data into the original file.

Usenet indexers constantly scans for new articles being posted on certain newsgroups and compiles them into NZB files for each release that is found. Indexers must also constantly scan PreDB sources (IRC, websites) for new scene release information such as name, sizes, and status (nuked or not).

PreDB and Hashed Filenames

PreDB sources include:

Filenames from scene releases are typically hashed. From nZEDb, the hashes that were used (but does not seem to be any more) are:

  • MD5(title)
  • MD5(MD5(title))
  • SHA1(title)
  • MD5(title.requestid)
  • MD5(title.requestid.requestid)

These hashes are then matched to the filenames that are posted on Usenet and the association is then used to generate NZB files for users to download and to obtain the scene release files.

In my vain effort to understand how the hashes are computed:

  • The hashes are probably not nested MD5/SHA1/MD4/RIPEMD128/TIGER128/HAVAL128 hashes (up to 100 deep) of just the release name (reversed, upper/lower cased, XOR encrypted with the group name as the key, base64 encoded, uuencoded, rot13)
    • Probably a secret key and/or date and time, concatenated with the release name. eg. md5(key . md5($release . $time) . $time . $release)
    • Not just a straight up hash of the release name, but possibly hashes of hahes of different values?
  • Different hashes exist for the same release. Same hash exists for different releases (?? Is this true?)
    • This means the hash includes more than just the release name
    • Might also mean that the post date isn't part of the hash... perhaps a counter of some sort?
  • The hashed content probably contains only information from PreDB, (ie. Release name, Size?, Date?)
    • Probably does not contain the random poster name, the news group name, the unix timstamp of the post date (because... how would you time it?)
    • Probably contains: PreDB release name, post date, secret key??

Hashed Examples

  • Godzilla.King.of.the.Monsters.2019.720p.BluRay.x264-SPARKS has hash:
    • e0283ea766d244dbb410cae9c2ef5c8d a.b.bloaf 2019-08-16 08:21:45 1xmY1W2Ovvtri@vWbDSk.7M2
    • 6ae22cd32a0343c49bd0e20a55761ccd a.b.multimedia 2019-08-16 06:51:54 HzqQ29TWq_LIGXccKrBTR@0wyQWFFhPU.pG1
    • 4cc5e6140b3742e1a6ffadda1e612356 a.b.amazing 2019-08-15 23:14:41 Z1Wm51T_eNG-AfqhnSP5riWpzQZ@r9o6dkIumq73A._60
  • Fear.the.Walking.Dead.S05E10.1080p.WEB.h264-TBS has hash 7c46447b6c3443f2bced84ba3bea5e6f and 4161d996c52a4a28bf7fe31124da29b7
    • different poster, same date
  • American.Horror.Story.S08E08.1080p.BluRay.x264-SHORTBREHD
    • b7eb500e245242df9e91674044948005 a.b.multimedia 2019-08-07 15:04:23 kK6qv1X460DmB27-IA9AxWg0BKT408x5ryjYNq7o@5Vq_IbQs4u8E.AZ0
    • Same file
  • The.Secret.Life.of.Pets.2.2019.720p.BluRay.x264-DRONES
    • f9840aa34a274ab9aff01fc4c1734036 a.b.multimedia 2019-08-21 04:16:35 DFO2_d2k1jJ43mqLT6gfIB@UenJFDy94K_.9BB
    • f6a84c99c32f4836b8507a97ef7d545c a.b.amazing 2019-08-20 21:13:49 -ZBjV3xNkDp@hVmr4.yjR


See Also

Providers

There are two main types of providers: Tier-1 provider that operates their own independent infrastructure (ie. backbone), and resellers that resell from a tier-1 provider. Understanding which backbone your provider is offering is important since it may affect:

  • Download speed (due to geographical location and its resulting latency),
  • DMCA (Digital Millennium Copyright Act) takedown or NTD (Notice & Take down) requests due to local laws the backbone is operated in
  • Article retention period

To help with download completion and avoid missing articles, select providers from different backbones. Block accounts from alternate backbones are useful as backup providers to ensure a single DMCA/NTD on a backbone won't cause a download to fail due to incomplete articles.

A full list of Tier-1 and resellers can be found at:

Usenet providers typically have sales which are usually at a large discount. Prices below are what I've paid mostly from sales. Block accounts are typically non-expiring and I've had some for more than 4 years.

Provider Service and Price
UsenetExpress $5.50 for 500GB Block
NewsDemon $3 for 500GB Block
NewsDemon $15 for 24 months, 500GB per month, 50 connections, VPN Access
XsUsenet Free, requires monthly renewal, 25GB per month, 1 connection, 2MBps
XennaNews (test) Free, 10 connection, 10Mbps, 60GB per month. Sometimes open for free registration.
NNTP.pw Free, 1 connection at ~7Mbps, unlimited bandwidth. Was usenet.pw and usenet.cat years ago. news service at 51.15.47.139.
free-usenet.com Free 3 hours of Usenet. Unlimited renewals. 1 connection, 1Mbps. ~1.2GB per 3 hours.

Services

Usenet Indexers

Here is a list of free usenet indexers. Typically, free accounts have limited number of API and downloads per day. Use NZBHydra to consolidate and rate limit hits to the APIs.

Here are some paid indexers:

A more up to date list can also be found on the /r/usenet wiki: