Usenet is a distributed discussion system that allows users to read and post messages (called articles) to different groups (called newsgroups). Discussions are threaded and posts are stored sequentially. Usenet is distributed among large servers that store and feed messages to each other. Since all Usenet servers replicate between each other, users may read or post messages to any server such as their local ISP or a commercial Usenet provider.

Overview[edit]

Usenet is a protocol used to send and receive articles between participating servers. Like a gossip network, articles posted on one server will be forwarded to neighboring servers using a flooding algorithm so that it is propagated across the network.

The NNTP or Network News Transfer Protocol operates on TCP port 119 for unencrypted and TCP port 563 for SSL encrypted connections.

Protocol[edit]

The protocol is similar to SMTP but with different commands and usage and can be played with using telnet.

An example session:

# telnet news.newsdemon.com 119
Trying 69.16.179.23...
Connected to news.newsdemon.com.
Escape character is '^]'.
200 Welcome (fx38.iad)
> authinfo user username
381 PASS required
> authinfo pass password
281 Authentication Successful!  Welcome to Usenet! (No Posting)
> article <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
220 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071
Organization: Usenet.Farm
Newsgroups: alt.binaries.newznzb.bravo
Date: Thu, 28 Jun 18 15:25:10 UTC
Path: not-for-mail
Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE-----
X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D
Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
From: nEwZ[NZB] <pr3d@NET.world>
X-Received-Bytes: 6481
X-Received-Body-CRC: 2720885501

=ybegin part=1 total=1 line=128 size=5071 name=serif.nfo
=ypart begin=1 end=5071
74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ
JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74
=yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e
.
> head <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
221 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
Subject: [PRiVATE] \439025696e\::b46dbf8654df93.bfebda2625d7cc61c14d12ed89b636.fde850f2::/c96337a734a4/ [newzNZB] [3/3] - yEnc (1/1) 5071
Organization: Usenet.Farm
Newsgroups: alt.binaries.newznzb.bravo
Date: Thu, 28 Jun 18 15:25:10 UTC
Path: not-for-mail
Verify-Signature-Newznzb: -----BEGIN newzNZB SIGNATURE-----NjdkNGNjYTExYjM0NWQxZGJiMTgzOTlkNTU5ZGJiNjRmYjgxMDg1Y2YyZTVmMGM1NDRmYzQ4ZWJiNjAxNWRjNjBhZDkzN2E0YzA0ZTFjYzgzZjhjMDJkNjA0NjdmZjYwZWMyYjE3YzZhNDU4ZmRkMzVhY2MxMzMwNzU1MzE5ZTRjNjJkZWFhYWI3YWEyYjAzMTVjM2MwNzQ1NjJjYjdhZg==-----END newzNZB SIGNATURE-----
X-Ufk: dUJnI0ho5OBMs71DufQubCpJ%2F0Puy6%2FNxhNuO%2BR3wX8PWSFkwmJJ0NJgwGVMKtHKz8hn9axDrdBOhX97uwNwDM4%2B%2Fx3ly9JQRME%2BfomPW1pbAn2rs5%2FJYVKMIQ5WQzL9RK3dyZzLMr3LeuzGiEy%2FSU5aDVWRMmXj6dXKB80c4OUoaUC7uu0D1T3cdk6FGyfp66ZrlqMhvOHvU6QXg5vpxxk6FSiKTYk%3D
Message-Id: <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
From: nEwZ[NZB] <pr3d@NET.world>
X-Received-Bytes: 6481
X-Received-Body-CRC: 2720885501
.
> body <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
222 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
=ybegin part=1 total=1 line=128 size=5071 name=serif.nfo
=ypart begin=1 end=5071
74JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ▒ULQV74JJJJJJJJJJJJJJJJJJJJJJJJJ▒VJJJJJJJJJJJJJJJJJJJJJJJJV▒QXJJJX74JJJJJ
JJJJJJ▒▒▒▒74JJJJJJJJJJJJJJY▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒74
=yend size=5071 part=1 pcrc32=6fc3637e crc32=6fc3637e
.
> stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
223 0 <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484291@PRiVATE>
> stat <IgLnZpOzUcUfNzGyQaCiZfVw-1530199484234>
430 No Such Article

Response Codes[edit]

As defined in RFC977, status response codes are 3 digit numeric codes with the first two digits defined as:

     1xx - Informative message
     2xx - Command ok
     3xx - Command ok so far, send the rest of it.
     4xx - Command was correct, but couldn't be performed for
           some reason.
     5xx - Command unimplemented, or incorrect, or a serious
           program error occurred.


     x0x - Connection, setup, and miscellaneous messages
     x1x - Newsgroup selection
     x2x - Article selection
     x3x - Distribution functions
     x4x - Posting
     x8x - Nonstandard (private implementation) extensions
     x9x - Debugging output

Some common response codes are:

Code Description
200 Server ready, with posting
201 Server ready, with no posting
220 Article Found
221 Headers follows
222 Data follows
223 Article found
430 Article not found
211 Group joined
411 No such group
215 Info follows
205 Closing connection
340 Continue posting
500 Command not recognized
501 Command syntax error
502 Access restriction or permission denied, eg. Connections exceeded
281 Auth accepted
381 Auth password needed
480 Auth required
481 Auth failed

Newsgroups[edit]

There are 9 major hierarchies that newsgroups fall under. The largest hierarchy is the alt.binaries which contains large binary posts used for sharing data or to facilitate piracy.

Binary Data[edit]

Prior to ~2001, binary content was encoded with uuencode which converts an 8-bit byte into a 7-bit ASCII representation. This was done because the underlying NNTP protocol was not designed to transmit arbitrary binary data. Later, when Usenet software allowed 8-bit extended ASCII, content began being encoded using yEnc that escapes certain characters (LF, CR, NUL, =) but otherwise encodes everything using extended ASCII character set.

Large files are typically compressed into multi-volume RAR archives along with Parchive files for error checking and to recreate missing data if files are missing from the server.

Retention[edit]

Each news server allocates a certain amount of storage space for each newsgroup. When the storage space is filled, old posts are deleted for the new content. The retention time of the server is how long a post will stay in storage before it is rolled over and deleted.

Most major Usenet providers have retention times of around 4 or more years.

Usage[edit]

Popular Newsgroups[edit]

Some of the more popular binary newsgroups are:

Group
a.b.teevee
a.b.inner-sanctum
a.b.multimedia.anime.highspeed
a.b.mp3
a.b.amazing
a.b.multimedia
a.b.bloaf
a.b.e-book.magazines
a.b.warez
a.b.triballs
a.b.ath
a.b.boneless
a.b.u4e
a.b.mom
a.b.sounds.flac
a.b.e-book.german
a.b.misc
a.b.movies.divx
a.b.etc
a.b.usenetrevolution
a.b.test
a.b.town
a.b.anime
a.b.moovee
a.b.british.drama
a.b.hdtv.x264
a.b.games

NZB File[edit]

A NZB file is a XML document that describes to a Usenet download client where articles making up a particular file are located. Information for each file includes the poster, groups, article name, size, and sequence number. Once all articles are downloaded, the Usenet downloader can then decode and recombine the binary data into the original file.

Usenet indexers constantly scans for new articles being posted on certain newsgroups and compiles them into NZB files for each release that is found. Indexers must also constantly scan PreDB sources (IRC, websites) for new scene release information such as name, sizes, and status (nuked or not).

PreDB and Hashed Filenames[edit]

PreDB sources include:

Filenames from scene releases are typically hashed. From nZEDb, the hashes that were used (but does not seem to be any more) are:

  • MD5(title)
  • MD5(MD5(title))
  • SHA1(title)
  • MD5(title.requestid)
  • MD5(title.requestid.requestid)

These hashes are then matched to the filenames that are posted on Usenet and the association is then used to generate NZB files for users to download and to obtain the scene release files.

In my vain effort to understand how the hashes are computed:

  • The hashes are probably not nested MD5/SHA1/MD4/RIPEMD128/TIGER128/HAVAL128 hashes (up to 100 deep) of just the release name (reversed, upper/lower cased, XOR encrypted with the group name as the key, base64 encoded, uuencoded, rot13)
    • Probably a secret key and/or date and time, concatenated with the release name. eg. md5(key . md5($release . $time) . $time . $release)
    • Not just a straight up hash of the release name, but possibly hashes of hahes of different values?
  • Different hashes exist for the same release. Same hash exists for different releases (?? Is this true?)
    • This means the hash includes more than just the release name
    • Might also mean that the post date isn't part of the hash... perhaps a counter of some sort?
  • The hashed content probably contains only information from PreDB, (ie. Release name, Size?, Date?)
    • Probably does not contain the random poster name, the news group name, the unix timstamp of the post date (because... how would you time it?)
    • Probably contains: PreDB release name, post date, secret key??

Hashed Examples[edit]

  • Godzilla.King.of.the.Monsters.2019.720p.BluRay.x264-SPARKS has hash:
    • e0283ea766d244dbb410cae9c2ef5c8d a.b.bloaf 2019-08-16 08:21:45 1xmY1W2Ovvtri@vWbDSk.7M2
    • 6ae22cd32a0343c49bd0e20a55761ccd a.b.multimedia 2019-08-16 06:51:54 HzqQ29TWq_LIGXccKrBTR@0wyQWFFhPU.pG1
    • 4cc5e6140b3742e1a6ffadda1e612356 a.b.amazing 2019-08-15 23:14:41 Z1Wm51T_eNG-AfqhnSP5riWpzQZ@r9o6dkIumq73A._60
  • Fear.the.Walking.Dead.S05E10.1080p.WEB.h264-TBS has hash 7c46447b6c3443f2bced84ba3bea5e6f and 4161d996c52a4a28bf7fe31124da29b7
    • different poster, same date
  • American.Horror.Story.S08E08.1080p.BluRay.x264-SHORTBREHD
    • b7eb500e245242df9e91674044948005 a.b.multimedia 2019-08-07 15:04:23 kK6qv1X460DmB27-IA9AxWg0BKT408x5ryjYNq7o@5Vq_IbQs4u8E.AZ0
    • Same file
  • The.Secret.Life.of.Pets.2.2019.720p.BluRay.x264-DRONES
    • f9840aa34a274ab9aff01fc4c1734036 a.b.multimedia 2019-08-21 04:16:35 DFO2_d2k1jJ43mqLT6gfIB@UenJFDy94K_.9BB
    • f6a84c99c32f4836b8507a97ef7d545c a.b.amazing 2019-08-20 21:13:49 -ZBjV3xNkDp@hVmr4.yjR


See Also[edit]

Services[edit]

Free Usenet Service[edit]

Here is a list of free Usenet providers, ordered from best to worse from my experience:

  • test.xennanews.com - Free test accounts from xennanews provide 10Mbit/s service. Only open for registration once in a blue moon.
  • xsusenet.com - Very reliable service. Monthly renewal required by logging in to their control panel. A limit of 30GB 25GB per month.
  • nntp.pw - Was usenet.pw and usenet.cat years ago. Free registration. Service has been reliable since around October 2019.
  • free-usenet.com - Free 3 hours of Usenet. Unlimited renewals. Speed capped at a measly 1Mbit/s, or about 1.2GB per 3 hours.

I cannot vouch for these provider's security. Assume everything is logged.

Usenet Indexers[edit]

Here is a list of free usenet indexers. Typically, free accounts have limited number of API and downloads per day. Use NZBHydra to consolidate and rate limit hits to the APIs.

Here are some paid indexers:

A more up to date list can also be found on the /r/usenet wiki:

Enable Dark Mode!