July 2016 OnwardsAt the end of July 2016, Gmane briefly closed and was acquired by Yomura in mid August.
Gmane is currenly being rebuilt in Python and PHP and uses ElasticSearch for its underlying document store. At the time of writing, we have just over 112M documents indexed.
The site uses a variety of technology provided by Yomura's subsidiary Delimiter:
- Dedicated Servers
- Cloud Compute
For the latest development information please head over to the Gmane Blog.
Free software is mainly developed on mailing lists. Mailing lists have many advantages over other forms of communication, but they have two weaknesses: It's difficult to follow discussions in a sensible way, and mailing list archives (when they exist) have a tendency to disappear over time.
Several mailing list archives exist, but these are all hidden under a web interface. Reading mail that way is not convenient. Reading mail as if it were news is convenient.
This is what Gmane offers. Mailing lists are funneled into news groups. This isn't a new idea; several mail-to-news gateways exist. What's new with Gmane is that no messages are ever expired from the server, and the gateway is bidirectional. You can post to some of these mailing lists without being subscribed to them yourself, depending on whether the mailing lists allow non-subscribers to post or not.
In addition, Gmane does spam detection, cross-post handling, has a TMDA-fueled encryption/forwarding service, a web interface, respects X-No-Archive, supplies RSS feeds, uses SPF, gathers traffic statistics, and has a real-time indexing search engine.
Not all mailing lists allow non-subscribers to post, and some are moderated. Gmane requires that users post to Gmane groups using a valid e-mail address, and requires a one-time authorization per group.
To read the mailing lists stored in Gmane, point your news reader to news.gmane.org.
Discussion about the Gmane hierarchy takes place on the gmane.discuss newsgroup.
At present, the Gmane hierarchy is heavily dominated by computer-related mailing lists, which reflects the interests of the initial user base. However, Gmane is not meant to be a service exclusively for IT people. Feel free to suggest non-computer-related mailing lists.
To get in touch with the Gmane administrators, send a mail to Lars Magne Ingebrigtsen.
Gmane has been in full operation since February 11th 2002, after a one month test period. And the "g" in "Gmane" is silent.
BackgroundI've tried to gather some thoughts on Gmane here. It's been running now for almost half a year, and it's about time to summarize the experience somewhat.
My main reason for setting up Gmane in the first place was my annoyance with the concept of mailing lists. I've always preferred newsgroups -- they have a nice interface; they can be archived in a convenient format; and they can be searched (via DejaNews/Google). Mailing lists have a lousy interface, and if they have an archive at all, it's always hidden behind some web-based monstrosity. And they never allow searching.
So I started Gmane, and gathered the mailing lists I was vaguely interested in. Many, many other lists were requested by other users and added by me and the other Gmane administrators. There's now more than two thousand lists here. If I'm looking for discussion on any given topic (for instance -- buffer overflows in the libc resolver library), I just have my news reader display all the Gmane groups, and, more often than not, all the relevant mailing lists are already there, and I find what I'm looking for.
People generally seem to respond pretty enthusiastically to Gmane, too, which is nice. However, Gmane is not an unproblematic project.
SpamThe first major problem is spam.
I hate spam. You hate spam. We all hate spam. I get something like 100-200 spam/virus mail messages per day. Many people have stopped publishing their real mail addresses on the web, and on Usenet newsgroups. But on mailing lists, they have to identify themselves. If they don't, they aren't allowed to subscribe to the lists.
Gmane makes it much easier for spam harvesters to gather these real, authentic mail addresses. Even though the Gmane web interface to the news spool obfuscates all addresses, a spam harvesting bot just has to point itself to the news interface to slurp down the entire spool. And there's not much I can do to stop that from happening.
I have implemented a scheme for rewriting addresses in a manner that would make it difficult for address harvesters to use the addresses. In short, it rewrites the address firstname.lastname@example.org to be larsi-SDHSGGHghsdyS@public.gmane.org, and then TMDA is used to provide a challenge/response thing before forwarding any mail messages to email@example.com. See gmane.discuss for a more in-depth discussion on this issue.
The idea is that mailing list administrators will themselves be able to say whether the Gmane group of that list should use address obfuscation or not. Each individual user will also be able to say so using a special header.
PrivacyThe second major problem is the loss of privacy, and is a bit more difficult to program around.
All the mailing lists archived in Gmane are open mailing lists. The subscription commands sent out by Gmane clearly identifies Gmane as a mail-to-news gateway, and any administrator running a closed mailing list should be able to refuse Gmane entry. Perhaps the subscription message should be made even more explicit, somehow.
Still, even open mailing lists are somewhat less public than, say, a Usenet newsgroup. The assumption on (some) mailing lists is that whatever happens on the list will only be read by the other (few hundred) participants, and won't be archived anywhere. It's one step up from IRC.
Pushing something that is somewhat public, but somewhat private, into the entirely public sphere is inherently problematic.
Some people on these somewhat-private lists who learn about Gmane get a bit miffed. During Gmane's first six months, there's been about one request for list removals per month. What almost all of these have in common is that they are chat-oriented lists, and not technical lists.
Of course, any request for removal is acted upon at once. All the requests were without incident, except one, which basically asked (among all the invective and not-so-veiled threats) -- what right do I have to do this?
RightsWhen Digital started Altavista back in the mid-90s, the same questions were raised. People had been putting stuff on their web pages, and not thinking too much about what they were putting out there. Then came Altavista which made it trivial to type in the name of anybody and getting a whole heap of information on that person back.
Some people were horrified. They wrote not very nice messages to Altavista and basically asked (among all the invective and not-so-veiled threats) -- what right did they have to do this?
Web spiders settled on using a robots.txt file to specify whether something was not supposed to be indexed. DejaNews settled on using X-No-Archive to say whether something wasn't supposed to be archived. Gmane does the same.
A StatementI grepped through quite a bit of the news spool looking for negative reactions from participants on mailing lists about being archived in Gmane. There weren't many, which might have something to do with the participants not knowing about Gmane, but on the lists where Gmane was being discussed, there were a couple. (And quite a few positive ones.) One writer expressed skepticism about Gmane because he couldn't find a privacy statement anywhere on the site.
I must confess that my first reaction was to roll my eyes a bit, and my second reaction was to be somewhat amused -- after all, only criminals swear that they aren't. Honest people take it for granted.
But he has a point. I could be evil.
So here's the statement: I won't use any of the contents of the Gmane news spool for spamming, for advertisement, for sending mass mail, for gathering profiling information, or anything of that kind, or willingly allow anybody else to do the same. In fact, I'll do my darnest to make it difficult for anybody else to do so, to the best of my abilities.
The FutureRunning Gmane is a bit of work, and I'm pretty sure I won't wish to run it for the rest of my life, but at the moment, it's ok.
Things that will be done to Gmane in the not-too-distant future includes running a spam cancellation bot over the Gmane spool. This will make reading many of the groups much more pleasant. Quite a few of the mailing lists receive an astounding amount of spam.
Gmane will also start feeding other sites a full gmane.* feed.
And Gmane will implement X-No-Archive and the like.
Hope you'll enjoy the improvements.
Lars Magne Ingebrigtsen