An Interview with Evan Prodromou, the Developer Behind the Open Source Twitter Clone

An Interview with Evan Prodromou, the Developer Behind the Open Source Twitter Clone

Programmer and self-described internet entrepreneur Evan Prodromou has been involved in starting open source and open content projects. His best-known is Wikitravel, a Wiki site for collaboratively edited travel guides.

Indenti.ca, his current project, is his attempt to develop a free network service using shared, open data. But to the uninitiated, the site and service look—and function very much—like a clone of Twitter. The big difference is that the software running Indenti.ca, which Prodromou has also been developing, Laconica, is free and open source. People can copy the code and use it on their own servers.

Originally from San Francisco, Prodromou is now based in Montreal where he has established a company, Control Yourself, to turn Indenti.ca into a business.

Howard Wen: What term would you use to describe this communication architecture—the Twitter or Facebook-style “user status update” system?

Evan Prodromou: If it [were] up to me, I would probably choose something like “short message hub” or “universal messaging hub.” But the name that’s really sticking with this kind of service is “microblogging.” I think that comes out of the fact that a lot of [these] services have a web interface and it looks like a blog. I don’t think it’s a fully accurate term, but it’s good enough, and it gets the idea across.

HW: Can you sum up how the architecture for microblogging basically works in a service like Twitter or Indenti.ca?

EP: Typically, you have a web service that serves as a hub for messages, and messages can go into this hub through many different media. They can be posted from a web page, but they could also be sent from an IM client, SMS or e-mail. They can be sent in through a web API, too. Twitter, for example, [has] a number of desktop clients that support their API.

Then each user in these networks can subscribe to messages from friends, from people they’re interested in. So the messaging hub does this switching process of saying, “Of all these messages coming in, where do I send them out to?” The messaging hub sends them out to subscribers. These messages go out over RSS, web pages, IM, SMS or to API clients.

So, it’s a “multiple channels in; multiple channels out” messaging system with a social networking aspect.

HW: To clarify, what’s the difference between the archicture of a microblogging system and that of an instant messenging service?

EP: IM tends to not have a lot of persistence. You don’t expect to be able to find a particular message or something that someone said in IM kept around on the web forever, and that’s something that does happen with microblogging. We expect things to be persistent.

There’s also an expectation of real-time conversation with IM. Whereas in microblogging, the conversation can happen over a period of days. So it’s a more extended conversation.

Finally, most IM conversations are one-to-one. With microblogging, even a relatively antisocial person will usually end up with 50, 60, 100 people listening to them. It’s common to get up into the triple digits fairly quickly, even for someone who’s not looking to add a lot of friends.

One funny thing is that in IM if I’m in a conversation with you and another person, we can all three “hear” each other. But in microblogging, I might send out a notice, and you and the other person can both hear what I’m saying, but can’t hear what each other are saying, because you’re not subscribed to each other’s messages. So there’s a disconnect. The way notices propagate is a little bit more fractured across time and across a social network.

HW: Why did you create an open source version of the Twitter system?

EP: As someone who’s very active in open source software and open source web software, I’m very interested in how much of our online life we are putting into the hands of services that are very proprietary. Some people call it the “roach motel model”—you put your data in and it won’t come out. I put tons of information about myself and my life into services like Facebook. Twitter was another one that I was participating in quite a bit. But if anything happened with Twitter, if Twitter had reliability problems, or if Twitter wasn’t on the web for a while, I couldn’t take my data, put it into my instance of that software, and run it somewhere else. I find that frustrating. I think a lot of other people find that frustrating, too: giving up control of your social life and your sociality online to someone who doesn’t necessarily have your best interests at heart.

It’s not that the people who start Facebook or MySpace or Twitter are bad or anything. But losing one user’s data, or dropping one user’s set of messages is not a big deal to them, whereas it’s a really big deal to me. So I wanted to experiment with providing web services where the user is in control.

I’m interested in doing this in different fields in the web services area. Microblogging just happens to be very popular. I wanted to get the idea in front of people who are interested in Twitter, and say maybe there’s another way that we can do this.

HW: What language and libraries did you use to create Laconica, and why?

EP: Laconica is written with PHP and MySQL. We have off-line processing daemons that do a lot of the same kind of work that the Twitter off-line daemons do. They do routing and they do sending stuff out over different channels. We don’t have a dedicated queuing server that’s built into the system right now, but that will be in an upcoming version.

Right now, we just use our MySQL database as kind of an ad-hoc queuing server. It’s not a very good way to do things and it kind of hurts our performance.

I think it’s a little more unusual to do long-running processing, like queue handlers in PHP, than it is to do it with other scripting languages, like Ruby or Python or Pearl. However, once we made PHP work that way, it’s been actually a pretty decent choice for implementation. And the fact that the web interface and the off-line daemons use the same language and libraries makes contributing to both a lot easier.

HW: How similar is Laconica’s technology to Twitter’s?

EP: Let me say that I only know what is public information about Twitter. I don’t know anything about their internal source code or anything like that.

Superficially, I’ve tried to track very closely the same kind of functionality that Twitter has. So my intention is to have someone who’s very familiar with Twitter’s function set be able to set up their own Laconica instance and be like, “I’ve been using Twitter for a while. I know how to add a friend. I know how to send direct messages. I know that I can get RSS feeds, etc..”

Also, early on, we cloned the Twitter API function-for-function. Twitter has a published API that they use, and there have been a number of desktop, web, and mobile clients that use this API. Because we cloned an identical API, that means third-party developers can simply change a few constants in their code and their code will work just as well with Identi.ca as it does with Twitter. In fact, it sometimes works a little better.

HW: How different is Laconica from Twitter?

EP: Under the covers, [Laconica] looks very different:

As far as I know, and again this is mostly hearsay, Twitter is implemented as a Ruby on Rails application [on] the web side. It also uses a number of off-line queue processing demons. So if you post a message on Twitter through the web site, the message goes into a queue, and the off-line processing demons, which I think are also written in Ruby, also process it. And they say, “This message goes to this person… this message might go out over IM… this message needs to go out over the web… this one’s going to get stored here… this one gets sent there.”

I believe they use a queuing manager called Starling, which, again, is written in Ruby. That is about the extent of my knowledge of the Twitter architecture.

Laconica’s architecture is a little bit different because we have different requirements. The principal difference in our environment is that I wanted to make sure that our software can be installed practically anywhere. You can install [Laconica] on a PHP and MySQL server, and it should work fine.

However, at the higher end, we have to support a big system like Identi.ca. At that level, we need to have a very similar architecture to the one that Twitter has, so requirements are to scale. When people talk about scaling, they usually talk about getting bigger. For Laconica, we need to cover the scale from the very small implementation to very big implementations.

HW: What are the challenges when it comes to developing or setting up a microblogging system?

EP: People expect this kind of system to have a utility-like reliability. If I’m sending out a message to all my friends, I would like to have that message go through quickly, efficiently, reliably. So it raises the level of requirements for getting real-time (or close to real-time) communication, and having everything working on lots of different protocols.

Each time that your number of messages and your number of actual communication message-sending points go up by squared as you get end users, your scale goes up very quickly. Your scale goes up very quickly end squared against end. You need to cover a lot of different channels; there’s lots of different protocols that you need to be working with.

HW: As a developer, what do you find technically interesting about microblogging?

EP: One of the things that’s been interesting for me is developing software for multiple protocols and multiple ways of speaking over the internet, mobile devices and routing. The success of our projects is going to depend on routing over as many channels as we possibly can.

Another thing I’ve found very interesting is the necessity for reliability. We need to be getting fast, quick service to people on a fairly large volume of messages. I’m learning a lot about building messaging architectures, trying to keep up with how our system is growing.

Fortunately for me, the open source community and open source development community have been forthcoming. Lots of people have been participating in our project. Rasmus Lerdorf did a code review on the software and came up with a lot of suggestions and ideas.

HW: What do you foresee will be the future of microblogging? Basically, is it just a fad, or will it evolve into something truly useful?

EP: I don’t know of many communication systems that have grown to this kind of popularity without eventually becoming fairly important. These kinds of markets grow for a reason and they meet a need that people have. There’s somewhere around 110 microblogging services out there, not counting about 100 Laconica instances. So I don’t think those services are serving a need that’s illusory. I think there’s something real there.

The fact that it turns communication into a worldwide conversation very quickly is really powerful and really forceful. You see in the conversations around the U.S. elections, the way that people respond to emergencies like the earthquake in California a few weeks ago, that there’s something here that people are looking for.

Where will it go in the future? I think we’ll probably see even more sharing of structured data, things like images, video, [and] audio. It’ll become a bit more like micro-podcasting or micro-videocasting. You might take a quick video, send it out to all your friends, and it gets to them wherever they are.

There’s also a good case to be made for using microblogging in [the] enterprise. When people in different departments in a company need to know what each is doing, it’s a great way to keep people up-to-date. I think corporate microblogging is going to become a big part of what people do in their company.

So I don’t think it’s just a fad. It’s going to be ubiquitous, and, if we’re careful and smart, we can make it into a protocol that everyone can use, that lots of different vendors can support and implement, and that we can build on top of, instead of having competing services.

devx-admin

devx-admin

Share the Post:
AI Software Development

ChatGPT is Now Making Video Games

Pietro Schirano’s foray into using ChatGPT, an AI tool for programming, has opened up new vistas in game and software development. As design lead at

Llama Codebot

Developers! Here’s Your Chatbot

Meta Platforms has recently unveiled Code Llama, a free chatbot designed to aid developers in crafting coding scripts. This large language model (LLM), developed using

Tech Layoffs

Unraveling the Tech Sector’s Historic Job Losses

Throughout 2023, the tech sector has experienced a record-breaking number of job losses, impacting tens of thousands of workers across various companies, including well-established corporations

Chinese 5G Limitation

Germany Considers Limiting Chinese 5G Tech

A recent report has put forth the possibility that Germany’s Federal Ministry of the Interior and Community may consider limiting the use of Chinese 5G

Modern Warfare

The Barak Tank is Transforming Modern Warfare

The Barak tank is a groundbreaking addition to the Israeli Defense Forces’ arsenal, significantly enhancing their combat capabilities. This AI-powered military vehicle is expected to

AI Software Development

ChatGPT is Now Making Video Games

Pietro Schirano’s foray into using ChatGPT, an AI tool for programming, has opened up new vistas in game and software development. As design lead at business finance firm Brex, Schirano

Llama Codebot

Developers! Here’s Your Chatbot

Meta Platforms has recently unveiled Code Llama, a free chatbot designed to aid developers in crafting coding scripts. This large language model (LLM), developed using Meta’s Llama 2 model, serves

Tech Layoffs

Unraveling the Tech Sector’s Historic Job Losses

Throughout 2023, the tech sector has experienced a record-breaking number of job losses, impacting tens of thousands of workers across various companies, including well-established corporations and emerging startups in areas

Chinese 5G Limitation

Germany Considers Limiting Chinese 5G Tech

A recent report has put forth the possibility that Germany’s Federal Ministry of the Interior and Community may consider limiting the use of Chinese 5G technology by local network providers

Modern Warfare

The Barak Tank is Transforming Modern Warfare

The Barak tank is a groundbreaking addition to the Israeli Defense Forces’ arsenal, significantly enhancing their combat capabilities. This AI-powered military vehicle is expected to transform the way modern warfare

AI Cheating Growth

AI Plagiarism Challenges Shake Academic Integrity

As generative AI technologies like ChatGPT become increasingly prevalent among students and raise concerns about widespread cheating, prominent universities have halted their use of AI detection software, such as Turnitin’s

US Commitment

US Approves Sustainable Battery Research

The US Department of Energy has revealed a $325 million commitment in the research of innovative battery types, designed to enable solar and wind power as continuous, 24-hour energy sources.

Netanyahu Musk AI

Netanyahu and Musk Discuss AI Future

On September 22, 2023, Israeli Prime Minister Benjamin Netanyahu met with entrepreneur Elon Musk in San Francisco prior to attending the United Nations. In a live-streamed discussion, Netanyahu lauded Musk

Urban Gardening

Creating Thriving Cities Through Urban Gardening

The rising popularity of urban gardening is receiving increased recognition for its numerous advantages, as demonstrated in a recent study featured in the Environmental Research Letters journal. Carried out by

What You Need to Know About Cloud Security Strategies

What You Need to Know About Cloud Security Strategies

Today, many businesses are adopting cloud computing services. As a result, it’s important to recognize that security measures for data in the cloud are different from those in traditional on-premises

Romanian Energy Security

Eastern Europe is Achieving Energy Security

Canada and Romania have solidified their commitment to energy security and independence from Russian energy exports by signing a $3-billion export development agreement. The deal is centered on constructing two

Seamless Integration

Unlocking Seamless Smart Home Integration

The vision of an intelligently organized and interconnected smart home that conserves time, energy, and resources has long been desired by many homeowners. However, this aspiration has often been hindered

New Algorithm

MicroAlgo’s Groundbreaking Algorithm

MicroAlgo Inc. has revealed the creation of a knowledge-augmented backtracking search algorithm, developed through extensive research in evolutionary computational techniques. The algorithm is designed to boost problem-solving effectiveness, precision, and

Poland Energy Future

Westinghouse Builds Polish Power Plant

Westinghouse Electric Company and Bechtel have come together to establish a formal partnership in order to design and construct Poland’s inaugural nuclear power plant at the Lubiatowo-Kopalino site in Pomerania.

EV Labor Market

EV Industry Hurting For Skilled Labor

The United Auto Workers strike has highlighted the anticipated change towards a future dominated by electric vehicles (EVs), a shift which numerous people think will result in job losses. However,

Soaring EV Quotas

Soaring EV Quotas Spark Battle Against Time

Automakers are still expected to meet stringent electric vehicle (EV) sales quotas, despite the delayed ban on new petrol and diesel cars. Starting January 2023, more than one-fifth of automobiles

Affordable Electric Revolution

Tesla Rivals Make Bold Moves

Tesla, a name synonymous with EVs, has consistently been at the forefront of the automotive industry’s electric revolution. The products that Elon Musk has developed are at the forefront because

Sunsets' Technique

Inside the Climate Battle: Make Sunsets’ Technique

On February 12, 2023, Luke Iseman and Andrew Song from the solar geoengineering firm Make Sunsets showcased their technique for injecting sulfur dioxide (SO₂) into the stratosphere as a means

AI Adherence Prediction

AI Algorithm Predicts Treatment Adherence

Swoop, a prominent consumer health data company, has unveiled a cutting-edge algorithm capable of predicting adherence to treatment in people with Multiple Sclerosis (MS) and other health conditions. Utilizing artificial

Personalized UX

Here’s Why You Need to Use JavaScript and Cookies

In today’s increasingly digital world, websites often rely on JavaScript and cookies to provide users with a more seamless and personalized browsing experience. These key components allow websites to display

Geoengineering Methods

Scientists Dimming the Sun: It’s a Good Thing

Scientists at the University of Bern have been exploring geoengineering methods that could potentially slow down the melting of the West Antarctic ice sheet by reducing sunlight exposure. Among these

why startups succeed

The Top Reasons Why Startups Succeed

Everyone hears the stories. Apple was started in a garage. Musk slept in a rented office space while he was creating PayPal with his brother. Facebook was coded by a