Thursday, December 16, 2010

The dirty tricks that monopolists play and why Level 3 needs to get into last mile access

[It is interesting to watch the dirty little games that Comcast is playing in the US to protect its video delivery monopoly and protect themselves from “Over the Top” (OTT) competitors like Google and Netflix.
I suspect they are not alone in this activity and other cablecos and telcos in North America’s duopoly are contemplating doing the same thing. For those who have been following the Comcast-Level 3 dispute, which is fundamentally a lot more than simply an argument about peering, the surprising news was that Comcast charges Content Distribution Networks (CDN) for access to their last mile network. Up to now most network providers welcomed CDNs to locate their equipment at their POPs at no charge, because CDNs significantly reduced their Internet transit costs. But to add insult to injury not only does Comcast charge CDN providers they also run their Internet transit links almost at 100% capacity as noted in a very interesting e-mail exchange on the NANOG (North American Network Operators Group) list. This insures that any over the top distributor, as well as any other provider, who has not paid Comcast’s shakedown fees is subject to significant packet loss and poor performance of delivery of their product to Comcast’s users. This is classic monopoly rent extortion, pure and simple. Don Vito Corleone would be proud.

While I am a firm believer that network neutrality is essential, I have little faith that our regulatory bodies will be able to accomplish anything meaningful in this area – not for lack of will, but because of the huge lobbying and legal obstacles that the incumbents will throw in their path. That is why I have long argued for the need for a National Public Internet (NPI) in the same vein that National Public Radio (NPR) and other national broadcasters provide an independent view from the “FOXification” of mainstream news channels. But in addition to NPI we still need a private sector solution to balance the incredible concentration of market power now exuded by Comcast-NBC and other North American content –pipe conglomerates.

I have always been an admirer of Level 3 and their business strategy as a carrier’s carrier (despite their current financial woes). But what I think they need to do is extend this business model to the last mile. As Beniot Felten has noted in his blog ( ) and many studies, fiber to the home only works financially if you can carry multiple service providers. Structural separation surprisingly makes sense from both a regulatory and business case. But as long as you can maintain your duopoly there is no compelling reason to move to this business model.

Unfortunately structurally separate FTTH is hugely capital intensive with a small rate of return (if any), especially if you are competing against the duopoly of cablecos and telcos. However there are business models that have succeeded in this market, especially where there is strong regulatory pressure. For example, in The Netherlands, pension funds are now investing in structurally separate FTTH projects and there are many companies that specialize in this business. The pension funds have recognized that once you have structural separation with multiple service providers, the investment return is not spectacular but solid and consistent, year after year. If one service provider stumbles for whatever reason, your ROI is protected by the returns from the other service providers.

Another business model that I have long advocated is to bundle FTTH with energy resale. Given that Level 3’s roots are in the energy business another possible business model is to partner with companies like Google and large energy companies to deploy FTTH as an “energy play” rather than as an telecom or Internet bundle. Not only would this provide a entire new revenue stream but it also might help address the biggest challenge facing the planet – global warming. For more details please see


Some noteworthy links:

European Investment Bank and 5 commercial banks support Reggefiber high speed optical fibre network in the Netherlands with EUR 285m loan

The roll out of a very high speed optical fibre network in the Netherlands received a significant boost today with the agreement of a EUR 285m bank loan for open access network provider Reggefiber. The loan from the European Investment Bank, the European Union’s long-term lending institution, and five commercial banks facilitates the further rollout of next generation fibre optic broadband services. The investment in the rollout of the network, with up and download speeds of more than 100 megabits-per-second, is in line with the EU’s goal to increase high-speed Internet access to half of all households by 2020.

Reggefiber’s open network model makes it possible for different telecom operators and service providers to bring high speed broadband applications and multi-room HD digital TV to 1.1 – 1.3 million homes across the country by 2012.


Some truth about Comcast - WikiLeaks style

Ever wonder what Comcast's connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.

1 day graphs:

Image #1:
Image #1 (Alternate Site):

Image #2:
Image #2 (Alternate Site):

Notice how those graphs flat-line at the top? That's because they're completely full for most of the day. If you were a Comcast customer attempting to stream Netflix via this connection, the movie would be completely unwatchable. This is how Comcast operates: They intentionally run their IP transit links so full that Content Providers have no other choice but to pay them (Comcast) for access. If you don't pay Comcast, your bits wont make it to their destination. Though they wont openly say that to anyone, the content providers who attempt to push bits towards their customers know it. Comcast customers however have no idea that they're being held hostage in order to extort money from content.

Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it's being used as an excuse to call foul of Level (3) and other content heavy networks. But this shoulnd't surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of the 21st century.

If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...

Image #3:
Image #3 (Alternate Site):

Comcast needs to be truthful with its customers, regulators and the public in general. The Level (3) incident only highlights the fact that Comcast is pinching content and backbone providers to force them to pay for uncongested access to Comcast customers. Otherwise, there's no way to send traffic to Comcast customers via the other paths on the Internet without hitting congested links.

Remember that this is not TATA's fault, Comcast is a CUSTOMER of TATA. TATA cannot force Comcast to upgrade its links, Comcast elects to simply not purchase enough capacity and lets them run full. When Comcast demanded that Level (3) pay them, the only choice Level (3) had was to give in or have its traffic (such as Netflix) routed via the congested TATA links. If Level (3) didn't agree to pay, that means Netflix and large portions of the Internet to browse would be simply unusable for the majority of the day for Comcast subscribers.

The big squeeze

The reaction of the pay-TV/programming industry to Netflix is fascinating. You can hear them trumpeting that the Netflix model doesn’t stand a chance - and you can see them squeezing Netflix in order to make sure that prediction comes true.
Squeeze No. 1 - acting collectively, the media oligopolists can charge an enormous amount to independent aggregators of online video.
Today’s Times quotes from Jeffrey Bewkes were choice. Mr. Bewkes has great faith in the solidarity of his industry. Netflix made a relatively-inexpensive deal with Starz in 2008 that began the process of getting Netflix into the streaming business in a big way. Such a deal!
It won’t happen again, Mr. Bewkes made clear to the Times. It won’t be possible, because Netflix won’t have enough money to pay what programmers and studios will charge - now that the content industry has wised up to Netflix’s possibly destructive role. “This has been an era of experimentation [with Netflix],” Mr. Bewkes told the Times, “and I think it’s coming to a close.”
Squeeze No. 2 - usage based pricing will discourage consumers from subscribing to independent video aggregators.
When the FCC Chairman on December 1 announced that he was circulating a proposed “open Internet” order, he said that usage based pricing would be a good idea: “Our work has also demonstrated the importance of business innovation to promote network investment and efficient use of networks, including measures to match price to cost such as usage-based pricing.”
Consumers strongly prefer flat-fee pricing for information goods, because they don’t want to worry about whether the next transaction they do online forces them into a higher pricing tier. Consumers regularly over-estimate how much transmission they will need, and want to insure themselves against surprisingly-large bills. Indeed, consumers are willing to pay more for flat-rate plans than for metered ones. For similar behavioral reasons, any kind of barrier to usage discourages usage. If something costs more based on use, we consumers will use it less. If we have a flat pricing plan, we’ll feel more comfortable and use it more.
If you want to discourage consumers from accessing independent online video, one way to do that is to ensure that it feels expensive to do it.
Comcast has a cap on Internet use of 250 gigabytes a month and says it doesn’t currently have any plans to move to usage-based pricing. But it will have the freedom to do so. If it combines that freedom with (”pay for priority”) tiers of service, it can migrate all the same slicing/dicing/bundling abilities it has in the pay-TV world - which are just about unlimited - into the online world. Its flexibility to tier would allow it to treat Netflix differently and apply a lower usage-limit to Netflix traffic; its flexibility to bundle and price its cable TV packages with its high-speed Internet packages, combined with usage caps, would allow it to deter some consumers from cutting the cord.
Craig Moffett of Bernstein Research saw the connection between usage-based pricing and the cable industry infrastructure immediately: “Usage-based pricing will preserve, and even enhance, the economics of cable’s infrastructure. . . even if consumers eventually get some, or even all, of their video content over the Web,” he wrote.
Squeeze No. 3 - make it expensive for interconnecting networks to reach your customers - so that those networks pass on the cost to Netflix.
Warm recommendation that you read Adam Rothschild’s post on this. “Comcast purchases commodity IP transit service from Tata, as a means of reaching networks it doesn’t maintain direct peering relationships with. Unlike Level 3 though, Comcast runs its ports to Tata at capacity, deliberately, as a means of degrading connectivity to networks which won’t peer with them or pay them money.” Users will gradually become disappointed with sites that don’t work as well for them, and will stick with those that do - those that pay networks to pay Comcast.
That’s a lot of squeeze. And we haven’t even started on the devices - we’ll think about AllVid tomorrow.

Internet Access Should Be Application-Agnostic

Julius Genachowski, the Chairman of the FCC, recently announced that he would ask the FCC to adopt rules to protect the open Internet at its open meeting on December 21st. We applaud the Chairman's effort, but we worry the proposed framework, as it is currently drafted, will not result in the free and open Internet that is his goal.

The proposed rule has several problems:
1. it prohibits only unjust and unreasonable discrimination but does not clearly define those terms,
2. broadband access providers are not prohibited from charging web services like Google, Facebook or Twitter a fee to reach consumers or to get faster access to consumers, and
3. users who access the Internet over wireless networks have few protections.
If these concerns are not addressed, access providers could use their ability to control access to the Internet to control the market for Internet applications and services.
I remember too well, the experience of investing in cable television programming start-ups back in the 90s when there was limited channel capacity on cable networks and the companies that controlled access to consumers made it very clear that they would need to own 20% of your company before they would agree to carry your programming on their network. The Internet we know today exists only because, until now, there have been no gatekeepers between consumers and service providers. We need to keep it that way.
The good news is that the FCC can balance the interests of web services innovators and consumers with those of telephone and cable companies without changing the substance of the proposed rule simply by defining application-specific discrimination as unreasonable.
Barbara van Schewick, a professor at the Stanford Law School, describes this approach here. She says the correct approach is:
"A non-discrimination rule that would ban all application-specific discrimination (i.e. discrimination based on applications or classes of applications), but would allow application-agnostic discrimination."
The brilliance of this approach is that it offers cable and telephone companies great flexibility to package and price their services and to manage their networks without harming investment and innovation in web services.
If a user wants more packets or less latency, an access provider should be able to sell that to them. But for that access service to meet the test of being application-agnostic, the choice of when to use these services and for which applications must be left to the user.
Similarly, if a user consumes a disproportionate share of packets at certain times of day, a network provider should be able to temporarily reduce that user's throughput to avoid degrading the experience of others. These actions would not threaten a free and open Internet because they are targeted at a consumer's use of network capacity, not a specific application.
On the other hand, if access providers throttled only the bandwidth available to BitTorrent to deal with congestion, that would clearly be application-specific discrimination. Blocking or throttling video would be discrimination against a class of applications.

This approach works equally well for wireless.
If an older wireless network does not have the capacity to handle lots of packets at peak times, it can reasonably limit the number of packets available to users. When congestion is eased it can open up the pipe again.
This is reasonable network management that does not distort the competitive market for web services. Blocking or discriminating against a specific web service like Skype or against a whole class of web services like streaming video would be prohibited under this framework.
If it is not possible to solve all network management problems on older wireless networks in an application-agnostic way, there could be an exception; but the presumption should be that network management would be as application-agnostic as possible.
If cable and telephone companies intend to use their control over consumer's access to the Internet to extract outsize profits from the innovative companies working in the dynamic and competitive market for Internet services, it should be pretty clear to the FCC that they cannot reconcile their interests with those of consumers and innovators. If, on the other hand, access providers are, as they say, concerned only about their ability to invest in their network and manage it responsibly, they will support this application-agnostic regulatory framework.
This is not just a problem for venture capital investors. There is a great post here that summarizes all of these issues from an entrepreneur's perspective.

We believe it is in everyone's interest to improve the current proposal by:
• defining any application-specific discrimination as unreasonable,
• extending that reasonableness test to include wireless Internet access, and
• making it clear that pay-to-play access fees (whether for access to users or faster access to users) are prohibited.
If you agree, we encourage you to write to Chairman Genachowski, Commissioner Copps, and, Commissioner Clyburn, Commisioner McDowell, and Commissioner Baker and urge them to work together to make this modest but important change before bringing the rule to a vote later this month.

We also encourage your you to exercise your own authority and influence, using the services that you use everyday to let the FCC know you understand the problem and will support their effort to create an application-agnostic regulatory framework:

1. Post your thoughts on your own blog.
2. Tell your friends on Facebook
3. Post this on Tumblr, and
4. Tweet this to your followers on Twitter.

A Scary Picture for the Future of the Wireless Web

The fight over Net Neutrality -- that fundamental principle that keeps the Internet open and free from discrimination -- can get pretty wonky.

It's sometimes hard to find the right words when you're trying to communicate to policymakers, geeks and the general public at the same time. How do crucial issues like "paid prioritization" harm the open Internet? What are the dangers of "specialized services"?

But a picture can be worth, well, you know. That's why I always liked this clever illustration about what an Internet without Net Neutrality would look like:

To most of us, that looks like a nightmare. But to big companies like AT&T and Verizon, it's a business plan.

Even as the Federal Communications Commission weighs new Net Neutrality rules, the dominant phone companies and their vendors are plotting a dystopian future for consumers that you won't see in their soft-focus ads.

But you will see it soon enough.

Tollbooth Policy

On Tuesday, Allot and Openet -- two companies that specialize, respectively, in "deep packet inspection" technology and billing software -- presented their plan to give wireless operators new ways to nickel-and-dime customers while dictating the development of mobile apps. I got my hands on a copy of their slides.

Their presentation unveiled a new hardware and software package that will enable mobile wireless providers to block, prioritize and discriminate against Internet content based on its source. The two companies -- one of which, Openet, already counts AT&T and Verizon as clients -- boasted that their new "policy control" solution "enables operators to monetize their networks, with a focus on online content-based charging of [mobile] applications."

One slide prepared by the companies depicts how a customer of a wireless company like Vodafone (or any company that deploys their product) would be charged different discriminatory rates based on the source of Internet content. The slide suggests that consumers can access content free of charge from their carrier, while paying 2 cents per MB for Facebook, $3 per month for Skype, and 50 cents per month for the privilege of accessing YouTube. Here's the illustration:

Look familiar?

That's not all. Another slide discusses how mobile operators will be able to implement "application-specific pricing," and "tiered services." Still another presents the benefits of "split billing." Imagine your customer is watching a movie on his iPhone, the vendors explain: "First 15 minutes of the movie [would be] streamed for free to user as a promotion. If user doesn't purchase movie, content provider is billed for the 15 minutes of network consumption. If user purchases movie, revenue is shared between operator and content provider."

That's their vision of a pay-to-play, metered-down-to-the-millisecond wireless future. Such blatant Net Neutrality violations are far from hypothetical: The scenarios clearly reflect the industry's active desire to engage in such behavior. In fact, the press release announcing the presentation stated that it was being conducted "due to the great interest shown in this joint solution."

Problem, Meet Solution

How can they get away with this, you ask?

Well, remember that the wireless market is increasingly uncompetitive. The two biggest carriers -- AT&T and Verizon -- already control more than 60 percent of the market and have locked down devices like the iPhone that you might actually want to use to go online. The top four companies account for 90 percent of the market.

The lack of competition and choice suggests we might need rules to keep these companies from exploiting customers and stifling innovation. But that brings us back to the FCC -- which according to numerous press reports is planning to exempt mobile wireless operators from Net Neutrality protections.

The schemes proposed by Allot and Openet will be the very real-world consequences of such a short-sighted decision by the FCC. Mobile wireless services will never be viable competitors to the phone-cable duopoly if their future looks like this, and these discriminatory practices will destroy the mobile Internet as an open platform for innovation.

When these companies talk to the press and politicians, they pinky swear that they would never interfere with the free-flowing Internet; they say that Net Neutrality is a "solution in search of a problem," and they claim "self-regulation" is enough to keep them in line. But of course these slides show (yet again) that when carriers talk to investors or among themselves, they tell a completely different story.

What will it take for the FCC to wake up? We already know that underserved communities and people of color are more likely to use mobile wireless services as their primary Internet connection. Having glimpsed at this version of the mobile Internet's future, does the FCC really want to adopt rules that create open networks only for those Americans who can afford them and to relegate low-income, minority and rural Americans to an Internet without free speech protections and cutting-edge applications?

We'll have a clearer picture of where the FCC stands next week when it meets to vote on Net Neutrality rules, but right now it doesn't look so pretty.

twitter: BillStArnaud
skype: Pocketpro