jebmo's posthaven

Warren Buffett: Bitcoin Is Pure FOMO

http://ift.tt/2DZFI2O

Heed Warren Buffett’s warning: bitcoin is pure FOMO - MarketWatch

To the surprise of nobody, billionaire investor Warren Buffett isn’t interested in bitcoin, the electronic currency that has zoomed higher in value over the past few weeks.

Bitcoin BTCUSD, -3.43% BTCF8, -2.69% is a complex idea. Simply put, it’s a virtual currency that is created, owned and traded entirely online in anonymous and unregulated settings. In theory, there is a limited number of these physically non-existent digital “coins,” though that limit hasn’t yet been reached. A few years ago, bitcoins were almost worthless; earlier in December, bitcoin’s valued briefly topped $19,000.

Buffett is having none of it. "You can't value bitcoin, because it's not a value-producing asset," he said recently. Earlier, in 2014, when bitcoin was worth much, much less, Buffett said, "Stay away from it. It's a mirage, basically,” adding, “The idea that it has some huge intrinsic value is just a joke, in my view."

So what drives the value of an essentially value-free asset? FOMO — fear of missing out.

The real danger

Who buys a home expecting it to double in value in a year? Who buys an unproven web startup certain that it will disrupt an entire industry in short order?

Think back to the housing bubble and, before that, the dot-com stock bubble. There was a time, early on, when these investments were nearly worthless.

Yet investors pounced on dot-coms and fixer-uppers. They feared missing out on a chance to get rich quickly.

As with dot-coms and real estate, the fuel driving the bitcoin fire is the continual entry of new investors, more and more people motivated by FOMO, fear of missing out. (Remember Beanie Babies? Exactly...)

This can happen to even large stocks, of course. Nearly every asset class has its heady moments. Yet stocks bounce back, over and over, thanks to Buffett’s notion of intrinsic value.

So what is intrinsic value? Imagine you buy stock in a company, say, Coca-Cola KO, +0.03% .

Huge numbers of people around the world consume Coke products. They pay cash, billions of dollars of cash, to buy and drink it. Whatever the value of drinking Coke might be, millions of people do so every day.

Now subtract the cost of making Coca-Cola, which is sugar, water and a few other ingredients, and you have a real stream of profits.

Waterfall of cash

Intrinsic value is this continuous waterfall of cash. It’s not the stock price of Coke but the actual cash flow of the business, after costs.

Buffett made his billions by divining when the gap is greatest between intrinsic value and a stock’s share price, then buying loads of shares, tickets to real cash flow other investors would want.

Given that bitcoin is supposed to replace cash, what is the ultimate source of cash flow from digital coins created on the internet? It’s dollars flowing from the pockets of buyers who want to own those coins.

Cut off the supply of new investors and the bitcoin craze ends.

Read: This is where we are in the life cycle of cryptocurrencies

The fact is, bitcoin has no intrinsic value at all. While many digital coin “investors” would argue that neither does a dollar, I counter that just about nobody thinks of American cash as an investment, except for perhaps currency speculators.

Rather, dollars are a temporary store of value, a means of transmitting that value from one person to another. As Buffett says, valuing bitcoin is like trying to value a paper check drawn on a bank. Pointless.

“But Mitch,” you might counter, “bitcoin is different. It’s a revolutionary new idea.” Sure, and it’s also an unregulated international marketplace rife with fraudulent actors. Just ask the Securities and Exchange Commission.

There is just one reason that bitcoin’s price currently is hovering around $15,000 rather than zero and that’s FOMO, plain and simple.

from Hacker News http://ift.tt/YV9WJO
via IFTTT

Sorry Tom Hanks, Steven Spielberg: If Daniel Ellsberg Is a Hero, So Is Edward Snowden

http://ift.tt/2lipkBU

If you need more proof that baby boomers, the generation born between 1946 and 1964, aren't quite up to living in the 21st century, look no further than this BBC interview with Tom Hanks, Steven Spielberg, and Meryl Streep, about the new movie The Post, which details the Washington Post brave and precedent-setting commitment to publishing the Pentagon's secret history of U.S. involvement in Vietnam, a.k.a. "The Pentagon Papers." Despite massive political and legal pressure not to do so, Ben Bradlee and Kay Graham (played by Hanks and Streep), pushed ahead with printing secret documents stolen by Daniel Ellsberg, winning a massive victory for press freedom and almost certainly shortening the Vietnam War.

Of course that was the right thing to do, Hanks tells Sam Asi. But when Asi brings up Edward Snowden, who in 2013 revealed massive, warrantless, secret surveillance of all sorts of electronic communications among American citizens, Hanks gets tongue-tied. Skip to about the five-minute mark:

After brushing aside the Snowden revelations—"I wasn't that surprised" that the government was spying, says Hanks—the Oscar winner plays social-media whataboutism. "Facebook makes money off of us based on what we're interested in," he says. "Do you think it's wrong that Google has an algorithm that can essentially you know, say, hey, Sam only likes black suits, so we're going to start sending you advertisements for black clothing." When asked whether Snowden is "a traitor or not," Hanks waves off the question, joking that it's about his "pay grade."

Asi makes the point that Hanks, along with Spielberg and the movie's other star Meryl Streep, were public supporters of Barack Obama, whose secret programs Snowden opposed, and Hillary Clinton, who called Snowden a traitor. He then asks Speilberg why Hollywood was silent during Obama's unprecedented reliance on the World War I-era Espionage Act. Obama pursued nine prosecutions, compared to just three since the law's start under Woodrow Wilson. Snowden's revelations were "different for me," says Spielberg, because Snowden simply had information about agencies with the capabilities of spying on individuals and infringing on our privacy, while "Daniel Ellsberg was trying to stop the Vietnam War." Ellsberg was a "hero," says Spielberg, who refuses to call Snowden by the same term. "I don't have the same information." Streep, speaking at the end of the clip, at least grants that it's good and "valid" that the programs and activities Snowden unmasked are now in public view.

Asi ends his segment by asking whether The Post is actually intended as a defense of a free press or "a warning" to President Trump that, you know, the media took down Richard Nixon, so watch out.

That's a provocative question precisely because Barack Obama was, in the words of one of the journalists prosecuted by him, "the greatest enemy of press freedom" to hold the Oval Office. Of course Hanks, Spielberg, and Streep are simply representative Hollywood liberals ready to give Democrats a pass regardless of their actual policies. But more important, they are also aging baby boomers who are quick to fetishize outlaw heroes of their youth while throwing shade on younger mavericks. To the extent that they are self-absorbed and morally certain about events from their own past, baby boomers are simply playing to type. Indeed, while promoting Saving Private Ryan, Spielberg reduced World War II to a footnote to the birth of his own generation, calling it the "key, the turning point of the whole century...It was as simple as this: The century either was going to produce the baby boomers or it was not going to produce the baby boomers."

Streep, Spielberg, Hanks|||20th Century Fox But such generational solipsism also renders them un-serious, if not next to useless, in a 21st-century in which Millennials now outnumber boomers. Although the circumstances in which Daniel Ellsberg leaked the Pentagon Papers will never be repeated exactly, there was nothing unique about what he did, and what's called for by all men and women of good will. What exactly has changed about the ways in which government, especially the surveillance state, functions since Lyndon Johnson opened up the "credibility gap" and today? In the wake of Vietnam, Watergate, the Church Committee, Iran-Contra, the revelations of William Binney, Thomas Drake, Chelsea Manning, Snowden, and others, is there any reason to believe that the federal government is not duplicitous? Not at all, and it's a shame to see three articulate, thoughtful people stumble on the altar of partisan loyalty when it comes to such a question. If they don't have enough information on which to evaluate Snowden, that's on them (read and watch his interview with Reason here).

To his credit, Daniel Ellsberg calls Snowden "a hero of mine." Speaking to Reason in our December 2017 issue, he said he hopes his legacy is something like this: "I would like others to believe that they have the power—and the obligation, really—as patriots, as human beings, to reveal what they themselves know are unjustified dangers to human existence. And not simply, for reasons of career and promises to superiors, to conceal dangers of that nature. In other words, to be truth tellers."

As a boomer myself (born near the end, in 1963), I understand that resisting generational and moral nostalgia is no easy thing. The past is past and it's easier and quieter not to worry so much about the current moment, I suppose. But for those of us who not only want to see the future but participate in it, keeping up is the first business of the day and partisanship should be left off the to-do list altogether. If Ellsberg is a hero (and he is), so too is Snowden and all the others who put their lives on hold in order to rein in powers that dare not show their hands.

from Hit & Run : Reason Magazine https://reason.com/blog
via IFTTT

Netflix: What Happens When You Press Play?

http://ift.tt/2jvTd4Q

This article is a chapter from my new book Explain the Cloud Like I'm 10. The first release was written specifically for cloud newbies. I've made some updates and added a few chapters—Netflix: What Happens When You Press Play? and What is Cloud Computing?—that level it up to a couple ticks past beginner. I think even fairly experienced people might get something out of it.

So if you are looking for a good introduction to the cloud or know someone who is, please take a look. I think you'll like it. I'm pretty proud of how it turned out.

I pulled this chapter together from dozens of sources that were at times somewhat contradictory. Facts on the ground change over time and depend who is telling the story and what audience they're addressing. I tried to create as coherent a narrative as I could. If there are any errors I'd be more than happy to fix them. Keep in mind this article is not a technical deep dive. It's a big picture type article. For example, I don't mention the word microservice even once :-)

Netflix seems so simple. Press play and video magically appears. Easy, right? Not so much.

Given our discussion in the What is Cloud Computing? chapter, you might expect Netflix to serve video using AWS. Press play in a Netflix application and video stored in S3 would be streamed from S3, over the internet, directly to your device.

A completely sensible approach…for a much smaller service.

But that’s not how Netflix works at all. It’s far more complicated and interesting than you might imagine.

To see why let’s look at some impressive Netflix statistics for 2017.

Netflix has more than 110 million subscribers.
Netflix operates in more than 200 countries.
Netflix has nearly $3 billion in revenue per quarter.
Netflix adds more than 5 million new subscribers per quarter.
Netflix plays more than 1 billion hours of video each week. As a comparison, YouTube streams 1 billion hours of video every day while Facebook streams 110 million hours of video every day.
Netflix played 250 million hours of video on a single day in 2017.
Netflix accounts for over 37% of peak internet traffic in the United States.
Netflix plans to spend $7 billion on new content in 2018.

What have we learned?

Netflix is huge. They’re global, they have a lot of members, they play a lot of videos, and they have a lot of money.

Another relevant factoid is Netflix is subscription based. Members pay Netflix monthly and can cancel at any time. When you press play to chill on Netflix, it had better work. Unhappy members unsubscribe.

We’re Going Deep

Netflix is a terrific example of all the ideas we’ve talked about, which is why this chapter goes into a lot more detail than the other cloud services we’ve covered.

One big reason for diving deeper into Netflix is they make much more information available than other companies.

Netflix holds communication as a central cultural value. Netflix more than lives up to its standards.

In fact, I’d like to thank Netflix for being so open about their architecture. Over the years, Netflix has given hundreds of talks and written hundreds of articles on the inner-workings of how they operate. The whole industry is better for it.

Another reason for going into so much detail on Netflix is Netflix is just plain fascinating. Most of us have used Netflix at one time or another. Who wouldn’t love peeking behind the curtain to see what makes Netflix tick?

Netflix operates in two clouds: AWS and Open Connect.

How does Netflix keep their members happy? With the cloud of course. Actually, Netflix uses two different clouds: AWS and Open Connect.

Both clouds must work together seamlessly to deliver endless hours of customer-pleasing video.

The three parts of Netflix: client, backend, CDN.

You can think of Netflix as being divided into three parts: the client, the backend, and the CDN.

The client is the user interface on any device used to browse and play Netflix videos. It could be an app on your iPhone, a website on your desktop computer, or even an app on your Smart TV. Netflix controls each and every client for each and every device.

Everything that happens before you hit play happens in the backend, which runs in AWS. That includes things like preparing all new incoming video and handling requests from all apps, websites, TVs, and other devices.

Everything that happens after you hit play is handled by Open Connect. Open Connect is Netflix’s custom global content delivery network (CDN). When you press play the video is served from Open Connect. Don’t worry; we’ll talk about what this means later.

Interestingly, at Netflix they don’t actually say hit play on video, they say clicking start on a title. Every industry has its own lingo.

By controlling all three areas—client, backend, CDN— Netflix has achieved complete vertical integration.

Netflix controls your video viewing experience from beginning to end. That’s why it just works when you click play from anywhere in the world. You reliably get the content you want to watch when you want to watch it.

Let’s see how Netflix makes that happen.

In 2008 Netflix Started Moving to AWS

Netflix launched in 1998. At first they rented DVDs through the US Postal Service. But Netflix saw the future was on-demand streaming video.

In 2007 Netflix introduced their streaming video-on-demand service that allowed subscribers to stream television series and films via the Netflix website on personal computers, or the Netflix software on a variety of supported platforms, including smartphones and tablets, digital media players, video game consoles, and smart TVs.

On a personal note, that streaming video-on-demand was the future might seem obvious. And it was. I worked at a couple of startups that tried to make a video-on-demand product. They failed.

Netflix succeeded. Netflix certainly executed well, but they were late to the game, and that helped them. By 2007 the internet was fast enough and cheap enough to support streaming video services. That was never the case before. The addition of fast, low-cost mobile bandwidth and the introduction of powerful mobile devices like smart phones and tablets, has made it easier and cheaper for anyone to stream video at any time from anywhere. Timing is everything.

Netflix Began by Running their Own Datacenters

EC2 was just getting started in 2007, about the same time Netflix’s streaming service started. There was no way Netflix could have launched using EC2.

Netflix built two datacenters, located right next to each other. They experienced all the problems we talked about in earlier chapters.

Building out a datacenter is a lot of work. Ordering equipment takes a long time. Installing and getting all the equipment working takes a long time. And as soon they got everything working they would run out of capacity, and the whole process had to start over again.

The long lead times for equipment forced Netflix to adopt what is known as a vertical scaling strategy. Netflix made big programs that ran on big computers. This approach is called building a monolith. One program did everything.

The problem is when you’re growing really fast like Netflix; it’s very hard to make a monolith reliable. And it wasn’t.

A Service Outage Caused Netflix to Move to AWS

For three days in August 2008, Netflix could not ship DVDs because of corruption in their database. This was unacceptable. Netflix had to do something.

The experience of building datacenters had taught Netflix an important lesson—they weren’t good at building datacenters.

What Netflix was good at was delivering video to their members. Netflix would rather concentrate on getting better at delivering video rather than getting better at building datacenters. Building datacenters was not a competitive advantage for Netflix, delivering video is.

At that time, Netflix decided to move to AWS. AWS was just getting established, so selecting AWS was a bold move.

Netflix moved to AWS because it wanted a more reliable infrastructure. Netflix wanted to remove any single point of failure from its system. AWS offered highly reliable databases, storage and redundant datacenters. Netflix wanted cloud computing, so it wouldn’t have to build big unreliable monoliths anymore. Netflix wanted to become a global service without building its own datacenters. None of these capabilities were available in its old datacenters and never would be.

A reason Netflix gave for choosing AWS was it didn’t want to do any undifferentiated heavy lifting. Undifferentiated heavy lifting are those things that have to be done, but don’t provide any advantage to the core business of providing a quality video watching experience. AWS does all the undifferentiated heavy lifting for Netflix. This lets Netflixians focus on providing business value.

It took more than eight years for Netflix to complete the process of moving from their own datacenters to AWS. During that period Netflix grew its number of streaming customers eightfold. Netflix now runs on several hundred thousand EC2 instances.

Netflix is More Reliable in AWS

It’s not like Netflix has never experienced down time on AWS, but on the whole, its service is much more reliable than it was before.

You don’t see complaints like this very often anymore:

or this:

Netflix is so reliable now because they’ve taken extraordinary steps to make their service reliable.

Netflix operates out of three AWS regions: one in North Virginia, one in Portland Oregon, and one in Dublin Ireland. Within each region, Netflix operates in three different availability zones.

Netflix has said there are no plans to operate out of more regions. It’s very expensive and complicated to add new regions. Most companies operate out of just one region, let alone two or three.

The advantage of having three regions is that any one region can fail, and the other regions will step in handle all the members in the failed region. When a region fails, Netflix calls this evacuating a region.

Let’s use an example. Let’s say you’re watching a new House of Cards episode in London England. Because it’s closest to London, chances are your Netflix device is connected to the Dublin region.

What happens if the entire Dublin region fails? Does that mean Netflix should stop working for you? Of course not!

Netflix, after detecting the failure, redirects you to Virginia. Your device would now talk to the Virginia region instead of Dublin. You might not even notice there was a failure.

How often does an AWS region fail? Once a month. Well, a region doesn’t actually fail every month. Netflix runs monthly tests. Every month Netflix causes a region to fail on purpose just to make sure its system can handle region level failures. A region can be evacuated in six minutes.

Netflix calls this their global services model. Any customer can be served out of any region. This is amazing. And it doesn’t happen automatically. AWS has no magic sauce for handling region failures or serving customers out of multiple regions. Netflix has done all this work on its own. Netflix is a pioneer in figuring out how to create reliable systems using multiple regions. I’m not aware of any other company that goes to these lengths to make their service so reliable.

Another advantage of being in these three regions is that it gives Netflix world-wide coverage. Netflix ran some tests and found if you use a Netflix application anywhere in the world, you’ll get fast service from one of these three regions.

Netflix Saves Money in AWS

This may surprise a lot of people, but AWS is cheaper for Netflix. The cloud costs per streaming view ended up being a fraction of the cost of its old datacenters.

Why? The elasticity of the cloud.

Netflix could add servers when it needed them and return them when it didn’t. Rather than have a lot of extra computers hanging around doing nothing just to handle peak load, Netflix only had to pay for what was needed, when it was needed.

All the stuff we talked about in the What is Cloud Computing? chapter.

What Happens in AWS Before you Press Play?

Anything that doesn’t involve serving video is handled in AWS.

This includes scalable computing, scalable storage, business logic, scalable distributed databases, big data processing and analytics, recommendations, transcoding, and hundreds of other functions.

Don’t worry, you don’t need to understand what all those things are, but since you may find it interesting, I’ll explain them briefly.

Scalable computing and scalable storage.

Scalable computing is EC2 and scalable storage is S3. Nothing new for us here.

Your Netflix device—iPhone, TV, Xbox, Android phone, tablet, etc.—talks to a Netflix service running in EC2.

View a list of potential videos to watch? That’s your Netflix device contacting a computer in EC2 to get the list.

Ask for more details about a video? That’s your Netflix device contacting a computer in EC2 to get the details.

It’s just like all the other cloud services we’ve talked about in the book.

Scalable distributed database.

Netflix uses both DynamoDB and Cassandra for their distributed databases. Not that these names should mean anything to you, they’re just high-quality database products.

Database. A database stores data. Your profile information, your billing information, all the movies you’ve ever watched, all that kind of information is stored in a database.

Distributed. Distributed means the database doesn’t run on one big computer, it runs on many computers. Your data is copied to multiple computers so if one or even two computers holding your data fail, your data will be safe. In fact, your data is copied to all three regions. That way, if a region fails your data will be there when the new region is ready to start using it.

Scalable. Scalable means the database can handle as much data as you ever want to put into it. That’s one major advantage of being a distributed database. More computers can be added as necessary to handle more data.

Big data processing and analytics.

Big data simply means there’s a lot of data. Netflix collects a lot of information. Netflix knows what everyone has watched when they watched it and where they were when they watched. Netflix knows which videos members have looked at but decided not to watch. Netflix knows how many times each video has been watched…and a lot more.

Putting all the data in a standard format is called processing.

Making sense of all that data is called analytics. Data is analyzed to answer specific questions.

Netflix personalizes artwork just for you.

Here’s a great example of how Netflix entices you to watch more videos using its data analytics capabilities.

When browsing around looking for something to watch on Netflix, have you noticed there’s always an image displayed for each video? That’s called the header image.

The header image is meant to intrigue you, to draw you into selecting a video. The idea is the more compelling the header image, the more likely you are to watch a video. And the more videos you watch, the less likely you are to unsubscribe from Netflix.

Here’s an example of different header images for Stranger Things:

You might be surprised to learn the image shown for each video is selected specifically for you. Not everyone sees the same image.

Everyone used to see the same header image. Here’s how it worked. Members were shown at a random one picture from a group of options, like the pictures in the above Stranger Things collage. Netflix counted every time the video was watched, recording which picture was displayed when the video was selected.

For our Stranger Things example, let’s say when the group picture in the center was shown, Stranger Things was watched 1,000 times. For all the other pictures, it was watched only once each.

Since the group picture was the best at getting members to watch, Netflix would make it the header image for Stranger Things forever.

This is called being data-driven. Netflix is known for being a data-driven company. Data is gathered—in this case, the number of views associated with each picture—and used to make the best decisions possible—in this case, which header image to select.

Clever, but can you imagine doing better? Yes, by using more data. That’s the theme of the future—solving problems by learning from data.

You and I are likely very different people. Do you think we are motivated by the same kind of header image? Probably not. We have different tastes. We have different preferences.

Netflix knows this too. That’s why Netflix now personalizes all the images they show you. Netflix tries to select the artwork highlighting the most relevant aspect of a video to you. How do they do that?

Remember, Netflix records and counts everything you do on their site. They know which kind of movies you like best, which actors you like the most, and so on.

Let’s say one of your recommendations is the movie Good Will Hunting. Netflix must choose a header image to show you. The goal is to show an image that lets you know about a movie you’ll probably be interested in. Which image should Netflix show you?

If you like comedies, Netflix will show you an image featuring Robin Williams. If you prefer romantic movies, Netflix will show you an image Matt Damon and Minnie Driver poised for a kiss.

By showing Robin Williams, Netflix is letting you know there’s likely to be humor in the movie and because Netflix knows you like comedies, this video is a good match.

The Matt Damon and Minnie Driver image conveys a completely different message. If you’re a comedy fan and saw this image, you might skip right on by.

That’s why selecting the right header image is so important. It sends a strong personalized signal indicating what a movie is about.

Here’s another example, Pulp Fiction.

If you’ve watched a lot of movies starring Uma Thurman, then you’re likely to see the header image featuring Uma. If you’ve watched a lot of movies starring John Travolta, then you’re likely to see the header image featuring John.

Can you see how choosing the best possible personalized artwork might make you more likely to watch a particular video?

Netflix appeals to your interests when selecting artwork, yet Netflix doesn’t want to lie to you either. They don’t want to show a clickbait image just to get you to watch a video you may not like. There’s no incentive in that. Netflix isn’t paid per video watched. Netflix tries to minimize regret. Netflix wants you to be happy with the videos you watch, so they pick the best header images they can for you.

This is just one small example of how data analysis is used by Netflix. Netflix uses these kind of strategies everywhere.

Recommendations.

Usually Netflix will show you only 40 to 50 video options, yet they have many thousands of videos available.

How does Netflix decide? Using machine learning.

That’s part of the big data processing and analytics we just talked about. Netflix looks at its data and predicts what you’ll like. In fact, everything you see see on a Netflix screen was chosen specifically for you using machine learning.

Transcoding From Source Media to What You Watch

Here’s where we start transitioning into how video is handled by Netflix.

Before you can watch a video on your favorite device of choice, Netflix must convert the video into a format that works best for your device. This process is called transcoding or encoding.

Transcoding is the process that converts a video file from one format to another, to make videos viewable across different platforms and devices.

Netflix encodes all its video in AWS on as many as 300,000 CPUs at one time. That’s larger than most super computers!

The source of source media.

Who sends video to Netflix? Production houses and studios. Netflix calls this video source media. The new video is given to the Content Operations Team for processing.

The video comes in a high definition format that’s many terabytes in size. A terabyte is big. Imagine 60 stacks of paper as tall as the Eiffel Tower. That’s a terabyte.

Before you can view a video, Netflix puts it through a rigorous multi-step process.

Validating the video.

The first thing Netflix does is spend a lot of time validating the video. It looks for digital artifacts, color changes, or missing frames that may have been caused by previous transcoding attempts or data transmission problems.

The video is rejected if any problems are found.

Into the media pipeline.

After the video is validated, it’s fed into what Netflix calls the media pipeline.

A pipeline is simply a series of steps data is put through to make it ready for use, much like an assembly line in a factory. More than 70 different pieces of software have a hand in creating every video.

It’s not practical to process a single multi-terabyte sized file, so the first step of the pipeline is to break the video into lots of smaller chunks.

The video chunks are then put through the pipeline so they can be encoded in parallel. In parallel simply means the chunks are processed at the same time.

Let’s illustrate parallelism with an example.

Let’s say you have one hundred dirty dogs that need washing. Which would be faster, one person washing the dogs one after another? Or would it be faster to hire one hundred dog washers and wash them all the same time?

Obviously, it’s faster to have one hundred dog washers working at the same time. That’s parallelism. And that’s why Netflix uses so many servers in EC2. They need a lot of servers to process these huge video files in parallel. It works too. Netflix says a source media file can be encoded and pushed to their CDN in as little as 30 minutes.

Once the chunks are encoded, they’re validated to make sure no new problems have been introduced.

Then the chunks are assembled back into a file and validated once again.

The result is a pile of files.

The encoding process creates a lot of files. Why? The end goal for Netflix is to support every internet-connected device.

Netflix started streaming video in 2007 on Microsoft Windows. Over time more devices were added—Roku, LG, Samsung Blu-ray, Apple Mac, Xbox 360, LG DTV, Sony PS3, Nintendo Wii, Apple iPad, Apple iPhone, Apple TV, Android, Kindle Fire, and Comcast X1.

In all, Netflix supports 2200 different devices. Each device has a video format that looks best on that particular device. If you’re watching Netflix on an iPhone, you’ll see a video that gives you the best viewing experience on the iPhone.

Netflix calls all the different formats for a video its encoding profile.

Netflix also creates files optimized for different network speeds. If you’re watching on a fast network, you’ll see higher quality video than you would if you’re watching over a slow network.

There are also files for different audio formats. Audio is encoded into different levels of quality and in different languages.

There are also files included for subtitles. A video may have subtitles in a number of different languages.

There are a lot of different viewing options for every video. What you see depends on your device, your network quality, your Netflix plan, and your language choice.

Just how many files are we talking about?

For The Crown, Netflix stores around 1,200 files!

Stranger Things season 2 has even more files. It was shot in 8K and has nine episodes. The source video files were many, many terabytes of data. It took 190,000 CPU hours to encode just one season.

The result? 9,570 different video, audio, and text files!

Let’s see how Netflix plays all that video.

Three Different Strategies for Streaming Video

Netflix has tried three different video streaming strategies its own small CDN; third-party CDNs; and Open Connect.

Let’s start by defining CDN. A CDN is a content distribution network.

Content for Netflix—is of course—the video files we discussed in the previous section.

Distribution means video files are copied from a central location, over a network and stored on computers located all over the world.

For Netflix, the central location where videos are stored is S3.

Why build a CDN?

The idea behind a CDN is simple: put video as close as possible to users by spreading computers throughout the world. When a user wants to watch a video, find the nearest computer with the video on it and stream to the device from there.

The biggest benefits of a CDN are speed and reliability.

Imagine you’re watching a video in London and the video is being streamed from Portland, Oregon. The video stream must pass through a lot of networks, including an undersea cable, so the connection will be slow and unreliable.

By moving video content as close as possible to the people watching it, the viewing experience will be as fast and reliable as possible.

Each location with a computer storing video content is called a PoP or point of presence. Each PoP is a physical location that provides access to the internet. It houses servers, routers, and other telecommunications equipment. We’ll talk more about PoPs later.

The First CDN Was Too Small

In 2007, when Netflix debuted its new streaming service, it had 36 million members in 50 countries, watching more than a billion hours of video each month, streaming multiple terabits of content per second.

To support the streaming service, Netflix built its own simple CDN in five different locations within the United States.

The Netflix video catalog was small enough at the time that each location contained all of its content.

The Second CDNs Were Too Big

In 2009, Netflix decided to use 3rd-party CDNs. Around this time, the pricing for 3rd-party CDNs was coming down.

Using 3rd-party CDNs made perfect sense for Netflix. Why spend all the time and effort building a CDN of your own when you can instantly reach the globe using existing CDN services?

Netflix contracted with companies like Akamai, Limelight, and Level 3 to provide CDN services. There’s nothing wrong with using third-party CDNs. In fact, pretty much every company does. For example, the NFL has used Akamai to stream live football games.

By not building out its own CDN, Netflix had more time to work on other higher priority projects.

Netflix put a lot of time and effort into developing smarter clients. Netflix created algorithms to adapt to changing networks conditions. Even in the face of errors, overloaded networks, and overloaded servers, Netflix wants members always viewing the best picture possible. One technique Netflix developed is switching to a different video source—say another CDN, or a different server—to get a better result.

At the same time, Netflix was also devoting a lot of effort into all the AWS services we talked about earlier. Netflix calls the services in AWS its control plane. Control plane is a telecommunications term identifying the part of the system that controls everything else. In your body, your brain is the control plane; it controls everything else.

Then Netflix thought it could do better by developing it’s own CDN.

Open Connect Was Just Right

In 2011, Netflix realized at its scale it needed a dedicated CDN solution to maximize network efficiency. Video distribution is a core competency for Netflix and could be a huge competitive advantage.

So Netflix started developing Open Connect, its own purpose-built CDN. Open Connect launched in 2012.

Open Connect has a lot of advantages for Netflix:

Less expensive. 3rd-party CDNs are expensive. Doing it themselves would save a lot of money.
Better quality. By controlling the entire video path—transcoding, CDN, clients on devices—Netflix reasoned it could deliver a superior video viewing experience.
More Scalable. Netflix has the goal of providing service everywhere in the world. Quickly supporting all those people while providing a quality video viewing experience required building its own system.

The 3rd-party CDNs must support users accessing any kind of content from anywhere in the world. Netflix has a much simpler job.

Netflix knows exactly who its users are because they must subscribe to Netflix. Netflix knows exactly which videos it needs to serve. Just knowing it only has to serve large video streams allows Netflix to make a lot of smart optimization choices other CDNs can’t make. Netflix also knows a lot about it members. The company knows which videos they like to watch and when they like to watch them.

With this kind of knowledge, Netflix built a really high-performing CDN. Let’s go into more details on how Open Connect works.

Open Connect Appliances

Remember how we said a CDN has computers distributed all over the world?

Netflix developed its own computer system for video storage. Netflix calls them Open Connect Appliances or OCAs.

Here’s what an early OCA installation in a site looked like:

There are many OCAs in the above picture. OCAs are grouped into clusters of multiple servers.

Each OCA is a fast server, highly optimized for delivering large files, with lots and lots of hard disks or flash drives for storing video.

Here’s what one of the OCA servers looks like:

There are several different kinds of OCAs for different purposes. There are large OCAs that can store Netflix’s entire video catalog. There are smaller OCAs that can store only a portion of Netflix’s video catalog. Smaller OCAs are filled with video every day, during off-peak hours, using a process Netflix calls proactive caching. We’ll talk more about how proactive caching works later.

From a hardware perspective, there’s nothing special about OCAs. They’re based on commodity PC components and assembled in custom cases by various suppliers. You could buy the same computers if you wanted to.

Notice how all Netflix’s computers are red? Netflix had their computers specially made to match their logo color.

From a software perspective, OCAs use the FreeBSD operating system and NGINX for the web server. Yes, every OCA has a web server. Video streams using NGINX. If none of these names make any sense, don’t worry, I’m just including them for completeness.

The number of OCAs on a site depends on how reliable Netflix wants the site to be, the amount of Netflix traffic (bandwidth) that is delivered from that site, and the percentage of traffic a site allows to be streamed.

When you press play, you’re watching video streaming from a specific OCA, like the one above, in a location near you.

For the best possible video viewing experience, what Netflix would really like to do is cache video in your house. But that’s not practical yet. The next best thing is to put a mini-Netflix as close to your house as they can. How do they do that?

Where does Netflix put Open Connect Appliances (OCAs)?

Netflix delivers huge amounts of video traffic from thousands of servers in more than 1,000 locations around the world. Take a look at this map of video serving locations:

Other video services, like YouTube and Amazon, deliver video on their own backbone network. These companies literally built their own global network for delivering video to users. That’s very complicated and very expensive to do.

Netflix took a completely different approach to building its CDN.

Netflix doesn’t operate its own network; it doesn’t operate its own datacenters anymore either. Instead, internet service providers (ISPs) agree to put OCAs in their datacenters. OCAs are offered free to ISPs to embed in their networks. Netflix also puts OCAs in or close to internet exchange locations (IXPs).

Using this strategy Netflix doesn’t need to operate its own datacenters, yet it gets all the benefits of being in a regular datacenter it’s just someone else’s datacenter. Genius!

Those last two paragraphs were pretty dense, so let’s break it down.

Using ISPs to build a CDN.

An ISP is your internet provider. It’s who you get your internet service from. It might be Verizon, Comcast, or thousands of other services.

The main point here is ISPs are located all around the world and they’re close to customers. By placing OCAs in ISP datacenters, Netflix is also all over the world, and close to its customers.

Using IXPs to build a CDN.

An internet exchange location is a datacenter where ISPs and CDNs exchange internet traffic between their networks. It’s just like going to a party to exchange Christmas presents with your friends. It’s easier to exchange presents if everyone is in one place. It’s easier to exchange network traffic if everyone is one place.

IXPs are located all over the world:

TeleGeography's Internet Exchange Map

Here’s what the London Internet Exchange looks like:

London Internet Exchange (LINX)

Drill down on those yellow fiber optic cables and what you’ll see is something like this from the AMS-IX Internet exchange point, in Amsterdam, Netherlands:

Wikimedia Commons

Each wire in the above picture connects one network to another network. That’s how different networks exchange traffic with each other.

An IXP is like a highway interchange, only using wires:

Wikimedia Commons

For Netflix, this is another win. IXPs are all over the world. So by putting their OCAs in IXPs, Netflix doesn’t have to run its own datacenters.

Video is Proactively Cached to OCAs Every Day

Netflix has all this video sitting in S3. They have all these video serving computers spread throughout the world. There’s just one thing missing: Video!

Netflix uses a process it calls proactive caching to efficiently copy video to OCAs.

What is a cache?

A cache is a hiding place, especially one in the ground, for ammunition, food and treasures.

You know how squirrels bury nuts for the winter?

Each location they bury nuts is a cache. During the winter, any squirrel can find a nut cache and chow down.

Arctic explorers sent small teams ahead to cache food, fuel and other supplies along the route they were taking. The larger team following behind would stop at every cache location and resupply.

Both the squirrels and Arctic explorers were being proactive; they were doing something ahead of time to prepare for later.

Each OCA is a video cache of what you’ll most likely want to watch.

Netflix caches video by predicting what you’ll want to watch.

Everywhere in the world, Netflix, knows to a high degree of accuracy what its members like to watch and when they like to watch it. Remember how we said Netflix was a data-driven company?

Netflix uses its popularity data to predict which videos members probably will want to watch tomorrow in each location. Here, location means a cluster of OCAs housed within an ISP or IXP.

Netflix copies the predicted videos to one or more OCAs at each location. This is called prepositioning. Video is placed on OCAs before anyone even asks.

This gives great service to members. The video they want to watch is already close to them, ready and available for streaming.

Netflix operates what is called a tiered caching system.

The smaller OCAs we talked about earlier are placed in ISPs and IXPs. These are too small to contain the entire Netflix catalog of videos. Other locations have OCAs containing most of Netflix’s video catalog. Still, other locations have big OCAs containing the entire Netflix catalog. These get their videos from S3.

Every night, each OCA wakes up and asks a service in AWS which videos it should have. The service in AWS sends the OCA a list of videos it’s supposed to have based on the predictions we talked about earlier.

Each OCA is in charge of making sure it has all the videos on its list. If an OCA in the same location has one of the videos it’s supposed to have, then it will copy the video from the local OCA. Otherwise, a nearby OCA with the video will be found and copied.

Since Netflix forecasts what will be popular tomorrow, there’s always a one day lead time before a video is required to be on an OCA. This means videos can be copied during quiet, off-peak hours, substantially reducing bandwidth usage for ISPs.

There’s never a cache miss in Open Connect. A cache miss would be asking for a specific video from an OCA and the OCA saying it doesn’t have it. Cache misses happen all the time on other CDNs because you can’t afford to copy content everywhere. Since Netflix knows all the videos it must cache, it knows exactly where each video is at all times. If a smaller OCA doesn’t have a video, then one of the larger OCAs is always guaranteed to have it.

Why doesn’t Netflix just copy all their video to every OCA in the world? Its video catalog is way too large to store everything at all locations. In 2013, the video catalog for Netflix was over 3 petabytes; I have no idea how large it is today, but I can only assume it’s significantly larger.

That’s why Netflix developed the method of choosing which videos to store on each OCA using data to predict what their members will want to watch.

Let’s take an example. House of Cards is a very popular show. Which OCAs should it be copied to? Probably every location because members worldwide will want to watch House of Cards.

What if a video isn’t as popular as House of Cards? Netflix decides which locations it should be copied to in order to best serve nearby member requests.

Within a location, a popular video like House of Cards is copied to many different OCAs. The more popular a video, the more servers it will be copied to. Why? If there was only one copy of a very popular video, streaming the video to members would overwhelm the server. As they say, many hands make light work.

A video isn’t considered live when it’s copied to just one OCA. Netflix wants to be able to play the same content at the same time everywhere in the world. Only when there are a sufficient number of OCAs with enough copies of the video to serve it appropriately, will the video be considered live and ready for members to watch.

Daredevil Season 2 in 2016, for example, was the first time Netflix released all episodes of a show, on all devices, in all countries, at the same time.

Hosting OCAs: What’s in it for ISPs?

Why would an ISP agree to put an OCA cluster inside their network? At first blush, it seems too generous, but you’ll be happy to know it’s rooted firmly in self-interest.

To understand why, we’ll need to talk about how networks work. I know throughout this book we’ve said cloud services are accessed over the internet. That’s not the case for Netflix, at least when watching a video. When using a Netflix app, it talks to AWS over the internet.

The internet is an interconnect of networks. You have an ISP that provides internet service. I get my internet service from Comcast. What that means is my house connects to Comcast’s network using a fiber optic cable. Comcast’s network is their network; it’s not the internet, the internet is something else.

Let’s say I want to do a Google search, and I type a query into my browser and hit enter.

My request to Google first flows over Comcast’s network. Google isn’t on Comcast’s network. At some point, my request has to go to Google’s network. That’s what the internet is for.

The internet connects Comcast’s network to Google’s network. There are these things called routing protocols that act like a traffic cop, directing where network traffic goes.

When my Google query is routed onto the internet it’s not on Comcast’s network anymore, and it’s not on Google’s network. It’s on what’s called the internet backbone.

The internet is woven together from many privately owned networks that choose to interoperate with each other. The IXPs we looked at earlier are one way networks connect with each other.

In the United States, here’s a map of the long haul fiber network:

InterTubes: A Study of the US Long-haul Fiber-optic Infrastructure

What Netflix has done with Open Connect is placed its OCA clusters inside the ISPs network. That means if I watch a Netflix video I’ll be talking to an OCA in Comcast’s network. All my video traffic is on Comcast’s network; it never hits the internet.

The key to scaling video delivery is to be as close to users as possible. When you’re doing that you’re not using the internet backbone. Requests are being satisfied on a local part of the network.

Why is this a good thing? Recall that we said Netflix already consumes more than 37% of the internet traffic in the United States. If ISPs didn’t cooperate, Netflix would use even more of the internet. The internet couldn’t handle all the video traffic. ISPs would have to add a lot more network capacity, and that’s expensive to build.

Right now, up to 100% of Netflix content is being served from within ISP networks. This reduces costs by relieving internet congestion for ISPs. At the same time, Netflix members experience a high-quality viewing experience. And network performance improves for everyone.

It’s a win-win.

Open Connect is Reliable and Resilient

Earlier we discussed how Netflix increased the reliability of its system by running out of three different AWS regions. The architecture of Open Connect accomplished the same goal.

What may not be immediately obvious is that the OCAs are independent of each other. OCAs act as self-sufficient video-serving archipelagos. Members streaming from one OCA are not affected when other OCAs fail.

What happens when an OCA fails? The Netflix client you’re using immediately switches to another OCA and resumes streaming.

What happens if too many people in one location use an OCA? The Netflix client will find a more lightly loaded OCA to use.

What happens if the network a member is using to stream video becomes overloaded? The same sort of thing. The Netflix client will find another OCA on a better performing network.

Open Connect is a very reliable and resilient system.

Netflix Controls the Client

Netflix handles failures gracefully because it controls the client on every device running Netflix.

Netflix develops its Android and iOS apps themselves, so you might expect them to control those. But even on platforms like Smart TVs, where Netflix doesn’t build the client, Netflix still has control because it controls the software development kit (SDK).

A SDK is a set of software development tools that allows the creation of applications. Every Netflix app makes requests to AWS and plays video using the SDK.

By controlling the SDK, Netflix can adapt consistently and transparently to slow networks, failed OCAs, and any other problems that might arise.

Finally: Here’s What Happens when You Press Play

It’s been a long road getting here. We’ve learned a lot. Here’s what we’ve learned so far:

Netflix can be divided into three parts: the backend, the client, and the CDN.
All requests from Netflix clients are handled in AWS.
All video is streamed from a nearby Open Connect Appliance (OCA) in the Open Connect CDN.
Netflix operates out of three AWS regions and can usually handle a failure in any region without members even noticing.
New video content is transformed by Netflix into many different formats so the best format can be selected for viewing based on the device type, network quality, geographic location, and the member’s subscription plan.
Every day, over Open Connect, Netflix distributes video throughout the world, based on what they predict members in each location will want to watch.

Here’s a picture of how Netflix describes the play process:

Now, let’s complete the picture:

You select a video to watch using a client running on some device. The client sends a play request, indicating which video you want to play, to Netflix’s Playback Apps service running in AWS.
We’ve not talked about this before, but a big part of what happens after you hit play has to do with licensing. Not every location in the world has a license to view every video. Netflix must determine if you have a valid license to view a particular video. We won’t talk about how that works—it’s really boring—but keep in mind it’s always happening. One reason Netflix started developing its own content is to avoid licensing issues. Netflix wants to release a show to everyone in the world all at the same time. Creating its own content is the easiest way for Netflix to avoid worrying about licensing problems.
Taking into account all the relevant information, the Playback Apps service returns URLs for up to ten different OCA servers. These are the same sort of URLs you use all the time in your web browser. Netflix uses your IP address and information from ISPs to identify which OCA clusters are best for you to use.
The client intelligently selects which OCA to use. It does this by testing the quality of the network connection to each OCA. It will connect to the fastest, most reliable OCA first. The client keeps running these tests throughout the video streaming process.
The client probes to figure out the best way to receive content from the OCA.
The client connects to the OCA and starts streaming video to your device.
Have you noticed when watching a video the picture quality varies? Sometimes it will look pixelated, and after awhile the picture snaps back to HD quality? That’s because the client is adapting to the quality of the network. If the network quality declines, the client lowers video quality to match. The client will switch to another OCA when the quality declines too much.

That’s what happens when you press play on Netflix. Who would have ever thought so simple a thing as watching a video was so complex?

from Hacker News http://ift.tt/YV9WJO
via IFTTT

How to Do Everything with Your New Nintendo Switch

http://ift.tt/2Db7xUS

Nintendo Switch How-To Guides - Nintendo Switch Wiki Guide - IGN

We have updated our PRIVACY POLICY and encourage you to read it by clicking here.

IGN uses cookies and other tracking technologies to customize online advertisements, and for other purposes. IGN supports the Digital Advertising Alliance principles. Learn More

from IGN Video Games http://ift.tt/1a4Rbxe
via IFTTT

It Seems Like Julian Assange Deleted His Twitter

http://ift.tt/2l5HCqb

It Seems Like Julian Assange Deleted His Twitter

@JulianAssange is now more.

For reasons unknown, the official Twitter account of Julian Assange, the leader of disgraced transparency organization Wikileaks, was been deleted earlier tonight for reasons unknown. The Internet Archive —or someone availing themselves of its services—appears to have ben preserving snapshots of Assange’s account once every hour since September 18 of this year. Based on that information, it’s likely the @JulianAssange account was deleted between 4am and 5am GMT.

The account does not appear to have been suspended by Twitter, and the most recent cached version of his account suggests he had not tweeted since Friday. Strangely, the official account of Wikileaks itself remains intact and has not commented on the disappearance of its founder from the platform.

We’ve reached out to Twitter (via email) and Wikileaks (via Twitter DM) for additional information and will update if we hear back. In the meantime, consider it an early Christmas present.

from Hacker News http://ift.tt/YV9WJO
via IFTTT

How I went from $100-an-hour programming to $X0,000-a-week consulting

http://ift.tt/QTx8bl

Clients Pay For Value, Not For Time

A few years ago, I was a much-put-upon grunt programmer at a large Japanese megacorp. I go home every Christmas to Chicago, so I was going to be in Chicago during December 2009.

I have an Internet buddy in Chicago named Thomas Ptacek. We met on Hacker News. He's the #1 poster by karma and I'm #2. Since we apparently share the same mental disease characterized by being totally unable to resist comment boxes, I decided to invite Thomas out to coffee. My agenda, such that it was, was to gossip about HN threads.

Thomas runs a very successful webapp security consultancy, Matasano. (Brief plug: they're hiring and if it weren't for this business thing I'd work there in a heartbeat: some of the smartest folks I know doing very, very interesting work which actually matters. If you can program they'll train you on the security stuff.)

Anyhow, after we got our coffee, Thomas invited me into their conference room. We talked shop for three hours: Thomas and his VP wanted to hear what I'd do to market their products and services offering. I had been writing about how I marketed Bingo Card Creator for a while, and started applying some of the lessons learned to their content creation strategy.

(The actual contents of the conversion are not 100% germane to the story, but I blogged a bit about it and Thomas posted his thoughts on HN. Long story short: programmers can do things which meaningfully affect marketing outcomes.)

At the end of the conversation, Thomas said something which, no exaggeration, changed my life.

Thomas: Some food for thought: If this hadn't been a coffee date, but rather a consulting engagement, I'd be writing you a check right now.

Me: Three hours at $100 an hour or whatever an intermediate programmer is worth would only be $300. Why worry about that?

Thomas: I got at least $15,000 of value out of this conversation.

You'll notice that I immediately thought the proposed transaction was time-for-money, but Thomas (the savvy business owner) saw the same conversation as an exchange of business-results-for-money. He correctly anticipated that Matasano would be able to take that advice and turn it into a multiple of $15,000. (They did, within two weeks, but that isn't my story to tell.)

This is, far and away, the most important lesson to learn as a consultant. People who are unsavvy about business, like me in 2009 or like most freelancers today, treat themselves like commodity providers of a well-understood service that is available in quantity and differentiated purely based on price. This is stunningly not the case for programming, due to how competitive the market for talent is right now, and it is even more acutely untrue for folks who can program but instead choose to offer the much-more-lucrative service "I solve business problems -- occasionally a computer is involved."

So after this conversation, I stopped saying "I don't think I could do that" when companies asked me to work with them... and I also stopped calling myself a programmer.

I could literally talk for several hours on properly pricing consulting services. Erm, strike that, I literally have. Ramit Sethi and I talked about it for a few hours (here and here, with transcript). It was also the topic of a recent podcast episode with my cohost Keith Perhac and Brennan Dunn, again available with transcript. Read those. People have told me that, in aggregate, they've made hundreds of thousands of dollars just by walking their rates up as a response to those interviews. (Brennan also writes a book about concrete strategies to do that. I bought a copy myself. It is worth your time.)

I wish I could tell you that after speaking to Thomas I had a sudden burst of enlightenment on this topic, but that would be a lie. When I started consulting several months later, I went straight back to $100 an hour "to get my feet wet", but did learn one thing from the experience: I charged weekly rates.

Charging Weekly: It Makes Everything Automatically Better

What's the difference between $100 an hour and $4,000 a week? Aren't they mathematically equivalent? No. Weekly billing strictly dominates hourly billing.

Weekly billing means you never waste time itemizing minute by minute invoices ("37 minutes: call with Bob about the new login page").
Weekly billing means you have uninterrupted schedulable consulting availability in weekly blocks, and non-billable overhead like prospecting or contract negotiations happens between the blocks (when you weren't billable anyhow) rather than during the workday (when, as an hourly freelancer, you are in principle supposed to be billing).
Weekly billing makes it easy to align units of work to quantifiable business goals, where those goals dwarf the rate charged.

Weekly billing also does wonderful things for pricing negotiations... because you'll stop having them. When I write a proposal for an engagement, I typically write a list of things we can do and my estimate for how many will fit into 1, 2, or 3 weeks. If clients don't have 3 weeks in the budget, we can compromise on scope rather than compromising on my rate.

If you quote hourly rates rather than weekly rates, that encourages clients to see you as expensive and encourages them to take a whack at your hourly just to see if it sticks. Think of anything priced per hour. $100 an hour is more than that costs, right? So $100 per hour, even though it is not a market rate for e.g. intermediate Ruby on Rails programmers, suddenly sounds expensive. Your decisionmaker at the client probably does not make $100 an hour, and they know that. So they might say "Well, the economy is not great right now, we really can't do more than $90." That isn't objectively true, the negotiator just wants to get a $10 win... and yet it costs you 10% of your income.

When you're charging weekly rates, the conversation goes something more like this: "So you don't have $12,000 in the budget for 3 weeks? OK. What is the budget? $10,000? Alright, what do you want us to cut?" You can then give the negotiator something to hang his cost-cutting hat on while still preserving your ability to charge your full rate in this engagement and all future engagements. (Word to the wise: no client, anywhere, likes giving up discounts after they've been given them. I have ridiculously successful client relationships where I, stupidly, cut them a discount years ago and I'm still paying for that decision.)

So I bet you're wondering how I got from $4,000 a week to $X0,000 a week. Sadly, no silver bullet: I just climbed a ladder of project importance, gradually (over ~20 engagements and ~2 years) accumulating wins and using each win to get me to the next rung of the ladder. Let's look at how.

Getting Clients: The Importance Of Social Proof

One of my first consulting clients was Fog Creek Software. They got in touch with me after having read my blog and forum comments for a while. We've done the odd gig together over the years, beginning with a relaunch of their marketing site for FogBugz (writeup here) and continuing with a very fun project that will be written up on their blog Very Soon Now (TM).

People occasionally tell me that my strategy is not replicable because I'm, air quotes, "Internet famous." Back when Fog Creek got in touch with me, my blog had a few hundred readers on a very good day. I never demonstrated (and don't possess) untouchable genius unmatched by anyone before or since... I simply talked openly about things that worked.

I always ask to follow a successful consulting engagement with a case study. My pitch is "This is a mutual win: you get a bit more exposure and I get a feather in my cap, for landing the next client." Case studies of successful projects with some of my higher profile consulting clients (like e.g. Fog Creek) helped me to get other desirable consulting clients. Very few clients turn down free publicity, particularly if you offer to do all the work in arranging it.

If you can't get a public case study (do all the work for them and just ask for their approval -- this makes it very easy to say "Yes, go for it."), there are intermediate options. One is to ask for them to just OK a one or two sentence testimonial about working together. Write this testimonial for them and ask if they want to make any changes to it.

Here's a bad testimonial: "Patrick is smart. We enjoy working with him. -- A Client" That testimonial does not resolve the #1 issue in your prospect's head: will this engagement make the business more than it costs?

Here's a much more persuasive testimonial:

Patrick's advice on starting a drip campaign for WPEngine was an epic win for us -- it permanently moved the needle on signups after just a week of work. And it's easy to measure and therefore to improve.

-- Jason Cohen, Founder & CEO, WPEngine

"Permanently moving the needle on signups" is one of the most persuasive things you could possibly offer a SaaS business. (It sure sounds better than "He wrote 8 emails. They were good!")

That testimonial is in Jason's own words, but I hinted in the direction of what would be useful for him to say. (The main change he did was striking a number that I had suggested.) Notice that this testimonial, even with that edit, remains specific, focused on a business result, and highly credible. In a discussion with a new prospect, their big worry is "Can Patrick do something valuable for me?" and "If I have him do it, will it generate a wildly positive result for the business?" Jason's testimonial assuages both worries.

If you can't get either a writeup or a public testimonial, at least ask for a private reference. I have a few from companies which I can't publicly mention as clients. They sometimes help sway people who are on-the-fence about hiring me, particularly companies which are at the upper end of places where I could reasonably work. (I'm being oblique here. Sorry, nature of the business. I can reasonably work for Fog Creek / WPEngine, and in fact have. I couldn't reasonably be entrusted with strategic level projects at, say, Google. There exist a few order-of-magnitude jumps in between those two. Periodically, I try making one of those order-of-magnitude jumps. When I successfully manage one, I will -- naturally -- ask to do a public case study.)

I occasionally joke that every time I get a new case study, my weekly rate gets another zero, but that is directionally accurate. My first rate was $4,000 a week. It is now several multiples of that, all justified by "A client very similar to you paid my prevailing rate and had $THIS_FABULOUS_RESULT. If I could do that for you, what would it be worth?", with rates moving up every time I was comfortable doing so. (Both in terms of "My pipeline is sufficiently stuffed such that losing engagements over price shock is not a problem" and "Even at 25% more than what I charge right now, I'm still confident customers will have a successful outcome from working with me.")

Scaling A Consulting Business

I don't do consulting full-time because I enjoy running my own businesses more. That said, there are a couple of ways to take a consultancy and scale it. They're fascinating, since you can layer so many different business models onto the base money-for-time offering.

If you do well by your clients, you'll soon have too much work to handle. You've got two options at this point: raising your rates and turning away work in excess of your capacity to deliver, or raising your capacity, typically by hiring people. (Working more hours has serious scaling challenges... another reason to do weekly rates, by the way, as many folks working hourly succumb to the temptation to bill "just another hour" and end up with miserable work/life balance.)

Anyhow, you can either hire other people on a per-engagement basis (a freelancer managing freelancers, like my good buddy Keith Perhac) or you can have full-time employees who you charge out to customers, like Brennan Dunn's Rails consultancy. We talked in-depth about these models in our podcast recently.

The big surprise to me, when I investigated this as an option, was that you have to be very careful to maintain your margins. I once had the bright idea, back when I was charging $5k a week, to bring in someone at $4k a week and pay them $80 a hour. $800 a week for doing nothing, right?

That's a catastrophically bad idea. In addition to having to cover your overhead, when you're sitting between a client and your employees' paychecks, you are absorbing significant risk of non-payment or delayed payment. Your payroll check needs to clear on Friday regardless of whether the customer has paid their invoice yet. There are occasionally terrifying cash flow swings in the consulting business... heck, that is practically the definition of the business.

Ooh, let's do story time. Earlier this year I got married. Weddings are expensive. You don't get any significant price break for buying two of them at once, which I did because my wife and I are from different countries and we wanted to include everybody. I recommend getting married, assuming you've found the right person, because how else are you going to have conversations like "You should pay $500 for a single balloon" and find yourself saying "That sounds perfectly reasonable, but only if we could pop the balloon immediately on receiving it."

One particular portion of the wedding cost $30k. I anticipated that $30k of receivables (work that had been done but hadn't been paid for yet) would turn into $30k of wire transfers prior to the wedding hall needing the money. They didn't. There was nothing unusual at all with this state of affairs: collecting invoices is like trying to dance the samba while simultaneously juggling, and I missed a ball or two. I ended up putting that on a credit card for a few weeks while waiting for my business to sort things out.

If you have two employees, you can easily have a wedding worth of cash go out the door every two weeks. Accordingly, you need to be better at cash flow management than I was. One way to do it is by charging your employees out at a rate substantially higher than you pay them, and then using the difference to build a cash-flow cushion. (Brennan recommends having $30k around per employee prior to scaling to your next hire. That strikes me as a great start.)

Concrete example: If you charge someone out at $4k a week, you should reasonably think of paying them on the order of $60 an hour (if they're freelancers).

Things get even more difficult if you hire full-time. You have to worry about maintaining utilization of your team -- i.e. always having work for them ready to go, because they get paid whether or not you've got them scheduled. In general, you should shoot for about 70% utilization, or them working 3 weeks a month. Let's say you hire an intermediate Rails developer at market rates: $8k per month. This costs you about $12k after you pay for taxes/health care/etc etc. (Employee benefits: welcome to being a business, please enjoy your stay.) You'll need to hire them out at in excess of $6k a week to accomplish that safely, at 70% utilization.

The economic returns to running a consultancy come largely from "leverage": being able to bill out your employees, as opposed to walking up your own rates as the principal. (Many principals find that their take-home goes down after hiring employees, at least for a while, while they iron out the kinks of cash flow and pipeline management.) After you're at 3 employees, even at a fairly healthy rate for yourself, you're probably making half your money on your billings and half on the margin between what your employees cost and what you successfully bill them for. This scales right on up, with the additional wrinkle that as one has more employees one spends more time on non-billable overhead (prospecting, HR, business administration, yadda yadda) and less on what you got into the business to do.

A necessary collorary to this: the principals of a technical consultancy do very well for themselves. I don't know if that is a secret but it certainly isn't well appreciated: nobody says Occupy Boutique Rails Consultancies, but the principals of them do end up in the 1%.

Hybridizing Consultancies With Product Businesses

Over the years I've seen a few people run product businesses concurrently with consultancies. (And I do it myself.) Some of the models are very interesting, but they're not obvious.

Probably the most common one is using the consulting revenues to underwrite product development. For example, 37signals and Fog Creek both started as web development consultancies. There are quite a few products produced along the same model these days: the principals hire themselves out at generous rates, only take modest salaries, use the difference to hire programmers at market rates, and have the programmers build the product they wish to sell. This has a lot to recommend it over e.g. bootstrapping the product from nothing (everyone gets a paycheck every two weeks) or e.g. taking investment (it is hard to close $20k in angel investment but easy to close $20k in consulting contracts, and if the product takes off, you can stop consulting but you can't conveniently forget ever having been invested in). Consulting also helps develop generic business skills (which are very applicable to product businesses) and helps expose you to the problems of companies which pay money for solutions to problems, which makes customer development for your products much easier. (You can even share sales channels.)

The recent course on lifecycle emails which I made was a variant of this: it is a productized consulting offering. Basically, I took an engagement that I've delivered five times now and said "How could I refactor this engagement such that I can deliver much of the value for near-zero marginal time investment?" The answer was collecting my expertise and experience in a package that customers could use to self-direct themselves through implementing the advice. It seems to have worked out really well for many customers: instead of paying me $X0,000 for a week they can pay $500 to hear much of the same advice. It worked out very, very well for my business, too: I got approximately 1~2 weeks of consulting revenue without needing 1~2 weeks of on-site availability, and I now know that it will work if I want to repeat the experiment.

Another popular offering for consulting firms is to offer training workshops. For example, if you are an expert in a particular field, many companies are willing to pay for you to train up internal experts at their company. You can either convince one client to pay for a 1-, 2-, or 3-day event, or sell tickets to one which you put on for yourself. Typical rates I see for internal training range from $5k to $15k per day depending on what is taught and to whom. For public training, price points in the $500 to $1,500 a ticket range are common. Why is that a hybrid product offering as opposed to being straight consulting with a different charging model? After you have the curriculum/courseware for the engagement mapped out, you can then scale delivery of it almost arbitrarily: the cost of a trainer to deliver a day-long workshop in e.g. optimizing Postgres performance is much cheaper than someone who can actually optimize Postgres performance at an arbitrary company, and they're (relatively) common. You could take your single expert's knowledge and then package it up in several dozen workshops delivered in-person or online, with delivery performed by folks hired for the purpose. Amy Hoy and Thomas Fuchs used to run in-person and then online Javascript performance training workshops. It worked out very well but the rest of their businesses basically ate their availability. Sometimes billing $25,000 for a day just isn't worth the opportunity costs. #firstworldproblems

Many consulting companies shut down their availability after successfully getting a product business going. That isn't a law of nature (Pivotal Labs' consulting arm survived Pivotal Tracker, for example), but after you've figured out scaling for a product business (highly non-trivial), they often scale so well that continuing with the consulting would be economically irrational.

Is This Interesting To You?

This is a bit off the beaten path for my writing, but a lot of you asked for it over email, and I seem to get strong responses when I post about it publicly. If you want to hear more like this, hit Reply and tell me what, specifically, you'd like to hear about. I tried to keep this fairly generic but if you want to hear a deep-dive into e.g. how to sell consulting clients on A/B testing or just want to hear what is in my bag-of-tricks I'm happy to oblige.

I also look forward to doing some more writing about software in the near future. If you've got a particular topic you want covered, drop me an email. I read them all and respond to most. (n.b. A belated Happy Halloween. Sorry for the few months of dead air -- six weeks of international travel plus the busy season for my business coincided, and that cut down a bit on my available time to write these.)

Until next time.

Regards,

Patrick McKenzie

P.S. Quick plug: Brennan Dunn has a podcast on consulting. They're at six episodes now and they're all fantastic and, moreover, actionable, like on how you can use content marketing as a lead generation tool. I'd definitely suggest listening to them at the gym or whatever.

from Hacker News http://ift.tt/YV9WJO
via IFTTT

BREAKING: Appeals court rules that Trump’s third travel ban is illegal

http://ift.tt/2kFZ6tI

The United States Court of Appeals for the 9th Circuit has ruled that Trump’s travel ban, the third he has proposed since becoming president, is illegal.

The third travel ban, issued in September, imposed “indefinite and significant restrictions and limitations on entry of nationals” from seven countries — Chad, Iran, Libya, North Korea, Syria, Venezuela, and Yemen.

From the decision:

The Government’s interpretation of 8 U.S.C. § 1182(f) not only upends the carefully crafted immigration scheme Congress has embodied in the INA, but it deviates from the text of the statute, legislative history, and prior executive practice as well. Further, the President did not satisfy the critical prerequisite Congress attached to his suspension authority: Before blocking entry, he must first make a legally sufficient finding that the entry of the specified individuals would be “detrimental to the interests of the United States.” The Proclamation once again conflicts with the INA’s prohibition on nationality-based discrimination in the issuance of immigrant visas.

The decision takes direct aim at Trump’s claim that, as President, he can do whatever he’d like with regard to banning people from any country for any reason.

The Government argues that the President, at any time and under any circumstances, could bar entry of all aliens from any country, and intensifies the consequences of its position by saying that no federal court—not a federal district court, nor our court of appeals, nor even the Supreme Court itself—would have Article III jurisdiction to review that matter because of the consular nonreviewability doctrine. Particularly in the absence of an explicit jurisdiction-stripping provision, we doubt whether the Government’s position could be adopted without running roughshod over the principles of separation of powers enshrined in our Constitution.

The court also takes on the xenophobia underlying Trump’s order:

In assessing the public interest, we are reminded of Justice Murphy’s wise words: “All residents of this nation are kin in some way by blood or culture to a foreign land.” Korematsu v. United States, 323 U.S. 214, 242 (Murphy, J., dissenting). It cannot be in the public interest that a portion of this country be made to live in fear.

The court issued a limited injunction blocking the ban as it applies to “foreign nationals who have a bona fide relationship with a person or entity in the United States.”

The full ruling is available here.

This is a breaking news post and will be updated.

from Yglesias http://ift.tt/2aOiKwJ
via IFTTT

Miss America pageant rocked by report of execs' emails ripping former winners

http://ift.tt/2kHudoA

Former Miss Americas call for resignation of organization's leadership - CNN

The report detailed email exchanges in which CEO Sam Haskell, current president Josh Randle, Lewis Friedman -- formerly a writer for MAO -- and board members Tammy Haddad and Lynn Weidner, wrote offensive emails about former Miss America winners, referencing their weight or their private sex lives.

Following the report, 49 former Miss Americas -- dating back to 1948 -- called for the resignation of MAO's leadership, including Haskell, Randle and Haddad.

Haddad is a former executive producer for CNN's "Larry King Live." That program stopped airing in 2010.

"As dedicated members of communities, businesses and families, and ambassadors for the Miss America program across the country, we stand firmly against harassment, bullying and shaming -- especially of women -- through the use of derogatory terms meant to belittle and demean," the former Miss Americans said in a statement.

"We collectively call for their immediate resignation."

Emails targeted former contestants

In some instances, Huffington Post reported, Haskell appeared to condone degrading comments others made about the women, or suggest that those comments were funny to him.

One example was an email from Friedman to Haskell, who wanted to refer to past pageant winners as "formers" instead of "forevers." Friedman said he'd changed "forevers" to "c****," to which Haskell replied: "Perfect...bahahaha"

The nonprofit organization, which hosts the competition, was founded in 1921, and claims to be a "movement of empowering young women everywhere to achieve their dreams," according to its website.

Mallory Hagan, who was crowned Miss America in 2013, was one of Haskell's targets, according to the report, which references one email exchange in which he and Friedman demean her for her weight after she won the pageant.

According to the report, Haskell was emailed a photo of Hagan and three other former pageant winners. "OMG she is huge ... and gross ... why does he want that?????" Haskell wrote, before forwarding the picture to Randle and repeating the comment.

"She's a healthy one!! Hahaha," Randle replied. Haddad also responded to the picture, saying Hagan was "barely recognizable."

In another exchange, Friedman and Haskell made derogatory comments about Hagan's sex life. "Ps. Are we four the only ones not to have f****d Mallory?" Friedman asked.

"It appears we are the only ones!" Haskell said.

In late 2015, Haddad also suggested Haskell hire a private investigator "to get something" on Hagan, because the CEO felt the former Miss America was "viciously and cruelly" attacking him, the report said.

At times, according to Huffington Post, the emails targeted Gretchen Carlson, the former Fox News anchor who won the pageant in 1989, when she declined to publicly defend Haskell after another pageant winner, Kate Shindle, released a book critical of the organization and the CEO's leadership.

Haddad, the Huffington Post reported, called Carlson a "snake" for refusing to attack Shindle.

Haddad told CNN Friday she was resigning, "effective immediately." She said she had told CEO Sam Haskell she planned to resign early this year.

"The women who put their hopes and dreams into this program are the best of America, as well as those who have worn the Miss America crown, traveling the country lifting hearts and spirits through their service above self," Haddad said.

CNN has attempted to reach Haskell, Randle, Friedman and Weidner for comment, but those efforts have been unsuccessful.

Miss America winner: Emails 'validated' me

On Friday, Hagan told CNN she wasn't "shocked" by the emails.

"I feel validated because for the longest time I had been trying to tell people that this thing is happening to me and I didn't have any proof and now I do," she said. "I feel a big weight off my shoulders."

Carlson -- who stunned the media world when she filed a suit in July 2016 against Roger Ailes, the late former Fox News chairman and CEO --

addressed the Huffington Post story on Twitter

"As a proud former Miss America & former member of (the) Board, I'm deeply saddened by disgusting statements about women attributed to leadership of MAO," she wrote. "No woman should be demeaned with such vulgar slurs. As I've learned, harassment and shaming of women is never acceptable."

"I still believe that Miss America has relevance and purpose in 2017 and beyond," Shindle, who CNN has attempted to reach for comment,

said in a statement posted on Twitter

. "But in order to achieve that purpose, the entire Board of Directors must immediately resign, including and especially Sam Haskell."

"Only then can the women of Miss America reclaim its rich history and catalyze what is a clearly necessary evolution."

MAO: Emails not representative

"The Miss America Organization ('MAO') is committed to upholding the highest standards of character and integrity in all facets of its national framework," the organization told CNN in a statement Friday. The statement says that after MAO's board of directors was made aware of "inappropriate language" in internal emails several months ago, the board "acted immediately, forming an investigative committee and retaining independent legal counsel to conduct a full investigation of the matter."

The statement adds the emails were "illegally procured," but do not represent the "character and integrity of MAO or its representatives."

When asked if MAO's statement also represented statements for the MAO employees named in the article, the spokeswoman for the pageant said its statement "comes from the organization but I have directed your inquiry to the other individuals."

Dick Clark Productions severed MAO ties months ago

Dick Clark Productions -- the iconic American production company that produces the Golden Globe Awards, American Music Awards and Billboard Music Awards, among others -- said the company severed ties with MAO months ago after becoming aware of portions of the emails the Huffington Post reported on.

"We were appalled by their unacceptable content and insisted, in the strongest possible terms, that the Miss America Organization board of directors conduct a comprehensive investigation and take appropriate action to address the situation," the production company said in a statement.

Soon after, Dick Clark Productions resigned its positions on the board and ended its relationship with MAO, the statement said.

The Huffington Post, citing sources close to Dick Clark executives, said nothing changed after Dick Clark Productions brought the emails to the attention of MAO's board, and that was when the production company ended its association with MAO.

CNN's Chloe Melas contributed to this report.

from CNN http://ift.tt/1d1low5
via IFTTT

How Facebook's Secret Unit Created Digital Propaganda Troll Armies To Influence Elections

http://ift.tt/2pdy0il

Authored by Shelley Kasli via GGI News,

Just days after GreatGameIndia exposed how American and Japanese companies could be hacking Indian elections, a recent Bloomberg report has revealed how a secret unit of Facebook has helped create troll armies for governments around the world including India for digital propaganda to influence elections. Under fire for Facebook Inc.’s role as a platform for political propaganda, co-founder Mark Zuckerberg has punched back, saying his mission is above partisanship.

But Facebook, it turns out, is no bystander in global politics. What he hasn’t said is that his company actively works with political parties and leaders including those who use the platform to stifle opposition—sometimes with the aid of “troll armies” that spread misinformation and extremist ideologies.

The initiative is run by a little-known Facebook global government and politics team led from Washington by Katie Harbath, a former Republican digital strategist who worked on former New York Mayor Rudy Giuliani’s 2008 presidential campaign as well as 2014 Indian elections.

Since Facebook hired Harbath to run their secret global goverment and politics unit three years later, her team has traveled the globe (including India helping political clients use the company’s powerful digital tools to create troll armies for digital propaganda.

In India (many other countries as well) the unit’s employees have become de facto campaign workers. And once a candidate is elected, the company in some instances goes on to train government employees or provide technical assistance for live streams at official state events.

In the U.S., the unit embedded employees in Trump’s campaign. In India, the company helped develop the online presence of Prime Minister Narendra Modi, who now has more Facebook followers than any other world leader.

At meetings with political campaigns, members of Harbath’s team sit alongside Facebook advertising sales staff who help monetize the often viral attention stirred up by elections and politics. They train politicians and leaders how to set up a campaign page and get it authenticated with a blue verification check mark, how to best use video to engage viewers and how to target ads to critical voting blocs. Once those candidates are elected, their relationship with Facebook can help extend the company’s reach into government in meaningful ways, such as being well positioned to push against regulations.

That problem is exacerbated when Facebook’s engine of democracy is deployed in an undemocratic fashion. A November report by Freedom House, a U.S.-based nonprofit that advocates for political and human rights, found that a growing number of countries are “manipulating social media to undermine democracy.” One aspect of that involves “patriotic trolling,” or the use of government-backed harassment and propaganda meant to control the narrative, silence dissidents and consolidate power.

In 2007, Facebook opened its first office in Washington. The presidential election the following year saw the rise of the world’s first “Facebook President” in Barack Obama, who with the platform’s help was able to reach millions of voters in the weeks before the election. The number of Facebook users surged around the Arab Spring uprisings in the Middle East around 2010 and 2011, demonstrating the broad power of the platform to influence democracy.

By the time Facebook named Harbath, the former Giuliani aide, to lead its global politics and government unit, elections were becoming major social-media attractions. Facebook began getting involved in electoral hotspots around the world.

Facebook has embedded itself in some of the globe’s most controversial political movements while resisting transparency. Since 2011, it has asked the U.S. Federal Election Commission for blanket exemptions from political advertising disclosure rules that could have helped it avoid the current crisis over Russian ad spending ahead of the 2016 election.

The company’s relationship with governments remains complicated. Facebook has come under fire in the European Union, including for the spread of Islamic extremism on its network. The company just issued its annual transparency report explaining that it will only provide user data to governments if that request is legally sufficient, and will push back in court if it’s not.

Facebook Troll Armies In India

India is arguably Facebook’s most important market recently edging out the U.S. as the company’s biggest. The number of users here is growing twice as fast as in the U.S. And that doesn’t even count the 200 million people who use the company’s WhatsApp messaging service in India, more than anywhere else on the globe.

By the time of India’s 2014 elections, Facebook had for months been working with several campaigns. Modi relied heavily on Facebook and WhatsApp to recruit volunteers who in turn spread his message on social media. Since his election, Modi’s Facebook followers have risen to 43 million, almost twice Trump’s count.

Within weeks of Modi’s election, Zuckerberg and Chief Operating Officer Sheryl Sandberg both visited India as it was rolling out a critical free internet service that was later curbed due to massive protests. Harbath and her team have also traveled here, offering a series of workshops and sessions that have trained more than 6,000 government officials.

As Modi’s social media reach grew, his followers increasingly turned to Facebook and WhatsApp to target harassment campaigns against his political rivals. India has become a hotbed for fake news, with one hoax story this year that circulated on WhatsApp leading to mob beatings resulting in several deaths. The nation has also become an increasingly dangerous place for opposition parties and reporters.

However its not just Modi or the Bharatiya Janata Party who has utilize Facebook’s services. The company says it offers the same tools and services to all candidates and governments regardless of political affiliation, and even to civil society groups that may have a lesser voice.

What is interesting is that Mark Zukerberg himself wants to be the President of US and has already employed in succession David Plouffe (campaign adviser to Barack Obama in 2008) and then Ken Mehlman (George Bush Jr.’s campaign adviser in 2004). He is currently employing Amy Dudley (Senator Tim Kaine’s former advisor), Ben LaBolt (Barack Obama’s former press adviser) and Joel Benenson (Hillary Clinton’s former campaign adviser in 2016).

How India’s Fake News Ecosystem Work

Most of us don’t give much thought to what we post on social media, and a lot of what we see on social media is pretty innocuous. However, it only seems that way at first glance. The truth is that what we post online has a frightening potential. According to recent research from the Pacific Northwest National Laboratory and the University of Washington, the things we post on social media could be utilized by software to predict future events – maybe even the next Prime Minister of India.

In a paper that’s just been published on Arxiv, the team of researchers found that social media can be used to “detect and predict offline events”. Twitter analysis can accurately predict civil unrest, for instance, because people use certain hashtags to discuss issues online before their anger bubbles over into the real world.

The most famous example of this came during the Arab Spring, when clear signs of the impending protests and unrest were found on social networks days before people took to the streets.

The reverse of this is also true. Meaning that anger can also be manufactured on social media & once it reaches an optimum level be targeted on to real life events on the streets as we have been witnessing since atleast a couple of years in India with cases of mob lynchings and such.

In India a massive fake news industry has sprung up exercising influence over traditional discourse of politics and has a potential in becoming a security challenge like the Arab Spring if not kept in check. As the debate over mob lynching in India is raging it should be understood that such incidents would not have had such a rapid and massive effect if the youth had not had access to Facebook, Twitter, YouTube, and other social media that allowed the fake news industry to organise and share made-up videos and information. The mob lynching since the past years are a direct result of the fake news industry spilling over from social media to the real world.

This takes a totally new dimension now that it has been revealed that Facebook & WhatsApp itself colluded with the establishment in creating such “troll arimes” for digital propaganda, resulting directly into violence on Indian soil. This is a clear textbook case of terrorism. Terrorism is defined as ‘the systematic use of terror or violence by any individual or group to achieve political goals’. In this case this terrorism is perpetrated by a foreign company Facebook on Indian soil using digital information warfare. What more are we waiting for to respond to such an act?

Fake News was used very effectively during US Presidential elections. It was part of the official campaign itself run in collaboration with tech companies and it is also being alleged that even the Russians also ran their own network. The same method was used to shape the Brexit debate as well. As we write this the fake news industry is spreading its tentacles in India as well. Many of India’s leading sportsmen, celebrities, economists, politicians have already fallen victim to this by disseminating such fake content. This is a dangerous trend and should be kept in check by our intelligence agencies to avert future disaster.

The way it works in short is like this. Numerous websites and portals of varying degree of legitimacy and funding are floated. Specific news contents are generated for different groups based on their region, ideology, age, religion etc. which is mixed with a heavy dose of soft porn to slowly blend in with their objective. These fake contents are than peddled in social media and specific groups targeted via analytics tools developed by tech companies. As a lot of such fake content is generated slowly it starts getting a momentum of itself and somewhere down the line it is picked up by any unsuspected person of influence – celebrities, politicians and even journalists themselves. What happens after this point is sheer madness.

Whether by choice or by ignorance even the mainstream media starts peddling this nonsense, dedicating their entire primetime news shows in analyzing the fake news, who said what and why and blah blah… instead of identifying where the fake content was generated in the first place and getting it shut. Due to the nature and sensationalism of the generated content and also because its echoed by persons of influence with time this fake worldview has the potential to spill over in the real world with physical casualties, as we have seen in so many lynching cases. If not kept in check it could capture and take over the entire national discourse. We will reach a point where it will be very difficult to keep track of what is fact or fiction and the entire society would be radicalized into different opposing camps all based on lies.

Facebook & Indian Elections

What is more is that the Election Commission of India itself has partnered with Facebook for voter registration during election process. Dr.Nasim Zaidi, Chief Election Commissioner, Election Commission of India, said “I am pleased to announce that the Election Commission of India is going to launch a ‘Special Drive to enrol left out electors, with a particular focus on first time electors. This is a step towards fulfilment of the motto of ECI that ‘NO VOTER TO BE LEFT BEHIND’. As part of this campaign, Facebook will run a voter registration reminder in multiple Indian languages to all the Facebook users in India. I urge all eligible citizens to enrol and VOTE i.e. Recognize your Right and Perform your Duty. I am sure Facebook will strengthen Election Commission of India’s enrolment campaign and encourage future voters to participate in the Electoral Process and become responsible Citizens of India.”

All 17 American Intelligence agencies have raised serious concern about the impact of this fake news industry on their election process and their society. According to a study conducted by the Pew Research Center a majority of Americans (a whopping 88%) believe that completely made-up news has left Americans confused about even the basic facts. And we in India are heading towards a worst scenario than this. Why? Because unlike India, the US Government and Intelligence community has publicly addressed this issue and are working towards resolving this menace. Will Indian govt address such meddling by Facebook in India’s internal affairs?

Committees after committees are being setup, senate hearings are being convened to reach to the bottom of this and new Units are being created to effectively counter this threat to their society. While Facebook is being investigated for meddling in US Presidential elections not much focus has been given to how Facebook’s secret unit has influenced elections in India. In light of these revelations Facebook’s interference in India’s elections should be thoroughly investigated. Ofcourse in order to do so first the Govt should have to acknowledge the existence of this fake news industry in order to take action against it.

Along with Facebook, American Microchip Inc and Japanese Renesas contracted by Election Commission for fusing secret EVM code should also be investigated for interfering in India’s elections and all those who colluded with them. It will be a grave mistake to take this threat of foreign companies interfering in India’s elections lightly.

from Zero Hedge http://ift.tt/qouXdu
via IFTTT

A Zero-Math Introduction to Markov Chain Monte Carlo Methods

http://ift.tt/2BYqyNb

For many of us, Bayesian statistics is voodoo magic at best, or completely subjective nonsense at worst. Among the trademarks of the Bayesian approach, Markov chain Monte Carlo methods are especially mysterious. They’re math-heavy and computationally expensive procedures for sure, but the basic reasoning behind them, like so much else in data science, can be made intuitive. That is my goal here.

So, what are Markov chain Monte Carlo (MCMC) methods? The short answer is:

MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space.

In this article, I will explain that short answer, without any math.

First, some terminology. A parameter of interest is just some number that summarizes a phenomenon we’re interested in. In general we use statistics to estimate parameters. For example, if we want to learn about the height of human adults, our parameter of interest might be average height in in inches. A distribution is a mathematical representation of every possible value of our parameter and how likely we are to observe each one. The most famous example is a bell curve:

Courtesy M. W. Toews

In the Bayesian way of doing statistics, distributions have an additional interpretation. Instead of just representing the values of a parameter and how likely each one is to be the true value, a Bayesian thinks of a distribution as describing our beliefs about a parameter. Therefore, the bell curve above shows we’re pretty sure the value of the parameter is quite near zero, but we think there’s an equal likelihood of the true value being above or below that value, up to a point.

As it happens, human heights do follow a normal curve, so let’s say we believe the true value of average human height follows a bell curve like this:

Clearly, the person with beliefs represented by this graph has been living among giants for years, because as far as they know, the most likely average adult height is 6'2" (but they’re not super confident one way or another).

Lets imagine this person went and collected some data, and they observed a range of people between 5" and 6". We can represent that data below, along with another normal curve that shows which values of average human height best explain the data:

In Bayesian statistics, the distribution representing our beliefs about a parameter is called the prior distribution, because it captures our beliefs prior to seeing any data. The likelihood distribution summarizes what the data is telling us, by representing a range of parameter values accompanied by the likelihood that each each parameter explains the data we are observing. Estimating the parameter value that maximizes the likelihood distribution is just answering the question: what parameter value would make it most likely to observe the data we have observed? In the absence of prior beliefs, we might stop there.

The key to Bayesian analysis, however, is to combine the prior and the likelihood distributions to determine the posterior distribution. This tells us which parameter values maximize the chance of observing the particular data that we did, taking into account our prior beliefs. In our case, the posterior distribution looks like this:

Above, the red line represents the posterior distribution. You can think of it as a kind of average of the prior and the likelihood distributions. Since the prior distribution is shorter and more spread out, it represents a set of belief that is ‘less sure’ about the true value of average human height. Meanwhile, the likelihood summarizes the data within a relatively narrow range, so it represents a ‘more sure’ guess about the true parameter value.

When the prior the the likelihood are combined, the data (represented by the likelihood) dominate the weak prior beliefs of the hypothetical individual who had grown up among giants. Although that individual still believes the average human height is slightly higher than just what the data is telling him, he is mostly convinced by the data.

In the case of two bell curves, solving for the posterior distribution is very easy. There is a simple equation for combining the two. But what if our prior and likelihood distributions weren’t so well-behaved? Sometimes it is most accurate to model our data or our prior beliefs using distributions which don’t have convenient shapes. What if our likelihood were best represented by a distribution with two peaks, and for some reason we wanted to account for some really wacky prior distribution? I’ve visualized that scenario below, by hand drawing an ugly prior distribution:

Visualizations rendered in Matplotlib, enhanced using MS Paint

As before, there exists some posterior distribution that gives the likelihood for each parameter value. But its a little hard to see what it might look like, and it is impossible to solve for analytically. Enter MCMC methods.

MCMC methods allow us to estimate the shape of a posterior distribution in case we can’t compute it directly. Recall that MCMC stands for Markov chain Monte Carlo methods. To understand how they work, I’m going to introduce Monte Carlo simulations first, then discuss Markov chains.

Monte Carlo simulations are just a way of estimating a fixed parameter by repeatedly generating random numbers. By taking the random numbers generated and doing some computation on them, Monte Carlo simulations provide an approximation of a parameter where calculating it directly is impossible or prohibitively expensive.

Suppose that we’d like to estimate the area of the follow circle:

Since the circle is inside a square with 10 inch sides, the area can be easily calculated as 78.5 square inches. Instead, however, we can drop 20 points randomly inside the square. Then we count the proportion of points that fell within the circle, and multiply that by the area of the square. That number is a pretty good approximation of the area of the circle.

Since 15 of the 20 points lay inside the circle, it looks like the circle is approximately 75 square inches. Not too bad for a Monte Carlo simulation with only 20 random points.

Now, imagine we’d like to calculate the area of the shape plotted by the Batman Equation:

Here’s a shape we never learned an equation for! Therefore, finding the area of the bat signal is very hard. Nevertheless, by dropping points randomly inside a rectangle containing the shape, Monte Carlo simulations can provide an approximation of the area quite easily!

Monte Carlo simulations aren’t only used for estimating the area of difficult shapes. By generating a lot of random numbers, they can be used to model very complicated processes. In practice, they’re used to forecast the weather, or estimate the probability of winning an election.

The second element to understanding MCMC methods are Markov chains. These are simply sequences of events that are probabilistically related to one another. Each event comes from a set of outcomes, and each outcome determines which outcome occurs next, according to a fixed set of probabilities.

An important feature of Markov chains is that they are memoryless: everything that you would possibly need to predict the next event is available in the current state, and no new information comes from knowing the history of events. A game like Chutes and Ladders exhibits this memorylessness, or Markov Property, but few things in the real world actually work this way. Nevertheless, Markov chains are powerful ways of understanding the world.

In the 19th century, the bell curve was observed as a common pattern in nature. (We’ve noted, for example, that human heights follow a bell curve.) Galton Boards, which simulate the average values of repeated random events by dropping marbles through a board fitted with pegs, reproduce the normal curve in their distribution of marbles:

Pavel Nekrasov, a Russian mathematician and theologian, argued that the bell curve and, more generally, the law of large numbers, were simply artifacts of children’s games and trivial puzzles, where every event was completely independent. He thought that interdependent events in the real world, such as human actions, did not conform to nice mathematical patterns or distributions.

Andrey Markov, for whom Markov chains are named, sought to prove that non-independent events may also conform to patterns. One of his best known examples required counting thousands of two-character pairs from a work of Russian poetry. Using those pairs, he computed the conditional probability of each character. That is, given a certain preceding letter or white space, there was a certain chance that the next letter would be an A, or a T, or a whitespace. Using those probabilities, Markov was ability to simulate an arbitrarily long sequence of characters. This was a Markov chain. Although the first few characters are largely determined by the choice of starting character, Markov showed that in the long run, the distribution of characters settled into a pattern. Thus, even interdependent events, if they are subject to fixed probabilities, conform to an average.

For a more useful example, imagine you live in a house with five rooms. You have a bedroom, bathroom, living room, dining room, and kitchen. Lets collect some data, assuming that what room you are in at any given point in time is all we need to say what room you are likely to enter next. For instance, if you are in the kitchen, you have a 30% chance to stay in the kitchen, a 30% chance to go into the dining room, a 20% chance to go into the living room, a 10% chance to go into the bathroom, and a 10% chance to go into the bedroom. Using a set of probabilities for each room, we can construct a chain of predictions of which rooms you are likely to occupy next.

Making predictions a few states out might be useful, if we want to predict where someone in the house will be a little while after being in the kitchen. But since our predictions are just based on one observation of where a person is in the house, its reasonable to think they won’t be very good. If someone went from the bedroom to the bathroom, for example, its more likely they’ll go right back to the bedroom than if they had come from the kitchen. So the Markov Property doesn’t usually apply to the real world.

Running the Markov chain for thousands of iterations, however, does give the long-run prediction of what room you’re likely to be in. More importantly, this prediction isn’t affected at all by which room the person began in! Intuitively, this makes sense: it doesn’t matter where someone is in the house at one point in time in order to simulate and describe where they are likely to be in the long-term, or in general. So Markov chains, which seem like an unreasonable way to model a random variable over a few periods, can be used to compute the long-run tendency of that variable if we understand the probabilities that govern its behavior.

With some knowledge of Monte Carlo simulations and Markov chains, I hope the math-free explanation of how MCMC methods work is pretty intuitive.

Recall that we are trying to estimate the posterior distribution for the parameter we’re interested in, average human height:

I am not a visualization expert, nor apparently am I any good at keeping my example within the bounds of common sense: my example of the posterior distribution seriously overestimates average human height.

We know that the posterior distribution is somewhere in the range of our prior distribution and our likelihood distribution, but for whatever reason, we can’t compute it directly. Using MCMC methods, we’ll effectively draw samples from the posterior distribution, and then compute statistics like the average on the samples drawn.

To begin, MCMC methods pick a random parameter value to consider. The simulation will continue to generate random values (this is the Monte Carlo part), but subject to some rule for determining what makes a good parameter value. The trick is that, for a pair of parameter values, it is possible to compute which is a better parameter value, by computing how likely each value is to explain the data, given our prior beliefs. If a randomly generated parameter value is better than the last one, it is added to the chain of parameter values with a certain probability determined by how much better it is (this is the Markov chain part).

To explain this visually, lets recall that the height of a distribution at a certain value represents the probability of observing that value. Therefore, we can think of our parameter values (the x-axis) exhibiting areas of high and low probability, shown on the y-axis. For a single parameter, MCMC methods begin by randomly sampling along the x-axis:

Red points are random parameter samples

Since the random samples are subject to fixed probabilities, they tend to converge after a period of time in the region of highest probability for the parameter we’re interested in:

Blue points just represent random samples after an arbitrary point in time, when convergence is expected to have occurred. Note: I’m stacking point vertically purely for illustrative purposes.

After convergence has occurred, MCMC sampling yields a set of points which are samples from the posterior distribution. Draw a histogram around those points, and compute whatever statistics you like:

Any statistic calculated on the set of samples generated by MCMC simulations is our best guess of that statistic on the true posterior distribution.

MCMC methods can also be used to estimate the posterior distribution of more than one parameter (human height and weight, say). For n parameters, there exist regions of high probability in n-dimensional space where certain sets of parameter values better explain observed data. Therefore, I think of MCMC methods as randomly sampling inside a probabilistic space to approximate the posterior distribution.

Recall the short answer to the question ‘what are Markov chain Monte Carlo methods?’ Here it is again as a TL;DR:

MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space.

I hope I’ve explained that short answer, why you would use MCMC methods, and how they work. The inspiration for this post was a talk I gave as part of General Assembly’s Data Science Immersive course in Washington, DC. The goals of that talk were to explain Markov chain Monte Carlo methods to a non-technical audience, and I’ve tried to do the same here. Leave a comment if you think this explanation is off the mark in some way, or if it could be made more intuitive.

from Hacker News http://ift.tt/YV9WJO
via IFTTT

In 2008 Netflix Started Moving to AWS

Netflix Began by Running their Own Datacenters

A Service Outage Caused Netflix to Move to AWS

Netflix is More Reliable in AWS

Netflix Saves Money in AWS

What Happens in AWS Before you Press Play?

Transcoding From Source Media to What You Watch

Three Different Strategies for Streaming Video

Why build a CDN?

The First CDN Was Too Small

The Second CDNs Were Too Big

Open Connect Was Just Right

Open Connect Appliances

Where does Netflix put Open Connect Appliances (OCAs)?

Video is Proactively Cached to OCAs Every Day

Hosting OCAs: What’s in it for ISPs?

Open Connect is Reliable and Resilient

Netflix Controls the Client

Finally: Here’s What Happens when You Press Play

Related Articles

Clients Pay For Value, Not For Time

Charging Weekly: It Makes Everything Automatically Better

Getting Clients: The Importance Of Social Proof

Scaling A Consulting Business

Hybridizing Consultancies With Product Businesses

Is This Interesting To You?

Emails targeted former contestants

Miss America winner: Emails 'validated' me

MAO: Emails not representative

Dick Clark Productions severed MAO ties months ago

Facebook Troll Armies In India

How India’s Fake News Ecosystem Work

Facebook & Indian Elections