Collecting the Internet So You Don't Have To

We work on the Internet. As such, we are constantly consuming information. Believe me, there is a lot of it out there. Sometimes we even forget things unless we write them down. Our blog covers everything from web standards to the muppets, php to comic books, music and everything else that we find interesting. Leave us a note when you drop by.

Shaw Customer Consultation Session

Technology
Dwayne Kristjanson
Dwayne Kristjanson Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:154
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Dwayne Kristjanson Indifference Engine

I attended a Shaw Customer Consultation Session on March 28th. It was somewhat informative but since Shaw's stated purpose was to gather data they could use to decide what type of billing model to introduce in the future, I didn't find out anything about what their plans are. Officially at this point they don't seem to have any. The key take-away phrase was "we're open to considering anything". Shaw had wanted to get 40 people per session, but they didn't get anywhere near that number. I didn't do a proper head-count, but there were perhaps 15 customers present and six or seven representatives from Shaw. Apparently this type of turnout is typical for these consultation sessions, although the earlier ones did have somewhat higher numbers. I suspect that there was a heavy selection bias in place, anyone who didn't really care one way or another about UBB might not bother to come. All of the other people I spoke with had requested to attend, rather than being asked by Shaw. That might not be representative, though, I only asked two or three of those present, and the Shaw representatives did indicate that they invited a cross section of light, medium, and heavy users to attend.

The session was supposed to be under "Chatham House Rule", according to the email I received. Which apparently means I can discuss what was said, but not who said it. I suspect roughly identifying Shaw vs. customer points is fair game. For the most part, Shaw seemed to be there to listen in any case.

The session itself:

The session began with an admission by the Shaw representatives that they'd screwed up in implementing Usage Based Billing. A large part of that was the timing, they started rolling out UBB right around the same time as the CRTC decision for Bell and the subsequent public outcry. I'm pretty sure that this particular decision, however, is not directly relevant to Shaw charging or not charging based on bandwidth usage. I believe the cable companies won the right to charge residential customers this way nearly a year ago in a separate decision. Unfortunately I can't find a handy reference right now. If they'd had UBB in place for several months when that happened, they could have simply shrugged it off and pointed out that their customers hadn't minded until then. That, or have already dealt with a backlash of their own. A lot would depend on  how many people were hit by the charges and for how much.

After the opening remarks things opened up to questions from the floor. A lot of questions were asked, and the Shaw people were fairly open with their answers, but didn't really have any hard numbers. People did want to know where the $1 or $2 per GB UBB charges had come from. The other customers present, like myself, thought those values seemed much higher than necessary. Shaw acknowledged that the wholesale cost per gigabyte had come down in recent years, and that this number seemed to have plateaued, but said that this cost alone couldn't be used to determine the true cost of bandwidth. I suspect that's true, but without access to a lot of data from Shaw it's not possible for anyone else to calculate what a fair cost of bandwidth would be. Hopefully they will release those numbers at some point, but I'm not going to hold my breath.

One of the main goals of Shaw was to receive feedback on possible alternative models they could use for billing. They indicated that since their current projections are for continued exponential growth in bandwidth usage, they forsee issues with all levels of their network. What is currently considered heavy use will eventually be considered normal use, and peak usage hours are widening over time. There was some joking that eventually peak hours will be 24 hours a day. That's probably an overstatement, but it's easy to see that bandwidth usage will only go up as we move forward. As Shaw sees it, there is no silver bullet to fix the issues they're facing short of spending more money on improving the network. Which in turn means collecting more money from their customers, since the number of internet subscribers isn't growing nearly as fast as bandwidth usage. Which makes sense, nearly everyone has broadband these days. Usage is going up because the existing users are using more, not because there are more users using the same amount.

The general consensus among the customers present was that any problems with Shaw's network were almost certainly due to peak usage. Shaw needs to build and maintain their network for peak traffic, and it's increases in peak traffic that will cause the most problems for other customers and incur the most additional costs for Shaw. No one seemed to think that per GB usage billing would directly affect this. 

Suggested payment models:

  • Shaw's original proposed plans but with rollover GB: Similar to Shaw's initial proposal, but allow rolling over unused GB from a previous month.
  • Flat rate with bandwidth cap and speed limit: If you go over your current bandwidth usage your speed will be throttled.
  • Quarterly or yearly usage caps, instead of monthly.
  • Bulk purchase of usage: Buy GB that can be used at any time. Possibly at a flat cost, possibly cheaper if you buy more at a time.
  • Different per GB costs depending on time: Have different rates for GB used in peak hours vs. off-peak hours.
  • Limit speeds to a guaranteed floor: (My suggestion.) Instead of two upper limits, one instantaneous in Mb/s and one in GB/month have a floor in Mb/s. Barring attack of backhoe, guarantee the customer that speed or better.
  • Cheaper plan for low-usage customers: Since lower instaneous speeds have less negative effect on the network, have a cheaper plan for customers willing to download very slowly. This might have either a higher usage cap or lower overage fees or both.
  • Automatically rising usage caps: Shaw tend to increase the costs of packages a couple times per year. It was suggested that the usage cap for each plan be increased at the same time so the plans would keep up with the expected increase in how much bandwidth a normal user consumes.

Things I hadn't thought of:

  • It was pointed out that over the last few years, while usage and package prices have been going up the usage caps on the various packages have actually been going down.
  • Does bandwidth directed toward your IP without your consent cost against your usage. i.e. A denial of service attack. The answer seems to be yes. I suspect you'd be able to talk your way out of such charges, but it would likely entail a lot of hassle.
  • It hadn't occurred to me before, but being at this session made me wonder whether re-sent packets would count against your usage. The answer is yes, since Shaw can't really tell if a packet was re-sent "just because", because of issues outside of Shaw's control, or because of issues with Shaw's network being congested.
  • It was pointed out that the very low rates for very high bandwidth found in, for example, Japan are due to heavy government subsidies. Apparently there are very few direct subsidies for telecommunications in Canada.

Conclusion

Overall, I suspect that we haven't seen the last of UBB from Shaw. It doesn't seem that enough people really care for them to drop this plan entirely. People don't like the idea of their bill going up, but as long as only some people are affected, I suspect Shaw will find that a few complaints are worth not leaving that money on the table. As was pointed out in the session, they don't really have a lot of competition. Especially here in Manitoba. The only real alternative is MTS, but they don't have comparable speeds. So heavy users don't really have a choice. Luckily, they're also aware that UBB isn't particularly popular, so hopefully they'll limit it so that most users will not have to deal with cell-phone style overage charges.

(Oh, and Wil, Shaw does do "network management" but don't look too closely at the packets. I think this means they do throttle torrent downloads. This wasn't directly applicable to UBB so I just quickly asked at the end.)

Sucker Punch

Event
Wil Alambre
Wil Alambre Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:152
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Wil Alambre Whiteboard Ninja

After visually beautiful movies like 300 and Watchmen, Zack Snyder's newest project looks ready to wow movie audiences again. It's been described as a mix of Alice In Wonderland and Kill Bill, it's one of the first big movies of 2011 that I'm looking forward to. Check out this trailer and try to disgree with me... I mean, it has a good-looking woman with a samurai sword attacking a giant robot firing a chain gun! Win! :D

Imitation is the Sincerest Form of Flattery

Technology
Dwayne Kristjanson
Dwayne Kristjanson Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:154
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Dwayne Kristjanson Indifference Engine

So, the other day we spent a fun couple of hours optimizing livethesheendream.com so it could keep people supplied with Charlie Sheen quotations without turning the server into a charred lump. It's always fun to work on something new, and afterwards we started keeping an eye out for other sites doing similar things. Maybe we'd learn something, or just be entertained with a few odd quotations. So when the Winnipeg Free Press posted a link to livethebettmandream.com we all took a look. It was very familiar. Very, very, familiar.

Now, this was just a simple bit of help on our part for Jarrett Moffatt, so I don't really care if I see that code again elsewhere on a different site. That's Jarrett's business. I haven't even checked if Jarrett was responsible for this site as well. I suspect not, since the Bettman version always loads up with "We always had confidence in the game and our fans.", so it's not using the server-side randomization that the Sheen version has. More telling, however, is what we see when we take a look at the source code. Specifically at the Google Analytics javascript. That should be different, since Google generates different access codes for different sites. But it's not.

Playing internet CSI is fun, but I'd rather this other site used it's own analytics code so the data collected for livethesheendream.com would only include visits to that site. If whoever put that site up is reading this: please get your own Google Analytics. You're polluting the data for the site you copied. (Oh, and right now most of your traffic is coming from the Winnipeg Free Press.)

Lessons Learned from Mega-Success: Optimize

Functionality
Julian Moffatt
Julian Moffatt CEO / Partner
Visual Lizard
work
1 (204) 957-5520 ext:1
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Julian Moffatt Purveyor of Good Times

Incase you have been living under a rock, Charlie Sheen has been running amok through all types of media since being dropped by CBS. Given his state of mind, there have been a few rambling and incoherent interviews over the weeks. It has been sort of like watching a real-time train-wreck and the general public is eating it up.

Given the media buzz, our friend, Jarrett Moffatt over at Velocity Branding, along with his good friend Chris Meisner in New York, had the idea to create a single serving website that delivered some choice quotes from Charlie Sheen. You may have heard of a little site called livethesheendream.com recently.

livethesheendream.com screenshot

The site itself is simple. Jarrett and Chris wanted to keep it clean, light and easy for people to get new quotes out of. Now neither of them are hardened web programmers, but for what they wanted to create, they didn't need to be. They figured that the site would get a little bit of buzz from their social media friends on Twitter and Facebook. It got more than a little buzz.

On the third day of being live, livethesheendream.com had popped up on Perez Hilton and Fred Durst's Twitter feeds. From there, traffic to the site jumped from a few hundred visitors over the first couple of days to millions. As a social media experiment, the awesome power of the internet was in full effect.

With all the traffic to the site, came the obvious load issues that high volume traffic brings. Given that Chris and Jarrett had just hacked together a little PHP code and some HTML to make the site, it was not optimized or ready for high volume traffic. On the end of the first day of being stampeded, Jarrett and I where discussing things, and we offered to help optimize the site to try and keep things running smoothly.

The following morning, traffic was through the roof with over 75,000 visitors an hour. We jumped into the code and identified the following things immediately:

  • each time a visitor wanted a new quote, the entire page would reload
  • the reload of the page was serving up all elements of the site, because caching was not being specifically instructed
  • the image of Charlie Sheen was 30KB

We looked for the fastest items we could address in order to speed the site up. We didn't want to miss any visitors if we could help it. Here is what we did:

  • compressed the image of Charlie Sheen to 8KB. It still looked good and saved 22KB per page load. Do the math on 22KB * 600,000 visitors on the peak day. That alone saved 1.25 GBs of data from being served up. We deployed this change immediately to help keep the server up, although it was under heavy load and it was not serving pages to all visitors.
  • we then moved all of the page requests for new quotes onto the visitor's browser. We took the array of quotes and loaded them all into a Javascript array the first time a visitor loaded up the site. We then applied an event to the "get a new quote" functionality that leverage Javascript rather than PHP. When looking at the Analytics data for the site, the average visitor was reading through 10-12 quotes, which was previously triggering 11 additional complete page loads for each visitor. With 600,000 visitors * 10 extra page loads, we removed 6,000,000 requests to the server.
  • within the actual HTML of the site, we did a little clean up, added a doctype, removed some extra tags and tidied up the CSS. We saved about 15KB total. 15KB doesn't sound like much, but it really adds up given the amount of traffic they were getting. 600,000 visitors X 15 KB means we saved 8.5 GBs of additional data with a little less than sixty minutes of programming.
  • on the server, we wrote a little .htaccess file that specifically instructed visiting web browsers to cache the picture of Charlie Sheen and the CSS files for a long period of time. Even if they reloaded the page, rather than clicking the floating head, their browser would pull the image and CSS files from their local hard drive if they existed. We can't calculate specifically how much bandwidth this saved, but it certainly didn't hurt.

Once all that was in place, we had to reboot the server. Jarrett and Chris had used MediaTemple and livethesheendream.com had been placed on a 500MB DV. Once the reboot finished, the traffic stampede continued, but this time the server load was not even peaking over 1.5. Previously it had been steadily in the 45-50 range, which the server was unable to sustain.

All told, we spent about 4 hours between three of us to make a huge difference in the code. It was fun being involved in an internet sensation. Jarrett and Chris had an amazing little idea that hit on a popular culture wave of interest.

The moral of this entire article is: CODE MATTERS. Never discount the possibility that lightning might strike your idea for a little web project. If lightning does strike, it is great to be prepared. Congratulations on your amazing success Jarrett and Chris!

Speedworld with Velocity

General
Wil Alambre
Wil Alambre Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:152
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Wil Alambre Whiteboard Ninja

Visual Lizard joined Velocity Branding in go-karting at Speedworld's indoor track, Le Mans-like. Though most of us had never driven a go-kart before, and many of us had never really driven fast before, everyone who gave it a try had a fantastically entertaining time. Click on any of the thumbnails below to enjoy a small sampling of the high-speed fun.

IMG_0113
IMG_0114
IMG_0115
IMG_0116
IMG_0117
IMG_0120
IMG_0121
IMG_0122
IMG_0123
IMG_0125
IMG_0126
IMG_0128
IMG_0129
IMG_0132

A Book Apart, Elements of Content Strategy

Event
Wil Alambre
Wil Alambre Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:152
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Wil Alambre Whiteboard Ninja

Erin Kissane is the author of A Book Apart's third release in their fantastic series of "brief books for people who make websites". This brief guide explores content strategy’s roots, and quickly and expertly demonstrates not only how it’s done, but how you can do it well. We already own the first two books, and both have been worth every penny. :)

Dropkick Murphys "Going Out In Style"

Event
Wil Alambre
Wil Alambre Senior Programmer
Visual Lizard
work
1 (204) 957-5520 ext:152
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Wil Alambre Whiteboard Ninja

Going Out In Style is the seventh studio album by the Dropkick Murphys. It's described as a concept album that traces the journey of a fictional character named Cornelius Larkin. Though there were one or two good tracks on Meanest Of Times, I haven't fully enjoyed a Dropkick Murphys disc since Blackout. Hopefully this one will bring back some thumping good times!

Max's Birthday

Event
Julian Moffatt
Julian Moffatt CEO / Partner
Visual Lizard
work
1 (204) 957-5520 ext:1
fax
1 (204) 957-5519
toll-free
1 (888) 237-9559
url
http://www.visuallizard.com
Julian Moffatt Purveyor of Good Times

Happy Birthday. Lunch is on us! Choose wisely :)