Over the summer, a small team of Engauge employees pulled off an extraordinary feat: in 6 weeks we produced a site for a major client, from scratch, that receives approximately 2.5 million page views per month.  At the outset, we were in an uncomfortable position for any project team: we had a tight deadline, no functional requirements, no visual/creative program established, and not much of an idea how the new site should behave.  There was only one way we could possibly produce a quality product in such a short amount of time: we would all start from zero and work in parallel, we would produce flexible assets and we would react to each other’s work as we progressed.

As Team Lead for the production effort, I felt obligated to identify a process that would allow team members from far-flung disciplines like graphic design, behavioral research, and programming to work in parallel without leading to chaos and disruption.  We borrowed heavily from the Agile software development process and came up with a strategy that we’ve coined “Interdisciplinary Agile.”

The tenants of the Agile philosophy are succinctly described (in under 70 words) in the Agile Manifesto. The success of an Agile project hinges on the quality of communication between internal production team members and between the production team and the client.

The most fundamental piece of our strategy was the 15-minute daily “standup” meeting. Every day the client services team would get together with anyone who was currently producing assets for the website (i.e. wireframes, user survey results, creative comps, or code) and each person answered three questions:

1. What did you do yesterday?
2. What are you doing today?
3. What roadblocks are preventing you from accomplishing your task?

Often, standups would precipitate small, quick break-out meetings between team members to address more complex issues.

Another important focus of our strategy involved building our deliverables within two-week “sprints.”  The idea is to define tasks in such a way that after two or three weeks, you’ve produced something you can demonstrate (and it doesn’t always have to be pretty).  We produced functioning database access code (demonstrated via an incomplete first-draft user interface) in the first two-week sprint and finalized the user interface in the following two-week sprint.  After the first four weeks, we had essentially completed the site; however, we provided another two week “client acceptance” period where we accommodated client requests for changes.

As a team, we benefited greatly from working concurrently; we managed ourselves effectively and took ownership of the work we had to do.  Great ideas that came out of the flow of the project could be incorporated into the site in short-order and everyone had an opportunity to provide new insight to the project and help guide our progress towards the final goal.

The Internet has already captured a majority of the wealth of human knowledge. Big ideas (both emerging and established) can be accessed and processed via an increasingly wide array of electronic devices. Still, this huge ocean of data is often only visible from the portals of our desktop or laptop screens. We are approaching a unique watershed moment where we will soon breach the Fourth Wall separating us from this information.

Imagine walking into a bookstore to find a title that focuses on a particular area of professional interest. You’re browsing through the “Software” section of the store because you’re interested in learning more about software development. You pick out a book that looks interesting and you snap a picture of it with your cell phone camera. Your phone will display a synopsis of the book, an average reader-rating score with further access to user reviews, and recommendations for further reading. You decide from this information that the book you’re holding in your hand is not the one you’re looking for, but given the recommendations you’ve just received, the book on the shelf above it will give you what you want.

This type of instant access to Internet intelligence is already possible and will one day be de rigueur.

Here is one of the necessary ingredients:

A two-dimensional barcode (also known as a QR Code) can store over two megabytes of data, more than enough storage capacity to hold things like book synopses and other detailed product information. A mobile phone user simply snaps a picture of the QR Code and the phone will extract the data (provided the phone has QR Code-reading software installed on it). The barcode can also transfer a URL to the phone, thus providing a bridge between the Internet and the physical world.

These QR Codes can be placed anywhere: billboards, magazine ads, TV ads, websites, etc. Accordingly, this technology will have a huge impact on marketing. My colleague Amy Griswold recently blogged about the dearth of website links displayed during Super Bowl ads. Part of the problem with displaying a URL during a TV spot is that unless the viewer is actively typing the link into a web-browser (granted, TiVo would make this easier), this information is effectively lost on them. However, snapping a picture of an on-screen QR Code is immediate and has the added benefit of storing the link into the cell phone for future retrieval.

This technology is poised to have a profound impact on the way we acquire information from the world around us. The combined data of a “QR-encoded” physical entity and the personalized information that can be stored on a cell phone provides vast potential for directed marketing opportunities. Major US mobile service providers are already advancing their own QR Code initiatives and we’ll soon start noticing more of these codes around us, providing avenues to a further digitally enhanced life.

In my previous post on complexity, I outlined some of the perils that can result from increasingly complex software. For part deux, we’ll examine a solution to the complexity problem.

One fundamental aspect of software development must be remembered during any conversation regarding the problems of software complexity: good software arises out of a process of gradual refinement. This simple premise is often overlooked by the stakeholders involved in a project (including developers). However, a single word has arisen out of the programmers’ lexicon into the mainstream that tersely communicates the state of a project currently undergoing some form of refinement: Beta. Beta is a powerful word. Beta communicates to the end-user (and client) that the application they are currently using has not yet reached its full potential.

Beta means different things for different applications. For some, Beta means that the application either has functionality that may not work at all under certain conditions, or that obvious functional points are missing but will soon be in place. In other words, it’s a means of providing coverage when things go wrong or when things are obviously missing. For Google’s increasingly popular GMail, Beta means “100% of the functionality we have provided will work 98% of the time, but there are many wonderful things we have in store for you that we will present on our terms.” It is thanks to Google, I think, that “Beta” has become an almost everyday word.

It is also thanks to Google that the process of gradual refinement in software, a process programmers call Iterative Development, can be packaged as a more palatable concept for stakeholders who drive software production (e.g. our clients at Spunlogic). Any application that contains an easily-identifiable amount of complexity (in either concept or execution) must go through an interim Beta phase until the project’s goal or the client’s vision has been met. Of course for large and complex projects, vision may be clouded from the outset. This provides all the more reason to develop and launch the project in iterative steps, refining the application while at the same time refining project goals when necessary. As more pieces of the application are launched, stakeholders can protect themselves from the storm of complexity by keeping the application under the “Beta umbrella.”

I’d like to return to Google and one of their flagship products, Google Maps, as an example of iterative development done right. Google Maps currently provides the means to view satellite data of a region with a road-map overlay, view real-time traffic data in major metropolitan areas, view building elevation in these areas, re-route driving directions via drag-and-drop interaction, etc., etc., across multiple countries. However, when Google Maps first launched, it did two things: provided driving directions (and not always reliable ones) on a slick drag-able street map view of the United States. Here are some highlights in Google Maps’ iterative evolution from a product that delivered two main pieces of functionality for one part of the world to a product that can do dozens of different things while representing many places across the globe (with credit to Wikipedia):

• February 8 2005: Google Maps first announced, providing street maps and driving directions covering much of the United States.
• Late June 2005: Google releases the Extensibility and Customisation API, laying the foundation for the Google Maps Mashup movement.
• Mid July 2005: Google releases Google Maps for Japan.
• January 2, 2006: Google Maps features road maps for the United States, Puerto Rico, Canada, the United Kingdom, Japan, and certain cities in the Republic of Ireland.
• On April 3, 2006: version 2 of the Maps API is released. Two months later, geocoding capabilities are added.
• February 2007: buildings and subway stops are displayed in Google Maps “map view” for parts of New York City, Washington, D.C., London, San Francisco. MARTA stops have since been added.
• February 28, 2007: Traffic information is officially launched to automatically include real-time traffic flow conditions for 30 major cities of the United States.
• May 30, 2007: Street View is added, providing a ground-level 360 degree view of streets in some major cities in United States.
• June 28, 2007: drag-able driving directions are introduced.
• November 27, 2007: “Terrain” view showing basic topographic features are added.

Obviously, Google Maps has become an extraordinarily complex application, and not all applications must deliver this degree of complexity. Regardless, the Google Maps development life cycle is a case-study in how to mitigate complexity through iterative development.

Assume that some of the great functionality present in the current iteration of Google Maps was not foreseen at the outset of the project, but grew organically from within the development process. At the same time, there must have been some functionality on the drawing-board that never made it out of iterative development (and certain things that did make it out have since been iteratively eliminated, like the “Hybrid” view). All the while, Google Maps lived in a Beta state throughout much of its life (it is no longer Beta).

We must take this model and apply it to our own complex projects. A Beta site launch will portend a better product for all stakeholders and iterative development will help solidify the product and eliminate distractions to the project goal.

“Complexity Kills” is a maxim that can never be overlooked by organizations that create software. Ray Ozzie, chief software architect at Microsoft, elaborates:

“Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges and it causes end-user and administrator frustration.”

The New York Times recently ran an article focusing on the pitfalls of software systems that have grown too complex. As we demand more from any given software system, it necessarily grows more complex. Paradoxically, this ever more complex system can suffer from these increasing demands, becoming less useful (and perhaps more dangerous) due to problems arising from complexity. These frail applications shut down; travelers then wait for hours at airports, phone systems vanish, vast power outages cripple major metropolitan areas. All the while, our way of life is becoming more dependent on complex software applications.

What are we to do?

The demand for more productive software will never cease. In order for software companies to continue building successful (and profitable) software, they must adhere to processes that mitigate the evils of complexity. Care needs to be applied to how all stakeholders in a project communicate and contribute, from the client to management to development. Developers need to focus on techniques that slice a difficult problem into smaller, more manageable components.

In the second part of this article, I’ll focus on specific ways that complexity can be managed. We’ll examine Google Maps as a case-study of an application that is both complex and successful.

On Wednesday, May 30 (coincidentally my birthday), Microsoft unveiled their newest attempt to expand the company’s consumer electronics business with Surface, a touch-sensitive computing system that somewhat resembles an old pac-man table-top video game (a relic of the 80’s that can still be found lingering in a few bars and pubs). While Microsoft has made previous attempts to leap from the confines of the personal computer (some more successful than others), Surface is perhaps their most intriguing venture yet.

Resembling the “Multi-touch” system first unveiled over a year ago by Jeff Han at NYU, the Surface screen allows multiple input points to be active at once. For example, if you want to enlarge the size of an image, simply put one index finger on a corner of the image, put your other index finger on the opposite corner, and move your two fingers apart. Magically, the image will grow to fit the dimensions of the imaginary rectangle formed by the diagonal distance between your two fingers. Likewise, move your two fingers together, and the image will shrink; pretty cool.

Microsoft has taken this idea even further by allowing Surface to not only interact with our hands, but also other types of peripheral gadgets that we may be fond of. To transfer songs between two MP3 players (Microsoft Zunes, perhaps), simply place the devices on the Surface surface (?) and begin dragging files between them with your fingers.

Another neat demonstration of Surface’s potential involves paying the check at a restaurant: you place your credit card on the table and your date places their credit card on the table (you’re going Dutch tonight) and you move pictures representing the food you’ve ordered onto your respective cards, thus divvying up the tab.

While this mode of interaction seems to open up a new world of possibilities for how people exchange information, I’m having trouble thinking of scenarios where I could immediately benefit from this technology. Myself and Mike, another developer here at Spunlogic, were considering ways we could use this device around the office. While I sat at my desk and Mike sat in the little papasan across from me, we couldn’t define a way that Surface would allow us to be more productive (as we were sitting around talking about Surface while at work, it’s safe to say that it has so far proven counter-productive).

Thus, I fear that Surface may become the Segway of computers; a really cool idea, perhaps too novel and ahead of its time to be immediately useful. I certainly see Surface providing a medium for the elusive convergence between devices that electronics manufacturers have been seeking for some time now, but until an ecosystem of Surface-friendly devices become available, it may not have much of a leg-up on the plain old PC.

From the non-developer’s perspective, the developer’s job is one that cannot be understood; it involves communicating in acronyms (XML, ASP .Net, MVC, etc.) and staring blankly at a computer screen decorated with bizarre combinations of words that mean nothing. Working in this manner to build a functioning application that anyone can use and understand (e.g. Google or Word) lends a certain amount of mystique to the work we do. However, strip away the layers of jargon and all of that messy code and you’ll find that a developer is nothing more than a puzzle-solver. Our puzzles are complicated, though, and to solve them we must learn a slew of technologies and special languages (both literally and metaphorically).

Regardless, the developer is not one who merely excels at writing inscrutable combinations of letters and phrases which we call “code” and non-developers call “confusing and nonsensical.” We have to be able to understand a variety of problems as they exist in the real world. We must be able to identify deficiencies in how these problems are defined and we must be able to invent solutions to these problems; only then can we perform the labor of writing code to implement the solution.To some degree, as developers we must also become bankers, doctors, house painters, etc. If we don’t understand the nature of the work that we are developing for, then we can’t understand what we are required to do. Being a doctor, banker, house painter, etc. is the most challenging aspect of being a developer.

We have to understand every aspect of what we call the “problem domain” in order to ensure that we produce the most reasonable solution. Thus, we must be very detail-oriented people, always requiring more information than we are initially given and asking a bunch of questions that tend to perplex some unfortunate project manager. Eventually, we will get all of those evasive details and through the powers of what I call “dev-magic,” we’ll produce a fully functioning (mostly) and bug-free (rarely) software application that will make everyone happy (never).

Let’s consider an example:

You’re a developer working in the “dev shop.” A frantic project manager approaches you and says “We need to figure out how quickly three people can paint a house… GO!”

…

“Well…”

“I have a few questions…”

“How big is the house? Who’s painting this house? Will they be painting more than one house? Do I need to accommodate the fact that they’ll be painting houses of different sizes? Do they paint at the same rate? Do we need to take weather into account? How do you want the total time displayed? How will the total time be entered?”

At this point, some poor soul (perhaps the developer) will have to contact the client and rattle off these questions. The two parties will then come to some mutual agreement about what is required (let’s call it “a moment of clarity”) and then the developer will embark on his or her dev-magical journey into the realm of a solution.

Here is what you find out:

All the houses are exactly the same (they just built a new sub-division in Canton), so we don’t care about differently-sized houses. Weather will impact the speed that each painter paints, but we don’t yet know how much (the client will provide the details later). Time will be entered and displayed as hours. Three people will be painting each house: Sally, Jimmy, and Sandy. They each paint at different rates. On days when weather does not interfere, Sally can paint a house in 3 hours, Jimmy can paint a house in 4 hours, and Sandy can paint a house in 5 hours.

You now have everything you need; you can finally begin to practice your dev-sorcery. First, you must restate the problem using only the most fundamental information required:

Sally can paint a house in 3 hours, Jimmy can paint a house in 4 hours, and Sandy can paint a house in 5 hours. If the three of them work together, how long will it take for them to paint the house?

A good ol’ fashioned word-problem clears things up for you. Now, you get out a little notepad and a pen from your pocket-protector (not really) and work out the math:

Every hour, Sally paints 1/3rd of the house, Jimmy paints 1/4th of the house, and Sandy paints 1/5th of the house. Let the number 1 represent a house that has been fully painted and let x represent the total number of hours required for every painter working together to finish the house.

x(1/3) + x(1/4) + x(1/5) = 1

The hard part is now over; you’ve gotten all the information you need, you’ve wracked your brain (and your little notepad) to come up with a basic algorithm (you developers love that word) that you can use to solve the problem.

The easiest part of the job is writing the code:

public decimal GetTotalPaintTimeFor3Painters(decimal sally_hours, decimal jimmy_hours, decimal sandy_hours)
{

//the following equation is derived from x(1/sally_ hours) + x(1/jimmy_ hours) + x(1/sandy_ hours) = 1

decimal total_time = (sally_hours * jimmy_hours * sandy_hours) / ((sally_hours + jimmy_hours) * sandy_hours + (sally_hours * jimmy_hours));

return total_time;

}

The above function will get used in your application like so:

decimal time = GetTotalPaintTimeFor3Painters(3, 4, 5);

You find that on an average day, these three painters can paint a house in 1.28 hours, or about an hour and seventeen minutes. When the client eventually tells you out how much longer it takes each painter to complete a house under adverse weather conditions, you can use this same function to find the correct answer.

The problem given in this example is far simpler than most of the problems we have to solve on a day-to-day basis. The problems we work on typically involve many different (sometimes competing) requirements that must be reconciled to produce the right solution. Additionally, we often have to leverage different development theories and technologies to produce an expected result. How we use these tools takes us into the realm of dev-magic, but there is certainly nothing magical or mysterious about the nature of our jobs. All we require is a clear understanding of the problem we’re asked to solve and a lot of Diet Coke (i.e. Spunlogic programmer fuel).