Community & Search Era

The setting

Hardware: PCs get cheaper and spread into homes; servers shift from expensive minicomputers to cheap x86 machines.
Network: ADSL broadband replaces dial-up with flat monthly pricing, so users can stay online for long stretches and upload content.
Capital: after the bust, capital turned cautious and looked for real revenue; large platforms began to consolidate.
Users: the number of internet users surges, shifting from "reading" to "writing": blogs, forums, comments, online shopping.
Policy: electronic signatures gained legal force in many countries; security and data-protection rules expanded.

The pressure

What needed to change

How do you turn free content and traffic into revenue?
Online transactions lack trust: buyers fear paying and getting no goods, sellers fear shipping and getting no payment.
Users and data volumes explode; a single server or single database can't cope, and big sales or hot events bring it down.
Every action on a web page reloads the whole page, an experience far worse than desktop software.
Browser compatibility is a serious problem (IE6 dominates and ignores standards), and writing a single drop-down menu means handling a lot of compatibility code.
Internet products keep getting more complex, and waterfall development takes a year or two to ship one version, failing to keep up with changing user needs.
Website vulnerabilities and worms run rampant (SQL injection, XSS, Code Red, Blaster, email worms), and a black-market industry chain begins to form.
Mobile internet is only WAP: crude pages, expensive data, slow speeds.

The building blocks

Search pay-per-click ads (keyword ads), online games, SMS value-added services -> pain point 1: the internet has a scalable revenue source for the first time.
PayPal-style escrow and marketplace ratings -> pain point 2: the platform holds the buyer's payment first and releases it to the seller only after the goods are confirmed received.
LAMP horizontal scaling, read/write splitting, caching (Memcached), CDNs -> pain point 3: replace one expensive machine with many cheap ones.
Google's GFS (2003) and MapReduce (2004) papers, and the open-source implementation Hadoop in 2006 -> pain point 3: storing and computing on massive data, the starting point of what later became "big data."
AWS S3 and EC2 (2006) -> pain point 3: rent servers on demand instead of buying machines and building your own data center, the starting point of cloud computing.
Ajax (named in 2005) -> pain point 4: partial page refreshes let web pages work like desktop software, such as Gmail and Google Maps.
The web standards movement, CSS layouts replacing tables, and JS libraries like jQuery (2006) -> pain point 5: one set of code smooths over browser differences, and "frontend" starts to become an independent technical track.
Agile development, rapid iteration, and user-experience (UED) teams -> pain point 6: small, fast steps and continuous revisions based on user feedback, which is how the product manager role took shape.
Web application firewalls, secure coding standards, MLPS assessments, and vulnerability response -> pain point 7.
iPhone (2007), App Store (2008), Android (2008) -> pain point 8: a genuinely usable mobile browser and app-distribution channel. They lead into the next era.

The people

FrontendFrontend Engineer
Split off from "web production." The daily work is writing standards-compliant HTML/CSS, building interactions with Ajax and jQuery, and handling IE6 compatibility. During this period, major domestic companies set up dedicated frontend or UED teams.
BackendBackend Engineer
Shifts toward high concurrency and distributed systems. The daily work is sharding databases and tables, adding caching, optimizing slow queries, and writing business logic in Java/PHP.
ProductProduct Manager
The product manager system spreads across internet companies. The daily work is writing requirement documents, drawing prototypes, following up on iterations, and reviewing user feedback.
DesignInteraction Designer
Splits into interaction design and visual design, doing user research and usability testing, and establishing the UED way of working.
OpsDevOps Engineer
System administrators (SA) and database administrators (DBA), responsible for managing hundreds or thousands of servers, monitoring, and releases.
SecuritySecurity Engineer
Web security engineers and penetration testers, doing vulnerability scanning, MLPS assessments, and incident response.
OperationsOperations Specialist
Content, community, and e-commerce operations took shape in this period, corresponding to the operations role page `/roles/operations/`.
Full-stack (precursor)Full-stack Engineer
Small teams built both the page and the server; LAMP stacks made one developer responsible for the whole request path.

Trust & safety

GlobalThe US COPPA (1998) required verifiable parental consent before collecting data from children under 13. Source
GlobalThe Budapest Convention (2001) harmonised cybercrime offences and cross-border evidence requests. Source

Selected moments

2001-01-15
Wikipedia launches
Sources: en.wikipedia.org
2001-02
The Manifesto for Agile Software Development is published
Sources: agilemanifesto.org
2001-07
The Code Red worm breaks out
Sources: en.wikipedia.org
2003-10
Google publishes the GFS paper
Sources: research.google
2004-10
The first Web 2.0 Conference; the term "Web 2.0" catches on
Sources: en.wikipedia.org
2004-12
Alipay begins operating independently; the same month Google publishes the MapReduce paper
Sources: zh.wikipedia.org · research.google
2005-02-18
Jesse James Garrett publishes an article coining the term Ajax
Sources: en.wikipedia.org
2006-01
jQuery is released
Sources: en.wikipedia.org
2006-03-14 / 08-25
AWS launches S3 and EC2 (beta)
Sources: en.wikipedia.org
2006
Hadoop becomes an independent open-source project
Sources: en.wikipedia.org
2007-01-09
Apple unveils the iPhone
Sources: en.wikipedia.org
2007-06-22
The Administrative Measures for Information Security Multi-Level Protection are issued
Sources: www.miit.gov.cn
2008-07-10
Apple's App Store launches
Sources: en.wikipedia.org
2008-09-02
Google Chrome is released; the V8 engine greatly improves JS performance
Sources: en.wikipedia.org
2008-09-23
The first Android phone, the HTC Dream (T-Mobile G1), is released
Sources: en.wikipedia.org
2004-02-04
Facebook launches from a Harvard dorm room
Sources: en.wikipedia.org
2005-02-14
YouTube launches
Sources: en.wikipedia.org
2006-03-21
Twitter launches
Sources: en.wikipedia.org

Looking forward

The business models and distributed architecture of the PC internet have both matured, but users can only get online sitting at a computer, and mobile internet is still hard to use. Smartphones and app stores have appeared; all that's missing is a cheap, fast-enough mobile network, and the 3G licenses of 2009 filled in that piece.

Dates and scope

Era boundaries

Boundary start: 2001 Wikipedia launches and the Agile Manifesto is published
Boundary end: 2008 the App Store launches and the first Android phone ships; mobile access begins to overtake the desktop

The setting

Hardware: PCs get cheaper and spread into homes; servers shift from expensive minicomputers to cheap x86 machines.

Network: ADSL broadband replaces dial-up with flat monthly pricing, so users can stay online for long stretches and upload content.

Capital: after the bust, capital turned cautious and looked for real revenue; large platforms began to consolidate.

Users: the number of internet users surges, shifting from "reading" to "writing": blogs, forums, comments, online shopping.

Policy: electronic signatures gained legal force in many countries; security and data-protection rules expanded.

What needed to change

How do you turn free content and traffic into revenue?

Online transactions lack trust: buyers fear paying and getting no goods, sellers fear shipping and getting no payment.

Users and data volumes explode; a single server or single database can't cope, and big sales or hot events bring it down.

Every action on a web page reloads the whole page, an experience far worse than desktop software.

Browser compatibility is a serious problem (IE6 dominates and ignores standards), and writing a single drop-down menu means handling a lot of compatibility code.

Internet products keep getting more complex, and waterfall development takes a year or two to ship one version, failing to keep up with changing user needs.

Website vulnerabilities and worms run rampant (SQL injection, XSS, Code Red, Blaster, email worms), and a black-market industry chain begins to form.

Mobile internet is only WAP: crude pages, expensive data, slow speeds.

The building blocks

Search pay-per-click ads (keyword ads), online games, SMS value-added services -> pain point 1: the internet has a scalable revenue source for the first time.

PayPal-style escrow and marketplace ratings -> pain point 2: the platform holds the buyer's payment first and releases it to the seller only after the goods are confirmed received.

LAMP horizontal scaling, read/write splitting, caching (Memcached), CDNs -> pain point 3: replace one expensive machine with many cheap ones.

Google's GFS (2003) and MapReduce (2004) papers, and the open-source implementation Hadoop in 2006 -> pain point 3: storing and computing on massive data, the starting point of what later became "big data."

AWS S3 and EC2 (2006) -> pain point 3: rent servers on demand instead of buying machines and building your own data center, the starting point of cloud computing.

Ajax (named in 2005) -> pain point 4: partial page refreshes let web pages work like desktop software, such as Gmail and Google Maps.

The web standards movement, CSS layouts replacing tables, and JS libraries like jQuery (2006) -> pain point 5: one set of code smooths over browser differences, and "frontend" starts to become an independent technical track.

Agile development, rapid iteration, and user-experience (UED) teams -> pain point 6: small, fast steps and continuous revisions based on user feedback, which is how the product manager role took shape.

Web application firewalls, secure coding standards, MLPS assessments, and vulnerability response -> pain point 7.

iPhone (2007), App Store (2008), Android (2008) -> pain point 8: a genuinely usable mobile browser and app-distribution channel. They lead into the next era.

The people

FrontendFrontend Engineer

Split off from "web production." The daily work is writing standards-compliant HTML/CSS, building interactions with Ajax and jQuery, and handling IE6 compatibility. During this period, major domestic companies set up dedicated frontend or UED teams.

BackendBackend Engineer

Shifts toward high concurrency and distributed systems. The daily work is sharding databases and tables, adding caching, optimizing slow queries, and writing business logic in Java/PHP.

ProductProduct Manager

The product manager system spreads across internet companies. The daily work is writing requirement documents, drawing prototypes, following up on iterations, and reviewing user feedback.

DesignInteraction Designer

Splits into interaction design and visual design, doing user research and usability testing, and establishing the UED way of working.

OpsDevOps Engineer

System administrators (SA) and database administrators (DBA), responsible for managing hundreds or thousands of servers, monitoring, and releases.

SecuritySecurity Engineer

Web security engineers and penetration testers, doing vulnerability scanning, MLPS assessments, and incident response.

OperationsOperations Specialist

Content, community, and e-commerce operations took shape in this period, corresponding to the operations role page `/roles/operations/`.

Full-stack (precursor)Full-stack Engineer

Small teams built both the page and the server; LAMP stacks made one developer responsible for the whole request path.

Selected moments

2001-01-15