Search technologies

NARRATION: I'd love to book a holiday. I'd like to visit Wales and go walking. I think I'll use the internet to research my trip. I said the internet there, but actually I mean something called the world wide web. Hang on, I hear you cry, isn’t the world wide web the same as the internet?Well they are connected, but they’re not the same thing.

In the 1970s scientists developed the system of computer networks we call the internet to allow them to send messages and share their work. They tried making information pages and sharing them on the network but the pages were difficult to find and very slow to load. It wasn’t much fun. Books and papers were still easier and faster to use. Then, in 1989 a scientist called Sir Tim Berners-Lee changed everything. Tim and his team invented a language called HTML to write better pages, a set of rules called HTTP to make sharing the pages easier and the URL, an address for each page so it could be found easily. He called his system the world wide web and it grew into the huge collection of websites we know today. No wonder he became Sir Tim - what a genius!

So how is the world wide web different to the internet? Well, you could think of them both as part of a big library. The internet is the library building and all of its infrastructure, rooms and spaces full of bookshelves and cupboards and storage racks, and the corridors, lifts and stairs that connect them. The world wide web is the content of the library - the books, videos and magazines on the shelves. Each book is a website, each page is a web page.

The world wide web contains billions of pages of information with text, images, photos, videos, data. In order to find the information we’re looking for, we need the help of a computer program called a search engine. We can think of search engines as the librarians of the world wide web. In a real library the librarians create and manage a detailed list of everything kept in the library. This is called an index. Librarians spend so much time updating the index that they know what books are most popular and where to find them.

Search engines do a very similar job on the world wide web. They send out little programs called web crawlers that visit all the pages on the world wide web, making a huge index as they go. For each web page they record the web page address, this is one of Sir Tim’s inventions, it’s called the URL. They make a note of what the page is about and list the important key words. They also count links from other pages as this shows how popular that page is. These clever little web crawlers are always on the move, looking for new web pages to add to their index, and removing web pages that are no longer available.

Let’s search for information about the walking routes in Wales. I’ll use 'best, walks, Wales’ as my key search words. The search engine will use the work of those web crawlers and check the millions of entries on the index. It looks at my key search words and matches them up to the index notes about each webpage. The search engine then shows me a list of websites from its index that it thinks will be useful. The trouble is, the search engine is so good at its job it might find thousands of websites with information about Welsh walking routes. How does it decide which ones to put at the top of the list? It uses a set of rules called a search algorithm. And if you don't know what an algorithm is then I've made a handy video guide on the BBC Teach website.

The search engine uses the algorithm rules to check the thousands of websites it found in the index. There might be 200 rules to check before it decides on a page ranking, or what order to put the websites in. Some of the rules might include: Do the website pages include all the key search words? Do lots of other websites have links to this one? Is the website updated regularly? Is it quick to load? Pages with a high page rank score go nearer the top, and should be the most relevant and useful. Each search engine uses their own algorithm so you might get a different list with each search engine despite using the same search words.

It’s important to think carefully about your key search words. If I just use ‘best walks’ without adding ‘Wales’ then the search engine will look for different walks all over the world. If I spell ‘Wales’ wrong and put ‘whales’ with an 'h', then I might end up with pages of underwater videos of giant sea mammals! Those hard working algorithms need as much help as you can give them.

One last thing to remember. Companies will often pay to make sure their website goes to the top of the search list. These pages at the top of the list will usually have the letters AD, short for advert, next to them to show you that they bought their place at the top rather than by being chosen by the search engine algorithm as being useful.

So there we have it - popping down to the internet library to search the world wide web bookshelves, chatting to the search engine librarians and catching up with our old friend the algorithm. Thanks again Sir Tim and your team!