Why is metadata relevant to your search listing?

For those who have never watched Halt and Catch Fire, it provides interesting insights into the rise of computers, servers, browsers, and AI (artificial intelligence). The series starts in the 1980's with a scrappy team sprinting to build the first laptop computer in their garage. While the characters are pursuing the next big idea, iconic miseries of the era included; from backing up files on floppies to shoulder pads and grunge jackets. Midway through season 4, they are in roughly 1996, and the battle between the first search engines are raging.

Search Engines at Conception

The two businesses are pursuing different approaches for capturing organizing the growing number of websites. One uses a manual approach (we assume is Yahoo!), in which site owners send in their URL to a team of category 'experts' to add them to a listing of categories like 'music' and 'cooking'. Through retrospection, we can easily identify their anticipation of thousands of websites as a severe miscalculation. As of April 2018, there are roughly 1.9 billion.

Meanwhile, the other business (we assume is Google), leverages an algorithm to scan sites and apply topics to them. While at the beginning inaccurate and unable of providing the human touch of recommending sites, it quickly evolves to 'impressive' search speeds of 35 seconds, bragging the average user 'finds what they are looking for in a minute and 30 seconds.' In 2018, the industry standard of page load speed is 3 seconds.

Search Engines Today

Other than the speed of search, it's surprising to realize how little things have evolved in search engine operation. Some obstacles the businesses encounter at an early stage are page titles not matching content on the page, adding information to index pages to help a user pick their desired site, and the infiltration of porn. Some of their visions include the ability to leverage advertising, 'the marketer’s dream', and the trying to make the index/search pages a destination, rather than a tool to keep users moving out of it. I foresee news feeds and email in the series future.

Help Search Engines Find and Know Your Site

One of the biggest solutions for the obstacles of search engines is they allow us to help them. There are three important pieces of information to add to your site for search (metadata components), and two main ways to help your website be found by a search engine.

Metadata Components

Page Title: Page titles should be the brief topic of a single page. These are shown in tabs of browsers, and on a search page it will be the first and largest text listed on a search results listing.

Meta Description: This is a short description (one or two sentences) giving a snapshot of what to expect when a user lands on the page. It should be a summary in sentence form including keywords to entice a user to select your page over the others on the listing page. It will be the bulk of the listing on a search results page.

Keywords: Although some search engines can probably identify and recommend more relevant keywords than most people, for more primitive search engines, it’s best that you tell the engine what topics are relevant on your page. These do not show on search engine listing pages.

Getting found by a search engine

Indexing: This is the more assertive and accurate way to have your site be found. For Google, an account needs to be set up in the Google Search Console, and a sitemap submitted in Google’s Index. This is you telling Google you have a new site, and you want it to be added to Google's queue of sites. Usually this is completed in a few days, and will refresh the site periodically for new content.

Bots (AKA crawlers or spiders): This is the more passive way, as it relies on these tools to eventually find your site, as well as all the pages within it. These will occasionally be rerun for new content.

Of course, there are several other considerations for getting your website found and prioritized in search. However, knowing the history of how these engines were built helps us understand why each of these components and tools are important to making sure your wesbite is included.