Setup SEO on Google and Bing - MVC Core


To set up IIS website for Search Engine Optimization (SEO) on Google and Bing, you need some setup on you servers, also some code change to make you website search friendly. There're some tips for IIS, AWS EC2 and MVC Core.

There're billions of websites on the Internet and half of the world population are online. You need help from search engines to index your contents and make them visible.

Google is dominating about 70% of all searches. Bing is remotely the second, but conversion rate on Bing is higher. Here introduce the setup on both platforms.


Steps to setup SEO on Google:

  1. Prepare your website to be search friendly

    Setup HTTP protocal, create a domain or alias start with "www." and create good text contents. Your articles should be more than 600 words long, better to be over 1000 words. Images won't be visible for search engines, however using some images are suggested. It's also important to have a start page that links to your other pages. Add structed data for your page for google referencing "Understand how structured data works".

  2. Setup HTTPS

    You can buy a SSL certificate for single domain, or a wildcard SSL for all domains with *.your.domain format. Godaddy has $1/month standard SSL plan for the first year. It also includes a McAfee security badge for low traffic websites. There're some detail in this blog, Install Godaddy SSL Certificate on AWS EC2 IIS

  3. Setup domain alias with "www."

    You need to setup an Alias record in your DNS. This is an example of AWS Route53 Hosted Zone setting for DNS


  4. Redirect naked domain to www. domain using URL Rewrite on IIS

    It is a common practice to redirect naked domain "your.domain" to "www.your.doman".
    Go to IIS Manager, click your website or "Defaut Web Site", find "URL Rewrite" in feature view. Or you can download from here and install.
    Open URL Rewrite, click "Canonical domain name", add a rule, enter the domain name with "www.your.domain", click "OK". Then try "your.domain" in a browser, it will be redirected to "www.your.doman". Click here for detailed steps.

  5. Create a sitemap.xml on your website root folder

    Sitemap file is an xml file with all the urls you want to share, it also have create dates to tell search engine if there's anything new, or it will be ignored from search engine indexing. There're 2 types of sitemap.xml. There're differentiated by schema url inside the xml file.

    This is the schema for all search engines,

    http://www.sitemaps.org/schemas/sitemap/0.9

    This is the schema only for google,

    http://www.google.com/schemas/sitemap/0.84

    Here's a tool can also generate the file by scanning your default page.

    https://www.xml-sitemaps.com/

    You can create sitemap.xml manually. Here is the standard of sitemap,

    https://www.sitemaps.org/protocol.html#validating

  6. Create robots.txt file in the root folder of your website

    This tells search engines you allow or deny to have your website indexed.

    This is an example to allow indexing. And it set the sitemap URL for search engine to crawl your site.

      User-agent: *
      Allow: /
      Sitemap: https://www.your.doman/sitemap.xml

  7. Create a ROBOTS meta tag in the HTML header

      <meta name="ROBOTS" content="INDEX, FOLLOW, ARCHIVE">

  8. Create searchable header tags for all your pages

    A. Create "title" tag in the html header, and display a single H1 title to inside the body tag.

    B. Create a meta "description" tag, google will search it and display in search result.

    C. Create a meta "keyword" tag, seperate keywords or search terms with commas.

    D. Create meaningful urls, or some people call them slugs.

  9. Setup accounts in Google Search Console

    Register an account on Google Search Console here, Google Search Console. Add 4 website properties that are with "www." or without, with "http" or "https".

  10. Create language tag

    Tells google what language your website is using. Allow users to use their language setting if his browser don't support the language of your site.

    Add lanaguage to html tags

      <html lang="en">

    Add the following lines to your web page header section,

      <link rel="alternate" href="<Your URL> " hreflang="x-default">
      <link rel="alternate" href="<Your URL>" hreflang="en" >

    Check this blog for more detail: How to set language tags for html web pages

  11. Create Canonical meta tags

    You can see from step 5, people can search your site with 4 different URLs, the canonical tags will let search engine knows they're the same site. This is important, because you want all clicks and impressions to be counted together, instead of seperately. Let take "About" page for example,

      <link rel="canonical" href="https://www.softwareblogs.com/Home/About">

    Here's a link tell you what's Clicks and Impressions,

      https://growtraffic.com/blog/2015/01/difference-website-impressions-clicks

  12. Set URLs in sitemap.xml to be Canonical urls

    Your sitemap needs to be consistent with your Canonical urls. Otherwise search engine may not able to index your site. Otherwise search engine may find another site it thought to be a Canonical site for your's.

  13. Set your preferred domain on Google Search Console

    You need to switch to old Google Search Console first. Select each properties, click the gear icon for settings. Select "Site Settings" in the dropdown. In the "Preferred domain" section, select the option you want. This needs to be consistent with Canonical settings described earlier.

  14. Submit the Sitemap and URLs to Google Search Console

    Select the property your want to submit first. Click "Sitemaps" menu on the left. Submit your sitemap.xml.
    Click "URL inspection" menu on the left, enter the URL of your webpage, click "Request Indexing". Repeat this for every new page you wnat Google to index. You need to wait for a few days before Google finish crawl and index your site.

  15. Sign up for Google Analytics

    Google Analytics will show all the traffic comes to your website in real time. After sign up, Google will give your some steps to verify your ownership. Then Google will give your instructions to put some code includes your ID inside your website. This will allow Google to track the traffic on your website.
    In Google Analytics you can setup filters to exclude requests from your own computer. Also remember to exclude traffic for seach engines bots.
    There's a lots of details to get your hands wet. You can start from here: Get started with Analytics
    Setup goals to track your website performance. Please reference Google's instruction, Create, edit, and share goals

  16. Test if your website is indexed by Google

    Search the full URL of your preferred domain with HTTP or HTTPS. If you can find your site, that means google had indexed your site. Otherwise check status on google search console, your website is probably still waiting to be indexed.
    In google search console, you can check the reports of Clicks and Impressions. This is not real time, there're a couple of days delay. You can also check page errors, mobile usability, etc.



Steps for setup SEO on Bing:

It's much easier to setup SEO on Bing. Beside many common steps have already been done when we setup for Google..

  1. Register Bing Webmaster account

    (https://www.bing.com/toolbox/webmaster/)

    You can choose to create your account using your Microsoft, Google or Facebook account. This will help to verify that you're the site owner.

  2. Add your website to Bing webmaster

    use the site matching your Canonical urls. And submit your site. Then your need to wait for some days before it's approved.

  3. Verify your website ownership on Bing

    There're 3 ways to verify your site. You can choose any of them that's available.

    A. Download BingSiteAuth.xml file from Bing Webmaster. Put it in your website root folder. The file includes your Bing User ID.

    B. Create a meta tag for Bing in the header of your web pages like this,

      <meta name = "msvalidate.01" content = "<Your Bing User ID>" />

    C. Create a 301 redirect from "<Your Bing User ID>.your.domain" to "verify.bing.com"

    Bing help file includes how to create redirect for many hostings. But if you're using AWS Route53, you will not find the instruction and will have some issues. I'll show your the work around.

    Since the domain name with the Bing User ID will be considered as an invalid format, you won't be able to create a redirect directly. You need to create a S3 bucket as a bridge. "<Your Bing User ID>.your.domain" will be a valid bucket name for S3. Create the S3 bucket, change "Static Web Hosting" setting to redirect to verify.bing.com. Then, inside Route53, Create an A record, choose the S3 bucket as the alias. (You need to finish the last step in Chrome. In IE and Edge, the S3 bucket is not listed for some reason.)

  4. Submit you sitemap and URLs to Bing

    Under "Configure My Site" on the left menu, you can find "Sitmaps" and "Submit URLs. You can submit multiple URLs together on Bing, so you can finish it much faster compares with on Google


It's exciting that we have finished all the steps above. Depends on how your site is structured, you may have technical issues in some ways. I'll share with you some of my tricks that may help.


Solution One: Put SEO settings in config file.

If you put title, description, keywords in a config file, it will save you a lots of time to change them later on, which happens when you need to change them to improve SEO. The reason to put in a config file is, content in config file will be loaded when startup, and will be cached to improve performance. And when config content change, the site will automatically detect the change.


For people using MVC Core, I'll show you the steps here.

  1. Create a seperated config file in you project root folder, something like pagetags.json

  2. {
      "metatags": {
        "Bing": {
          "name": "msvalidate.01",
          "content": "<your Bing User ID>"
        },
        "Site": {
          "Url": "https://www.softwareblogs.com"
        },
        "ROBOTS": {
          "Settings": "INDEX, FOLLOW, ARCHIVE"
        },
        "About": {
          "Title": "Software Development Blog",
          "Description": "Software Development Consulting, tutorials and knowledge share.",
          "Keywords": "ASP.NET, C#, MVC, Web API, WCF, SQL Server, Mobile, Xamarin, Microsoft, Java, database, application, software, development, aws, Angular, jQuery"
        },
        .........

      }
    }


  3. Change Program.cs to include the new config file,

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) => WebHost.CreateDefaultBuilder(args)
    .ConfigureAppConfiguration((hostingContext, config) =>
    {
      config.AddJsonFile("pagetags.json", optional: false, reloadOnChange: true);
    })
    .UseStartup<Startup>();


  4. Change ConfigureServices method in Startup.cs file, add the following line to the end,

    services.AddSingleton<IConfiguration>(Configuration);

  5. Add a BaseController.cs class. Add the following code to set ViewBag attributes for meta tags and titles

    (Note: "?" is called "Null Conditional Operator". "??" is called "Null Coalescing Operator". These operators make your code more solid and easier to maintain.)

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Text.RegularExpressions;
    using System.Threading.Tasks;
    using Microsoft.AspNetCore.Mvc;
    using Microsoft.Extensions.Configuration;

    namespace SoftBlog.Controllers
    {
      public class BaseController : Controller
      {

        protected IConfiguration _conf;

        public void SetBingMetaTags()
        {

          string bing_name = _conf?["metatags:Bing:Name"] ?? "";
          string bing_content = _conf?["metatags:Bing:Content"] ?? "";
          ViewData["BingMetaTags"] = "<meta name = \"" + bing_name + "\"
          content = \"" + bing_content + "\" />";
        }

        public string GetCanonialUrl()
        {
          return HttpContext.Request.Path;
        }

        public string GetSiteUrl()
        {
          return _conf?["metatags:Site:Url"] ?? "";

        }

        protected string GetDefaultPagePath()
        {
          return _conf?["metatags:Default:Path"] ?? "";
        }

        protected void SetTags(string name)
        {
          ViewData["Title"] = _conf?["metatags:" + name + ":Title"] ?? "";
          ViewData["Description"] = _conf?["metatags:" + name + ":Description"] ?? "";
          ViewData["Keywords"] = _conf?["metatags:" + name + ":Keywords"] ?? "";
          ViewData["ROBOTS"] = _conf?["metatags:ROBOTS:Settings"] ?? "";
        }
      }
    }


  6. Inherit the BaseControl class in your controllers and inject config parameter in your controller constructor

  7. public class HomeController : BaseController
    {
      public HomeController(IHostingEnvironment env, IConfiguration config)
      {
      _conf = config;
      }
      ...........
    }


  8. Call the base class methods to set Tags

  9. public IActionResult About()
    {
      SetTags("About");
      return View();
    }


Solution Two: Add Canonical URLs using a ActionFilterAttribute class

There are 2 reasons to setup Canonical URLs on your website. First is to let search engine knows that the requests with or without "www.", with "http" or "https" can be for the same website. Second reason is you may reconstructed your site's file structure. In this case you want search engines know that a page in a new location is the same page with a file in an old location. Therefore even the pages moved or name changed, the search engines will still be able to add the counts correctly the same page.


Here I'll show you the solution using MVC Core. We'll create a ActionFilterAttribute class. This filter will will allow you to set a certain Canonical URL path to an action, or when you pass an empty URL, it will call a methond to get the Canonical URL with the same path. This will be a answer for both reasons we discussed earlier.

  1. Create a Filters folder and create a class ActionFilterAttribute.cs with following code,
    (This example also includes SetBingMetaTags(), it's for a different purpose that we already discussed)

  2. using Microsoft.AspNetCore.Mvc;
    using Microsoft.AspNetCore.Mvc.Filters;
    using SoftBlog.Controllers;

    public class CanonicalURL : ActionFilterAttribute
    {
      public string Url { get; private set; }

      public CanonicalURL(string url)
      {
        Url = url;
      }

      public override void OnResultExecuting(ResultExecutingContext filterContext)
      {
        string path = filterContext.HttpContext.Request.Path;
        var controller = filterContext.Controller as BaseController;

        if (controller == null) return;

        if (string.IsNullOrWhiteSpace(this.Url)){
          controller.ViewBag.CanonicalUrl = controller.GetSiteUrl() + path;
        }
        else
        {
          controller.ViewBag.CanonicalUrl = controller.GetSiteUrl() + this.Url;
        }

        if (string.IsNullOrWhiteSpace(path) || "/".Equals(path))
        {
          controller.SetBingMetaTags();
        }

        base.OnResultExecuting(filterContext);
      }
    }


  3. Add following code into your _Layout.cshtml header

  4. <link rel="canonical" href="@ViewBag.CanonicalUrl" />

  5. Add the following declaration to each your controller actions
    (The SetTags call is for a different reason that we discussed earlier)

  6. [CanonicalURL(null)]
    public IActionResult About()
    {
      SetTags("About");
      return View();
    }


"[CanonicalURL(null)]" will set Canonical URL to start with default Domain setting that we set in pagetags.json
(metatags -> Sites -> Url -> http://www.softwareblogs.com)

If you use "[CanonicalURL("/Home/OldAbout")]", it will match an old action called "OldAbout" instead.