A WebCrawler demonstrating the Beauty of TPL Dataflow

[2012-Oct-13 Updated downloadable code to wait for cancel to complete]

Für mehr Informationen zu diesem Thema in Deutsch siehe meinen Artikel “Gleichzeitig zum Erfolg, Parallele Programmierung und Dataflow mit .NET” im dotnetpro Magazin, Heft 6/2013.

This post demonstrates the beauty of TPL Dataflow by implementing a simple web crawler. For an overview about Dataflow see:

This Dataflow web crawler implements the following features:

  • Download web pages asynchronously.
  • Download max 4 web pages in parallel.
  • Traverse  the web pages links tree.
  • Parse for links to images.
  • Download jpg images to disk.
  • Download the images using the new async I/O TAP method Stream.CopyToAsync.

Dataflow Web Crawler Architecture

Sub Main()
    'Define Blocks
    Dim downLoader As New TransformBlock(Of String, String)(
        Async Function(url)
            WriteLineInColor(url, ConsoleColor.White)
            Return Await New HttpClient().GetStringAsync(url).ConfigureAwait(False)
        End Function, New ExecutionDataflowBlockOptions With {.MaxDegreeOfParallelism = 4})

    Dim contentBroadcaster As New BroadcastBlock(Of String)(Function(html) html)

    Dim linkParser As New TransformManyBlock(Of String, String)(
            Dim doc As New HtmlDocument()
            Return From n In doc.DocumentNode.Descendants("a")
               Where n.Attributes.Contains("href")
               Let url = n.GetAttributeValue("href", "")
               Where url.StartsWith("http://")
               Select url
        End Function)

    Dim imageParser As New TransformManyBlock(Of String, String)(
            Dim doc As New HtmlDocument()
            Return From n In doc.DocumentNode.Descendants("img")
                               Where n.Attributes.Contains("src")
                               Let url = n.GetAttributeValue("src", "")
                               Where url.StartsWith("http://")
                               Select url
        End Function)

    Dim linkBroadCaster As New BroadcastBlock(Of String)(Nothing)

    Dim imageProcessor As New ActionBlock(Of String)(
       Async Function(url)
               Dim uri As New Uri(url)
               Dim fileUrl = If(String.IsNullOrWhiteSpace(uri.Query), uri.AbsoluteUri, uri.AbsoluteUri.Replace(uri.Query, ""))
               WriteLineInColor(fileUrl, ConsoleColor.Yellow)

               Dim webStream = Await (New HttpClient().GetStreamAsync(url)).ConfigureAwait(False)
               Dim filePath = New DirectoryInfo(Path.Combine(My.Application.Info.DirectoryPath, "..\..\..\Images", IO.Path.GetFileName(fileUrl))
               Using fileStream = IO.File.OpenWrite(filePath)
                   Await webStream.CopyToAsync(fileStream).ConfigureAwait(False)
               End Using
       End Function)

    'Handle unexpected errors
    downLoader.Completion.ContinueWith(Sub(ant) WriteLineInColor(ant.Exception.GetBaseException.ToString, ConsoleColor.Red),
    imageProcessor.Completion.ContinueWith(Sub(ant) WriteLineInColor(ant.Exception.GetBaseException.ToString, ConsoleColor.Red),

    'Link Blocks
    '.jpg just to demonstrate a link filter
    linkBroadCaster.LinkTo(imageProcessor, Function(url) url.EndsWith(".jpg"))

    'Let's start

    PromptUser("Crawling... Press  to abort:", ConsoleColor.White)

In BroadcastBlocks one should generally consider cloning the message to avoid consuming blocks working on the same instance. In this example that is not necessary, because the message is of type String and thus immutable. Thus in contentBroadcaster I simply pass the html. linkBroadCaster shows that one can omit the Action completely.

Download sample code.

This web crawler example was inspired by Bnaya Eshet’s post TPL Dataflow walkthrough.

Über Peter Meinl

Perpetual Traveller, IT Consultant
Dieser Beitrag wurde unter Computers and Internet abgelegt und mit , , , verschlagwortet. Setze ein Lesezeichen auf den Permalink.

2 Antworten zu A WebCrawler demonstrating the Beauty of TPL Dataflow

  1. svick schreibt:

    Few comments:

    1. Why are you using ConfigureAwait()? It has no effect in console applications.
    2. contentBroadcaster could have use Nothing, instead of identity lambda, same as linkBroadcaster.
    3. I think downloading to a file would be simpler using WebClient.DownloadFileTaskAsync(). HttpClient is newer, but that doesn’t mean you have to use it everywhere.
    4. Why is the code in the archive slightly different than the code you posted here?

  2. Peter Meinl schreibt:

    thanks for your quick and valuable comments.

    I am trying to make it a habit using ConfigureAwait(False) whenever I do not need the continuation to be executed on the original context. This way I avoid accidentally having the overhead of ConfigureAwait(True) when copying code to a library or application where it runs in a context.

    After reading „It’s all about the synchronization context“ http://msdn.microsoft.com/en-us/magazine/gg598924.aspx I am not sure if there really is not performance penalty in a console app, because the default synchronization context seems to be used. What do you think?

    Additionally I use ConfigureAwait in this example to trigger me talking with colleagues about the mere existence of it.

    Yes, I could use Nothing for the Action in both BroadcastBlocks. In my more detailled article (not published yet) I am using the lambda (Function(html) html) to explain that one commonly clones the message in the broadcast block. I now added a comment to this post too.

    You caught me here – always wanting to use the new stuff:-) Yes, WebClient.DownloadFileTaskAsync() would be simpler, I did not think of it.

    However, by default I prefer using the new HttpClient in async Scenarios, because WebClient sometimes seems to block the calling thread while initializing the async I/O. If I remember correctly when passing an invalid URL like http://www.micorosoft.com or when the network is disconnected.

    I sometimes keep the code shown in posts and articles simpler than in the downloadable version. In this post the download includes cancellation which I found would unnecessarily bloat the listing in the post.

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:


Du kommentierst mit Deinem WordPress.com-Konto. Abmelden /  Ändern )

Google Foto

Du kommentierst mit Deinem Google-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Twitter-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Facebook-Konto. Abmelden /  Ändern )

Verbinde mit %s