A WebCrawler demonstrating the Beauty of TPL Dataflow

[2012-Oct-14 Updated downloadable code to wait for cancel to complete]

Für mehr Informationen zu diesem Thema in Deutsch siehe meinen in Heft 6/2013 kommenden Artikel “Gleichzeitig zum Erfolg, Parallele Programmierung und Dataflow mit .NET” im dotnetpro Magazin.

This post demonstrates the beauty of TPL Dataflow by implementing a simple web crawler. For an overview about Dataflow see:

This Dataflow web crawler implements the following features:

  • Download web pages asynchronously.
  • Download max 4 web pages in parallel.
  • Traverse  the web pages links tree.
  • Parse for links to images.
  • Download jpg images to disk.
  • Download the images using the new async I/O TAP method Stream.CopyToAsync.

Dataflow Web Crawler Architecture

Continue reading

Posted in Computers and Internet | Tagged , , , | 2 Comments

Managing BLOBs using SQL Server FileStream via EF and WCF streaming

Für mehr Informationen zu diesem Thema in Deutsch siehe meinen Artikel “Stream das BLOB” im dotnetpro Magazin 6/2012.

With FileStream SQL Server 2008+ stores BLOBs in the NTFS file system instead of its table store and offers fast streaming access.

This post shows how to manage FileStream BLOBs via the Entity Framework (EF 4.1) DbContext API and how to stream them via Windows Communication Foundation (WCF).
Download sample code.
View UI demo video.

The main options for storing large objects (BLOBs) in SQL Server are:

  • Store BLOB in a DB table.
  • Store BLOB via FileStream in NTFS.
  • Store BLOB via FileTable in NTFS.
  • Store File in NTFS with a reference to its path in a DB Table.

If you prefer to manually store your BLOBs as NFTS Files consider adding transaction support, see Using Transactional NTFS (TxF) with SQL Server in .NET.

BLOB storage options

Continue reading

Posted in Computers and Internet | Tagged , , , , , , | 4 Comments

How to move from Hotmail and Gmail to Office 365 as a Server for Outlook desktop and Windows Phone 8

When moving to Office 365 I found it difficult to stop my Windows Phone syncing its calendar and contacts with Hotmail. Here is how I finally did it.

For the reasons I was using Hotmail and Gmail see Synchronize Windows Phone with Outlook Emails, Contacts, Calendar and Tasks.

Hotmail was the primary Windows account on my phone and Windows Phone does not allow changing the primary account without a hard reset (loosing some local information).
The phone’s settings under “Email+Account” for the primary account only allow to select if emails is synced. There are no checkboxes for calendar or contacts:

wp_ss_20130423_0010

Configuring Windows Phone to use Office 365 instead of Hotmail

To deselect syncing the Hotmail calendar use the Calendar app settings:

wp_ss_20130423_0009

Selecting “Show to-dos…” syncs Office 365 Tasks with the phone’s calendar.

To deselect syncing Hotmail contacts use the phone’s People app setting “filter my contact list”:

wp_ss_20130423_0007

Copying contacts, emails and tasks via copy&paste

During the migration process I had to move calendar entries, tasks and many (5000+) emails from one Server to another. I did this by configuring all server accounts in a Outlook desktop app and simply using copy&paste to copy the items.

With the calendar items and tasks I first filtered for the ones I wanted (Ex. NOT category:holiday) an then used CTRL-A to select all, CTRL-C to copy them and CTRL-V to paste them. There is no right-click menu item for pasting in Outlook. With some email folders I simply did a CTRL-A to select all messages and copied them.

The Outlook desktop app automatically synced the copied items to the Office 365 server. Even after the status bar said all folders were up to date and the status message “Updating sent items” no longer showed, the number of messages was not correct for a while. After several hours all messages were synced and my desktop and notebook Outlook showed the same number of items in all folders.

Posted in Computers and Internet | Tagged , , | Leave a comment

Installing Windows 8 on an old X61s ThinkPad

[11 Nov 2012: added Bluetooth support]
[25 Feb 2013: Add warning against installing the newer ThinkPad Hotkey Features Integration driver for Win8]

Windows 8 installed fine on my old ThinkPad X61s and runs nicely and fast (the little old thing has an SSD though :-) ).

Upgrade

First I did an upgrade install by selecting “Keep Windows settings, personal files and applications”.

  • I enabled Bluetooth before starting the upgrade because I read somewhere that it could not be  enabled in Win8 otherwise.
  • Win8 Setup asked to delete some incompatible applications (nothing I currently need), including MS Security Essentials.
  • Most of these applications could be uninstalled from within Win8 Setup by simply clicking a button.
  • A few I had to manually uninstall using “Programs and Features”.
  • Because I could not find a way to uninstall “Lenovo Auto Scroll Utility”, I uninstalled the ThinkPad TrackPoint driver. That stopped Win8 Setup complaining and it installed.
  • I had to manually install my EFS certificates.
  • I deleted the windows.old backup using the “Free up disk space…” to free disk space.

After these steps everything worked fine including the Fn keys. Only the battery manager reported to be not compatible

Refresh

Experimenting with Win8 I successfully did a Refresh:

  • Refresh took quite long (1+ h, hanging at 5% for about 30 min).
  • It removed all desktop application like Office, VS, SQL Sever, …
    And placed a RemovedApps.html file on the desktop.
  • It removed all ThinkPad specific drivers.
  • After deleting the windows.old backup using the “Free up disk space…” App (selection System files) there was a lot of free disk space.
  • After Refresh the X61 worked fine,
    including the TrackPoint and its buttons, the speaker buttons and the wireless switch.
  • Device Manager showed the device “Video Controller” not working.
    And connecting external monitors or beamers  via Win+P was not supported.
  • I installed Intel Chipset Support for Windows 8.
  • After this Window Update installed the chipset drivers.
  • Device Manager “search automatically” now found a driver for the Video Controller.
    Win+P now offers to connect external displays.
  • Bluetooth did not work yet.
    Because I wanted to use a ThinkPad laser mouse without an extra Bluetooth dongle I decided to get Bluetooth working. Because Lenovo did not offer Win8 hotkey drivers I installed the X61s Hotkey Features Integration for Windows 7.
    I set the compatibility of the setup.exe to Win7 and deselected the Lenovo Auto Scroll Utility” because Win8 reported compatibility issues with this. Win8 automatically installed Bluetooth with enhanced Data Rate. With FN+F5 working now I can enable Bluetooth my laser mouse works fine.

Issues

  • Win+”.” to split the Start Screen does not work.
    This is by Win8 design because of the low 1024 screen resolution of the X61s.
  • Installing the newer ThinkPad Hotkey Features Integration driver for Win8  3.84.1500 which was offered by Lenovo System Update did disable the hardware WiFi switch. I to revert to the older Win7 driver mentioned above by restoring to an older restore point using system restore.
Posted in Computers and Internet | Tagged , | 16 Comments

WP7 Background Agent Pitfalls and App/Agent Synchronization

To put this post into perspective see Windows Phone 7 App Development Helpers and Time Savers.

Background agents are a very important feature for WP7 apps. Agents allow you to run code in background while the app is not running – and also while it is running.

Some pitfalls you should bear in mind:

  • Beware of misleading agent memory usage reported under the debugger.
  • Create an appropriate VS solution structure to comply with unsupported APIs.
  • Web request default timeouts might be too long for agent duration constraint.
  • App/Agent communication must be synchronized.
  • VB.NET agent template bug.

Misleading agent memory usage under the debugger

PeriodicAgents have memory limit of 6MB. Agent duration and memory constraints are not enforced when debugging. With the debugger attached DeviceStatus reports a way higher memory usage then without the debugger. For an empty PeriodicTask DeviceStatus.ApplicationPeakMemoryUsage reports 3.5 MB with the debugger and 1.8 MB without the debugger. To get your agents real memory usage write to IsolatedStorage or a Tile or show a Toast.

VS solution structure

Some APIs like Phone.Shell are unsupported in Agents. Because the submission process enforces this using static code analysis, your submission will not only fail when using an unsupported API, but also when you simply reference it. To avoid painful refactorings, you should plan your solution structure accordingly from the beginning on.

The solution structure of the demo application is optimized to adhere to the agent API and resource constraints. The projects are numbered to show the reference hierarchy.

Solution structure optimized for Agent API and memory constraints

VB.NET agent template bug

With VB.NET Visual Studio, Add Agent Project creates a Class ScheduledAgent with an invalid constructor. I wrongly names the constructor Sub ScheduledAgent(), correctly it  should be Sub New(). This bug results in your error handler never being registered and might lead to your agent being disabled on the users phone without you ever knowing.

Web request default timeouts might be too long for agent duration constraint

PeriodicAgents are constraint to 25 sec duration. WebClient and HttpWebRequest seem to have a default timeout of approx. 20 sec. Because both do not support to set at timeout you should consider implementing a timeout manually. When doing so, don’t forget the ignore WebExceptions with WebExceptionStatus.RequestCanceled that might arrive after cancelling.

App/Agent communication must be synchronized

Microsoft’s recommendations for app/agent communications are very strict: For one-direction communication where the foreground application writes and the agent only reads, we recommend using an isolated storage file with a Mutex. … We recommend that you do not use IsolatedStorageSettings to communicate between processes because it is possible for the data to become corrupt“.

The following sample uses a Mutex for synchronization:

Public Class IsolatedStorageFile
    Public Shared SerializationFormatter As SerializationFormatter = SerializationFormatter.Binary
    'Beware! XmlSerializer can not seriallize TimeSpan
    'Public Shared SerializationFormatter As SerializationFormatter = Threading.SerializationFormatter.DataContract

    Private Const _timoutToPreventBlocking As Integer = 5 * 1000

    Public Shared Sub WriteSynchronized(filePath As String, fileContent As Object)
        filePath = SetFileNameExtension(filePath)

        Dim mutex As New Mutex(False, filePath)
        Try
            If mutex.WaitOne(_timoutToPreventBlocking) Then
                Using isoFile = IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication(),
                        isoStream = isoFile.OpenFile(filePath, FileMode.Create)
                    Select Case SerializationFormatter
                        Case SerializationFormatter.Binary
                            SilverlightSerializer.Serialize(fileContent, isoStream)
                        Case SerializationFormatter.XML
                            Dim serializer = New XmlSerializer(fileContent.GetType)
                            serializer.Serialize(isoStream, fileContent)
                        Case SerializationFormatter.DataContract
                            Dim serializer = New DataContractSerializer(fileContent.GetType)
                            serializer.WriteObject(isoStream, fileContent)
                    End Select
                End Using
            Else
                'This should never happen. The phone runtime even releases mutexes for aborted processes. But better safe than sorry!
                Dim ex As New ThreadStateException("Unable to acquire mutex for app/agent sync")
                Trace.Error(ex.Message)
                Throw ex
            End If
        Finally
            mutex.ReleaseMutex()
        End Try
    End Sub

    Public Shared Function ReadSynchronized(Of contentType)(filePath As String) As contentType
...
    End Function

End Class
Posted in Computers and Internet | Tagged , | Leave a comment

Windows Phone 7 App Development Helpers and Time Savers

Für mehr Informationen zu diesem Thema in Deutsch siehe meine kommenden Artikel “Phone, Schweiß und Tränen” und “Aus App-Gründen ans Licht” im dotnetpro Magazin, Ausgaben 9 und 10/2012.

The Windows Phone is a fascinating app platform and Microsoft’s Visual Studio / Blend toolset is extremely powerful, but Silverlight can be a huge time sink and developing for the Windows Phone adds some. To  help streamlining your WP7 app development I am sharing some helpful experiences from developing my Currencies app.
Download sample code.
View Currencies demo video.

Balsamiq UI Sketch and Screenshot of Published App

Links to all posts of this series [I will add the missing posts as I find time to do so]:

  • User Experience
  • App Architecture
  • No MVVM
  • Development Tools
  • Fast Launch and Fluid UX
  • Asynchronous Web Requests
  • Use Tracing to understand your Apps Event Sequences
  • Good Error Handling
  • Agent Pitfalls and App/Agent Synchronization
  • Keypad (SIP) Tips
Posted in Computers and Internet | Tagged , | Leave a comment

Using Transactional NTFS (TxF) with SQL Server in .NET

I just stumbled across the very new Transactional NTFS (TxF) .NET managed wrapper by Pietro Partescano and gave it a try.

TxF offers transactional protection for NTFS operations:

  • Atomic operations on single files.
    Makes writing files more robust.
  • Transactions for multi-file operations.
    Concurrent readers only see committed operations.
    Works with remote file shares too.
  • The TxF resource manager can participate in heterogeneous distributed transactions.
    Ex: spanning NFTS, SQL Server and MSMQ operations.

MSDN: About Transactional NTFS
MSDN Magazine article: Enhance Your Apps With File System Transactions
Wikipedia: Transactional NTFS

Sample using NTFS Transaction with SQL Server

CREATE TABLE [dbo].[DocumentLink](
	[DocumentID] [uniqueidentifier] ROWGUIDCOL primary key  NOT NULL,
	[Name] [varchar](255) UNIQUE  NOT NULL,
	[Type] [varchar](5) NULL)
Imports System.Transactions

Module Module1
    Private _testFolderPath = IO.Path.Combine(IO.Directory.GetParent(My.Application.Info.DirectoryPath).Parent.FullName, "TestFiles")

    Sub Main()
        Try
            Using tsc As New TransactionScope(TransactionScopeOption.RequiresNew,
                                              New TransactionOptions With {.IsolationLevel = IsolationLevel.ReadCommitted}),
                  DB As New FileStreamTestEntities

                'Create file in NTFS
                Dim filePath = IO.Path.Combine(_testFolderPath, "TestFile1.txt")
                Dim content = System.Text.Encoding.UTF8.GetBytes("My test data")
                Dim fileHandle = TxF.File.CreateAndWriteFile(filePath, TxF.File.CreationDisposition.CreatesNewfileAlways, content)

                'Insert link to file in database table
                DB.DocumentLinks.Add(New DocumentLink With {.DocumentID = Guid.NewGuid, .Name = filePath})
                DB.SaveChanges()

                tsc.Complete()
            End Using
            Console.WriteLine("done")
        Catch ex As Exception
            Console.WriteLine(ex.ToString)
        Finally
            Console.ReadLine()
        End Try
    End Sub

End Module
Posted in Computers and Internet | Tagged , , , , | Leave a comment