A WebCrawler demonstrating the Beauty of TPL Dataflow

[2012-Oct-13 Updated downloadable code to wait for cancel to complete]

Für mehr Informationen zu diesem Thema in Deutsch siehe meinen in Heft 6/2013 kommenden Artikel “Gleichzeitig zum Erfolg, Parallele Programmierung und Dataflow mit .NET” im dotnetpro Magazin.

This post demonstrates the beauty of TPL Dataflow by implementing a simple web crawler. For an overview about Dataflow see:

• TPL Dataflow Tour Video
• Introduction to TPL Dataflow Whitepaper
• Dataflow (Task Parallel Library) MSDN Overview
• Inside TPL Dataflow Video

This Dataflow web crawler implements the following features:

• Download web pages asynchronously.
• Download max 4 web pages in parallel.
• Traverse  the web pages links tree.
• Parse for links to images.
• Download jpg images to disk.
• Download the images using the new async I/O TAP method Stream.CopyToAsync.

Dataflow Web Crawler Architecture

Managing BLOBs using SQL Server FileStream via EF and WCF streaming

Für mehr Informationen zu diesem Thema in Deutsch siehe meinen Artikel “Stream das BLOB” im dotnetpro Magazin 6/2012.

With FileStream SQL Server 2008+ stores BLOBs in the NTFS file system instead of its table store and offers fast streaming access.

This post shows how to manage FileStream BLOBs via the Entity Framework (EF 4.1) DbContext API and how to stream them via Windows Communication Foundation (WCF).
Download sample code.
View UI demo video.

The main options for storing large objects (BLOBs) in SQL Server are:

• Store BLOB in a DB table.
• Store BLOB via FileStream in NTFS.
• Store BLOB via FileTable in NTFS.
• Store File in NTFS with a reference to its path in a DB Table.

If you prefer to manually store your BLOBs as NFTS Files consider adding transaction support, see Using Transactional NTFS (TxF) with SQL Server in .NET.

BLOB storage options

My Windows 8.1 upgrade and usage experiences

This post describes my experiences with upgrading from Win7 to Win81 and using it for a few days. I hope this will help others to avoid some problems and make them aware of (silently) missing functionality in Win81. I posted this information for discussion in the Windows 8.1 IT Pro forum too.

Trying to do a backup in Win7 before the upgrade resulted in error “not enough disk space to create shadow copy.” This was caused be a test install of Win81 Preview as dual-boot which made a change to the common hidden System Reserved partition which did break Win7 Backup.
I fixed this using fsutil usn deletejournal /N /D F: as suggested here.

To keep my Win7 settings and desktop applications I did first upgrade to Win8 because MS removed keeping these with the Win81 upgrade.

The first upgrade rolled back with errors. Digging through the hidden error log C:\$Windows.~BT\Sources\Rollback\setupact.err revealed some drivers belonging to VMWare and DAEMON Tools causing problems. After uninstalling VMWare and DAEMON the upgrades worked fine. I remember some sort of compatibility check with previous test installs of Win8 and do not know if they were not executed during this upgrade or do not detect VMware and DAEMON generally.

Win 8.1 does (in contrast to Win7 and Win8) not(!) include a solution for a complete file based backup.

• Image backup is still available, but difficult to find: A Link on the bottom left of “Control Panel\All Control Panel Items\File History”, or you can use wbAdmin on a command line.
• Windows 7 Files Backup was removed in Win 8.1.
• Win81 File history does (in contrast to Win8) not backup SkyDrive, it does still backup SkyDrive Pro though.
  Remember: syncing to the cloud is no replacement for a backup.

File History silently(!) skips EFS encrypted files. The event log shows the following “funny” error “…If you want it to be protected, remove encryption …”. Silently skipping the most important files is totally unacceptable, there should at least be a warning in action center.

With these limitations of File History sadly we are forced to use 3rd party backup solutions.
As feared my first try, using Acronis True Image 2014, was ugly. TI does not officially support Win81 yet (one of the problems why I prefer using built-in tools for basic tasks like backup). While TI worked fine with EFS and SkyDrive a reboot did hang for a night with the message “…operations pending…”. This is a known Acronis problem maybe not even related to Win81. Because the machine at this stage of shutdown did not allow me to remote login to kill the Acronis processes I was forced to do a hard reset – yuck.

SkyDrive and SkyDrive Pro do not support EFS encrypted files. The explanations for this are questionable at least. Why can’t SkyDrive just do a binary copy when syncing (like robocopy \efsraw does)? And yes, if one would not have the EFS certificate installed on each machine or even loose it the files would not be readable – security does not come for free. Remember: Private files should only be stored in the cloud when encrypted with a key controlled by the user only (transport encryption and encryption in the cloud is not sufficient). Not supporting EFS or offering an alternative solution forces us to trust yet another party like BoxCryptor. With SkyDrive being tightly integrated into Win81 and MS urging us to use it user controlled encryption is mandatory and no longer just nice to have.

There are reports of people having their SkyDrive account locked by Microsoft because the content or even the file names only did violate Microsoft’s very wide code of conduct. This happened with files that were private only, not shared with others.

The search charm is too slow, see post.
With Win7 / Start Menu or Win81 / Classic Shell searches for applications are instantaneous.

The reminder popups of the Mail app obscure the search box when searching right after logging in.

Unlocking BitLocker drives with a right-click form File Explorer does not work. Error “Application not found”.
The Fix it did not solve this for me. Editing the registry manually worked.

The All Apps view of the Start Menu is unusable for me.
I my case it shows a desert with hundreds of icons. Such a view is good for a folder with photo thumbnails but not for a list of apps. The icons are only distracting without helping to find something. The grouping titles are too pale. We need a filterable text-only view here. I just discovered a helpful but difficult to discover view of the all Apps view. In Apps view click on the minus in the bottom right corner an the following group view shows:

My Officejet Pro 8500 no longer supports scanning. No fix from HP available.

NUMLOCK is always disabled after startup. Switching off “fast startup” as suggested in the forums is not a viable solution. The registry change suggestions I found do not solve this problem.

The desktop task bar was annoyingly transparent. The only way I was able to change this was via Classic Shell settings. I hope this change persists if I decide to uninstall Classic Shell.

After playing with the new navigation setting “Show my desktop background on Start” the desktop icon labels showed with drop shadows which looks very ugly with solid color backgrounds. Had to edit the registry to get rid of this. Removing the drop shadows only works with”Show my desktop background on Start” disabled. The problem still surfaces from time to time and logging off and on again fixes it.

Some Windows updates (KB2887595, KB2837642) did hang causing countless reboots and forcing me to hard reset the machine. I never managed to install the cumulative update KB2887595.

Windows Update leaks disk space. After using Win 8.1 for some days my system SSD was full. 40+ GB free space had disappeared. This was solved by the Windows update rollbacks caused by the update hangs mentioned above.

Hyper-V seems to leak memory. Even after shutting down all VMs and applications Win 8.1 still uses nearly all of my 8GB RAM. Only a reboot frees it.

As a Windows Phone developer one is forced to migrate to Win8 because the phone emulator requires Hyper-V.
Hyper-V (needlessly?) requires the special hardware visualization feature SLAT, forcing me the abandon otherwise perfectly good notebooks.

VMWare does not install on a machine with Hyper-V enabled. This forces me to move from VMWare to Hyper-V on all machines sharing VM Images. Hyper-V is cumbersome to use: It is overly complicated to copy or move VMs. Without clicking on settings one cannot see where the files reside. The UI does not have end-user quality. Setting to share USB and other host devices difficult to discover, sometimes simply not reachable.

Copying passwords, credit card infos etc. from my password manager 1Password to Modern Apps is cumbersome.

Things I like in Win 8.1:

• Works fine on my old Thinkpad X61s notebook
• Improved file management
  • Quicker delete, especially when bypassing recycle bin using <Shift>.
  • Copying EFS files to remote folders is much quicker than in Win 7.
  • More robust external drive handling. Win 8.1 still does not allow to eject USB sticks sporadically.
  • Less problems with “file already exists” when copying large amounts of files.

How to move from Hotmail and Gmail to Office 365 as a Server for Outlook desktop and Windows Phone 8

When moving to Office 365 I found it difficult to stop my Windows Phone syncing its calendar and contacts with Hotmail. Here is how I finally did it.

For the reasons I was using Hotmail and Gmail see Synchronize Windows Phone with Outlook Emails, Contacts, Calendar and Tasks.

Hotmail was the primary Windows account on my phone and Windows Phone does not allow changing the primary account without a hard reset (loosing some local information).
The phone’s settings under “Email+Account” for the primary account only allow to select if emails is synced. There are no checkboxes for calendar or contacts:

Configuring Windows Phone to use Office 365 instead of Hotmail

To deselect syncing the Hotmail calendar use the Calendar app settings:

Selecting “Show to-dos…” syncs Office 365 Tasks with the phone’s calendar.

To deselect syncing Hotmail contacts use the phone’s People app setting “filter my contact list”:

Copying contacts, emails and tasks via copy&paste

During the migration process I had to move calendar entries, tasks and many (5000+) emails from one Server to another. I did this by configuring all server accounts in a Outlook desktop app and simply using copy&paste to copy the items.

With the calendar items and tasks I first filtered for the ones I wanted (Ex. NOT category:holiday) an then used CTRL-A to select all, CTRL-C to copy them and CTRL-V to paste them. There is no right-click menu item for pasting in Outlook. With some email folders I simply did a CTRL-A to select all messages and copied them.

The Outlook desktop app automatically synced the copied items to the Office 365 server. Even after the status bar said all folders were up to date and the status message “Updating sent items” no longer showed, the number of messages was not correct for a while. After several hours all messages were synced and my desktop and notebook Outlook showed the same number of items in all folders.

Installing Windows 8 on an old X61s ThinkPad

[11 Nov 2012: added Bluetooth support]
[25 Feb 2013: Add warning against installing the newer ThinkPad Hotkey Features Integration driver for Win8]

Windows 8 installed fine on my old ThinkPad X61s and runs nicely and fast (the little old thing has an SSD though :-)).

Upgrade

First I did an upgrade install by selecting “Keep Windows settings, personal files and applications”.

• I enabled Bluetooth before starting the upgrade because I read somewhere that it could not be  enabled in Win8 otherwise.
• Win8 Setup asked to delete some incompatible applications (nothing I currently need), including MS Security Essentials.
• Most of these applications could be uninstalled from within Win8 Setup by simply clicking a button.
• A few I had to manually uninstall using “Programs and Features”.
• Because I could not find a way to uninstall “Lenovo Auto Scroll Utility”, I uninstalled the ThinkPad TrackPoint driver. That stopped Win8 Setup complaining and it installed.
• I had to manually install my EFS certificates.
• I deleted the windows.old backup using the “Free up disk space…” to free disk space.

After these steps everything worked fine including the Fn keys. Only the battery manager reported to be not compatible

Refresh

Experimenting with Win8 I successfully did a Refresh:

• Refresh took quite long (1+ h, hanging at 5% for about 30 min).
• It removed all desktop application like Office, VS, SQL Sever, …
  And placed a RemovedApps.html file on the desktop.
• It removed all ThinkPad specific drivers.
• After deleting the windows.old backup using the “Free up disk space…” App (selection System files) there was a lot of free disk space.
• After Refresh the X61 worked fine,
  including the TrackPoint and its buttons, the speaker buttons and the wireless switch.
• Device Manager showed the device “Video Controller” not working.
  And connecting external monitors or beamers  via Win+P was not supported.
• I installed Intel Chipset Support for Windows 8.
• After this Window Update installed the chipset drivers.
• Device Manager “search automatically” now found a driver for the Video Controller.
  Win+P now offers to connect external displays.
• Bluetooth did not work yet.
  Because I wanted to use a ThinkPad laser mouse without an extra Bluetooth dongle I decided to get Bluetooth working. Because Lenovo did not offer Win8 hotkey drivers I installed the X61s Hotkey Features Integration for Windows 7.
  I set the compatibility of the setup.exe to Win7 and deselected the Lenovo Auto Scroll Utility” because Win8 reported compatibility issues with this. Win8 automatically installed Bluetooth with enhanced Data Rate. With FN+F5 working now I can enable Bluetooth my laser mouse works fine.

Issues

• Win+”.” to split the Start Screen does not work.
  This is by Win8 design because of the low 1024 screen resolution of the X61s.
• Installing the newer ThinkPad Hotkey Features Integration driver for Win8  3.84.1500 which was offered by Lenovo System Update did disable the hardware WiFi switch. I to revert to the older Win7 driver mentioned above by restoring to an older restore point using system restore.

WP7 Background Agent Pitfalls and App/Agent Synchronization

To put this post into perspective see Windows Phone 7 App Development Helpers and Time Savers.

Background agents are a very important feature for WP7 apps. Agents allow you to run code in background while the app is not running – and also while it is running.

Some pitfalls you should bear in mind:

• Beware of misleading agent memory usage reported under the debugger.
• Create an appropriate VS solution structure to comply with unsupported APIs.
• Web request default timeouts might be too long for agent duration constraint.
• App/Agent communication must be synchronized.
• VB.NET agent template bug.

Misleading agent memory usage under the debugger

PeriodicAgents have memory limit of 6MB. Agent duration and memory constraints are not enforced when debugging. With the debugger attached DeviceStatus reports a way higher memory usage then without the debugger. For an empty PeriodicTask DeviceStatus.ApplicationPeakMemoryUsage reports 3.5 MB with the debugger and 1.8 MB without the debugger. To get your agents real memory usage write to IsolatedStorage or a Tile or show a Toast.

VS solution structure

Some APIs like Phone.Shell are unsupported in Agents. Because the submission process enforces this using static code analysis, your submission will not only fail when using an unsupported API, but also when you simply reference it. To avoid painful refactorings, you should plan your solution structure accordingly from the beginning on.

The solution structure of the demo application is optimized to adhere to the agent API and resource constraints. The projects are numbered to show the reference hierarchy.

Solution structure optimized for Agent API and memory constraints

VB.NET agent template bug

With VB.NET Visual Studio, Add Agent Project creates a Class ScheduledAgent with an invalid constructor. I wrongly names the constructor Sub ScheduledAgent(), correctly it  should be Sub New(). This bug results in your error handler never being registered and might lead to your agent being disabled on the users phone without you ever knowing.

Web request default timeouts might be too long for agent duration constraint

PeriodicAgents are constraint to 25 sec duration. WebClient and HttpWebRequest seem to have a default timeout of approx. 20 sec. Because both do not support to set at timeout you should consider implementing a timeout manually. When doing so, don’t forget the ignore WebExceptions with WebExceptionStatus.RequestCanceled that might arrive after cancelling.

App/Agent communication must be synchronized

Microsoft’s recommendations for app/agent communications are very strict: For one-direction communication where the foreground application writes and the agent only reads, we recommend using an isolated storage file with a Mutex. … We recommend that you do not use IsolatedStorageSettings to communicate between processes because it is possible for the data to become corrupt“.

The following sample uses a Mutex for synchronization:

Public Class IsolatedStorageFile
    Public Shared SerializationFormatter As SerializationFormatter = SerializationFormatter.Binary
    'Beware! XmlSerializer can not seriallize TimeSpan
    'Public Shared SerializationFormatter As SerializationFormatter = Threading.SerializationFormatter.DataContract

    Private Const _timoutToPreventBlocking As Integer = 5 * 1000

    Public Shared Sub WriteSynchronized(filePath As String, fileContent As Object)
        filePath = SetFileNameExtension(filePath)

        Dim mutex As New Mutex(False, filePath)
        Try
            If mutex.WaitOne(_timoutToPreventBlocking) Then
                Using isoFile = IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication(),
                        isoStream = isoFile.OpenFile(filePath, FileMode.Create)
                    Select Case SerializationFormatter
                        Case SerializationFormatter.Binary
                            SilverlightSerializer.Serialize(fileContent, isoStream)
                        Case SerializationFormatter.XML
                            Dim serializer = New XmlSerializer(fileContent.GetType)
                            serializer.Serialize(isoStream, fileContent)
                        Case SerializationFormatter.DataContract
                            Dim serializer = New DataContractSerializer(fileContent.GetType)
                            serializer.WriteObject(isoStream, fileContent)
                    End Select
                End Using
            Else
                'This should never happen. The phone runtime even releases mutexes for aborted processes. But better safe than sorry!
                Dim ex As New ThreadStateException("Unable to acquire mutex for app/agent sync")
                Trace.Error(ex.Message)
                Throw ex
            End If
        Finally
            mutex.ReleaseMutex()
        End Try
    End Sub

    Public Shared Function ReadSynchronized(Of contentType)(filePath As String) As contentType
...
    End Function

End Class

Windows Phone 7 App Development Helpers and Time Savers

Für mehr Informationen zu diesem Thema in Deutsch siehe meine kommenden Artikel “Phone, Schweiß und Tränen” und “Aus App-Gründen ans Licht” im dotnetpro Magazin, Ausgaben 9 und 10/2012.

The Windows Phone is a fascinating app platform and Microsoft’s Visual Studio / Blend toolset is extremely powerful, but Silverlight can be a huge time sink and developing for the Windows Phone adds some. To  help streamlining your WP7 app development I am sharing some helpful experiences from developing my Currencies app.
Download sample code.
View Currencies demo video.

Balsamiq UI Sketch and Screenshot of Published App

Links to all posts of this series [I will add the missing posts as I find time to do so]:

• User Experience
• App Architecture
• No MVVM
• Development Tools
• Fast Launch and Fluid UX
• Asynchronous Web Requests
• Use Tracing to understand your Apps Event Sequences
• Good Error Handling
• Agent Pitfalls and App/Agent Synchronization
• Keypad (SIP) Tips