Thoughts and Experiments on Cloud Encryption

[ Dies ist ein Auszug aus meinem Artikel “Wolkentresor” im dotnetpro Magazin 9.2015 mit einigen ergänzenden Abbildungen ]

Private files should only be stored in the cloud being encrypted via a key controlled and known by the user solely (= end-to-end encryption). Transport encryption and encryption in the cloud provided by storage providers are not sufficient here.

Attack Surface

The safety of encryption depends on good passwords. Passwords are good if they are not only strong but also easy to use. For an interesting discussion about better passwords see Toward Better Master Passwords.

Some cloud storage providers like Tresorit support automatic end-to-end encryption out of the box. For an overview and reviews of cloud storage providers and VPN services see Cloudwards. Many widely used services like DropBox and the deeply in Windows integrated OneDrive are sadly lacking this support. For these one can use additional encryption tools like:

boxcryptor: Good commercial tool.
CryptSync: Free open source tool based on 7-zip encryption.
Allows decrypting files even without CrypSync using 7-zip apps.

Global Error Handling for UWP-Apps

[ Für Details zur Fehlerbehandlung in .NET im Allgemeinen, unter Berücksichtigung von von multi-threaded Programmierung (insbesondere mit Task und Await)  siehe meinen Artikel „Lass da mal was sein“ im dotnetpro Magazin 3/2013.
Für Details zur Fehlerbehandlung in UWP-Apps siehe meinen Artikel „Was war da los?“ im dotnetpro Magazin 12/2016. ]

This post is the result of my experiments with handling unexpected errors in UWP-Apps.
For my approach about tracing see Tracing for UWP-Apps

It is debatable if showing information about unexpected exceptions to users is good idea. I have had Mail, Translator, Calculator, the Store app and other UWP-Apps abort without any feedback on crash or at restart. I personally find it annoying when apps simply disappear without any hint. The „little  Watson“ solution of displaying error info on the next app start feels weird to me. Both behaviors doe not conform with common user expectations.

If crash error details can be helpful for end users depends on the app type and the target audience. For enterprise apps I know from long experience that error notifications (even with deep technical infos) are definitely helpful. Users are often able to circumvent problems and keep working with defective apps without even contacting support (while support might have been made aware of the problem automatically in the background…).

Options to inform users about unexpected exceptions are:

• Never notify users about exceptions.
• Show dialog in App.UnhandledException
• Show dialog on next app start.
• Show exception details like Message and HResult
• Show „Sorry…“ only.

This following code sample implements the main options, configurable via _notifyUserAboutCrash.

Tracing for UWP-Apps

There are several tools available for tracing in UWP Apps:

• MetroLog
• PortiLog
• Event Tracing for Windows (ETW)
  via Diagnostics.FileLoggingSession

I wanted to use ETW via Diagnostics.FileLoggingSession but it proved to be unstable and there was no way to convert the etl files to human readable form to allow app users to examine the content before sending them as email attachments.

My main requirements for tracing tools are:

• Tool errors must never crash the app.
• Minimal impact on app performance.
• Ability to specify maximum file storage usage.
• Inject trace location (method name, line number).
• Unobtrusive trace calls that do not obscure the app code.

I make a distinction between tracing the program flow and variable values for developer usage and logging unexpected exceptions and important status messages for system management.

None of the tools available fulfilled all these requirement to my liking. Thus I wrote a little class Trace with the following features:

• Stores traces in alternating text files in ApplicationData.
• Short static trace methods.
• Trace levels Verbose, Debug, Info, Time, Status, Warning, Error, Fatal, Off
  The level „Status“ is intended to filter messages for logging errors and important status messages like started and stopped. This level is missing in many tracing frameworks and leads to either flooding logs intended for system management with info messages or not logging important status messages.
• Inject trace location via compiler generated parameter values like CallerMemberName and CallerLineNumber. These point to the correct source code location even in release. mode (when the app was compiled with the .NET Native toolchain.
• Buffer traces in a TPL BatchBlock to improve performance when writing to storage.
• CyclicBuffer to allow to trace the last n messages on error.
  Experimental feature only. Have not used it in production yet. Don’t know if it still works.

Download complete code.

Sample Trace Output

160628 18:57:47.837 Trace.Init@75|========== Travel Currencies V1.1.42.0
160628 18:57:47.837 Trace.Init@76|traceOutputLevel=Debug: traceTragets=Debug, BufferedFile traceSource=BackgroundTask
160628 18:57:47.837 Trace.Init@77|==========
160628 18:57:47.837 RefreshRatesTask.Run@24|->
160628 18:57:48.484 RefreshRatesTask.Run@39|Refreshing rates. RatesSource=Web
160628 18:57:48.488 License.GetLicenseInfo@37|
160628 18:57:48.495 License.GetLicenseInfo@42|LicenseInformation: isTrial=True TrialExpiration=12/31/9999 1:00:00 AM +01:00 isActive=True 
160628 18:57:48.498 Globals.CheckLicensingInformUser@35|
160628 18:57:48.503 PublisherCacheFile..ctor@13| folder.Path=C:\Users\pit\AppData\Local\Publishers\bmpfaqh7gxjd4\Currencies
160628 18:57:48.503 CurrentRatesSource.GetRatesFromFileAsync@40|
160628 18:57:48.511 PublisherCacheFile.ReadObjectAsync@38|->
160628 18:57:48.682 PublisherCacheFile.ReadObjectAsync@52|<- 160628 18:57:48.687 _RatesSourceBase.LoadRatesFromFileAsync@34|Found cached rates. RatesAsOfDate=6/28/2016 4:00:21 PM +00:00 160628 18:57:48.688 CurrentRatesSource.GetRatesFromWebAsync@51| 160628 18:57:48.688 PublisherCacheFile..ctor@13| folder.Path=C:\Users\pit\AppData\Local\Publishers\bmpfaqh7gxjd4\Currencies 160628 18:57:48.691 WebRatesSource.RefreshRatesFromWebAsync@10|->
160628 18:57:49.396 WebRatesSource.RefreshRatesFromWebAsync@27|RatesAsOfDate Utc=6/28/2016 4:00:21 PM +00:00 RatesAsOfDate Local=6/28/2016 6:00:21 PM +02:00
160628 18:57:49.396 WebRatesSource.RefreshRatesFromWebAsync@33|<-
160628 18:57:49.396 RatesRefresher.RefreshRatesToFile@64|Got new rates as of UTC 6/28/2016 4:00:21 PM +00:00

...

Quirks with date and number formats in UWP Apps and Windows 10

In Windows 10 users can configure several languages, a region, a Regional format and additionally several on-screen keyboards (OSK) in different languages. How to switch between OSK and their handling of region-specific decimal separators is not immediately obvious. UWP apps do by design ignore the Regional format configured by the user and there is a debate if developers should work around this.

Don’t Make the User Think

When displaying dates and numbers and designing data entry it is important to make intuitively clear what the data format is. We must avoid making users think to determine if a number is a day or a month or if a . (dot) is a decimal or a thousand separator.

Good Gear and Tools for Digital Nomads (with DE shopping links)

[ April 11, 2016: created ]
Für einige Überlegungen zum digitalen Nomadetum siehe meinen Artikel “.NET unter Palmen” im dotnetpro Magazin 3.2016.
[ Jan 22, 2018: Digital Nomad 35 backpack added ]

Digital nomads need robust and flexible gear to perform their work duties. Because I frequently get asked for good traveling stuff this post recommends hardware and software for the Windows ecosystem and other things helpful for working while traveling.

Good Gear and Tools for Digital Nomads (with US shopping links)

[ April 11, 2016: created ]
Für einige Überlegungen zum digitalen Nomadetum siehe meinen Artikel “.NET unter Palmen” im dotnetpro Magazin 3.2016.
[ Jan 22, 2018: Digital Nomad 35 backpack added ]

Digital nomads need robust and flexible gear to perform their work duties. Because I frequently get asked for good traveling stuff this post recommends hardware and software for the Windows ecosystem and other things helpful for working while traveling.

How to completely clean install Windows 10 on a UEFI PC

With Windows 10 I generally do recommend to simply upgrade which preserves your apps (modern and desktop), data and settings. In contrast to previous Windows versions the Win 10 upgrade is robust and has worked fine for me on several machines, so I consider a clean install not being worth the time needed.

With my Surface Pro 3 (SP3) however I wanted to do a clean install, including removing Windows 8.1 partitions no longer needed. A Win 10 upgrade from Win 8.1 left my Surface with the partitions shown in the picture below. I could not find any reliable information on which partitions are safe to delete.

Partitions after upgrading from Win 8.1 to Win 10

A clean install as described below removed 2 unneeded partitions and freed 5+ GB SSD space:

Partitions after Win 10 clean install

Windows Service Worker Options

[Für Details siehe meinen Artikel „Perfekter Service“ im dotnetpro Magazin 1/2016.]

Implementation patterns used for Windows Services can seriously influence performance, reactivity, computing resource consumption, stability and energy consumption of our systems. This post shares alternative patterns for creating Windows Services. Download complete code.

Design Aspects

When designing Windows Services consider the following aspects:

• How and when should the service be started?
• If OnStart() takes > 30 sec the SCM will abort the service.
• Choose a suitable pattern to implement processing. Ex: Event-driven, polling loops, poll-timers.
• Maintain thread hygiene.
• If OnStop() takes >~90 sec the SCM will abort the service. Exact time limit is undocumented and Windows version specific.
• Implement graceful shutdown.
• Windows Services must not have a direct UI. You can however create a UI and control the service via its OnCustomCommand method or implement a custom WCF-Interface, see Simple WCF-Services.
• Robust error handling.
• Sufficient tracing and logging.
• Windows Services must be installed.
• Windows Services cannot be started directly in the IDE. See test console app below.

Simple WCF

[Für Details siehe meinen kommenden Artikel “WCF Leicht und robust” im dotnetpro Magazin 1/2016.]

Choosing WCF options and configuring them can be quite cumbersome and demanding. This post shares a very simple approach for using WCF services. Download complete code. My simple approach uses:

• Shared Interface-DLL
• Self-hosting in a Windows Service.
  See Windows Service Worker Options.
• Channel proxies instead of ClientBase proxies.
• Simplified WCF configuration.
• Custom method CreateChannel().
  Caching ChannelFactories.
• Custom method CloseChannel().
  Handling Channel failures.

I am using a new Channel instance for each WCF call. This improves robustness against communication problems and against „The socket connection was aborted“ exceptions after a client idle time > 10 min (= default ReceiveTimeout) with netTcpBinding even when using InstanceContextMode=PerCall.

Tamed FileSystemWatcher

[Für Details siehe meinen Artikel “Gezähmte Beobachter” im Windows Developer Magazin, Sept 2015]

[July 11, 2015: Fixed bugs in code. See changelog in readme.txt]
[Feb 21, 2018: Fixed bug in code: Filter no being applied with OrderByOldestFirst=false]
[Mar 02, 2018: Fixed existing files event argument fileName passing full path with OrderByOldestFirst=false]
[Jan 14, 2019: Fixed with IncludeSubdirectories=true path for preexisting files was SearchPath instead of subdirectory path.
Fixed IncludeSubdirectories=true was ignored with OrderByOldestFirst=false.
[Jan 18, 2019: Fixed Deleted handler not working.]
[May 0518, 2020: Fixed FilePoller download.]

This post shares robust wrappers around the standard FileSystemWatcher (FSW)  fixing problems commonly encountered when using it to monitor the file system in real-world applications.

Buffering and Recovering FSW

Simply replace the standard FSW with my BufferingFileSystemWatcher and you no longer need to worry about InternalBufferOverflowExceptions. Use my RecoveringFileSystemWatcher to automatically recover from typical transient watch path accessibility problems. Download complete code. For a file system watcher using polling instead of file system events see my FilePoller. To process files detected in either way I recommend using TPL DataFlow ActionBlocks. They allow you to easily process files without having to spawn a Thread or create a Task yourself and allow to configure the degree of parallelism desired. For tips about handling lots of files and using contig.exe to defragment NTFS indexes see NTFS performance and large volumes of files and directories.

Typical FileSystemWatcher Problems

If used properly the standard FileSystemWatcher (FSW) is way better than its reputation. However, there are typical problems one may encounter when first using the FSW:

• Unexpected events.
• Lost events.
• InternalBufferOverflowExceptions.
• No option to report files existing before the FSW started.

The standard FileSystemWatcher:

• Reports exceptions via its Error event. Not via raising exceptions!
• Does not report files that existed before .EnableRaisingEvents =True.
• Does detect network disruptions, but does not automatically recover from them.
• Does automatically handle renames of its watch path.