Building an Application’s Server with F#

Introduction

On May 24^th, 2017 the project kickoff for Nikeza was live streamed. The kickoff involved several motivated developers that I’d met during my campaign to learn functional programming. I attempted to assemble these developers into a team so that I could not only advance my campaign to learn Functional Programming, but also to build an application by applying what I’d learned.

I was already familiar with F#. Hence, I had performed several code katas and had even built some sample mobile applications with it using Xamarin tools. However, Xamarin relied on Object Oriented Programming which conflicted with my campaign to learn Functional Programming. Thus, I wanted to build an actual application by applying the lessons I’d learned from Functional Programming. In this article, I will discuss the server that I built using F#.

Here’s a screenshot of the app:

Requirements

The requirements for the application were the following:

• Users
  • Browse experts
  • Subscribe to an expert’s activity feed
  • Access expert’s content (via topic)
• Experts
  • Set featured links
  • Set featured topics
  • Set data sources (i.e. YouTube, WordPress, StackOverflow, etc.)
• System
  • Periodically checks for new content from an expert’s data source collection

Based on the requirements defined above, this application was a typical CRUD app. In addition, this app wasn’t mission critical, didn’t handle money, and did not have long running workflows. In other words, there was no need for audit controls. Therefore, I didn’t have to entertain an architecture dependent on sagas or event sourcing. However, decisions were made for the app to be future proof.

For business and marketing reasons, the team decided that Nikeza should be a .Net Core application. Hence, we weren’t sure if the .Net framework was going to be considered legacy in the next five years. However, we soon realized that the combination of using .Net Core with F# would likely result in not-so-spectacular code at the Data Access Layer of the system. Hence, the SQL TypeProvider wasn’t yet ready for .Net Core when we decided to build Nikeza. We also considered Entity Framework Core as an alternative to using the Sql TypeProvider. Unfortunately, we learned that Entity Framework Core didn’t provide 1^st-class support for F# either. As a result, we surrendered our ambitions and resorted to using ADO.Net.

The overall architecture looks something like this:

Commands

The following commands support user requirements:

type Command =
    | UpdateProfile   of ProfileRequest
    | UpdateThumbnail of UpdateThumbnailRequest

    | Follow          of FollowRequest
    | Unsubscribe     of UnsubscribeRequest
  
    | AddLink         of Link
    | RemoveLink      of RemoveLinkRequest
    | FeatureLink     of FeatureLinkRequest
    | ObserveLinks    of ObservedLinks
  
    | UpdateTopics    of FeaturedTopicsRequest
  
    | AddSource       of DataSourceRequest
    | RemoveSource    of RemoveDataSourceRequest
    | SyncSource      of DataSourceRequest


let execute = function
    | UpdateProfile   info -> info |> updateProfile
    | UpdateThumbnail info -> info |> updateThumbnail

    | Follow          info -> info |> follow
    | Unsubscribe     info -> info |> unsubscribe

    | AddLink         info -> info |> addLink
    | RemoveLink      info -> info |> removeLink 
    | FeatureLink     info -> info |> featureLink
    | ObserveLinks    info -> info |> observeLinks

    | UpdateTopics    info -> info |> featureTopics

    | AddSource       info -> info |> addDataSource
    | RemoveSource    info -> info |> removeDataSource
    | SyncSource      info -> info |> syncDataSource

Queries

Querying data followed a different approach compared to the command approach I used. Hence, instead of leveraging a choice type (aka: Discriminated Union), I instead wrote arbitrary functions. In hindsight and for the future reference, I would stay consistent with the command approach that I used and leverage a discriminated union (aka: Choice Type) to model queries as I did commands.

The following is a high-level example of how I would refactor my Store module:

type Query =
    | LinksByProvider         of ProfileId
    | LinksLatestByProvider   of ProfileId
    | LinksByProviderPlatform of ProfileId * Platform

    | Source                  of SourceId
    | SourceByProvider        of ProfileId

    | SourceQuery             of SourceId  * Sql
    | SourcesAll

    | Profile                 of ProfileId
    | ProfileByEmail          of Email
    | ProfilesQuery           of ProfileId * Sql

    | Topics                  of ProfileId


let query = function

    | LinksLatestByProvider   info -> info |> infolinksLatestByProvider
    | LinksByProvider         info -> info |> infolinksByProvider
    | LinksByProviderPlatform info -> info |> infolinksByProviderPlatform

    | Source                  info -> info |> source
    | SourceByProvider        info -> info |> sourceByProvider
    | SourceByQuery           info -> info |> sourceByQuery
    | SourcesAll              info -> info |> sourcesAll
    
    | Profile                 info -> info |> getProfile
    | ProfileByEmail          info -> info |> profileByEmail
    | ProfilesQuery           info -> info |> queryProfile
    
    | Topics                  info -> info |> getTopics

I believe the example above is more expressive and maintainable compared to my current implementation (that I rather not show). Each query option that the server is required to support is explicit and has a function that’s mapped to it. Hence, my original implementation has arbitrary functions littered throughout the module with no meta-data on how they’re mapped to requirements. Contrary to my actual implementation, the example above has each requirement represented as a choice for executing a query. Thus, the query function above would be the main interface for querying data.

Content Discovery

The application that I wanted to build not only needed to provide users with the capability to access content from experts, but it also needed to discover the latest content from experts once their platforms and corresponding access ids were saved into the system. This meant that an expert should only be required to point Nikeza to the platforms that they produce content on. Nikeza would then check for new content periodically on those platforms and update subscribers accordingly.

To support this requirement, I wrote a Platforms module that served as an interface for platform types.

The following components were arranged to support content discovery:

Here’s the abstract code for discovering new content:

getAllSources() |> List.iter (fun s -> s |> syncDataSource |> ignore)

Here’s the implementation details:

let syncDataSource (info:DataSourceRequest) =

        let notInDatabase link =
            getLink link.Title |> List.isEmpty

        getLastSynched info.Id 
         |> function
            | Some lastSynched ->
                let newLinks = info |> dataSourceToPlatformUser 
                                    |> newPlatformLinks lastSynched
                                    |> List.filter(fun l -> l |> notInDatabase)

                let updatedSource = newLinks |> updateSourceRequest info

                updatedSource.Links 
                 |> List.ofSeq 
                 |> List.iter (fun link -> link |> addSourceLink updatedSource |> ignore )

                updateSyncHistory info.Id |> ignore

                info.Id  |> string

            | None -> addSyncHistory info.Id |> ignore
                      info.Id |> string

The Platforms module is as follows:

module Nikeza.Server.Platforms

open ...

let PlatformToString = function
    | YouTube       -> "youtube"
    | WordPress     -> "wordpress"
    | StackOverflow -> "stackoverflow"
    | Medium        -> "medium"
    | RSSFeed       -> "rss feed"
    | Other         -> "other"

let platformFromString (platform:string) =
    platform.ToLower() |> function
    | "youtube"       -> YouTube
    | "wordpress"     -> WordPress
    | "stackoverflow" -> StackOverflow
    | "medium"        -> Medium
    | "rss feed"      -> RSSFeed
    | "other"         -> Other
    | _               -> Other

let getKey = function
    | YouTube       -> File.ReadAllText(Path.Combine(Directory.GetCurrentDirectory(),KeyFile_YouTube))
    | StackOverflow -> File.ReadAllText(Path.Combine(Directory.GetCurrentDirectory(),KeyFile_StackOverflow))
    | WordPress     -> KeyNotRequired
    | Medium        -> KeyNotRequired
    | RSSFeed       -> KeyNotRequired
    | Other         -> KeyNotRequired

let getThumbnail accessId platform = platform |> function
    | YouTube       -> YouTube       .getThumbnail accessId <| getKey platform
    | StackOverflow -> StackOverflow .getThumbnail accessId <| getKey platform
    | WordPress     -> WordPress     .getThumbnail accessId
    | Medium        -> Medium        .getThumbnail accessId
    | RSSFeed       -> DefaultThumbnail
    | Other         -> DefaultThumbnail

let platformLinks (platformUser:PlatformUser) =

    let user =  platformUser.User
    
    platformUser.Platform |> function
    | YouTube       -> platformUser |> youtubeLinks
    | StackOverflow -> platformUser |> stackoverflowLinks
    | WordPress     -> user         |> wordpressLinks
    | Medium        -> user         |> mediumLinks
    | RSSFeed       -> user         |> rssLinks
    | Other         -> []

let newPlatformLinks (lastSynched:DateTime) (platformUser:PlatformUser) =

    let user =  platformUser.User
    
    platformUser.Platform |> function
    | YouTube       -> platformUser |> newYoutubeLinks       lastSynched
    | StackOverflow -> platformUser |> newStackoverflowLinks lastSynched
    | WordPress     -> user         |> newWordpressLinks     lastSynched
    | Medium        -> user         |> newMediumLinks        lastSynched
    | RSSFeed       -> user         |> newRssLinks           lastSynched

Each function above is essentially a Strategy pattern that’s dependent on the platform type provided.

API

The web services were implemented in Giraffe. The decision to use that particular framework was based on the team’s desire to use a lightweight framework that embraced idiomatic F# syntax.

The services are as follows:

let webApp: HttpHandler = 
    choose [
        GET >=>
            choose [
                route "/"                   >=> htmlFile "index.html"
                route  "/options"           >=> setHttpHeader "Allow" "GET, OPTIONS, POST" // CORS support
                routef "/syncsources/%s"        syncSources
                routef "/bootstrap/%s"          fetchBootstrap
                routef "/providers/%s"          fetchProviders
                routef "/links/%s"              fetchLinks
                routef "/suggestedtopics/%s"    fetchSuggestedTopics
                routef "/recent/%s"             fetchRecent
                routef "/followers/%s"          fetchFollowers
                routef "/subscriptions/%s"      fetchSubscriptions
                routef "/sources/%s"            fetchSources
                routef "/thumbnail/%s/%s"       fetchThumbnail
                routef "/provider/%s"           fetchProvider
                routef "/removesource/%s"       removeSourceHandler
            ]
        POST >=> 
            choose [
                route "/register"        >=> registrationHandler
                route "/login"           >=> loginHandler
                route "/logout"          >=> signOff AuthScheme >=> text "logged out"
                route "/follow"          >=> followHandler
                route "/unsubscribe"     >=> unsubscribeHandler
                route "/featurelink"     >=> featureLinkHandler
                route "/updateprofile"   >=> updateProfileHandler
                route "/updateprovider"  >=> updateProviderHandler
                route "/addsource"       >=> addSourceHandler
                route "/addlink"         >=> addLinkHandler
                route "/removelink"      >=> removeLinkHandler
                route "/updatethumbnail" >=> updateThumbnailHandler
                route "/featuredtopics"  >=> featuredTopicsHandler
            ]
            
        setStatusCode 404 >=> text "Not Found" ]

Conclusion

In conclusion, I wanted to continue my campaign of learning functional programming. To support my learning, I built a web app using Elm and F#. I then discussed why the team (at the time) chose .Net Core, ADO.Net, and Giraffe as the core frameworks to build Nikeza. Lastly, I discussed the app’s architecture as well as some implementation details for commands and queries. I must admit, Nikeza is still a prototype and requires significant work for my vision to be realized. However, I learned a lot working on it.