Verified Commit 7b38cd34 authored by Justin Andresen's avatar Justin Andresen
Browse files

Initial commit

parents
# Comparision of Haskell Effect Libraries
## Libraries
On Hackage, there are a lot of different effect libraries for Haskell.
The following is an alphabetically sorted list of packages that I've considered.
- [`effect-handlers`][]
- [`effect-monad`][]
- [`effects`][]
- [`effin`][]
- [`extensible`][]
- [`extensible-effects`][]
- [`freer`][]
- [`freer-effects`][]
- [`freer-simple`][]
- [`fused-effects`][]
- [`polysemy`][]
- [`simple-effects`][]
There are even more libraries such as [`eff`][] for example.
I've not considered [`eff`][] in my analysis since it is still under construction and hasn't been published to Hackage yet.
It would be infeasible to compare all of these libraries with each other.
Thus, I'll employ the *wisdom of the crowd* to determine which library it is worth to take a closer look at.
For this purpose, I've transferred usage statistics from Hackage into a [Google Sheets document][data].
A copy of the data can be found in the `data.csv` file.
To put the data points into perspective, I've also recorded data for the monad transformer library [`mtl`][].
## Library Popularity
### Most Downloads
The first data point I've considered is the total number of downloads.
The number of downloads will give us a rough estimate of which libraries are used the most.
The majority of packages don't seem to be very widespread with **between 1,500 and 7,500 downloads**.
The top three packages by total number of downloads are as follows.
They account for 58% of all downloads.
1. [`extensible-effects`][] with a total of **29,470 downloads**
2. [`extensible`][] with a total of **22,451 downloads**
3. [`simple-effects`][] with a total of **13,268 downloads**
For comparison, the [`mtl`][] library has **461,739 downloads** in total.
### Most Recent Downloads
The total number of downloads only tells us whether a package once has been popular (or available for a long time).
However, it does not tell us how popular the package is right now.
The current popularity of a package is better estimated by the number of downloads within the last 30 days (*recent downloads* hereinafter).
For example, while the [`extensible-effects`][] and [`simple-effects`][] libraries account for 38% of all downloads, they were recently downloaded just 48 and 36 times, respectively, which corresponds to less than 4% of all recent downloads.
This could be an indicator that the packages have been superseded by other libraries.
The following are the top three libraries by recent downloads.
1. [`fused-effects`][] with **697 recent downloads**
2. [`polysemy`][] with **674 recent downloads**
3. [`extensible`][] with **546 recent downloads**
With a **total of 1,917 recent downloads**, they account for 81% of all recent downloads.
Combined they are thus more popular than [`mtl`][], which was **downloaded 1,535 times** in the last 30 days.
Current usage of the other libraries seems to be neglectable.
### Most Votes
Hackage provides a voting system to rate packages which could in theory provide more insights into the actual popularity of packages.
Unfortunately, this feature does not seem to be widely accepted throughout the Hackage community.
For example, the [`mtl`][] package received a measly **15 votes** despite its popularity.
Thus, we cannot expect any statistically relevant results from the rating.
In fact, most packages were not voted for at all.
However, what I found very interesting is that [`fused-effects`][] and [`polysemy`][] (which scored best in the *Most Recent Downloads* test) **received 7 and 6 votes**, respectively.
Considering that [`mtl`][] has been downloaded over 30 times more often than both packages combined, I think that it is remarkable that they almost received the same number of votes.
This suggests that interest in these two packages is high even though their actual usage is still low.
### Most Stargazers
Another voting mechanism with better acceptance is provided by GitHub's starring feature.
Since all packages link to their GitHub pages on their Hackage page, we can analyze how many people have starred the packages.
1. [`polysemy`][] with **713 stargazers**
2. [`fused-effects`][] with **441 stargazers**
3. [`freer-simple`][] with **181 stargazers**
4. [`extensible-effects`][] with **160 stargazers**
All other libraries have **less than 100 stargazers**.
Interestingly, the `mtl` library has only **216 stargazers** which confirms our hypothesis from the analysis of Hackage votes that `polysemy` and `fused-effects` are extremely popular in the Haskell community right now.
An analysis of watchers and forks shows similar results.
It is surprising that [`simple-effects`][] received only **4 stars** considering its high number of total downloads.
The bad ranking is probably due to the usage of GitLab instead of GitHub as a hosting service.
### Summary
In summary, [`extensible-effects`][] seems to be the most widely used effect library at the moment.
However, interest in this library recently seems to have dropped in favor of [`fused-effects`][] and [`polysemy`][].
The [`extensible`][] library also seems to be used widely.
The high number of recent downloads suggests that it is still in use today.
[`freer-simple`][] and [`simple-effects`][] have medium popularity.
## Library Maintenance
In addition to popularity, I think that it is also important to evaluate how well maintained a package is.
We neither want to adopt a library, that has been abandoned by its authors nor a library whose API is going to change frequently.
### Most Recently Updated
The [`fused-effects`][], [`extensible`][], and [`polysemy`][] libraries have last been **updated this year**.
Whereas [`freer-simple`][], [`effects`][] and [`extensible-effects`][] have been **updated in 2019** for the last time.
All other packages are over two years old.
However, the [`mtl`][] library also has been **updated in February 2018**.
Thus, it may also be a sign that an API has stabilized if the package hasn't been updated for a long time.
### Most Recently Committed
Since the last update does not seem to tell us whether a package is still maintained, we have to look at the Git activity.
There is a commit on the default branch of the following repositories that is predated by the last commit to [`mtl`][] (i.e., commits after October 2019).
- [`polysemy`][] with last commit from **June 2020**.
- [`fused-effects`][] with last commit from **June 2020**.
- [`freer-simple`][] with last commit from **May 2020**.
- [`extensible-effects`][] with last commit from **April 2020**.
- [`extensible`][] with last commit from **March 2020**.
- [`simple-effects`][] with last commit from **January 2020**.
### Summary
All of the libraries that performed best in the popularity analysis above are still actively maintained.
Except for [`polysemy`][] and [`fused-effects`][], library updates on Hackage are infrequent, though.
## Library Features and Compatibility
Finally, we have to find out whether the libraries provide the features that we need and whether they are compatible with our setup.
We will only consider the libraries that were identified as actively maintained in the test above.
### Compatibility
In our setup we are using GHC 8.6.5.
The effect library should be compatible with that version of GHC as well as all libraries that we are using already.
Thus, the libraries have been added as a dependency to the `.cabal` file.
Since the plan is to use the same effect library for the [Free Compiler][freec] and the Free Compiler has more dependencies than `haskell-src-transformations`, this test was conducted against the Free Compiler directly.
In order to avoid side effects, each library was tested in its own Docker container.
No incompatibilities were found.
I also didn't notice any significant differences in build time.
The scripts that I've used to perform the compatibility check can be found in the `compatibility` directory.
### Features Comparision
Most of the libraries seem to provide very similar features.
They mostly differ in the concrete syntax for effect sets, how new effects are declared and their internal representation of effects.
#### [`polysemy`][] and [`fused-effects`][]
The advantage of [`polysemy`][] and [`fused-effects`][] over other effect libraries seems to be their performance and support for higher-order effects.
Prior to GHC 8.8 the [`polysemy`][] is outperformed by [`fused-effects`][].
In GHC 8.9+ [`polysemy`][] can take advantage of compiler optimizations to archive "zero-cost".
Since we are using GHC 8.6.5, we would not be able to take advantage of [`polysemy`][]'s performance advantages.
However, [`polysemy`][] reportedly provides better type errors and defining new effects requires less boilerplate code.
One drawback of [`polysemy`][] is that it needs a lot more language extensions than [`fused-effects`][].
Additionally, [`polysemy`][] can use a compiler plugin to improve type inference.
#### [`freer-simple`][], [`extensible-effects`][] and [`simple-effects`][]
The other libraries can only model first-order effects.
Thus functions like `catchError` and `local` cannot be implemented as effects and must be given a fixed interpretation.
I still think that they are sufficient for our purposes.
[`freer-simple`][] and [`simple-effects`][] require a similar number of language extensions to [`fused-effects`][].
The language extensions required by [`extensible-effects`][] is not documented.
Similar to [`polysemy`][], [`freer-simple`][] also provides custom compile-time errors and has `TemplateHaskell` integration to reduce the amount of boilerplate code that has to be written to define new effects.
#### [`extensible`][]
Event though [`extensible`][] still claims to provide "extensible, efficient, optics-friendly data types and effects", the effects have been moved to [a separate package][`extensible-skeleton`] six months ago.
Unfortunately, the library is not documented very well.
I'll not consider it in my further analysis.
### Summary
[`polysemy`][] and [`fused-effects`][] sound interesting but we probably don't need higher-order effects and performance is not critical in our case.
However, improved compile-time error messages and better type inference of [`polysemy`][] could be useful for us.
## Test Projects
The `test-projects` directory contains Cabal projects for the different libraries for a direct comparison.
The test projects can be executed as follows.
```haskell
$> cd ./test-projects/<library>/
$> cabal new-repl
λ> :m MyProgs
λ> runMyProg
(42,True)
λ> logMyProg
Info: 84
Info: 42
(42,True)
```
[data]:
https://docs.google.com/spreadsheets/d/1Zmy6E5ji8wGEKBWcYYUtj89r7a2pIrUCyGhCwiptxas/edit?usp=sharing
"Effect Libraries — Google Sheets"
[freec]:
https://github.com/FreeProving/free-compiler
"Free Compiler on GitHub"
[`effect-handlers`]:
https://hackage.haskell.org/package/effect-handlers
"effect-handlers: A library for writing extensible algebraic effects and handlers. Similar to extensible-effects but with deep handlers."
[`effect-monad`]:
https://hackage.haskell.org/package/effect-monad
"effect-monad: Embeds effect systems and program logics into Haskell using graded monads and parameterized monads"
[`effects`]:
https://hackage.haskell.org/package/effects
"effects: Computational Effects"
[`eff`]:
https://github.com/hasura/eff
"eff — screaming fast extensible effects for less"
[`effin`]:
https://hackage.haskell.org/package/effin
"effin: A Typeable-free implementation of extensible effects"
[`extensible`]:
https://hackage.haskell.org/package/extensible
"extensible: Extensible, efficient, optics-friendly data types and effects"
[`extensible-effects`]:
https://hackage.haskell.org/package/extensible-effects
"extensible-effects: An Alternative to Monad Transformers"
[`extensible-skeleton`]:
https://hackage.haskell.org/package/extensible-skeleton
"extensible-skeleton: Operational-based extensible effect library"
[`freer`]:
https://hackage.haskell.org/package/freer
"freer: Implementation of the Freer Monad"
[`freer-effects`]:
https://hackage.haskell.org/package/freer-effects
"freer-effects: Implementation of effect system for Haskell."
[`freer-simple`]:
https://hackage.haskell.org/package/freer-simple
"freer-simple: Implementation of a friendly effect system for Haskell."
[`fused-effects`]:
https://hackage.haskell.org/package/fused-effects
"fused-effects: A fast, flexible, fused effect system."
[`mtl`]:
https://hackage.haskell.org/package/mtl
"mtl: Monad classes, using functional dependencies"
[`polysemy`]:
https://hackage.haskell.org/package/polysemy
"polysemy: Higher-order, low-boilerplate, zero-cost free monads."
[`simple-effects`]:
https://hackage.haskell.org/package/simple-effects
"simple-effects: A simple effect system that integrates with MTL"
#!/bin/bash
libraries=(
"polysemy"
"fused-effects"
"freer-simple"
"extensible-effects"
"extensible"
"simple-effects"
)
for library in "${libraries[@]}"; do
echo "Testing $library compatibility ..."
echo "---------------------------------------------"
log_file="$library.log"
docker run --rm -it \
-v $(pwd):/data \
haskell:8.6.5 \
/data/test.sh "$library" \
| tee "$log_file"
status_code="$?"
echo "============================================"
echo "Test finished with status code $status_code!"
echo "Log has been written to $log_file"
echo "============================================"
echo
done
#!/bin/bash
library="$1"
# Get the free compiler.
git clone https://github.com/FreeProving/free-compiler.git
cd free-compiler
# Update the Cabal build configuration.
echo "Adding dependency on $library ..."
sed -i 's/build-depends: base/build-depends: '"$library"', base/' free-compiler.cabal
echo "--- [ Start at $(date) ] -------------------------"
cat free-compiler.cabal
echo "--- [ End at $(date) ] -------------------------"
# Build everything.
echo "Fetching latest package list ... "
echo "--- [ Start at $(date) ] -------------------------"
cabal new-update
echo "--- [ End at $(date) ] -------------------------"
echo "Building ... "
echo "--- [ Start at $(date) ] -------------------------"
time cabal new-build freec freec-unit-tests
status_code="$?"
echo "--- [ End at $(date) ] -------------------------"
echo "Running Tests ... "
echo "--- [ Start at $(date) ] -------------------------"
time ./tool/test.sh
status_code="$?"
echo "--- [ End at $(date) ] -------------------------"
echo
echo "Exited with status code $status_code"
exit "$status_code"
Name,Reverse Dependencies,Downloads (total),% Downloads (total),Downloads (last month),% Downloads (last month),Last Updated,Rating,Votes,Platform,Watchers,Stargazers,Forks,Last Commit,First Commit
effect-handlers,n/a,6133,5.45%,9,0.38%,April 2016,0,0,GitHub,5,16,2,"August 28, 2016","October 30, 2014"
effect-monad,n/a,4012,3.56%,63,2.67%,January 2018,2,1,GitHub,7,72,5,"June 24, 2018","September 14, 2013"
effects,1,4785,4.25%,85,3.61%,April 2019,0,0,GitHub,3,18,1,"August 14, 2019","October 15, 2011"
effin,3,7405,6.58%,133,5.64%,August 2017,0,0,GitHub,4,26,1,"August 3, 2017","May 7, 2014"
extensible,9,22451,19.95%,546,23.17%,March 2020,0,0,GitHub,15,91,11,"March 14, 2020","December 31, 2014"
extensible-effects,13,29470,26.18%,48,2.04%,January 2019,2.25,2,GitHub,16,160,24,"April 16, 2020","September 25, 2013"
freer,n/a,5757,5.11%,13,0.55%,November 2016,2.25,2,GitLab,n/a,15,10,"March 14, 2017","September 12, 2015"
freer-effects,3,1484,1.32%,12,0.51%,April 2017,2,1,GitHub,10,62,11,"May 10, 2017","September 12, 2015"
freer-simple,9,3496,3.11%,41,1.74%,October 2019,0,0,GitHub,14,181,9,"May 19, 2020","September 12, 2015"
fused-effects,8,6787,6.03%,697,29.57%,June 2020,2.75,7,GitHub,29,441,29,"June 17, 2020","September 9, 2018"
mtl,3784,461739,0.00%,1535,0.00%,February 2018,2.75,15,GitHub,17,216,40,"October 1, 2019","January 25, 2012"
polysemy,13,7514,6.68%,674,28.60%,February 2020,2.5,6,GitHub,39,713,48,"June 18, 2020","February 13, 2019"
simple-effects,3,13268,11.79%,36,1.53%,November 2018,2,1,GitLab,n/a,4,2,"January 27, 2020","July 21, 2016"
### Files and directories generated by Cabal ###
dist
dist-newstyle
.ghc.environment.*
Copyright (c) 2020, Justin Andresen <jan.dresen.95@gmail.com>, Kiel University
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the copyright holder nor the names of other
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import Distribution.Simple
main = defaultMain
-- Cabal configuration file for this package.
packages: mapro-polysemy-test.cabal
-- Configure this package to compile only if there are no compile time
-- warnings in production mode. To disable fatal warnings during development
-- pass `--ghc-option=-Wwarn` to `cabal`.
package mapro-polysemy-test
ghc-options: -Werror
cabal-version: 2.4
-- Initial package description 'mapro-effect-library-tests.cabal' generated
-- by 'cabal init'. For further documentation, see
-- http://haskell.org/cabal/users-guide/
name: mapro-polysemy-test
version: 0.1.0.0
synopsis: Cabal test project for the polysemy library.
-- description:
-- bug-reports:
license: BSD-3-Clause
license-file: LICENSE
author: Justin Andresen
maintainer: jan.dresen.95@gmail.com
copyright: (c) 2020, Justin Andresen, Kiel University
-- category:
library
-- exposed-modules:
-- other-extensions:
build-depends: base ^>=4.12.0.0
, polysemy
, polysemy-plugin
hs-source-dirs: src
default-language: Haskell2010
other-modules: MyEffects
, MyHandlers
, MyProgs
{-# LANGUAGE TemplateHaskell, ScopedTypeVariables #-}
{-# LANGUAGE LambdaCase, BlockArguments #-}
{-# LANGUAGE GADTs, FlexibleContexts, TypeOperators, DataKinds, PolyKinds #-}
module MyEffects where
import Polysemy
data MyState s m a where
MyGet :: MyState s m s
MyPut :: s -> MyState s m ()
makeSem ''MyState
data MyLog m a where
MyLog :: Show b => b -> MyLog m ()
makeSem ''MyLog
{-# OPTIONS_GHC -fplugin=Polysemy.Plugin #-}
{-# LANGUAGE TemplateHaskell, ScopedTypeVariables #-}
{-# LANGUAGE LambdaCase, BlockArguments #-}
{-# LANGUAGE GADTs, FlexibleContexts, TypeOperators, DataKinds, PolyKinds #-}
{-# LANGUAGE AllowAmbiguousTypes #-}
module MyHandlers where
import Polysemy
import Polysemy.Internal.Combinators
import MyEffects
runMyState :: s -> Sem (MyState s ': r) a -> Sem r (s, a)
runMyState = stateful $ \case
MyGet -> \s -> return (s, s)
MyPut s' -> \_ -> return (s', ())
logMyState :: (Members '[MyState s, MyLog] r, Show s) => Sem r a -> Sem r a
logMyState = intercept $ \case
MyGet -> myGet
MyPut s' -> do
myLog s'
myPut s'
printLog :: Member (Embed IO) r => Sem (MyLog ': r) a -> Sem r a
printLog = interpret $ \case
MyLog msg -> embed $ putStrLn ("Info: " ++ show msg)
{-# OPTIONS_GHC -fplugin=Polysemy.Plugin #-}
{-# LANGUAGE TemplateHaskell, ScopedTypeVariables #-}
{-# LANGUAGE LambdaCase, BlockArguments #-}
{-# LANGUAGE GADTs, FlexibleContexts, TypeOperators, DataKinds, PolyKinds #-}
{-# LANGUAGE TypeApplications #-}
module MyProgs where
import Polysemy
import MyEffects
import MyHandlers
myProg :: Member (MyState Int) r => Sem r Bool
myProg = do
n <- myGet
myPut (2 * n)
m <- myGet
myPut (m - n)
n' <- myGet
return (n' == n)
myInitialState :: Int
myInitialState = 42
runMyProg :: IO (Int, Bool)
runMyProg = runM @IO . runMyState myInitialState $ myProg
logMyProg :: IO (Int, Bool)
logMyProg =
runM @IO . printLog . runMyState myInitialState . logMyState $ myProg
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment