Note: This is a work in progress draft document
OCaml Labs is an initiative within the Cambridge Computer Laboratory started by Anil Madhavapeddy in 2011 to promote research, growth and collaboration within the wider OCaml community. We contribute to management of the day-to-day OCaml maintenance load, and align research agendas with real-world projects to help progress the language and make it available and applicable to a larger audience. Building on 40 years of language development, together with INRIA our goal is to freely release and integrate all work upstream, allowing all prospective users access to the efficient, expressive language of OCaml.
The OCaml Labs team is a distributed group of 50+ people, comprised of researchers at the University of Cambridge Computer Laboratory, industrial partners, student interns and individual collaborators. From a combination of research grant and industrial funding we are able to fund a small full-time team at the University of Cambridge, 2-4 student interns each academic year and regular community events to encourage contribution and collaboration.
This report details the work carried out by our group in 2016 and outlines prospective projects for 2017. You can see posts from previous years, in .html and .md format.
The Platform combines the core OCaml compiler with a coherent set of libraries, tools and documentation. We look to large-scale users of OCaml to guide the direction of development in this area, including Jane Street, MirageOS, Docker and Facebook. 2016 has seen a focus on the developer experience; tooling improvements around packaging, build and test infrastructure; and support for different operating systems and infrastructures.
Mention users of Platform:
Unikernel Systems formed in 2016 and built on work initiated by the OCaml Labs group over the last 5 years. Unikernel Systems was successfully acquired by container giant Docker, and progress on those original components and libraries has continued throughout the year.
Mention Jane Street, ARM, Microsoft, Citrix, Facebook, Aesthetic Integration, Hitachi
The OCaml Platform is a collaborative effort across the OCaml community led by OCaml Labs, Jane Street and OCamlPro. The goal of the Platform is to provide an integrated development environment with a comprehensive set of tooling and support for a wide range of operating systems and infrastructures to allow for the fun and efficient development of OCaml.
Opam 2.0 preview release: September 2016 with details of additions - link to work with FB on Reason that led to discussions and subsequent changes
Compilers as packages
Command wrappers
File installation tracking
Migrating the OCaml repository
Virtual OCaml packages
Continuous Integration
The defacto editor tool for OCaml providing support for Emacs, Vim, VSCode, Atom, Eclipse and Sublime Text.
What will go here? - topkg in mirage section...
Documentation has been a focus for the past few years, with direct effort made in 2016 by OCaml Labs and Jane Street to provide a working implementation for full documentation support.
Work on improving the documentation toolchain started in 2015 with the alpha release of codoc 0.2.0. The Platform team outlined the different stages of documentation and the difficulties in providing a feature-complete implementation. 2016 saw further development in the form of odoc, an OCaml API documentation tool that takes interface files and outputs cross-referenced module documentation.
The aim is that odoc will work correctly on functor-heavy code bases (where ocamldoc has previously struggled) and will link identifiers across packages. Once the rendering is improved it will replace ocamldoc as the default documentation path used by odig (a library to mine installed OCaml packages).
cmti files using odig ocamldocOdoc the documentation generator is comprised of:
doc-ock to extract documentation from OCaml files; doc-ock-html for HTML generation; doc-ock-xml and XML printer and parser; and Octavius an ocamldoc syntax parser.
Windows
ARM
Overview
MirageOS is a library operating system written in OCaml that can be used to build unikernels for secure, high-performance network applications across a variety of cloud computing and mobile platforms.
MirageOS applications are highly portable since they are parameterised across their operating environment. Rather than making assumptions about the presence of an operating system, they can depend on a set of small OCaml module signatures. These signatures provide services such as consoles, network or block devices or entropy. The mirage CLI tool parses a configuration file that assembles concrete implementations of these signatures to generate an executable. MirageOS supports targets ranging from standard POSIX ELF binaries to x86_64/ARM Xen and KVM unikernels, and more experimental targets such as QubesOS, JavaScript and hardware FPGAs.
There is a vibrant and growing ecosystem around constructing libraries for MirageOS that are built in pure OCaml with minimal external dependencies. This unlocks that functionality in all of the target architectures that MirageOS supports.
MirageOS depends on a number of components that are developed and maintained by OCaml Labs, some of which are detailed below.
2016 Changes
Mirage 3.0 is set for release in early 2017 and provides the groundwork for increased flexibility and stability of unikernels in production.
Overview
Irmin provides Git-like distributed, branchable storage. It was first released in 2013, but the first post detailing its features was in 2014. Irmin is a library component of MirageOS and seeks to provide an OS-independent, portable structure for data persistence.
2016 Changes
2016 saw a large API change to allow easier use of new and efficient Irmin backends, increased Windows support, new functors for metadata, improved documentation and the release of Irmin 0.12 which includes irmin-watcher to replace the slow and resource-intensive default polling.
Previous watches on the datastore had latency issues and were expensive as they scanned the full storage directory every second to detect changes. Applications relying on Irmin such as DataKit which has thousands of tags and Canopy which requires low latency are examples of the user-driven changes that took place this year.
Previous attention given to OCaml's package and release story was diverted from Assemblage to a more flexible system with a strong build-system model. The result is a combination of components that improve the process significantly, and which are used with the vast majority of our OCaml and MirageOS packages.
Overview
Topkg is a packager for distributing OCaml software that provides an API to describe the conventions applied to given build configurations, distribution creation and publication procedures. It is a library that is added as a build dependency and comes with an optional tool (topkg-care) to help manage your package and releases.
Odig is a library and command line tool to mine installed OCaml packages which supports metadata lookups and package description documentation. When used together with ocamldoc or odoc it generates per-package API documentation sets with inter-package cross-references. See the OCaml platform section for more details on documentation.
Framework for building directory structures: the next piece of the package and build puzzle will focus on making it quick and easy to set up new software projects and provide licensing and source boilerplate code during program development.
Overview
Functoria is a DSL to organise functor applications. Using Functoria, you can describe a set of modules and functors, their types and their application. It uses keys to define parameters and to switch implementation at configure time.
2016 Changes
Functoria was released in February 2016, with a new release due in 2017.
The web ecosystem has steadily been improving in OCaml, and OCaml Labs has made contributions towards the various protocol implementations, as well higher level frameworks to make web programming more pleasant. Most of these libraries are in pure OCaml and so directly compatible with MirageOS.
Databox, UCN, personal data
CueKeeper
Mr Mime: Email Parsing
Repo | Paper
Mr Mime is a pure OCaml library that targets parsing multipart email. It uses the Continuation Passing Style (CPS) to assemble small parser combinators on-the-fly to build a parser based on the mail header contents. This can be combined with MirageOS unikernels that send, receive, store and process emails. Future work may include unikernel deployments such as SMTP proxies and spam filters.
OWL: Data Analysis
Docs | Repo | Tutorial | Under development
Owl is an OCaml numerical library that supports both dense and sparse matrix operations, linear algebra, regressions, and many advanced mathematical and statistical functions. Owl is still in its infancy and under current development, with dense and sparse matrix functions and advance mathematical and statistical functions close to completion. Future work will feature support for linear algebra, regression functions and machine learning.
We regularly host community events based around functional programming and OCaml. 2016 was a busy year for us and we hosted lots of events including 4 compiler development sessions in Cambridge; the first MirageOS hackathon in Marrakech, followed by a local Cambridge MirageOS event; the 2016 OpenBSD summit; and the first International Metaprogramming Summer School at Robinson College.
Together with academic appearances, members of the OCaml Labs team present their work to the wider community including programming groups, students, financial groups and other development teams - most notably Facebook this year.
Encouraging community contribution and engagement has been a focus of this year, with consideration given to onboarding new users, surfacing and documenting small-scale projects, and developing strategies for efficient maintainership of our shared repositiories and code. We started the OCaml Labs Slack group to enable easy communication amongst our widely distributed group, and we have >100 active users.
February, May, June and November 2016
Initiated in 2013 by Leo White and Jeremy Yallop as a time and place to focus on their compiler projects, these events have continued every few months, with 4 evening sessions over 2016. The events are open to anyone with an interest in improving the OCaml toolchain, and there's always a handful of OCaml specialists to help new users get started and answer questions. We provide a list of mini projects for attendees to work on, and often host presentations of new releases and features.
During a recent push towards the upcoming MirageOS 3.0 release, we diversified by designating our most recent session in November as a combination OCaml compiler/MirageOS hack evening. We followed up with an activity summary which displayed an encouraging mix of first-ever commits, and OCaml and MirageOS specific features from a range of users. Feedback was positive, and we will likely combine some of our future hack events in similar ways.
11th-16th March 2016
OCaml Labs sponsored the first MirageOS hackathon - an international, community-organised event - at Rihad Priscilla in Marrakech, which provided a perfect venue for community building and hacking. Hannes Mehnert and David Kaloper started the OCaml-TLS project at Aftas beach house in Mirleft back in 2014, and were keen to encourage others to experience Morocco for themselves. Hackathon attendees included the MirageOS core team, complete newcomers and long standing community members - many of whom wrote trip reports detailing their experiences.
Event outputs included:
17th July 2016
Following the success of the Marrakech event, we ran a day-long MirageOS hack event at Darwin College, Cambridge.
Projects included:
8th-12th August 2016
Together with Microsoft Research and Facebook, OCaml Labs sponsored the first International Summer School for Metaprogramming which was held at Robinson College, Cambridge. The goal of the summer school was to explore the various forms of metaprogramming and its practical application. The school was aimed at graduate students of programming languages and related areas, but was open to researchers, practitioners interested in the subject.
Talks and speakers included:
Lightning talks from attendees:
30th August-5th September 2016
Improving OCaml OS and distribution support is always on our agenda, and we decided to address this directly by hosting the 2016 OpenBSD summit at the Computer Laboratory, Cambridge. It coincided with the 6.0 release and the last release to be accompanied by original songs and pressed onto CD - future releases will be online only.
More needed here....
It was interesting to see 60+ volunteer developer team operate, and discuss distributed community management with them.
22nd September 2016
We ran this year's OCaml Tutorial using IOCaml Notepad which provides a REPL within a browser with a friendly interface. The session lasted the afternoon and started with a brief introduction to OCaml followed by a tour through fundamental constructs needed to eventually write an implementation of the popular 2048 game. 21 people attended the tutorial, with varying abilities and experience levels, including some Haskell developers and one person with no prior programming experience. It was an open atmosphere for questions and discussion with the resources remaining available online . Most people managed to complete the task, and had some form of working game at the end of the tutorial, and providing a follow up tutorial for those who have completed this would be a useful next step.
23rd September 2016
To celebrate the upcoming OCaml 4.04.0 release, we designed and printed some OCaml t-shirts featuring the new memory profiler Spacetime. We hope this is the start of many adventures for our cartoon camel.
Towards the end of the summer we started using Canopy, the git-blogging unikernel to capture a range of talks that members of our team attended. It was a success on the whole, and we hope to continue using it for future events as a complement to videos and slides.
18th-24th September
Knowing we would be unable to attend all of the talks, we distributed our team across the different topic tracks and ran a liveblog unikernel to try and cover as many talks and workshops as we could. This would also allow others to enjoy talks they could not attend. There were some wifi issues, but we managed to log 54 talks, including:
7th-8th October
The Docker for Mac and Windows application utilises 40+ OCaml libraries, many of which were developed by the Unikernel Systems team stemming from OCaml Labs.
We used Canopy to liveblog the Docker Distributed Summit in Berlin and covered all of the talks, with two talks specifically related to work at OCaml Labs and MirageOS:
Talking TUF - Securing Software Distribution
The Update Framework (TUF) provides a specification and library that can be used universally to secure software update systems. Our OPAM signing proposal, Conex is based on TUF with additional features specific to OPAM.
Unikernels: The rise of the library hypervisor in MirageOS
This talk focused on the notion of breaking the hypervisor down into libraries and extending the "kit" model, whilst introducing the discrete components that form the Docker for Mac and Windows application: HyperKit, DataKit and VPNKit - all of which feature OCaml and MirageOS contributions.
14th January 2016
UCL Security Group Presentation - London, UK
Unikernels: Rise of the library operating system, and how to engineer not-quite-so-broken software
Anil Madhavapeddy
22nd January 2016
CIF @ Scale14X - California, USA
Knock, knock: Unikernels Calling!
Richard Mortier
23rd January 2016
/dev/winter - Cambridge, UK
Surfacing deep magic with library operating systems
Mindy Preston
My first unikernel with MirageOS
Matthew Gray
28th January 2016
Facebook TechTalk - California, USA
Concurrent and Multicore OCaml: A deep dive
KC Sivaramakrishnan & Stephen Dolan
4th February 2016
Compose - New York, USA
Composing Network Operating Systems
Mindy Preston
19th February 2016
MSR Seminar - Cambridge, UK
Anil Madhavapeddy
19th February 2016
BobKonf - Berlin, Germany
Jackline: A secure instant messaging application, functional from the ground up
Hannes Mehnert
21st February 2016
TRON Workshop @ NDSS - California, USA
Not-quite-so-broken TLS 1.3: Mechanised Conformance Checking
Hannes Mehnert
14th June 2016
LDN Functionals - Jane Street London, UK
The functional innards of Docker for Mac and Windows
Anil Madhavapeddy
2nd August 2016
LDN Functionals - Jane Street London, UK
OCaml Multicore and Programming with Reagents
KC Sivaramakrishnan
14th September 2016
London Facebook Faculty Summit - Facebook London, UK
Effective parallelism with Reagents
KC Sivaramakrishnan
23rd September 2016
OCaml Workshop - Nara, Japan
Improving the OCaml webstack: motivations and progress
Spiros Eliopoulos
29th September 2016
Rustat Conference - University of Cambridge, UK
Inner workings of DataKit and DataBox
Anil Madhavapeddy, KC Sivaramakrishnan
4th November 2016
CodeMesh - London, UK
Distributed Consensus: Making the Impossible Possible
Heidi Howard
8th November 2016
LCFS - University of Edinburgh, UK
Polymorphism, Subtyping and Type Inference in MLsub
Stephen Dolan, Alan Mycroft
9th November 2016
SPLS - University of Edinburgh, UK
Irrelevant Classical Logic in Agda
Stephen Dolan
15th December 2016 Opodis - Madrid, Spain
Flexible Paxos: Quorum Intersection Revisited
Heidi Howard, Dahlia Malkhi and Alexander Spiegelman
The following presentations take place at the Cambridge Computer Laboratory during term time.
Lent Term 2016
University of Cambridge Computer Laboratory - Cambridge, UK
Advanced Functional Programming
Jeremy Yallop, Anil Madhavapeddy and guest lecturers Alan Mycroft, Leo White & KC Sivaramakrishnan
28th January 2016
An overview of Bigraphs, their applications and future research directions
Michele Sevegnani (University of Glasgow)
2nd February 2016
Unanimous Revisited: Distributed consensus for geo-replication
Heidi Howard
10th February 2016
Not-quite-so-broken TLS 1.3 mechanised conformance checking
Hannes Mehnert
16th Feburary 2016
Managing Infrastructure as code with Puppet
Gareth Rushgrove (Puppet)
17th February 2016
Compiling algebraic effects to JavaScript in js_of_ocaml
Armaël Guéneau
25th February 2016
Is every Java program a Unikernel? An introduction to IncludeOS' design and foundations
Alfred Bratterud (HiOA)
17th March 2016
On PCIe Performance
Rolf Neugebauer (Docker)
26th April 2016
Towards Compilation of Affine Algebraic Effect Handlers Daniel Hillerström (University of Ediburgh)
16th June 2016
Ios: Why work when you can delegate?
Heidi Howard
12th July 2016
Unikernels and Beyond: The future of application containers in the cloud
Dan Williams (IBM Research, NY)
19th July 2016
Approximate Computing
Philip Dexter (Binghamton University)
11th August 2016
FlashBlade: Hardware and Software Co-design
Alex Ho (Pure Storage)
4th October 2016
Second Chances: Every second packet doesn't count
Jon Crowcroft
11th October 2016
Trip Report from S-REPLS 4
Nik Sultana
19th October 2016
Unik: A platform for automating unikernel compilation and deployment
Idit Levine (EMC)
10th November 2016
Flexible Paxos: Reaching agreement without majorities
Heidi Howard
15th November 2016
Higher Network Performance on MirageOS
Takayuki Imada (Hitachi)
Docker
Jane Street
ARM
Microsoft
Facebook - Reason ML
Sean Grove
Aesthetic Integration
Citrix
Hitachi
A comprehensive list of publications in 2016 submitted by members of the OCaml Labs group.
Staging Generic Programming
Partial Evaluation and Program Manipulation (PEPM), January 2016 Winner of Best Paper Award
Jeremy Yallop
Not-quite-so-broken TLS 1.3 mechanised conformance checking
To be published in the TRON Workshop Technical Programme as part of the NDSS Symposium, February 2016
David Kaloper-Mersinjak, Hannes Mehnert
Declarative foreign function binding through generic programming
To be published in the proceedings of the Thirteenth International Symposium on Functional and Logic Programming (FLOPS), March 2016
Jeremy Yallop, David Sheets and Anil Madhavapeddy
Representation without Taxation: A Uniform, Low-Overhead and High-Level Interface to Eventually Consistent Key-Value Stores
IEEE Data Engineering Bulletin, March 2016 39(1): 52-64
KC Sivaramakrishnan, Gowtham Kaki and Suresh Jagannathan
Composable Scheduler Activations for Haskell
Journal of Functional Programming (JFP), April 2016
KC Sivaramakrishnan, Tim Harris, Simon Marlow and Simon Peyton Jones
Flexible Paxos: Quorum Intersection Revisited
Heidi Howard, Dahlia Malkhi and Alexander Spiegelman
Compiling Links Effect Handlers to the OCaml Backend
ML Workshop, September 2016
Daniel Hillerström, Sam Lindley and KC Sivaramakrishnan
Eff Directly in OCaml
ML Workshop, September 2016
Oleg Kiselyov and KC Sivaramakrishnan
Migrating MultiMLton to the Cloud
ML Workshop, September 2016
KC Sivaramakrishnan, Lukasz Ziarek and Suresh Jagannathan
Lock-free programming for the masses
OCaml Workshop, September 2016
KC Sivaramakrishnan and Théo Laurent
Effective Concurrency with Algebraic Effects
OCaml Workshop, September 2016
Stephen Dolan, Leo White, KC Sivaramakrishnan, Jeremy Yallop and Anil Madhavapeddy
The State of the OCaml Platform
OCaml Workshop, September 2016
Louis Gesbert, Anil Madhavapeddy on behalf of the Platform team
Conex: Establishing trust into data repositories
OCaml Workshop, September 2016
Hannes Mehnert, Louis Gesbert
OCaml Inside: a drop-in replacement for libtls
OCaml Workshop, September 2016
Enguerrand Decorne, Jeremy Yallop, David Kaloper-Mersinjak
Who's got your mail? Mr. Mime!
OCaml Workshop, September 2016
Romain Calascibetta
Multicore - where it's going + effects, link
Modular Implicits
Macros
Libraries: PiFaceCAD; ODBC
Merlin
Parsing and integration with Menhir. Plans
Mac OSX Support
Performance profiling, Spacetime
MirageOS perf (taka)
MirageOS/Databox/Docker - Bushel
MirageOS storage - Gabriel
Git/etc Romain
nicolas internship
cloud infrastructure from Microsoft, hardware from ARM
matt harrison - personal data silos
repo maintainership/management
rwo
teaching
internship/outreachy programs
ocaml.org
more events: compiler hacking (feb), mirageos marrakech (march), databox open source (march)
Mention funding - grants, epsrc, at end - acknowledgements
List of contributors
