Food Recipes: Semantic representation, Formal Language and Visualization

I always enjoyed the precision that programming languages commands give instructions to computers and I have been wondering if that precision could be applied to food recipes.
If computers could understand semantically a recipe, they would be able to perform many interesting tasks.
For example a computer could

• Scale the recipe: Not only multiplying ingredients but also adjusting the time and the difficulty
• Merge recipes fitting one into another taking advantage of the idle moments. It would be possible to plan several recipes with a certain deadline. Something like "I want to prepare Risotto alle Fragole and Arista for 3 people and everything should be ready by 7 pm". This is as example of simplified merging process:
  
• Combine recipes to come out with new variation. There is the "Computer Cooking Contest" that is trying to achieve this result. The event is organised every year, see in the resources section for links.
• Using a version control system, such as Git, where could have people to add variations of the recipe creating branch, merging, committing, etc.
• Computers could, with the help of a machine or robot, cook for you

This is an example of annotated recipe from the CURD database. This recipe is annotated using the MILK language:

Visualization

If the recipe is semantically represented inside the computer, it could be automatically visualised using some algorithm. A viable format could be similar to:

Other formats:

Video could also be used. Animation could fit better because could be generated in CG. I personally like this point of view in recipe video:

And as a final note, this is a video of my mother that I took several years ago. Not a good example of clarity but... a good recipe:

Resources

1. A method for extracting major workflow composed of ingredients, tools, and actions from cooking procedural text

   By Yoko Yamakata, Shinji Imahori, Hirokuni Maeta, Shinsuke Mori (2016)
   A method for extracting a major workflow of cooking procedure from a Japanese recipe on the Web. It is utilized for various applications including recipe search, summarization, and visualization.
   http://www.ar.media.kyoto-u.ac.jp/publications/yamakata-CEA16.pdf

2. Training the PR2 in Culinary Arts. A Natural Language Model for Parsing Recipes

   By Jessica Zhao, Alejandro Bordallo, Subramanian Ramamoorthy (2016)
   a culinary language model for the kitchen by abstracting cooking instructions into a generalized tripartite form of ACTION, TARGET, TOOL.
   http://stanford.edu/~jesszhao/files/PR2cooking.pdf

3. Predicting the Structure of Cooking Recipes

   By Jermsak Jermsurawong and Nizar Habash (2015)
   An ingredient-instruction dependency tree data structure to represent recipes. The proposed representation allows for more refined comparison of recipes and recipe - parts, and is a step towards semantic representation of recipes
   http://www.aclweb.org/anthology/D/D15/D15-1090.pdf

4. Flow Graph Corpus from Recipe Texts

   By Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, Tetsuro Sasada4 (2015)
   An attempt at annotating procedural texts with a flow graph as a representation of understanding.The domain we focus on is cooking recipe.
   https://pdfs.semanticscholar.org/7f88/398e8928d32366ebf725eb457cb2fa3a94bf.pdf

5. Mise en Place: Unsupervised Interpretation of Instructional Recipes

   By Chloe Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi (2015)
   Automatically mapping instructional recipes to action graphs, which define what actions should be performed on which objects and in what order
   http://gthandavam.in/emnlp15_cooking.pdf

6. Construction of a Cooking Ontology from Cooking Recipes and Patents

   By Hidetsugu Nanba, Toshiyuki Takezawa, Yoko Doi, Kazutoshi Sumiya, Miho Tsujita (2015)
   A cooking ontology by means of pattern matching, statistical natural language processing techniques, and manual steps to identify hyponymy, synonymy, attributes, and meronymy.
   http://delivery.acm.org/10.1145/2650000/2641328/p507-nanba.pdf

7. KUSK Dataset: Toward a Direct Understanding of Recipe Text and Human Cooking Activity

   By Atsushi Hashimoto, Shinsuke Mori, Tetsuro Sasada, Michihiko Minoh, Yoko Yamakata (2014)
   Multimodal dataset for understanding cooking activities. To build the dataset, we instructed the subjects to perform cooking according to instructional texts shown on a display one by one. The instructional texts were generated from flow graphs, which were automatically extracted from recipes sampled from a Web site
   http://plata.ar.media.kyoto-u.ac.jp/publications/hashimoto-CEA14.pdf

8. A Framework for Procedural Text Understanding

   A framework for procedural text understanding. We tested our framework on cooking recipe texts annotated with a directed acyclic graph as their meaning.
   http://www.aclweb.org/anthology/W15-2206

9. Learning to Read Recipes: Activity Diagramming with Narrative Event Chains

   By Ganesa Thandavam Ponnuraj (2014)
   Generate activity diagrams of the recipe text automatically, with the help of narrative event chains
   http://www3.cs.stonybrook.edu/~gponnuraj/thesis.pdf

10. Cooking with Semantics

    By Jon Malmaud, Earl J. Wagner, Nancy Chang, Kevin Murphy (2014)
    How to extend [automatic interpretation of how-to instructions, such as cooking recipes,] using a model of pragmatics, based on a rich representation of world state
    https://www.cs.ubc.ca/~murphyk/Papers/acl2014.pdf

11. Proceedings of the Cooking with Computers workshop (CwC)

    By Am´elie Cordier and Emmanuel Nauer (2012)
    The “Cooking with Computers” workshop aims at gathering researchers from as many as possible fields of AI. A core application domain, which is cooking, is fixed and the main objective of this workshop is to show how some AI existing approaches could be used to solve problems in this domain.
    http://www.lirmm.fr/ecai2012/images/stories/ecai_doc/pdf/workshop/W29_proceedingsCWC2012.pdf

12. CookIIS – A Case-Based Recipe Advisor

    By Norman Ihle, R´egis Newo, Alexandre Hanft, Kerstin Bach, Meike Reichle (2010)
    CookIIS is a Case-Based Reasoning (CBR) system that provides recipe suggestions. The suggestions are created based on a given set of recipes that the system modifies according to a user’s specification.
    http://engineers-pool.com/members/27/publications/2010-03-11_13.05.29_CCC09_INHBR.pdf

13. TAAABLE: Text Mining, Ontology Engineering, and Hierarchical Classification for Textual Case-Based Cooking

    By Fadi Badra, Rokia Bendaoud, Rim Bentebibel, Pierre-Antoine Champin, Julien Cojan, Am´elie Cordier, Sylvie Despr´es, St´ephanie Jean-Daubias, Jean Lieber, Thomas Meilender, et al. (2008)
    Taaable is a Case-Based Reasoning (CBR, the process of solving new problems based on the solutions of similar past problems) system that uses a recipe book as a case base to answer cooking queries.Taaable participates in the Computer Cooking Contest since 2008. Its success is due, in particular, to a smart combination of various methods and techniques from knowledge - based systems: cbr, knowledge representation, knowledge acquisition and discovery, knowledge management, and natural language processing.
    https://hal.inria.fr/hal-00912767/document

14. SOUR CREAM: Toward Semantic Processing of Recipes

    By Dan Tasse and Noah A. Smith (2008)
    Preliminary work on SOUR CREAM (System to Organize and Understand Recipes, Capacitating Relatively Exciting Applications Meanwhile). The aim of this project is to develop new techniques for semantic parsing by focusing on the domain of cooking recipes. This report details the MILK meaning representation language and CURD, a database of recipes annotated in the MILK language. We also detail preliminary efforts at semantic processing using this dataset.
    http://www.lti.cs.cmu.edu/sites/default/files/cmulti08005.pdf

15. Chef

    By David Morgan-Mar (2003)
    Chef is a programming language in which programs look like recipes.
    http://www.dangermouse.net/esoteric/chef.html

16. Structural Analysis of Cooking Preparation Steps in Japanese

    By Reiko HAMADA, Ichiro IDE, Shuichi SAKAI, Hidehiko TANAKA (2000)
    https://pdfs.semanticscholar.org/2cc8/d5f093697e191eadb9d83f3023afee00666c.pdf

17. Computer Cooking Competition (Computer Cooking Contest)

    Given a restricted set of ingredients, the task is to cook something“ that tastes good”. These events are usually holded within the International Conference on Case-Based Reasoning (ICCBR)
    
    2016 (9th) Atlanta, USA, 31 October to 2 November
    2015 (8th) Frankfurt, Germany, 28 September
    2014 (7th) Cork, Ireland, September
    2013 (6th) Beijing, China, 3 to 9 August
    2012 (5th) Lyon, France, 3 September
    2011 (4th) Greenwich, London, UK, 12 to 15 September
    2010 (3rd) Alessandria, Italy, 19 to 22 July
    2009 (2nd) Seattle, USA, 21 July
    2008 (1st) Trier, Germany, 1 September
    http://cbrwiki.fdi.ucm.es/mediawiki/index.php/Computer_Cooking_Competition

18. Froglingo

    Enter ingredients, cooking method, and/or origin
    http://www.froglingo.com/ccc/index.html