---
title: "Introduction to the implied package"
author: "Jonas C. Lindstrøm"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    toc: true
vignette: >
  %\VignetteIndexEntry{Introduction to the implied package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

This package contains functions that convert between bookmaker odds and probabilities. The function implied_probabilities() convert bookmaker odds into proper probabilities. The function implied_odds() does the inverse conversion, it turns proper probabilities into bookmaker odds. Several methods are available, with different assumptions regarding the underlying mechanism the bookmakers convert their probabilities into odds. The main focus of this introduction is present how the package works and the methods that convert bookmaker odds into probabilities and. Towards the end is a small demonstration on how to convert probabilities to bookmaker odds. 

A naive conversion of bookmaker odds into probabilities has two main problems. The first is that the probabilities are not proper probabilities, since they sum to more than 1. The excess probability is called the bookmakers margin. The second problem is that the probabilities, even if the margin is removed, will be biased in several ways, usually because of what is called the [favorite-longshot bias](https://en.wikipedia.org/wiki/Favourite-longshot_bias). The methods in this package remove the bookmaker margin and some of them also adjust for favorite-longshot bias.

In version 0.5 a new feature was introduced. It is now possible to convert odds to probabilities with multiple winners, which means that the probabilities should sum to something greater than 1. One example of this is when you have odds for different teams/players to finish top 3 in a league, in which case the probabilities should sum to 3 instead of 1. The details are explained towards the end of this document.


# Methods 
## The basic method

The default method used by the function implied_probabilities() is called the basic method. This is the simplest and most common method for converting bookmaker odds into probabilities, and is obtained by dividing the naive probabilities (the inverted odds) by the sum of the inverted odds. If <i>p<sub>i</sub></i> is the true underlying probability for outcome <i>i</i>,  and <i>r<sub>i</sub></i> is the corresponding inverted odds, then the probabilities are computed as 

<i>p<sub>i</sub></i> = <i>r<sub>i</sub></i> / sum(<i>r</i>)

This method tend to be the least accurate of the methods in this package. I have also seen this normalization method been referred to as the multiplicative method.

The implied_probabilities() function return a list with the proper probabilities (as a matrix) and the bookmaker margins.

In the examples below are three sets of bookmaker odds from three football matches.

```{r basic}

library(implied)

# One column for each outcome, one row for each race or match.
my_odds <- rbind(c(4.20, 3.70, 1.95),
                 c(2.45, 3.70, 2.90),
                 c(2.05, 3.20, 3.80))
colnames(my_odds) <- c('Home', 'Draw', 'Away')

res1 <- implied_probabilities(my_odds)

res1$probabilities

res1$margin

```


## Margin Weights Proportional to the Odds

This method is from [Joseph Buchdahl's Wisom of the Crowds document](https://www.football-data.co.uk/wisdom_of_crowd_bets), and assumes that the margin applied by the bookmaker for each of the outcome is proprtional to the probabilitiy of the outcome. In other words, the excessive probabilties are unevenly applied in a way that is reflects the favorite-longshot bias. 

The probabilities are calculated from the bookmaker odds <i>O</i> using the following formula

<i>p<sub>i</sub></i> = (n - M * O<sub>i</sub>) / n * O<sub>i</sub>

where n is the number of outcomes, and M is the bookmaker margin.

```{r wpo}
res2 <- implied_probabilities(my_odds, method = 'wpo')

res2$probabilities

# The margins applied to each outcome.
res2$specific_margins
```

## The odds ratio method

The odds ratio method is also from the Wisdom of the Crowds document, but is originally from an [article by Keith Cheung](https://www.sportstradingnetwork.com/article/fixed-odds-betting-traditional-odds/). This method models the relationship between the proper probabilities and the improper bookmaker probabilties using the odds ratio (OR) function:

OR = <i>p<sub>i</sub></i>  (1 - <i>r<sub>i</sub></i>) / <i>r<sub>i</sub></i> (1 - <i>p<sub>i</sub></i>)

This gives the probabilities 

<i>p<sub>i</sub></i> = <i>r<sub>i</sub></i> / OR + <i>r<sub>i</sub></i> - (OR * <i>r<sub>i</sub></i>)

where the odds ratio OR is selected so that sum(<i>p<sub>i</sub></i>) = 1.


```{r or}
res3 <- implied_probabilities(my_odds, method = 'or')

res3$probabilities

# The odds ratios converting the proper probablities to bookmaker probabilities.
res3$odds_ratios
```

## The power method

The power method models the bookmaker probabilities as a power function of the proper probabilities. This method is also described in the Wisdom of the Crowds document, where it is referred to as the logarithmic method.

<i>p<sub>i</sub></i> = <i>r<sub>i</sub></i><sup>(1/k)</sup>

where <i>k</i> is selected so that sum(<i>p<sub>i</sub></i>) = 1.

```{r power}
res4 <- implied_probabilities(my_odds, method = 'power')

res4$probabilities

# The inverse exponents (n) used to convert the proper probablities to bookmaker probabilities.
res4$exponents
```

## The additive method

The additive method removes the margin from the naive probabilities by subtracting an equal amount of of the margin from each outcome. The formula used is

<i>p<sub>i</sub></i> = <i>r<sub>i</sub></i> - ((sum(<i>r</i>) - 1) / n)

If there are only two outcomes, the additive method and Shin's method are equivalent.


```{r additive1}

res5 <- implied_probabilities(my_odds, method = 'additive')

res5$probabilities

```

One problem with the additive method is that it can produce negative probabilities, escpecially for outcomes with low probabilties. This can often be the case when there are many outcomes, for example in racing sports. If this happens, you will be given a warning. Here is an example taken from Clarke et al (2017):

```{r additive2}

my_odds2 <- t(matrix(1/c(0.870, 0.2, 0.1, 0.05, 0.02, 0.01)))
colnames(my_odds2) <- paste('X', 1:6, sep='')

res6 <- implied_probabilities(my_odds2, method = 'additive')

res6$probabilities

```



 
## Balanced books and Shin's method

The two methods referred to as "balanced book" and Shin's method are based on the assumption that there is a small proportion of bettors that actually knows the outcome (called inside traders), and the rest of the bettors reflect the otherwise "true" uncertainty about the outcome. The proportion of inside traders is denoted Z. 

The two methods differ in what assumptions they make about how the bookmakers react to the pressence of inside traders. Shin's method is derived from the assumption that the bookmakers tries to maximize their profits when there are inside traders. The balanced books method assumes the bookmakers tries to minimize their losses in the worst case scenario if the least likely outcome were to acctually occur.  
 
We can not know what the insiders know, but both methods gives an estimate of the proportion of insiders. 


```{r shin}
res7 <- implied_probabilities(my_odds, method = 'shin')

res7$probabilities

# The estimated proportion of inside traders.
res7$zvalues
```


```{r bb}
# Balanced books
res8 <- implied_probabilities(my_odds, method = 'bb')

res8$probabilities

# The estimated proportion of inside traders.
res8$zvalues
```


## The Jensen–Shannon distance method

This method sees the improper bookmaker probabilities as a noisy version of the true underlying probabilities, and uses the [Jensen–Shannon (JS) distance](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) as a measure of how noisy the bookmaker probabilities are.

For the sake of finding the denoised probabilities <i>p<sub>i</sub></i>, each outcome i is modeled as a binomial variable, with outcomes i and NOT i. These have probabilities <i>p<sub>i</sub></i> and 1-<i>p<sub>i</sub></i>, with corresponding improper bookmaker probabilities <i>r<sub>i</sub></i> and 1-<i>r<sub>i</sub></i>. For a given noise-level D, as measued by the symmetric JS distance, the underlying probabilities can be found by solving the JS distance equation for <i>p<sub>i</sub></i>:


D = 0.5 * BKL(<i>p<sub>i</sub></i>, <i>m<sub>i</sub></i>) + 0.5 * BKL(<i>r<sub>i</sub></i>, <i>m<sub>i</sub></i>) 

where <i>m<sub>i</sub></i> = (<i>p<sub>i</sub></i> + <i>r<sub>i</sub></i>) / 2

and 

BKL(x, y) = <i>x</i> * log(<i>x</i>/<i>y</i>) + (1-<i>x</i>) * log((1-<i>x</i>)/(1-<i>y</i>))) + <i>y</i> * log(<i>y</i>/<i>x</i>) + (1-<i>y</i>) * log((1-<i>y</i>)/(1-<i>y</i>))

is the "binomial" Kullback–Leibler divergence.

The solution is found numerically by finding the value of of <i>D</i> so that sum(<i>p<sub>i</sub></i>) = 1.

The method was developed by Christopher D. Long (twitter: @octonion), and described in a series of Twitter postings.

```{r jsd}
# Balanced books
res9 <- implied_probabilities(my_odds, method = 'jsd')

res9$probabilities

# The estimated noise (JS distance)
res9$distance
```


# Multiple winning outcomes

In the examples above it has been assumed that the probabilities should sum to 1. This is the correct approach when only 1 of the possible outcomes occur, but this is not correct when multiple outcomes occur. One example of this are odds for players/teams to reach the final in a tournament. In this case the probabilities should sum to 2, as two of the outcomes will be considered a win. Another example is placing in the top 5 in a league, in which case the probabilities should sum to 5. 

You can change the </i>target_probability</i> to something other than 1, and this works for most methods. 

```{r target_prob}
# Example odds.
odds_reach_final <- c(1.6, 2.63, 3.3, 3.7, 5.6, 7.1, 12.5, 16.5, 25)

res10 <- implied_probabilities(odds_reach_final, method = 'or', target_probability = 2)

res10$probabilities

sum(res10$probabilities)

```


# Converting probabilities to odds

There is also a function that can do the opposite what the implied_probabilities() function does, namely the implied_odds() function. This function converts probabilities to odds, for a given margin, the inverse of the methods as described above. Not all methods have been implemented yet. Take a look at the help file for the function for more details.

In the code example below we use take the results of converting the odds to probabilities using the power method, and convert them back to odds again, with the same margin. We pretty much recover the original odds, except for some small numerical inaccuracy.


```{r imp_odds1}

res_odds1 <- implied_odds(res4$probabilities[1,], 
                     margin = res4$margin[1], 
                     method = 'power')

res_odds1$odds

# The exponents.
res_odds1$exponents

# Compare to the exponent from the odds-to-probability conversion.
res4$exponents[1]

```


# Other packages
The [odds.converter](https://cran.r-project.org/package=odds.converter) package can convert between different odds formats, including to decimal odds, that this package requires.


# Literature
Here are some relevant references and links:

* Joseph Buchdahl - USING THE WISDOM OF THE CROWD TO FIND VALUE IN A FOOTBALL MATCH BETTING MARKET [Link](https://www.football-data.co.uk/wisdom_of_crowd_bets)

* Keith Cheung (2015) Fixed-odds betting and traditional odds [Link](https://www.sportstradingnetwork.com/article/fixed-odds-betting-traditional-odds/)

* Stephen Clarke, Stephanie Kovalchik & Martin Ingram (2017) Adjusting Bookmaker’s Odds to Allow for Overround [Link](http://www.sciencepublishinggroup.com/journal/paperinfo?journalid=155&doi=10.11648/j.ajss.20170506.12)

* Hyun Song Shin (1992) Prices of State Contingent Claims with Insider Traders, and the Favourite-Longshot Bias [Link](https://doi.org/10.2307/2234526)

*  Hyun Song Shin (1993) Measuring the Incidence of Insider Trading in a Market for State-Contingent Claims [Link](https://doi.org/10.2307/2234526)

* Bruno Jullien & Bernard Salanié (1994) Measuring the incidence of insider trading: A comment on Shin [Link](https://doi.org/10.2307/2235458)

*  John Fingleton & Patrick Waldron (1999) Optimal Determination of Bookmakers' Betting Odds: Theory and Tests.[Link](https://www.semanticscholar.org/paper/Optimal-Determination-of-Bookmakers'-Betting-Odds%3A-Fingleton-Waldron/e576f3b103e0ba041ae072a9201b948059c7806e)


