# What is Reinforcement Learning? ## In its literal sense Reinforcement: It is defined as a consequence that follows a behavior that increases (or attempts to increase) the likelihood of that response occurring in the future[[source](https://psychology.uiowa.edu/comparative-cognition-laboratory/glossary/reinforcement)]. There are two types of reinforcement, known as positive reinforcement and negative reinforcement; positive is where a reward is offered on expression of the wanted behaviour and negative is taking away an undesirable element (or giving penalty), whenever the desired behaviour is achieved. Learning: Acquiring knowledge and skills and having them readily available from memory so you can make sense of future problems and opportunities[[source](https://pdfs.semanticscholar.org/4fc5/9769aa78266810ba1add6f0ed3e730e8efa6.pdf)]. ## RL within ML paradigm > "A computer program is said to learn from experience E with respect to some task T and some performance P, if its performance on T, as measured by P, improves with experience E."

- Mitchell 1997

```{figure} Images/Types_of_ML.jpeg --- width: 80% align: center name: types_of_ml --- Broad types of ML ``` Fitting RL into this definition gives, 1. **Task** (T) ![\Large x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}](https://latex.codecogs.com/svg.latex?\Large&space;\longrightarrow) Decision making strategy 2. **Performance** (P) ![\Large x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}](https://latex.codecogs.com/svg.latex?\Large&space;\longrightarrow) Cumulative rewards 3. **Experience** (E) ![\Large x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}](https://latex.codecogs.com/svg.latex?\Large&space;\longrightarrow) Interaction with environment/system ## The fundamental block of RL > Reinforcement Learning (RL) is the science of decision making for a pre-defined environment - mapping situations to actions - maximizing a numerical reward signal. ```{figure} Images/RL_Model.png --- width: 80% align: center name: rl_model --- A fundamental RL block ``` A typical formulation as follows can be considered, 1. Observe current state of environment, ![S_t](https://latex.codecogs.com/svg.latex?S_t), and current reward ![R_t](https://latex.codecogs.com/svg.latex?R_t) 2. Agent decides on the action, ![A_t](https://latex.codecogs.com/svg.latex?A_t) 3. Agent performs action, through interaction with environment 4. Environment shifts into the new state, ![S_t+1](https://latex.codecogs.com/svg.latex?S_{t+1}) 5. Reward feedback, ![R_t+1](https://latex.codecogs.com/svg.latex?R_{t+1}) ## Pacman as an RL example ```{figure} Images/RL_example_pacman.png --- width: 80% align: center name: rl_pacman --- Pacman as an RL example ``` Pacman is one of the most commonly known arcade games, which almost everyone has played at least once. Using it as an RL example, 1. State Space ![S](https://latex.codecogs.com/svg.latex?S): All possible configurations in the game, including the positions of the points, positions of the ghosts, and positions of the player, etc. 2. Action Space ![A](https://latex.codecogs.com/svg.latex?A): A set of all allowed actions {left, right, up, down, or stay}. 3. Policy ![$\pi$ : $S \longrightarrow A$](https://latex.codecogs.com/svg.latex?\pi%20\text{:}S%20\longrightarrow%20A): A mapping from the state space to the action space. The objective of the player is to take a series of optimal actions to maximize the total reward that it could obtain at the end of the game.[[source](http://cs229.stanford.edu/proj2017/final-reports/5241109.pdf)] ## Using RL on OpenAI Gym Atari ```{figure} Images/RL_atari.gif --- width: 80% align: center name: rl_atari --- RL on Atari environments ``` RL algorithms can be used on different types of atari game environments, where the environment definitions, player objectives, allowable actions, etc are all distinct. ## RL Agent learns to walk ```{figure} Images/RL_Learn_to_walk.gif --- width: 80% align: center name: rl_learn_to_walk --- RL Agent learns to walk ``` Against simple decision making of an arcade game, RL agents can learn even highly complex tasks, like bipedal walking, though it might take much longer to train. ## Many Faces of RL RL finds application at the intersection of various disciplines and fields as illustrated below, ```{figure} Images/many_faces_of_RL.PNG --- width: 80% align: center name: rl_many_faces --- Many Faces of RL ``` ## Distinctive Characteristics of RL What makes RL different from other ML paradims? * Not strictly supervised as instead of pre-labelled data, the data is generated through interaction with environment. * Unlike unsupervised, the goal isn't finding relationships within data, but maximize some reward function. * Reward feedback determines the quality of performance. * Time is a key component, as process is sequential (non i.i.d data) with delayed feedback. * Each action agent makes affects the subsequent data it receives. ## Summarizing * RL is a science of **learning to make decisions** from interaction with environment. * Solving any problem through RL makes us think about, * time * long-term action consequences * effective and efficient gathering of experience * predicting the future states * dealing with uncertainty