COMP 424 - Artificial Intelligence Lecture 3: Search for optimization problems

Instructors:

Joelle Pineau (jpineau@cs.mcgill.ca) Sylvie Ong (song@cs.mcgill.ca)

Class web page: www.cs.mcgill.ca/~jpineau/comp424

Overview
Uninformed search:
Assumes no knowledge about the problem. BFS, DFS, Iterative deepening

Informed search:
Use knowledge about the problem, in the form of a heuristic. Best-first search, heuristic search, A* (and extensions)

Search for optimization problems:

What is an optimization problem?

Today!

Iterative improvement algorithms: hill climbing, simulated annealing Genetic algorithms

COMP-424: Artificial intelligence

Joelle Pineau

Optimization problems
Typically characterized by: Large (continuous, combinatorial) state space. Searching all possible solutions is infeasible. A (non-uniform) cost function, which we want to optimize. We are satisfied to achieve a good solution. In some cases, constraints have to be satisfied.

Mathematical optimization is a field of its own. Here, we focus on those problems which arise most frequently in AI.
COMP-424: Artificial intelligence 3 Joelle Pineau

Example 1
Traveling salesman problem:

Many applications: scheduling, chip layout, graph coloring, learning control

Often easy to find some solution to the problem. Often provably very hard (NP-complete) to find the best solution. But we still want a good solution!

COMP-424: Artificial intelligence

Joelle Pineau

Example 2
Robot arm control:

Objective: find a configuration of joints such as to reach the goal.

COMP-424: Artificial intelligence

Joelle Pineau

Example
Stochastic optimization of robotic tissue manipulation:
http://www.youtube.com/watch?v=yFbUvmsNXX4

COMP-424: Artificial intelligence

Joelle Pineau

Search for optimization problems

Two classes of methods: Constructive methods: Start from scratch and build up a solution.
In TSP: start at the start city, add cities until a complete tour is formed.

Iterative improvement/repair methods: Start with a solution (which may be broken / suboptimal) and improve it.
In TSP: start with a complete tour and keep swapping cities to improve cost.

In both cases: the search is local

Have one solution in mind, and keep on looking for alternatives in its vicinity.

COMP-424: Artificial intelligence

Joelle Pineau

Constructive methods for optimization

Do we know how to do this? A* search

But: Cost in A* depends on final solution (i.e. path cost), not on how we got it (i.e. search cost). For many optimization problems, the cost of finding a solution is non-trivial (i.e. exorbitant!)

COMP-424: Artificial intelligence

Joelle Pineau

Iterative improvement algorithm

Intuition: Consider all possible solutions laid out on a landscape. We want to find the highest (or lowest) point.

This landscape is often very high-dimensional.

COMP-424: Artificial intelligence

Joelle Pineau

Sketch of iterative improvement

Start at random point on the landscape. Generate a set of points to move to. If the set is not empty:
Choose a point and move there.

If the set is empty:

You are stuck, so re-start.

Different algorithms have different ways of generating successors and choosing among them.
COMP-424: Artificial intelligence 10 Joelle Pineau

What moves should we consider?

Case 1: Robot planning
Start with initial state = random position. Move to an adjacent position. Terminate when goal is reached.

Case 2: Traveling Salesman Problem

Start with initial state = a random (possibly incomplete/illegal) tour. Swap cities to obtain a new partial tour. Terminate when constraints are met.

COMP-424: Artificial intelligence

11

Joelle Pineau

Hill climbing (Gradient ascent)

Main idea:
Always take a step in the direction that improves the current solution value the most.

Variation of best-first search. Very popular for learning algorithms.

COMP-424: Artificial intelligence

12

Joelle Pineau

Hill climbing
Algorithm: 1. Start at initial configuration S. 2. Let V be the value of S (high is good) . 3. Let Si, i=1 .. n be the set of neighboring configurations and Vi be the corresponding values. 4. Let Vmax = maxi Vi be the value of the best successor configuration and imax = argmaxi Vi be the index of the best configuration. 5. If Vmax V, return S (we are at a local optimum). 6. Let S Simax and V Vmax. Go to 3. What are the pros and cons of this algorithm?
COMP-424: Artificial intelligence 13 Joelle Pineau

Good things about hill climbing

Trivial to program! Requires no memory of where weve been (because theres no backtracking.) It is important to have a good set of moves (not too many, not too few.)

COMP-424: Artificial intelligence

14

Joelle Pineau

Problems with hill climbing

Can get stuck in a local maximum, or plateau.

Relies heavily on having a good evaluation (i.e. one with an easy-tonavigate landscape).

Russell and Norvig: Resembles climbing Mount Everest in thick fog with amnesia.
COMP-424: Artificial intelligence 15 Joelle Pineau

Improvements to hill climbing

Quick fix: Whenever stuck in a plateau or local maximum, use random restarts.

Better fix: Instead of picking the next best move, pick any move that produces an improvement.

COMP-424: Artificial intelligence

16

Joelle Pineau

Simulated annealing
Similar to hill climbing, but: allows some bad moves in the hope of escaping local maxima. decrease size and frequency of bad moves over time. Algorithm: 1. Start at initial configuration S. 2. Let V be the value of S (high is good). 3. Let i be a random move and Si the next configuration. 4. Let Vi be the value of Si. 5. If V < Vi then set S Si and V Vi . Else with some probability p, still accept the move: S Si and V Vi 6. Go to 3 (as long as you want.)
COMP-424: Artificial intelligence 17 Joelle Pineau

What value should we use for p?

Some fixed value. A value that decays to 0 over time. A value that decays to 0, but also gives similar chance to similarly bad moves.

COMP-424: Artificial intelligence

18

Joelle Pineau

Selecting moves in simulated annealing

If the new value Vi is better than the old value V, definitely move to the neighboring solution. If the new value is worse (Vi<V) then move to the neighboring solution with probability: e-(V-Vi)/T [Boltzmann distribution] T>0 is a parameter called the temperature, which typically starts high, then decreases over time towards 0. If T is very close to 0, the probability of moving to a worse solution is almost 0.

COMP-424: Artificial intelligence

19

Joelle Pineau

Properties of simulated annealing

What happens when T is high?
Algorithm is in an exploratory phase (even bad moves have a high chance of being picked).

What happens when T is low?

Algorithm is in an exploitation phase (the ``bad" moves have very low probability).

If T is decreased slowly enough, simulated annealing is guaranteed to reach the optimal solution.

But it may take an infinite number of moves!

COMP-424: Artificial intelligence

20

Joelle Pineau

Parallel search
Run many separate searches (hill-climbing or simulated annealing) in parallel. Keep the best solution found. Search speed can be greatly improved by using many processors (including, most recently, GPUs). Especially useful when actions have non-deterministic outcomes (more on this later).
Also known as Monte Carlo search.

21

Joelle Pineau

Evolutionary computing
Refers generally to computational procedures patterned after biological evolution. Nature looks for the best individual (i.e. the fittest). Many solutions (individuals) exist in parallel. Evolutionary search procedures are also parallel, perturbing probabilistically several potential solutions.

COMP-424: Artificial intelligence

22

Joelle Pineau

Genetic algorithms
A candidate solution is called an individual.
In a traveling salesman problem, an individual is a tour

Each individual has a fitness

fitness = numerical value proportional to quality of that solution

A set of individuals is called a population. Populations change over generations, by applying operations to individuals.
operations = {selection, mutation and crossover}

Individual typically represented by a binary string:

allows operations to be carried out easily.

COMP-424: Artificial intelligence

23

Joelle Pineau

Typical genetic algorithm

GA(Fitness, threshold, p, r, m) Initialize: P p random individuals Evaluate: for each h P, compute Fitness(h) While maxh Fitness(h) < threshold
Select: Probabilistically select (1-r)p members of P to include in Ps Crossover: Probabilistically select 0.5*r*p pairs of individuals from $P$. For each pair (h1, h2), produce two offspring by applying the Crossover operator. Include all offspring in Ps. Mutate: Invert a randomly selected bit in m*p members of Ps Update: P Ps Evaluate: for each h P, compute Fitness(h)

Return the individual from P that has the highest fitness.

24 Joelle Pineau

COMP-424: Artificial intelligence

Elitism
The best solution can "die" during evolution In order to prevent this, the best solution ever encountered can always be "preserved" on the side If the "genes" from the best solution should always be present in the population, it can also be copied in the next generation automatically, bypassing the selection process.

COMP-424: Artificial intelligence

25

Joelle Pineau

Selecting the most fit individual

Fitness proportionate selection:
Can lead to crowding (multiple copies being propagated).

Tournament selection:
Pick i, j at random with uniform probability. With probability p select the fitter one. Only requires comparing two individuals.

Rank selection:
Sort all hypothesis by fitness. Probability of selection is proportional to rank.

Softmax (Boltzman) selection:

COMP-424: Artificial intelligence

26

Joelle Pineau

Crossover
Combine parts of individuals to create new individuals. Single-point crossover:
Choose a crossover point, cut individuals there, swap the pieces. E.g. 101|0101 011|1110 101|1110 011|0101

Implementation:
Use a crossover mask, m, which is a binary string E.g. m = 1110000

Now think of crossover in terms of arbitrary crossover masks.

COMP-424: Artificial intelligence

27

Joelle Pineau

Mutation
Mutation is a way of generating desirable features that are not present in the original population. Typically mutation just means changing a 0 to a 1 (and vice versa). We can allow mutation in all individuals, or just in the offspring generated after crossover.

COMP-424: Artificial intelligence

28

Joelle Pineau

Operators for genetic algorithms

COMP-424: Artificial intelligence

29

Joelle Pineau

Genetic algorithms as search

States: possible solutions Search operators: mutation, crossover, selection Parallel search, since several solutions are maintained in parallel Hill-climbing on the fitness function, but without following the gradient Mutation and crossover should allow us to get out of local minima Very related to simulated annealing.

COMP-424: Artificial intelligence

30

Joelle Pineau

Example
Genetic algorithms for the multiple traveling salesman problem:
http://www.youtube.com/watch?v=OC7URrCbceA

COMP-424: Artificial intelligence

31

Joelle Pineau

Summary
Optimization problems are widespread and important. It is unfeasible to enumerate lots of solutions. Goal is to get a reasonable (not necessarily optimal) solution. Apply a local search and move in a promising direction.
Hill climbing (a.k.a. gradient ascent/descent) always moves in the (locally) best direction. Simulated annealing allows some moves towards worse solutions. Genetic algorithms allow a parallel search in all directions.

Search for optimization is a large field, with many variants on the algorithms described today.

COMP-424: Artificial intelligence

32

Joelle Pineau

Notes
Textbooks have been ordered, and will be in the bookstore in 2 weeks. Homework 1 will be published on Wednesday, due 1 week later. Any questions?

33

Joelle Pineau